2023-11-08T04:37:39.335 INFO:root:teuthology version: 0.0.1.dev203+g6899cd2 2023-11-08T04:37:39.336 DEBUG:teuthology.run:Teuthology command: teuthology --archive /home/teuthworker/archive/yuriw-2023-11-07_19:42:25-rbd-wip-yuri3-testing-2023-11-07-0801-pacific-distro-default-smithi/7451368 --name yuriw-2023-11-07_19:42:25-rbd-wip-yuri3-testing-2023-11-07-0801-pacific-distro-default-smithi --verbose --description rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-low-osd-mem-target policy/simple rbd-mirror/four-per-cluster supported-random-distro$/{centos_8} workloads/rbd-mirror-snapshot-stress-workunit-exclusive-lock} --owner scheduled_yuriw@teuthology -- /home/teuthworker/archive/yuriw-2023-11-07_19:42:25-rbd-wip-yuri3-testing-2023-11-07-0801-pacific-distro-default-smithi/7451368/orig.config.yaml 2023-11-08T04:37:39.383 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-11-08T04:37:39.471 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2023-11-07_19:42:25-rbd-wip-yuri3-testing-2023-11-07-0801-pacific-distro-default-smithi/7451368 branch: wip-yuri3-testing-2023-11-07-0801-pacific description: rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-low-osd-mem-target policy/simple rbd-mirror/four-per-cluster supported-random-distro$/{centos_8} workloads/rbd-mirror-snapshot-stress-workunit-exclusive-lock} email: yweinste@redhat.com first_in_suite: false job_id: '7451368' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi meta: - desc: run two ceph clusters and install rbd-mirror - desc: configure the permissions for client.mirror - desc: 2 ceph clusters with 1 mon and 3 osds each - desc: run four rbd-mirror daemons per cluster - desc: run the rbd_mirror_stress.sh workunit to test the rbd-mirror daemon name: yuriw-2023-11-07_19:42:25-rbd-wip-yuri3-testing-2023-11-07-0801-pacific-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 3 size: 30 os_type: centos os_version: 8.stream overrides: admin_socket: branch: wip-yuri3-testing-2023-11-07-0801-pacific ceph: conf: client: debug rbd: 20 debug rbd_mirror: 15 log to stderr: false rbd default features: 125 rbd mirror image policy type: simple client.mirror.0: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid client.mirror.1: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid client.mirror.2: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid client.mirror.3: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid global: mon client directed command retry: 5 ms inject socket failures: 5000 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd memory target: 1610612736 osd objectstore: bluestore flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME sha1: cfb6fd52113be6a0af3fe7fb89b2e08474eadc0f ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: cfb6fd52113be6a0af3fe7fb89b2e08474eadc0f selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-yuri3-testing-2023-11-07-0801-pacific sha1: cfb6fd52113be6a0af3fe7fb89b2e08474eadc0f owner: scheduled_yuriw@teuthology priority: 75 repo: https://github.com/ceph/ceph-ci.git roles: - - cluster1.mon.a - cluster1.mgr.x - cluster2.mgr.x - cluster1.osd.0 - cluster1.osd.1 - cluster1.osd.2 - cluster1.client.0 - cluster2.client.0 - - cluster2.mon.a - cluster2.osd.0 - cluster2.osd.1 - cluster2.osd.2 - cluster1.client.mirror - cluster1.client.mirror.0 - cluster1.client.mirror.1 - cluster1.client.mirror.2 - cluster1.client.mirror.3 - cluster1.client.mirror.4 - cluster1.client.mirror.5 - cluster1.client.mirror.6 - cluster2.client.mirror - cluster2.client.mirror.0 - cluster2.client.mirror.1 - cluster2.client.mirror.2 - cluster2.client.mirror.3 - cluster2.client.mirror.4 - cluster2.client.mirror.5 - cluster2.client.mirror.6 seed: 9776 sha1: cfb6fd52113be6a0af3fe7fb89b2e08474eadc0f sleep_before_teardown: 0 subset: 111/120000 suite: rbd suite_branch: wip-yuri3-testing-2023-11-07-0801-pacific suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_cfb6fd52113be6a0af3fe7fb89b2e08474eadc0f/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: cfb6fd52113be6a0af3fe7fb89b2e08474eadc0f targets: smithi077.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGSQ/WLcJdpJ0ClJVyBsFrcYIQ2QRbAwp2zUvz0ifkJpShDNVGje+V/Jdm6AZOafWAIrxu2EWTj6axjmMfZQozM= smithi087.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBD+XKU8wa6pBVq4mhdEwRNPaK1Sl9uWyjisTURFaLby1G+ogJ5JiOLEZ7VuzQt8PXKmmvgJJqHhmd6n+97zE5BE= tasks: - install: extra_packages: - rbd-mirror - ceph: cluster: cluster1 - ceph: cluster: cluster2 - exec: cluster1.client.mirror.0: - sudo ceph --cluster cluster1 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd' - sudo ceph --cluster cluster1 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster1 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster1 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster1 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd' - rbd-mirror: client: cluster1.client.mirror.0 thrash: true - rbd-mirror: client: cluster1.client.mirror.1 thrash: true - rbd-mirror: client: cluster1.client.mirror.2 thrash: true - rbd-mirror: client: cluster1.client.mirror.3 thrash: true - rbd-mirror: client: cluster2.client.mirror.0 thrash: true - rbd-mirror: client: cluster2.client.mirror.1 thrash: true - rbd-mirror: client: cluster2.client.mirror.2 thrash: true - rbd-mirror: client: cluster2.client.mirror.3 thrash: true - rbd-mirror-thrash: cluster: cluster1 - rbd-mirror-thrash: cluster: cluster2 - workunit: clients: cluster1.client.mirror: - rbd/rbd_mirror_stress.sh env: CEPH_ARGS: '' MIRROR_IMAGE_MODE: snapshot MIRROR_POOL_MODE: image RBD_IMAGE_FEATURES: layering,exclusive-lock RBD_MIRROR_INSTANCES: '4' RBD_MIRROR_USE_EXISTING_CLUSTER: '1' RBD_MIRROR_USE_RBD_MIRROR: '1' timeout: 6h teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 6899cd26fceddb2fec83dc1a1349394b28c8998e timestamp: 2023-11-07_19:42:25 tube: smithi user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.3590714 2023-11-08T04:37:39.472 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_cfb6fd52113be6a0af3fe7fb89b2e08474eadc0f/qa; will attempt to use it 2023-11-08T04:37:39.473 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_cfb6fd52113be6a0af3fe7fb89b2e08474eadc0f/qa/tasks 2023-11-08T04:37:39.473 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-11-08T04:37:39.475 INFO:teuthology.task.internal:Checking packages... 2023-11-08T04:37:39.499 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'cfb6fd52113be6a0af3fe7fb89b2e08474eadc0f' 2023-11-08T04:37:39.499 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-11-08T04:37:39.501 INFO:teuthology.packaging:ref: None 2023-11-08T04:37:39.502 INFO:teuthology.packaging:tag: None 2023-11-08T04:37:39.502 INFO:teuthology.packaging:branch: wip-yuri3-testing-2023-11-07-0801-pacific 2023-11-08T04:37:39.502 INFO:teuthology.packaging:sha1: cfb6fd52113be6a0af3fe7fb89b2e08474eadc0f 2023-11-08T04:37:39.503 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-yuri3-testing-2023-11-07-0801-pacific 2023-11-08T04:37:39.643 INFO:teuthology.task.internal:Found packages for ceph version 16.2.14-129.gcfb6fd52 2023-11-08T04:37:39.643 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-11-08T04:37:39.649 INFO:teuthology.task.internal:no buildpackages task found 2023-11-08T04:37:39.650 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-11-08T04:37:39.700 INFO:teuthology.task.internal:Saving configuration 2023-11-08T04:37:39.722 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-11-08T04:37:39.743 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-11-08T04:37:39.769 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi077.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-11-07_19:42:25-rbd-wip-yuri3-testing-2023-11-07-0801-pacific-distro-default-smithi/7451368', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-11-08 04:32:25.547376', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGSQ/WLcJdpJ0ClJVyBsFrcYIQ2QRbAwp2zUvz0ifkJpShDNVGje+V/Jdm6AZOafWAIrxu2EWTj6axjmMfZQozM='} 2023-11-08T04:37:39.792 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi087.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-11-07_19:42:25-rbd-wip-yuri3-testing-2023-11-07-0801-pacific-distro-default-smithi/7451368', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-11-08 04:32:25.549289', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBD+XKU8wa6pBVq4mhdEwRNPaK1Sl9uWyjisTURFaLby1G+ogJ5JiOLEZ7VuzQt8PXKmmvgJJqHhmd6n+97zE5BE='} 2023-11-08T04:37:39.792 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-11-08T04:37:39.801 INFO:teuthology.task.internal:roles: ubuntu@smithi077.front.sepia.ceph.com - ['cluster1.mon.a', 'cluster1.mgr.x', 'cluster2.mgr.x', 'cluster1.osd.0', 'cluster1.osd.1', 'cluster1.osd.2', 'cluster1.client.0', 'cluster2.client.0'] 2023-11-08T04:37:39.801 INFO:teuthology.task.internal:roles: ubuntu@smithi087.front.sepia.ceph.com - ['cluster2.mon.a', 'cluster2.osd.0', 'cluster2.osd.1', 'cluster2.osd.2', 'cluster1.client.mirror', 'cluster1.client.mirror.0', 'cluster1.client.mirror.1', 'cluster1.client.mirror.2', 'cluster1.client.mirror.3', 'cluster1.client.mirror.4', 'cluster1.client.mirror.5', 'cluster1.client.mirror.6', 'cluster2.client.mirror', 'cluster2.client.mirror.0', 'cluster2.client.mirror.1', 'cluster2.client.mirror.2', 'cluster2.client.mirror.3', 'cluster2.client.mirror.4', 'cluster2.client.mirror.5', 'cluster2.client.mirror.6'] 2023-11-08T04:37:39.802 INFO:teuthology.run_tasks:Running task console_log... 2023-11-08T04:37:39.927 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f57b0d8aaf0>, signals=[15]) 2023-11-08T04:37:39.928 INFO:teuthology.run_tasks:Running task internal.connect... 2023-11-08T04:37:39.936 INFO:teuthology.task.internal:Opening connections... 2023-11-08T04:37:39.936 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi077.front.sepia.ceph.com 2023-11-08T04:37:39.939 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi077.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-08T04:37:40.008 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi087.front.sepia.ceph.com 2023-11-08T04:37:40.009 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi087.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-08T04:37:40.075 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-11-08T04:37:40.081 DEBUG:teuthology.orchestra.run.smithi077:> uname -m 2023-11-08T04:37:40.103 INFO:teuthology.orchestra.run.smithi077.stdout:x86_64 2023-11-08T04:37:40.103 DEBUG:teuthology.orchestra.run.smithi077:> cat /etc/os-release 2023-11-08T04:37:40.164 INFO:teuthology.orchestra.run.smithi077.stdout:NAME="CentOS Stream" 2023-11-08T04:37:40.164 INFO:teuthology.orchestra.run.smithi077.stdout:VERSION="8" 2023-11-08T04:37:40.164 INFO:teuthology.orchestra.run.smithi077.stdout:ID="centos" 2023-11-08T04:37:40.165 INFO:teuthology.orchestra.run.smithi077.stdout:ID_LIKE="rhel fedora" 2023-11-08T04:37:40.165 INFO:teuthology.orchestra.run.smithi077.stdout:VERSION_ID="8" 2023-11-08T04:37:40.165 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM_ID="platform:el8" 2023-11-08T04:37:40.165 INFO:teuthology.orchestra.run.smithi077.stdout:PRETTY_NAME="CentOS Stream 8" 2023-11-08T04:37:40.165 INFO:teuthology.orchestra.run.smithi077.stdout:ANSI_COLOR="0;31" 2023-11-08T04:37:40.166 INFO:teuthology.orchestra.run.smithi077.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2023-11-08T04:37:40.166 INFO:teuthology.orchestra.run.smithi077.stdout:HOME_URL="https://centos.org/" 2023-11-08T04:37:40.166 INFO:teuthology.orchestra.run.smithi077.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-11-08T04:37:40.166 INFO:teuthology.orchestra.run.smithi077.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2023-11-08T04:37:40.166 INFO:teuthology.orchestra.run.smithi077.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-11-08T04:37:40.167 INFO:teuthology.lock.ops:Updating smithi077.front.sepia.ceph.com on lock server 2023-11-08T04:37:40.194 DEBUG:teuthology.orchestra.run.smithi087:> uname -m 2023-11-08T04:37:40.217 INFO:teuthology.orchestra.run.smithi087.stdout:x86_64 2023-11-08T04:37:40.218 DEBUG:teuthology.orchestra.run.smithi087:> cat /etc/os-release 2023-11-08T04:37:40.279 INFO:teuthology.orchestra.run.smithi087.stdout:NAME="CentOS Stream" 2023-11-08T04:37:40.279 INFO:teuthology.orchestra.run.smithi087.stdout:VERSION="8" 2023-11-08T04:37:40.279 INFO:teuthology.orchestra.run.smithi087.stdout:ID="centos" 2023-11-08T04:37:40.280 INFO:teuthology.orchestra.run.smithi087.stdout:ID_LIKE="rhel fedora" 2023-11-08T04:37:40.280 INFO:teuthology.orchestra.run.smithi087.stdout:VERSION_ID="8" 2023-11-08T04:37:40.280 INFO:teuthology.orchestra.run.smithi087.stdout:PLATFORM_ID="platform:el8" 2023-11-08T04:37:40.280 INFO:teuthology.orchestra.run.smithi087.stdout:PRETTY_NAME="CentOS Stream 8" 2023-11-08T04:37:40.280 INFO:teuthology.orchestra.run.smithi087.stdout:ANSI_COLOR="0;31" 2023-11-08T04:37:40.281 INFO:teuthology.orchestra.run.smithi087.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2023-11-08T04:37:40.281 INFO:teuthology.orchestra.run.smithi087.stdout:HOME_URL="https://centos.org/" 2023-11-08T04:37:40.281 INFO:teuthology.orchestra.run.smithi087.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-11-08T04:37:40.281 INFO:teuthology.orchestra.run.smithi087.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2023-11-08T04:37:40.281 INFO:teuthology.orchestra.run.smithi087.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-11-08T04:37:40.282 INFO:teuthology.lock.ops:Updating smithi087.front.sepia.ceph.com on lock server 2023-11-08T04:37:40.310 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-11-08T04:37:40.320 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-11-08T04:37:40.351 INFO:teuthology.task.internal:Checking for old test directory... 2023-11-08T04:37:40.352 DEBUG:teuthology.orchestra.run.smithi077:> test '!' -e /home/ubuntu/cephtest 2023-11-08T04:37:40.356 DEBUG:teuthology.orchestra.run.smithi087:> test '!' -e /home/ubuntu/cephtest 2023-11-08T04:37:40.377 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-11-08T04:37:40.383 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-11-08T04:37:40.384 DEBUG:teuthology.orchestra.run.smithi077:> test -z $(ls -A /var/lib/ceph) 2023-11-08T04:37:40.416 DEBUG:teuthology.orchestra.run.smithi087:> test -z $(ls -A /var/lib/ceph) 2023-11-08T04:37:40.449 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-11-08T04:37:40.506 INFO:teuthology.run_tasks:Running task kernel... 2023-11-08T04:37:40.522 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2023-11-08T04:37:40.523 INFO:teuthology.task.kernel:config {'cluster1.mon.a': {'kdb': True, 'sha1': 'distro'}, 'cluster2.mon.a': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2023-11-08T04:37:40.524 DEBUG:teuthology.orchestra.run.smithi077:> test -f /run/.containerenv -o -f /.dockerenv 2023-11-08T04:37:40.525 DEBUG:teuthology.orchestra.run.smithi087:> test -f /run/.containerenv -o -f /.dockerenv 2023-11-08T04:37:40.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-08T04:37:40.545 DEBUG:teuthology.orchestra.run.smithi087:> uname -r 2023-11-08T04:37:40.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-08T04:37:40.546 DEBUG:teuthology.orchestra.run.smithi077:> uname -r 2023-11-08T04:37:40.605 INFO:teuthology.orchestra.run.smithi087.stdout:4.18.0-489.el8.x86_64 2023-11-08T04:37:40.606 INFO:teuthology.task.kernel:Running kernel on smithi087: 4.18.0-489.el8.x86_64 2023-11-08T04:37:40.606 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum install -y kernel 2023-11-08T04:37:40.608 INFO:teuthology.orchestra.run.smithi077.stdout:4.18.0-489.el8.x86_64 2023-11-08T04:37:40.608 INFO:teuthology.task.kernel:Running kernel on smithi077: 4.18.0-489.el8.x86_64 2023-11-08T04:37:40.608 DEBUG:teuthology.orchestra.run.smithi077:> sudo yum install -y kernel 2023-11-08T04:37:44.440 INFO:teuthology.orchestra.run.smithi087.stdout:CentOS Stream 8 - AppStream 27 MB/s | 34 MB 00:01 2023-11-08T04:37:44.627 INFO:teuthology.orchestra.run.smithi077.stdout:CentOS Stream 8 - AppStream 24 MB/s | 34 MB 00:01 2023-11-08T04:37:53.369 INFO:teuthology.orchestra.run.smithi077.stdout:CentOS Stream 8 - BaseOS 20 MB/s | 53 MB 00:02 2023-11-08T04:37:53.421 INFO:teuthology.orchestra.run.smithi087.stdout:CentOS Stream 8 - BaseOS 19 MB/s | 53 MB 00:02 2023-11-08T04:38:01.373 INFO:teuthology.orchestra.run.smithi087.stdout:CentOS Stream 8 - Extras 41 kB/s | 18 kB 00:00 2023-11-08T04:38:01.435 INFO:teuthology.orchestra.run.smithi077.stdout:CentOS Stream 8 - Extras 35 kB/s | 18 kB 00:00 2023-11-08T04:38:01.848 INFO:teuthology.orchestra.run.smithi087.stdout:CentOS Stream 8 - Extras common packages 24 kB/s | 6.9 kB 00:00 2023-11-08T04:38:01.878 INFO:teuthology.orchestra.run.smithi077.stdout:CentOS Stream 8 - Extras common packages 26 kB/s | 6.9 kB 00:00 2023-11-08T04:38:02.158 INFO:teuthology.orchestra.run.smithi087.stdout:Copr repo for python3-asyncssh owned by ceph 23 kB/s | 3.5 kB 00:00 2023-11-08T04:38:02.163 INFO:teuthology.orchestra.run.smithi077.stdout:Copr repo for python3-asyncssh owned by ceph 26 kB/s | 3.5 kB 00:00 2023-11-08T04:38:05.888 INFO:teuthology.orchestra.run.smithi077.stdout:Extra Packages for Enterprise Linux 4.5 MB/s | 16 MB 00:03 2023-11-08T04:38:06.009 INFO:teuthology.orchestra.run.smithi087.stdout:Extra Packages for Enterprise Linux 4.3 MB/s | 16 MB 00:03 2023-11-08T04:38:09.680 INFO:teuthology.orchestra.run.smithi077.stdout:lab-extras 419 kB/s | 24 kB 00:00 2023-11-08T04:38:09.942 INFO:teuthology.orchestra.run.smithi087.stdout:lab-extras 406 kB/s | 24 kB 00:00 2023-11-08T04:38:13.221 INFO:teuthology.orchestra.run.smithi077.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2023-11-08T04:38:13.221 INFO:teuthology.orchestra.run.smithi077.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2023-11-08T04:38:13.484 INFO:teuthology.orchestra.run.smithi077.stdout:Dependencies resolved. 2023-11-08T04:38:13.485 INFO:teuthology.orchestra.run.smithi077.stdout:================================================================================ 2023-11-08T04:38:13.485 INFO:teuthology.orchestra.run.smithi077.stdout: Package Architecture Version Repository Size 2023-11-08T04:38:13.486 INFO:teuthology.orchestra.run.smithi077.stdout:================================================================================ 2023-11-08T04:38:13.486 INFO:teuthology.orchestra.run.smithi077.stdout:Installing: 2023-11-08T04:38:13.486 INFO:teuthology.orchestra.run.smithi077.stdout: kernel x86_64 4.18.0-521.el8 baseos 10 M 2023-11-08T04:38:13.486 INFO:teuthology.orchestra.run.smithi077.stdout:Installing dependencies: 2023-11-08T04:38:13.487 INFO:teuthology.orchestra.run.smithi077.stdout: kernel-core x86_64 4.18.0-521.el8 baseos 43 M 2023-11-08T04:38:13.487 INFO:teuthology.orchestra.run.smithi077.stdout: kernel-modules x86_64 4.18.0-521.el8 baseos 36 M 2023-11-08T04:38:13.487 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-11-08T04:38:13.487 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction Summary 2023-11-08T04:38:13.487 INFO:teuthology.orchestra.run.smithi077.stdout:================================================================================ 2023-11-08T04:38:13.488 INFO:teuthology.orchestra.run.smithi077.stdout:Install 3 Packages 2023-11-08T04:38:13.488 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-11-08T04:38:13.488 INFO:teuthology.orchestra.run.smithi077.stdout:Total download size: 89 M 2023-11-08T04:38:13.489 INFO:teuthology.orchestra.run.smithi077.stdout:Installed size: 96 M 2023-11-08T04:38:13.489 INFO:teuthology.orchestra.run.smithi077.stdout:Downloading Packages: 2023-11-08T04:38:13.655 INFO:teuthology.orchestra.run.smithi087.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2023-11-08T04:38:13.656 INFO:teuthology.orchestra.run.smithi087.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2023-11-08T04:38:13.933 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2023-11-08T04:38:14.041 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2023-11-08T04:38:14.041 INFO:teuthology.orchestra.run.smithi087.stdout: Package Architecture Version Repository Size 2023-11-08T04:38:14.041 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2023-11-08T04:38:14.042 INFO:teuthology.orchestra.run.smithi087.stdout:Installing: 2023-11-08T04:38:14.042 INFO:teuthology.orchestra.run.smithi087.stdout: kernel x86_64 4.18.0-521.el8 baseos 10 M 2023-11-08T04:38:14.042 INFO:teuthology.orchestra.run.smithi087.stdout:Installing dependencies: 2023-11-08T04:38:14.042 INFO:teuthology.orchestra.run.smithi087.stdout: kernel-core x86_64 4.18.0-521.el8 baseos 43 M 2023-11-08T04:38:14.043 INFO:teuthology.orchestra.run.smithi087.stdout: kernel-modules x86_64 4.18.0-521.el8 baseos 36 M 2023-11-08T04:38:14.043 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T04:38:14.043 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction Summary 2023-11-08T04:38:14.043 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2023-11-08T04:38:14.043 INFO:teuthology.orchestra.run.smithi087.stdout:Install 3 Packages 2023-11-08T04:38:14.044 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T04:38:14.044 INFO:teuthology.orchestra.run.smithi087.stdout:Total download size: 89 M 2023-11-08T04:38:14.044 INFO:teuthology.orchestra.run.smithi087.stdout:Installed size: 96 M 2023-11-08T04:38:14.044 INFO:teuthology.orchestra.run.smithi087.stdout:Downloading Packages: 2023-11-08T04:38:15.235 INFO:teuthology.orchestra.run.smithi087.stdout:(1/3): kernel-4.18.0-521.el8.x86_64.rpm 8.3 MB/s | 10 MB 00:01 2023-11-08T04:38:15.890 INFO:teuthology.orchestra.run.smithi077.stdout:(1/3): kernel-4.18.0-521.el8.x86_64.rpm 8.4 MB/s | 10 MB 00:01 2023-11-08T04:38:16.725 INFO:teuthology.orchestra.run.smithi087.stdout:(2/3): kernel-modules-4.18.0-521.el8.x86_64.rpm 13 MB/s | 36 MB 00:02 2023-11-08T04:38:17.494 INFO:teuthology.orchestra.run.smithi087.stdout:(3/3): kernel-core-4.18.0-521.el8.x86_64.rpm 12 MB/s | 43 MB 00:03 2023-11-08T04:38:17.495 INFO:teuthology.orchestra.run.smithi087.stdout:-------------------------------------------------------------------------------- 2023-11-08T04:38:17.495 INFO:teuthology.orchestra.run.smithi087.stdout:Total 25 MB/s | 89 MB 00:03 2023-11-08T04:38:17.773 INFO:teuthology.orchestra.run.smithi077.stdout:(2/3): kernel-modules-4.18.0-521.el8.x86_64.rpm 12 MB/s | 36 MB 00:03 2023-11-08T04:38:18.167 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction check 2023-11-08T04:38:18.491 INFO:teuthology.orchestra.run.smithi077.stdout:(3/3): kernel-core-4.18.0-521.el8.x86_64.rpm 11 MB/s | 43 MB 00:03 2023-11-08T04:38:18.491 INFO:teuthology.orchestra.run.smithi077.stdout:-------------------------------------------------------------------------------- 2023-11-08T04:38:18.491 INFO:teuthology.orchestra.run.smithi077.stdout:Total 18 MB/s | 89 MB 00:05 2023-11-08T04:38:18.590 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction check succeeded. 2023-11-08T04:38:18.590 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction test 2023-11-08T04:38:19.167 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction check 2023-11-08T04:38:19.581 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction check succeeded. 2023-11-08T04:38:19.581 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction test 2023-11-08T04:38:19.808 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction test succeeded. 2023-11-08T04:38:19.827 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction 2023-11-08T04:38:20.817 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction test succeeded. 2023-11-08T04:38:20.976 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction 2023-11-08T04:38:22.606 INFO:teuthology.orchestra.run.smithi087.stdout: Preparing : 1/1 2023-11-08T04:38:23.108 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : kernel-core-4.18.0-521.el8.x86_64 1/3 2023-11-08T04:38:23.755 INFO:teuthology.orchestra.run.smithi077.stdout: Preparing : 1/1 2023-11-08T04:38:24.428 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : kernel-core-4.18.0-521.el8.x86_64 1/3 2023-11-08T04:38:26.679 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: kernel-core-4.18.0-521.el8.x86_64 1/3 2023-11-08T04:38:27.066 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : kernel-modules-4.18.0-521.el8.x86_64 2/3 2023-11-08T04:38:28.039 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: kernel-core-4.18.0-521.el8.x86_64 1/3 2023-11-08T04:38:28.428 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : kernel-modules-4.18.0-521.el8.x86_64 2/3 2023-11-08T04:38:30.499 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: kernel-modules-4.18.0-521.el8.x86_64 2/3 2023-11-08T04:38:30.780 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : kernel-4.18.0-521.el8.x86_64 3/3 2023-11-08T04:38:31.860 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: kernel-modules-4.18.0-521.el8.x86_64 2/3 2023-11-08T04:38:32.149 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : kernel-4.18.0-521.el8.x86_64 3/3 2023-11-08T04:38:52.579 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: kernel-core-4.18.0-521.el8.x86_64 3/3 2023-11-08T04:38:52.602 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: kernel-modules-4.18.0-521.el8.x86_64 3/3 2023-11-08T04:38:52.931 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: kernel-4.18.0-521.el8.x86_64 3/3 2023-11-08T04:38:52.931 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : kernel-4.18.0-521.el8.x86_64 1/3 2023-11-08T04:38:52.932 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : kernel-core-4.18.0-521.el8.x86_64 2/3 2023-11-08T04:38:53.702 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : kernel-modules-4.18.0-521.el8.x86_64 3/3 2023-11-08T04:38:53.703 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T04:38:53.703 INFO:teuthology.orchestra.run.smithi087.stdout:Installed: 2023-11-08T04:38:53.703 INFO:teuthology.orchestra.run.smithi087.stdout: kernel-4.18.0-521.el8.x86_64 kernel-core-4.18.0-521.el8.x86_64 2023-11-08T04:38:53.703 INFO:teuthology.orchestra.run.smithi087.stdout: kernel-modules-4.18.0-521.el8.x86_64 2023-11-08T04:38:53.704 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T04:38:53.704 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2023-11-08T04:38:53.845 DEBUG:teuthology.orchestra.run.smithi087:> rpm -q kernel | sort -rV | head -n 1 2023-11-08T04:38:53.977 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: kernel-core-4.18.0-521.el8.x86_64 3/3 2023-11-08T04:38:53.999 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: kernel-modules-4.18.0-521.el8.x86_64 3/3 2023-11-08T04:38:54.103 INFO:teuthology.orchestra.run.smithi087.stdout:kernel-4.18.0-521.el8.x86_64 2023-11-08T04:38:54.104 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-521.el8.x86_64 2023-11-08T04:38:54.104 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-521.el8.x86_64 2023-11-08T04:38:54.104 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-11-08T04:38:54.105 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-11-08T04:38:54.105 INFO:teuthology.task.kernel:Installing distro kernel on cluster2.mon.a... 2023-11-08T04:38:54.105 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi087.front.sepia.ceph.com, path=None, version=distro) 2023-11-08T04:38:54.105 DEBUG:teuthology.orchestra.run.smithi087:> rpm -q kernel | sort -rV | head -n 1 2023-11-08T04:38:54.314 INFO:teuthology.orchestra.run.smithi087.stdout:kernel-4.18.0-521.el8.x86_64 2023-11-08T04:38:54.315 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-521.el8.x86_64 2023-11-08T04:38:54.315 DEBUG:teuthology.orchestra.run.smithi087:> sudo rpm -qi grub2-tools 2023-11-08T04:38:54.349 INFO:teuthology.orchestra.run.smithi087.stdout:Name : grub2-tools 2023-11-08T04:38:54.349 INFO:teuthology.orchestra.run.smithi087.stdout:Epoch : 1 2023-11-08T04:38:54.350 INFO:teuthology.orchestra.run.smithi087.stdout:Version : 2.02 2023-11-08T04:38:54.350 INFO:teuthology.orchestra.run.smithi087.stdout:Release : 129.el8 2023-11-08T04:38:54.350 INFO:teuthology.orchestra.run.smithi087.stdout:Architecture: x86_64 2023-11-08T04:38:54.350 INFO:teuthology.orchestra.run.smithi087.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2023-11-08T04:38:54.350 INFO:teuthology.orchestra.run.smithi087.stdout:Group : System Environment/Base 2023-11-08T04:38:54.351 INFO:teuthology.orchestra.run.smithi087.stdout:Size : 9265586 2023-11-08T04:38:54.351 INFO:teuthology.orchestra.run.smithi087.stdout:License : GPLv3+ 2023-11-08T04:38:54.351 INFO:teuthology.orchestra.run.smithi087.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2023-11-08T04:38:54.351 INFO:teuthology.orchestra.run.smithi087.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2023-11-08T04:38:54.351 INFO:teuthology.orchestra.run.smithi087.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2023-11-08T04:38:54.352 INFO:teuthology.orchestra.run.smithi087.stdout:Build Host : kbuilder.bsys.centos.org 2023-11-08T04:38:54.352 INFO:teuthology.orchestra.run.smithi087.stdout:Relocations : (not relocatable) 2023-11-08T04:38:54.352 INFO:teuthology.orchestra.run.smithi087.stdout:Packager : CentOS BuildSystem 2023-11-08T04:38:54.352 INFO:teuthology.orchestra.run.smithi087.stdout:Vendor : CentOS 2023-11-08T04:38:54.352 INFO:teuthology.orchestra.run.smithi087.stdout:URL : http://www.gnu.org/software/grub/ 2023-11-08T04:38:54.353 INFO:teuthology.orchestra.run.smithi087.stdout:Summary : Support tools for GRUB. 2023-11-08T04:38:54.353 INFO:teuthology.orchestra.run.smithi087.stdout:Description : 2023-11-08T04:38:54.353 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T04:38:54.353 INFO:teuthology.orchestra.run.smithi087.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-11-08T04:38:54.353 INFO:teuthology.orchestra.run.smithi087.stdout:customizable bootloader with modular architecture. It supports a rich 2023-11-08T04:38:54.354 INFO:teuthology.orchestra.run.smithi087.stdout:variety of kernel formats, file systems, computer architectures and 2023-11-08T04:38:54.354 INFO:teuthology.orchestra.run.smithi087.stdout:hardware devices. 2023-11-08T04:38:54.354 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T04:38:54.354 INFO:teuthology.orchestra.run.smithi087.stdout:This subpackage provides tools for support of all platforms. 2023-11-08T04:38:54.355 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: kernel-4.18.0-521.el8.x86_64 3/3 2023-11-08T04:38:54.356 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : kernel-4.18.0-521.el8.x86_64 1/3 2023-11-08T04:38:54.356 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : kernel-core-4.18.0-521.el8.x86_64 2/3 2023-11-08T04:38:54.356 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-11-08T04:38:54.357 INFO:teuthology.task.kernel:Updating grub on smithi087 to boot 4.18.0-521.el8.x86_64 2023-11-08T04:38:54.357 DEBUG:teuthology.orchestra.run.smithi087:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-11-08T04:38:55.116 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : kernel-modules-4.18.0-521.el8.x86_64 3/3 2023-11-08T04:38:55.116 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-11-08T04:38:55.116 INFO:teuthology.orchestra.run.smithi077.stdout:Installed: 2023-11-08T04:38:55.117 INFO:teuthology.orchestra.run.smithi077.stdout: kernel-4.18.0-521.el8.x86_64 kernel-core-4.18.0-521.el8.x86_64 2023-11-08T04:38:55.117 INFO:teuthology.orchestra.run.smithi077.stdout: kernel-modules-4.18.0-521.el8.x86_64 2023-11-08T04:38:55.117 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-11-08T04:38:55.117 INFO:teuthology.orchestra.run.smithi077.stdout:Complete! 2023-11-08T04:38:55.271 DEBUG:teuthology.orchestra.run.smithi077:> rpm -q kernel | sort -rV | head -n 1 2023-11-08T04:38:55.488 INFO:teuthology.orchestra.run.smithi077.stdout:kernel-4.18.0-521.el8.x86_64 2023-11-08T04:38:55.489 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-521.el8.x86_64 2023-11-08T04:38:55.489 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-521.el8.x86_64 2023-11-08T04:38:55.489 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-11-08T04:38:55.490 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-11-08T04:38:55.490 INFO:teuthology.task.kernel:Installing distro kernel on cluster1.mon.a... 2023-11-08T04:38:55.490 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi077.front.sepia.ceph.com, path=None, version=distro) 2023-11-08T04:38:55.490 DEBUG:teuthology.orchestra.run.smithi077:> rpm -q kernel | sort -rV | head -n 1 2023-11-08T04:38:55.698 INFO:teuthology.orchestra.run.smithi077.stdout:kernel-4.18.0-521.el8.x86_64 2023-11-08T04:38:55.699 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-521.el8.x86_64 2023-11-08T04:38:55.699 DEBUG:teuthology.orchestra.run.smithi077:> sudo rpm -qi grub2-tools 2023-11-08T04:38:55.732 INFO:teuthology.orchestra.run.smithi077.stdout:Name : grub2-tools 2023-11-08T04:38:55.733 INFO:teuthology.orchestra.run.smithi077.stdout:Epoch : 1 2023-11-08T04:38:55.733 INFO:teuthology.orchestra.run.smithi077.stdout:Version : 2.02 2023-11-08T04:38:55.733 INFO:teuthology.orchestra.run.smithi077.stdout:Release : 129.el8 2023-11-08T04:38:55.733 INFO:teuthology.orchestra.run.smithi077.stdout:Architecture: x86_64 2023-11-08T04:38:55.733 INFO:teuthology.orchestra.run.smithi077.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2023-11-08T04:38:55.734 INFO:teuthology.orchestra.run.smithi077.stdout:Group : System Environment/Base 2023-11-08T04:38:55.734 INFO:teuthology.orchestra.run.smithi077.stdout:Size : 9265586 2023-11-08T04:38:55.734 INFO:teuthology.orchestra.run.smithi077.stdout:License : GPLv3+ 2023-11-08T04:38:55.734 INFO:teuthology.orchestra.run.smithi077.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2023-11-08T04:38:55.734 INFO:teuthology.orchestra.run.smithi077.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2023-11-08T04:38:55.735 INFO:teuthology.orchestra.run.smithi077.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2023-11-08T04:38:55.735 INFO:teuthology.orchestra.run.smithi077.stdout:Build Host : kbuilder.bsys.centos.org 2023-11-08T04:38:55.735 INFO:teuthology.orchestra.run.smithi077.stdout:Relocations : (not relocatable) 2023-11-08T04:38:55.735 INFO:teuthology.orchestra.run.smithi077.stdout:Packager : CentOS BuildSystem 2023-11-08T04:38:55.735 INFO:teuthology.orchestra.run.smithi077.stdout:Vendor : CentOS 2023-11-08T04:38:55.735 INFO:teuthology.orchestra.run.smithi077.stdout:URL : http://www.gnu.org/software/grub/ 2023-11-08T04:38:55.736 INFO:teuthology.orchestra.run.smithi077.stdout:Summary : Support tools for GRUB. 2023-11-08T04:38:55.736 INFO:teuthology.orchestra.run.smithi077.stdout:Description : 2023-11-08T04:38:55.736 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-11-08T04:38:55.736 INFO:teuthology.orchestra.run.smithi077.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-11-08T04:38:55.736 INFO:teuthology.orchestra.run.smithi077.stdout:customizable bootloader with modular architecture. It supports a rich 2023-11-08T04:38:55.737 INFO:teuthology.orchestra.run.smithi077.stdout:variety of kernel formats, file systems, computer architectures and 2023-11-08T04:38:55.737 INFO:teuthology.orchestra.run.smithi077.stdout:hardware devices. 2023-11-08T04:38:55.737 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-11-08T04:38:55.737 INFO:teuthology.orchestra.run.smithi077.stdout:This subpackage provides tools for support of all platforms. 2023-11-08T04:38:55.738 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-11-08T04:38:55.738 INFO:teuthology.task.kernel:Updating grub on smithi077 to boot 4.18.0-521.el8.x86_64 2023-11-08T04:38:55.738 DEBUG:teuthology.orchestra.run.smithi077:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-11-08T04:38:56.184 INFO:teuthology.orchestra.run.smithi087.stderr:Generating grub configuration file ... 2023-11-08T04:38:57.280 INFO:teuthology.orchestra.run.smithi077.stderr:Generating grub configuration file ... 2023-11-08T04:38:58.319 INFO:teuthology.orchestra.run.smithi087.stderr:done 2023-11-08T04:38:58.329 DEBUG:teuthology.orchestra.run.smithi087:> mktemp 2023-11-08T04:38:58.365 INFO:teuthology.orchestra.run.smithi087.stdout:/tmp/tmp.KQEPq5gVDn 2023-11-08T04:38:58.365 DEBUG:teuthology.orchestra.run.smithi087:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.KQEPq5gVDn 2023-11-08T04:38:58.439 DEBUG:teuthology.orchestra.run.smithi087:> sudo chmod 0666 /tmp/tmp.KQEPq5gVDn 2023-11-08T04:38:58.586 DEBUG:teuthology.orchestra.remote:smithi087:/tmp/tmp.KQEPq5gVDn is 6KB 2023-11-08T04:38:58.635 DEBUG:teuthology.orchestra.run.smithi087:> rm -fr /tmp/tmp.KQEPq5gVDn 2023-11-08T04:38:58.653 DEBUG:teuthology.orchestra.run.smithi087:> sudo /bin/ls /boot/loader/entries || true 2023-11-08T04:38:58.741 INFO:teuthology.orchestra.run.smithi087.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2023-11-08T04:38:58.741 INFO:teuthology.orchestra.run.smithi087.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2023-11-08T04:38:58.742 INFO:teuthology.orchestra.run.smithi087.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2023-11-08T04:38:58.742 INFO:teuthology.orchestra.run.smithi087.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-521.el8.x86_64.conf 2023-11-08T04:38:58.743 DEBUG:teuthology.orchestra.run.smithi087:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-521.el8.x86_64 2023-11-08T04:38:58.863 DEBUG:teuthology.orchestra.run.smithi087:> sudo shutdown -r now 2023-11-08T04:39:00.070 INFO:teuthology.orchestra.run.smithi077.stderr:done 2023-11-08T04:39:00.072 DEBUG:teuthology.orchestra.run.smithi077:> mktemp 2023-11-08T04:39:00.098 INFO:teuthology.orchestra.run.smithi077.stdout:/tmp/tmp.Lpzjcn3gvW 2023-11-08T04:39:00.099 DEBUG:teuthology.orchestra.run.smithi077:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.Lpzjcn3gvW 2023-11-08T04:39:00.172 DEBUG:teuthology.orchestra.run.smithi077:> sudo chmod 0666 /tmp/tmp.Lpzjcn3gvW 2023-11-08T04:39:00.328 DEBUG:teuthology.orchestra.remote:smithi077:/tmp/tmp.Lpzjcn3gvW is 6KB 2023-11-08T04:39:00.378 DEBUG:teuthology.orchestra.run.smithi077:> rm -fr /tmp/tmp.Lpzjcn3gvW 2023-11-08T04:39:00.397 DEBUG:teuthology.orchestra.run.smithi077:> sudo /bin/ls /boot/loader/entries || true 2023-11-08T04:39:00.485 INFO:teuthology.orchestra.run.smithi077.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2023-11-08T04:39:00.485 INFO:teuthology.orchestra.run.smithi077.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2023-11-08T04:39:00.486 INFO:teuthology.orchestra.run.smithi077.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2023-11-08T04:39:00.486 INFO:teuthology.orchestra.run.smithi077.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-521.el8.x86_64.conf 2023-11-08T04:39:00.488 DEBUG:teuthology.orchestra.run.smithi077:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-521.el8.x86_64 2023-11-08T04:39:00.600 DEBUG:teuthology.orchestra.run.smithi077:> sudo shutdown -r now 2023-11-08T04:39:28.894 INFO:teuthology.task.kernel:Checking client cluster2.mon.a for new kernel version... 2023-11-08T04:39:28.895 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi087.front.sepia.ceph.com' 2023-11-08T04:39:28.896 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi087.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-08T04:39:30.606 INFO:teuthology.task.kernel:Checking client cluster1.mon.a for new kernel version... 2023-11-08T04:39:30.607 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi077.front.sepia.ceph.com' 2023-11-08T04:39:30.607 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi077.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-08T04:39:47.326 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.87 2023-11-08T04:39:49.118 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.77 2023-11-08T04:39:56.334 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi087.front.sepia.ceph.com' 2023-11-08T04:39:56.335 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi087.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-08T04:39:58.121 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi077.front.sepia.ceph.com' 2023-11-08T04:39:58.121 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi077.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-08T04:39:59.390 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.87 2023-11-08T04:40:01.178 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.77 2023-11-08T04:40:11.402 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi087.front.sepia.ceph.com' 2023-11-08T04:40:11.403 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi087.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-08T04:40:13.181 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi077.front.sepia.ceph.com' 2023-11-08T04:40:13.181 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi077.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-08T04:40:29.822 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.87 2023-11-08T04:40:31.614 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.77 2023-11-08T04:40:44.834 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi087.front.sepia.ceph.com' 2023-11-08T04:40:44.835 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi087.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-08T04:40:45.116 DEBUG:teuthology.orchestra.run.smithi087:> true 2023-11-08T04:40:45.436 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi087.front.sepia.ceph.com' 2023-11-08T04:40:45.436 INFO:teuthology.task.kernel:Checking kernel version of cluster2.mon.a, want "4.18.0-521.el8.x86_64"... 2023-11-08T04:40:45.437 DEBUG:teuthology.orchestra.run.smithi087:> uname -r 2023-11-08T04:40:45.453 INFO:teuthology.orchestra.run.smithi087.stdout:4.18.0-521.el8.x86_64 2023-11-08T04:40:45.453 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-521.el8.x86_64 vs 4.18.0-521.el8.x86_64 2023-11-08T04:40:45.454 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-11-08T04:40:45.454 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2023-11-08T04:40:46.455 INFO:teuthology.task.kernel:Enabling kdb on cluster2.mon.a... 2023-11-08T04:40:46.456 DEBUG:teuthology.orchestra.run.smithi087:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-11-08T04:40:46.616 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi077.front.sepia.ceph.com' 2023-11-08T04:40:46.616 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi077.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-08T04:40:47.220 INFO:teuthology.orchestra.run.smithi087.stdout:ttyS1 2023-11-08T04:40:47.256 DEBUG:teuthology.parallel:result is None 2023-11-08T04:40:47.984 DEBUG:teuthology.orchestra.run.smithi077:> true 2023-11-08T04:40:48.321 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi077.front.sepia.ceph.com' 2023-11-08T04:40:48.322 INFO:teuthology.task.kernel:Checking kernel version of cluster1.mon.a, want "4.18.0-521.el8.x86_64"... 2023-11-08T04:40:48.322 DEBUG:teuthology.orchestra.run.smithi077:> uname -r 2023-11-08T04:40:48.380 INFO:teuthology.orchestra.run.smithi077.stdout:4.18.0-521.el8.x86_64 2023-11-08T04:40:48.381 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-521.el8.x86_64 vs 4.18.0-521.el8.x86_64 2023-11-08T04:40:48.381 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-11-08T04:40:48.381 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2023-11-08T04:40:49.382 INFO:teuthology.task.kernel:Enabling kdb on cluster1.mon.a... 2023-11-08T04:40:49.383 DEBUG:teuthology.orchestra.run.smithi077:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-11-08T04:40:49.450 INFO:teuthology.orchestra.run.smithi077.stdout:ttyS1 2023-11-08T04:40:49.462 DEBUG:teuthology.parallel:result is None 2023-11-08T04:40:49.463 INFO:teuthology.run_tasks:Running task internal.base... 2023-11-08T04:40:49.470 INFO:teuthology.task.internal:Creating test directory... 2023-11-08T04:40:49.471 DEBUG:teuthology.orchestra.run.smithi077:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-11-08T04:40:49.473 DEBUG:teuthology.orchestra.run.smithi087:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-11-08T04:40:49.497 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-11-08T04:40:49.513 INFO:teuthology.run_tasks:Running task internal.archive... 2023-11-08T04:40:49.547 INFO:teuthology.task.internal:Creating archive directory... 2023-11-08T04:40:49.547 DEBUG:teuthology.orchestra.run.smithi077:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-11-08T04:40:49.550 DEBUG:teuthology.orchestra.run.smithi087:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-11-08T04:40:49.596 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-11-08T04:40:49.605 INFO:teuthology.task.internal:Enabling coredump saving... 2023-11-08T04:40:49.606 DEBUG:teuthology.orchestra.run.smithi077:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-11-08T04:40:49.619 DEBUG:teuthology.orchestra.run.smithi087:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-11-08T04:40:49.652 INFO:teuthology.orchestra.run.smithi077.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-08T04:40:49.662 INFO:teuthology.orchestra.run.smithi077.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-08T04:40:49.677 INFO:teuthology.orchestra.run.smithi087.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-08T04:40:49.688 INFO:teuthology.orchestra.run.smithi087.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-08T04:40:49.689 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-11-08T04:40:49.698 INFO:teuthology.task.internal:Configuring sudo... 2023-11-08T04:40:49.699 DEBUG:teuthology.orchestra.run.smithi077:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-11-08T04:40:49.705 DEBUG:teuthology.orchestra.run.smithi087:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-11-08T04:40:49.736 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-11-08T04:40:49.745 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-11-08T04:40:49.746 DEBUG:teuthology.orchestra.run.smithi077:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-11-08T04:40:49.774 DEBUG:teuthology.orchestra.run.smithi087:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-11-08T04:40:49.794 DEBUG:teuthology.orchestra.run.smithi077:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-11-08T04:40:49.879 DEBUG:teuthology.orchestra.run.smithi077:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-11-08T04:40:49.927 DEBUG:teuthology.orchestra.run.smithi077:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-11-08T04:40:50.013 DEBUG:teuthology.orchestra.run.smithi077:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-11-08T04:40:50.054 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2023-11-08T04:40:50.054 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-11-08T04:40:50.120 DEBUG:teuthology.orchestra.run.smithi087:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-11-08T04:40:50.169 DEBUG:teuthology.orchestra.run.smithi087:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-11-08T04:40:50.218 DEBUG:teuthology.orchestra.run.smithi087:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-11-08T04:40:50.305 DEBUG:teuthology.orchestra.run.smithi087:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-11-08T04:40:50.348 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2023-11-08T04:40:50.349 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-11-08T04:40:50.418 DEBUG:teuthology.orchestra.run.smithi077:> sudo service rsyslog restart 2023-11-08T04:40:50.420 DEBUG:teuthology.orchestra.run.smithi087:> sudo service rsyslog restart 2023-11-08T04:40:50.476 INFO:teuthology.orchestra.run.smithi077.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-11-08T04:40:50.518 INFO:teuthology.orchestra.run.smithi087.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-11-08T04:40:50.997 INFO:teuthology.run_tasks:Running task internal.timer... 2023-11-08T04:40:51.007 INFO:teuthology.task.internal:Starting timer... 2023-11-08T04:40:51.007 INFO:teuthology.run_tasks:Running task pcp... 2023-11-08T04:40:51.056 INFO:teuthology.run_tasks:Running task selinux... 2023-11-08T04:40:51.086 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2023-11-08T04:40:51.087 DEBUG:teuthology.orchestra.run.smithi077:> sudo service auditd rotate 2023-11-08T04:40:51.194 INFO:teuthology.orchestra.run.smithi077.stdout:Rotating logs: 2023-11-08T04:40:51.195 DEBUG:teuthology.orchestra.run.smithi087:> sudo service auditd rotate 2023-11-08T04:40:51.293 INFO:teuthology.orchestra.run.smithi087.stdout:Rotating logs: 2023-11-08T04:40:51.295 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-11-08T04:40:51.295 DEBUG:teuthology.orchestra.run.smithi077:> /usr/sbin/getenforce 2023-11-08T04:40:51.321 INFO:teuthology.orchestra.run.smithi077.stdout:Permissive 2023-11-08T04:40:51.321 DEBUG:teuthology.orchestra.run.smithi087:> /usr/sbin/getenforce 2023-11-08T04:40:51.376 INFO:teuthology.orchestra.run.smithi087.stdout:Permissive 2023-11-08T04:40:51.377 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi077.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi087.front.sepia.ceph.com': 'permissive'} 2023-11-08T04:40:51.377 DEBUG:teuthology.orchestra.run.smithi077:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-11-08T04:40:51.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-08T04:40:51.405 DEBUG:teuthology.orchestra.run.smithi087:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-11-08T04:40:51.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-08T04:40:51.434 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-11-08T04:40:51.434 DEBUG:teuthology.orchestra.run.smithi077:> sudo /usr/sbin/setenforce permissive 2023-11-08T04:40:51.477 DEBUG:teuthology.orchestra.run.smithi087:> sudo /usr/sbin/setenforce permissive 2023-11-08T04:40:51.504 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-11-08T04:40:51.517 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2023-11-08T04:40:51.522 INFO:teuthology.repo_utils:Fetching main from origin 2023-11-08T04:40:51.684 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-11-08T04:40:51.701 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-11-08T04:40:51.702 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi077.front.sepia.ceph.com,smithi087.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-11-08T04:48:08.994 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi077.front.sepia.ceph.com'), Remote(name='ubuntu@smithi087.front.sepia.ceph.com')] 2023-11-08T04:48:08.996 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi077.front.sepia.ceph.com' 2023-11-08T04:48:08.997 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi077.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-08T04:48:09.067 DEBUG:teuthology.orchestra.run.smithi077:> true 2023-11-08T04:48:09.136 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi077.front.sepia.ceph.com' 2023-11-08T04:48:09.137 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi087.front.sepia.ceph.com' 2023-11-08T04:48:09.137 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi087.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-08T04:48:09.204 DEBUG:teuthology.orchestra.run.smithi087:> true 2023-11-08T04:48:09.272 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi087.front.sepia.ceph.com' 2023-11-08T04:48:09.272 INFO:teuthology.run_tasks:Running task clock... 2023-11-08T04:48:09.283 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2023-11-08T04:48:09.284 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-11-08T04:48:09.284 DEBUG:teuthology.orchestra.run.smithi077:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-11-08T04:48:09.287 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-11-08T04:48:09.287 DEBUG:teuthology.orchestra.run.smithi087:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-11-08T04:48:09.317 INFO:teuthology.orchestra.run.smithi077.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-11-08T04:48:09.331 INFO:teuthology.orchestra.run.smithi077.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-11-08T04:48:09.344 INFO:teuthology.orchestra.run.smithi087.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-11-08T04:48:09.357 INFO:teuthology.orchestra.run.smithi087.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-11-08T04:48:09.395 INFO:teuthology.orchestra.run.smithi077.stderr:sudo: ntpd: command not found 2023-11-08T04:48:09.406 INFO:teuthology.orchestra.run.smithi077.stdout:506 Cannot talk to daemon 2023-11-08T04:48:09.419 INFO:teuthology.orchestra.run.smithi087.stderr:sudo: ntpd: command not found 2023-11-08T04:48:09.420 INFO:teuthology.orchestra.run.smithi077.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-11-08T04:48:09.430 INFO:teuthology.orchestra.run.smithi087.stdout:506 Cannot talk to daemon 2023-11-08T04:48:09.433 INFO:teuthology.orchestra.run.smithi077.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-11-08T04:48:09.443 INFO:teuthology.orchestra.run.smithi087.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-11-08T04:48:09.456 INFO:teuthology.orchestra.run.smithi087.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-11-08T04:48:09.510 INFO:teuthology.orchestra.run.smithi077.stderr:bash: ntpq: command not found 2023-11-08T04:48:09.514 INFO:teuthology.orchestra.run.smithi077.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-11-08T04:48:09.514 INFO:teuthology.orchestra.run.smithi077.stdout:=============================================================================== 2023-11-08T04:48:09.514 INFO:teuthology.orchestra.run.smithi077.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-08T04:48:09.514 INFO:teuthology.orchestra.run.smithi077.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-08T04:48:09.515 INFO:teuthology.orchestra.run.smithi077.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-08T04:48:09.515 INFO:teuthology.orchestra.run.smithi077.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-08T04:48:09.522 INFO:teuthology.orchestra.run.smithi087.stderr:bash: ntpq: command not found 2023-11-08T04:48:09.526 INFO:teuthology.orchestra.run.smithi087.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-11-08T04:48:09.526 INFO:teuthology.orchestra.run.smithi087.stdout:=============================================================================== 2023-11-08T04:48:09.527 INFO:teuthology.orchestra.run.smithi087.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-08T04:48:09.527 INFO:teuthology.orchestra.run.smithi087.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-08T04:48:09.527 INFO:teuthology.orchestra.run.smithi087.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-08T04:48:09.527 INFO:teuthology.orchestra.run.smithi087.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-08T04:48:09.528 INFO:teuthology.run_tasks:Running task install... 2023-11-08T04:48:09.536 DEBUG:teuthology.task.install:project ceph 2023-11-08T04:48:09.537 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'cfb6fd52113be6a0af3fe7fb89b2e08474eadc0f'}} 2023-11-08T04:48:09.537 DEBUG:teuthology.task.install:config {'extra_packages': ['rbd-mirror'], 'flavor': 'default', 'sha1': 'cfb6fd52113be6a0af3fe7fb89b2e08474eadc0f'} 2023-11-08T04:48:09.537 INFO:teuthology.task.install:Using flavor: default 2023-11-08T04:48:09.544 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2023-11-08T04:48:09.544 INFO:teuthology.task.install:extra packages: ['rbd-mirror'] 2023-11-08T04:48:09.545 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': ['rbd-mirror'], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'cfb6fd52113be6a0af3fe7fb89b2e08474eadc0f', 'tag': None, 'wait_for_package': False} 2023-11-08T04:48:09.545 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=cfb6fd52113be6a0af3fe7fb89b2e08474eadc0f 2023-11-08T04:48:09.547 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': ['rbd-mirror'], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'cfb6fd52113be6a0af3fe7fb89b2e08474eadc0f', 'tag': None, 'wait_for_package': False} 2023-11-08T04:48:09.548 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=cfb6fd52113be6a0af3fe7fb89b2e08474eadc0f 2023-11-08T04:48:09.709 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-11-07-0801-pacific/cfb6fd52113be6a0af3fe7fb89b2e08474eadc0f/centos/8/flavors/default/ 2023-11-08T04:48:09.709 INFO:teuthology.task.install.rpm:Package version is 16.2.14-129.gcfb6fd52 2023-11-08T04:48:09.772 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-11-07-0801-pacific/cfb6fd52113be6a0af3fe7fb89b2e08474eadc0f/centos/8/flavors/default/ 2023-11-08T04:48:09.773 INFO:teuthology.task.install.rpm:Package version is 16.2.14-129.gcfb6fd52 2023-11-08T04:48:09.877 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-11-07-0801-pacific/cfb6fd52113be6a0af3fe7fb89b2e08474eadc0f/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-11-07-0801-pacific/cfb6fd52113be6a0af3fe7fb89b2e08474eadc0f/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-11-07-0801-pacific/cfb6fd52113be6a0af3fe7fb89b2e08474eadc0f/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-11-08T04:48:09.878 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2023-11-08T04:48:09.878 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-11-08T04:48:09.906 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, rbd-mirror on remote rpm x86_64 2023-11-08T04:48:09.906 DEBUG:teuthology.orchestra.run.smithi087:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/cfb6fd52113be6a0af3fe7fb89b2e08474eadc0f/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-11-08T04:48:09.939 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-11-07-0801-pacific/cfb6fd52113be6a0af3fe7fb89b2e08474eadc0f/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-11-07-0801-pacific/cfb6fd52113be6a0af3fe7fb89b2e08474eadc0f/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-yuri3-testing-2023-11-07-0801-pacific/cfb6fd52113be6a0af3fe7fb89b2e08474eadc0f/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-11-08T04:48:09.939 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2023-11-08T04:48:09.940 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-11-08T04:48:09.968 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, rbd-mirror on remote rpm x86_64 2023-11-08T04:48:09.969 DEBUG:teuthology.orchestra.run.smithi077:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/cfb6fd52113be6a0af3fe7fb89b2e08474eadc0f/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-11-08T04:48:09.973 DEBUG:teuthology.orchestra.run.smithi087:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-11-08T04:48:10.037 DEBUG:teuthology.orchestra.run.smithi077:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-11-08T04:48:10.049 DEBUG:teuthology.orchestra.run.smithi087:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-11-08T04:48:10.115 DEBUG:teuthology.orchestra.run.smithi077:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-11-08T04:48:10.115 INFO:teuthology.orchestra.run.smithi087.stdout:check_obsoletes = 1 2023-11-08T04:48:10.117 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum clean all 2023-11-08T04:48:10.181 INFO:teuthology.orchestra.run.smithi077.stdout:check_obsoletes = 1 2023-11-08T04:48:10.183 DEBUG:teuthology.orchestra.run.smithi077:> sudo yum clean all 2023-11-08T04:48:10.384 INFO:teuthology.orchestra.run.smithi087.stdout:76 files removed 2023-11-08T04:48:10.412 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install ceph-radosgw 2023-11-08T04:48:10.452 INFO:teuthology.orchestra.run.smithi077.stdout:76 files removed 2023-11-08T04:48:10.479 DEBUG:teuthology.orchestra.run.smithi077:> sudo yum -y install ceph-radosgw 2023-11-08T04:48:11.108 INFO:teuthology.orchestra.run.smithi087.stdout:ceph packages for x86_64 259 kB/s | 76 kB 00:00 2023-11-08T04:48:11.223 INFO:teuthology.orchestra.run.smithi077.stdout:ceph packages for x86_64 226 kB/s | 76 kB 00:00 2023-11-08T04:48:11.425 INFO:teuthology.orchestra.run.smithi087.stdout:ceph noarch packages 55 kB/s | 15 kB 00:00 2023-11-08T04:48:11.523 INFO:teuthology.orchestra.run.smithi077.stdout:ceph noarch packages 58 kB/s | 15 kB 00:00 2023-11-08T04:48:11.801 INFO:teuthology.orchestra.run.smithi087.stdout:ceph source packages 4.1 kB/s | 1.4 kB 00:00 2023-11-08T04:48:11.882 INFO:teuthology.orchestra.run.smithi077.stdout:ceph source packages 4.3 kB/s | 1.4 kB 00:00 2023-11-08T04:48:14.642 INFO:teuthology.orchestra.run.smithi087.stdout:CentOS Stream 8 - AppStream 12 MB/s | 34 MB 00:02 2023-11-08T04:48:14.741 INFO:teuthology.orchestra.run.smithi077.stdout:CentOS Stream 8 - AppStream 12 MB/s | 34 MB 00:02 2023-11-08T04:48:23.250 INFO:teuthology.orchestra.run.smithi077.stdout:CentOS Stream 8 - BaseOS 23 MB/s | 53 MB 00:02 2023-11-08T04:48:23.660 INFO:teuthology.orchestra.run.smithi087.stdout:CentOS Stream 8 - BaseOS 19 MB/s | 53 MB 00:02 2023-11-08T04:48:31.233 INFO:teuthology.orchestra.run.smithi077.stdout:CentOS Stream 8 - Extras 41 kB/s | 18 kB 00:00 2023-11-08T04:48:31.676 INFO:teuthology.orchestra.run.smithi077.stdout:CentOS Stream 8 - Extras common packages 26 kB/s | 6.9 kB 00:00 2023-11-08T04:48:31.768 INFO:teuthology.orchestra.run.smithi087.stdout:CentOS Stream 8 - Extras 34 kB/s | 18 kB 00:00 2023-11-08T04:48:32.319 INFO:teuthology.orchestra.run.smithi087.stdout:CentOS Stream 8 - Extras common packages 19 kB/s | 6.9 kB 00:00 2023-11-08T04:48:32.503 INFO:teuthology.orchestra.run.smithi077.stdout:CentOS Stream 8 - PowerTools 9.8 MB/s | 6.6 MB 00:00 2023-11-08T04:48:33.045 INFO:teuthology.orchestra.run.smithi087.stdout:CentOS Stream 8 - PowerTools 12 MB/s | 6.6 MB 00:00 2023-11-08T04:48:33.937 INFO:teuthology.orchestra.run.smithi077.stdout:Copr repo for python3-asyncssh owned by ceph 25 kB/s | 3.5 kB 00:00 2023-11-08T04:48:34.463 INFO:teuthology.orchestra.run.smithi087.stdout:Copr repo for python3-asyncssh owned by ceph 30 kB/s | 3.5 kB 00:00 2023-11-08T04:48:35.648 INFO:teuthology.orchestra.run.smithi087.stdout:Extra Packages for Enterprise Linux 15 MB/s | 16 MB 00:01 2023-11-08T04:48:38.929 INFO:teuthology.orchestra.run.smithi077.stdout:Extra Packages for Enterprise Linux 3.3 MB/s | 16 MB 00:04 2023-11-08T04:48:39.514 INFO:teuthology.orchestra.run.smithi087.stdout:lab-extras 509 kB/s | 24 kB 00:00 2023-11-08T04:48:42.762 INFO:teuthology.orchestra.run.smithi077.stdout:lab-extras 466 kB/s | 24 kB 00:00 2023-11-08T04:48:44.066 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2023-11-08T04:48:44.068 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2023-11-08T04:48:44.069 INFO:teuthology.orchestra.run.smithi087.stdout: Package Arch Version Repository Size 2023-11-08T04:48:44.069 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2023-11-08T04:48:44.069 INFO:teuthology.orchestra.run.smithi087.stdout:Installing: 2023-11-08T04:48:44.069 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-radosgw x86_64 2:16.2.14-129.gcfb6fd52.el8 ceph 12 M 2023-11-08T04:48:44.069 INFO:teuthology.orchestra.run.smithi087.stdout:Upgrading: 2023-11-08T04:48:44.070 INFO:teuthology.orchestra.run.smithi087.stdout: librados2 x86_64 2:16.2.14-129.gcfb6fd52.el8 ceph 3.7 M 2023-11-08T04:48:44.070 INFO:teuthology.orchestra.run.smithi087.stdout: librbd1 x86_64 2:16.2.14-129.gcfb6fd52.el8 ceph 4.2 M 2023-11-08T04:48:44.070 INFO:teuthology.orchestra.run.smithi087.stdout:Installing dependencies: 2023-11-08T04:48:44.070 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-base x86_64 2:16.2.14-129.gcfb6fd52.el8 ceph 6.1 M 2023-11-08T04:48:44.071 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-common x86_64 2:16.2.14-129.gcfb6fd52.el8 ceph 24 M 2023-11-08T04:48:44.071 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-selinux x86_64 2:16.2.14-129.gcfb6fd52.el8 ceph 26 k 2023-11-08T04:48:44.071 INFO:teuthology.orchestra.run.smithi087.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2023-11-08T04:48:44.071 INFO:teuthology.orchestra.run.smithi087.stdout: leveldb x86_64 1.22-1.el8 epel 181 k 2023-11-08T04:48:44.071 INFO:teuthology.orchestra.run.smithi087.stdout: libcephfs2 x86_64 2:16.2.14-129.gcfb6fd52.el8 ceph 824 k 2023-11-08T04:48:44.072 INFO:teuthology.orchestra.run.smithi087.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2023-11-08T04:48:44.072 INFO:teuthology.orchestra.run.smithi087.stdout: librabbitmq x86_64 0.9.0-4.el8 baseos 47 k 2023-11-08T04:48:44.072 INFO:teuthology.orchestra.run.smithi087.stdout: libradosstriper1 x86_64 2:16.2.14-129.gcfb6fd52.el8 ceph 493 k 2023-11-08T04:48:44.072 INFO:teuthology.orchestra.run.smithi087.stdout: librdkafka x86_64 0.11.4-3.el8 appstream 354 k 2023-11-08T04:48:44.072 INFO:teuthology.orchestra.run.smithi087.stdout: librgw2 x86_64 2:16.2.14-129.gcfb6fd52.el8 ceph 3.8 M 2023-11-08T04:48:44.073 INFO:teuthology.orchestra.run.smithi087.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2023-11-08T04:48:44.073 INFO:teuthology.orchestra.run.smithi087.stdout: lttng-ust x86_64 2.8.1-11.el8 appstream 259 k 2023-11-08T04:48:44.073 INFO:teuthology.orchestra.run.smithi087.stdout: python3-ceph-argparse x86_64 2:16.2.14-129.gcfb6fd52.el8 ceph 45 k 2023-11-08T04:48:44.073 INFO:teuthology.orchestra.run.smithi087.stdout: python3-ceph-common x86_64 2:16.2.14-129.gcfb6fd52.el8 ceph 99 k 2023-11-08T04:48:44.073 INFO:teuthology.orchestra.run.smithi087.stdout: python3-cephfs x86_64 2:16.2.14-129.gcfb6fd52.el8 ceph 215 k 2023-11-08T04:48:44.074 INFO:teuthology.orchestra.run.smithi087.stdout: python3-prettytable noarch 0.7.2-14.el8 appstream 44 k 2023-11-08T04:48:44.074 INFO:teuthology.orchestra.run.smithi087.stdout: python3-rados x86_64 2:16.2.14-129.gcfb6fd52.el8 ceph 390 k 2023-11-08T04:48:44.074 INFO:teuthology.orchestra.run.smithi087.stdout: python3-rbd x86_64 2:16.2.14-129.gcfb6fd52.el8 ceph 369 k 2023-11-08T04:48:44.074 INFO:teuthology.orchestra.run.smithi087.stdout: python3-rgw x86_64 2:16.2.14-129.gcfb6fd52.el8 ceph 114 k 2023-11-08T04:48:44.074 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T04:48:44.075 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction Summary 2023-11-08T04:48:44.075 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2023-11-08T04:48:44.075 INFO:teuthology.orchestra.run.smithi087.stdout:Install 21 Packages 2023-11-08T04:48:44.075 INFO:teuthology.orchestra.run.smithi087.stdout:Upgrade 2 Packages 2023-11-08T04:48:44.075 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T04:48:44.076 INFO:teuthology.orchestra.run.smithi087.stdout:Total download size: 57 M 2023-11-08T04:48:44.076 INFO:teuthology.orchestra.run.smithi087.stdout:Downloading Packages: 2023-11-08T04:48:45.266 INFO:teuthology.orchestra.run.smithi087.stdout:(1/23): ceph-base-16.2.14-129.gcfb6fd52.el8.x86 12 MB/s | 6.1 MB 00:00 2023-11-08T04:48:45.341 INFO:teuthology.orchestra.run.smithi087.stdout:(2/23): ceph-selinux-16.2.14-129.gcfb6fd52.el8. 343 kB/s | 26 kB 00:00 2023-11-08T04:48:45.416 INFO:teuthology.orchestra.run.smithi087.stdout:(3/23): libcephfs2-16.2.14-129.gcfb6fd52.el8.x8 11 MB/s | 824 kB 00:00 2023-11-08T04:48:45.483 INFO:teuthology.orchestra.run.smithi087.stdout:(4/23): libradosstriper1-16.2.14-129.gcfb6fd52. 7.3 MB/s | 493 kB 00:00 2023-11-08T04:48:45.750 INFO:teuthology.orchestra.run.smithi087.stdout:(5/23): librgw2-16.2.14-129.gcfb6fd52.el8.x86_6 14 MB/s | 3.8 MB 00:00 2023-11-08T04:48:45.800 INFO:teuthology.orchestra.run.smithi087.stdout:(6/23): python3-ceph-argparse-16.2.14-129.gcfb6 909 kB/s | 45 kB 00:00 2023-11-08T04:48:45.850 INFO:teuthology.orchestra.run.smithi087.stdout:(7/23): python3-ceph-common-16.2.14-129.gcfb6fd 1.9 MB/s | 99 kB 00:00 2023-11-08T04:48:45.909 INFO:teuthology.orchestra.run.smithi087.stdout:(8/23): python3-cephfs-16.2.14-129.gcfb6fd52.el 3.6 MB/s | 215 kB 00:00 2023-11-08T04:48:45.976 INFO:teuthology.orchestra.run.smithi087.stdout:(9/23): python3-rados-16.2.14-129.gcfb6fd52.el8 5.8 MB/s | 390 kB 00:00 2023-11-08T04:48:46.026 INFO:teuthology.orchestra.run.smithi087.stdout:(10/23): python3-rbd-16.2.14-129.gcfb6fd52.el8. 7.2 MB/s | 369 kB 00:00 2023-11-08T04:48:46.108 INFO:teuthology.orchestra.run.smithi087.stdout:(11/23): python3-rgw-16.2.14-129.gcfb6fd52.el8. 1.4 MB/s | 114 kB 00:00 2023-11-08T04:48:46.552 INFO:teuthology.orchestra.run.smithi087.stdout:(12/23): librdkafka-0.11.4-3.el8.x86_64.rpm 797 kB/s | 354 kB 00:00 2023-11-08T04:48:46.652 INFO:teuthology.orchestra.run.smithi087.stdout:(13/23): lttng-ust-2.8.1-11.el8.x86_64.rpm 2.5 MB/s | 259 kB 00:00 2023-11-08T04:48:46.736 INFO:teuthology.orchestra.run.smithi087.stdout:(14/23): python3-prettytable-0.7.2-14.el8.noarc 527 kB/s | 44 kB 00:00 2023-11-08T04:48:47.011 INFO:teuthology.orchestra.run.smithi087.stdout:(15/23): ceph-radosgw-16.2.14-129.gcfb6fd52.el8 5.2 MB/s | 12 MB 00:02 2023-11-08T04:48:47.045 INFO:teuthology.orchestra.run.smithi087.stdout:(16/23): librabbitmq-0.9.0-4.el8.x86_64.rpm 152 kB/s | 47 kB 00:00 2023-11-08T04:48:47.247 INFO:teuthology.orchestra.run.smithi077.stdout:Dependencies resolved. 2023-11-08T04:48:47.249 INFO:teuthology.orchestra.run.smithi077.stdout:================================================================================ 2023-11-08T04:48:47.249 INFO:teuthology.orchestra.run.smithi077.stdout: Package Arch Version Repository Size 2023-11-08T04:48:47.250 INFO:teuthology.orchestra.run.smithi077.stdout:================================================================================ 2023-11-08T04:48:47.250 INFO:teuthology.orchestra.run.smithi077.stdout:Installing: 2023-11-08T04:48:47.250 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-radosgw x86_64 2:16.2.14-129.gcfb6fd52.el8 ceph 12 M 2023-11-08T04:48:47.250 INFO:teuthology.orchestra.run.smithi077.stdout:Upgrading: 2023-11-08T04:48:47.251 INFO:teuthology.orchestra.run.smithi077.stdout: librados2 x86_64 2:16.2.14-129.gcfb6fd52.el8 ceph 3.7 M 2023-11-08T04:48:47.251 INFO:teuthology.orchestra.run.smithi077.stdout: librbd1 x86_64 2:16.2.14-129.gcfb6fd52.el8 ceph 4.2 M 2023-11-08T04:48:47.251 INFO:teuthology.orchestra.run.smithi077.stdout:Installing dependencies: 2023-11-08T04:48:47.251 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-base x86_64 2:16.2.14-129.gcfb6fd52.el8 ceph 6.1 M 2023-11-08T04:48:47.252 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-common x86_64 2:16.2.14-129.gcfb6fd52.el8 ceph 24 M 2023-11-08T04:48:47.252 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-selinux x86_64 2:16.2.14-129.gcfb6fd52.el8 ceph 26 k 2023-11-08T04:48:47.252 INFO:teuthology.orchestra.run.smithi077.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2023-11-08T04:48:47.252 INFO:teuthology.orchestra.run.smithi077.stdout: leveldb x86_64 1.22-1.el8 epel 181 k 2023-11-08T04:48:47.252 INFO:teuthology.orchestra.run.smithi077.stdout: libcephfs2 x86_64 2:16.2.14-129.gcfb6fd52.el8 ceph 824 k 2023-11-08T04:48:47.253 INFO:teuthology.orchestra.run.smithi077.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2023-11-08T04:48:47.253 INFO:teuthology.orchestra.run.smithi077.stdout: librabbitmq x86_64 0.9.0-4.el8 baseos 47 k 2023-11-08T04:48:47.253 INFO:teuthology.orchestra.run.smithi077.stdout: libradosstriper1 x86_64 2:16.2.14-129.gcfb6fd52.el8 ceph 493 k 2023-11-08T04:48:47.253 INFO:teuthology.orchestra.run.smithi077.stdout: librdkafka x86_64 0.11.4-3.el8 appstream 354 k 2023-11-08T04:48:47.254 INFO:teuthology.orchestra.run.smithi077.stdout: librgw2 x86_64 2:16.2.14-129.gcfb6fd52.el8 ceph 3.8 M 2023-11-08T04:48:47.254 INFO:teuthology.orchestra.run.smithi077.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2023-11-08T04:48:47.254 INFO:teuthology.orchestra.run.smithi077.stdout: lttng-ust x86_64 2.8.1-11.el8 appstream 259 k 2023-11-08T04:48:47.254 INFO:teuthology.orchestra.run.smithi077.stdout: python3-ceph-argparse x86_64 2:16.2.14-129.gcfb6fd52.el8 ceph 45 k 2023-11-08T04:48:47.254 INFO:teuthology.orchestra.run.smithi077.stdout: python3-ceph-common x86_64 2:16.2.14-129.gcfb6fd52.el8 ceph 99 k 2023-11-08T04:48:47.255 INFO:teuthology.orchestra.run.smithi077.stdout: python3-cephfs x86_64 2:16.2.14-129.gcfb6fd52.el8 ceph 215 k 2023-11-08T04:48:47.255 INFO:teuthology.orchestra.run.smithi077.stdout: python3-prettytable noarch 0.7.2-14.el8 appstream 44 k 2023-11-08T04:48:47.255 INFO:teuthology.orchestra.run.smithi077.stdout: python3-rados x86_64 2:16.2.14-129.gcfb6fd52.el8 ceph 390 k 2023-11-08T04:48:47.255 INFO:teuthology.orchestra.run.smithi077.stdout: python3-rbd x86_64 2:16.2.14-129.gcfb6fd52.el8 ceph 369 k 2023-11-08T04:48:47.255 INFO:teuthology.orchestra.run.smithi077.stdout: python3-rgw x86_64 2:16.2.14-129.gcfb6fd52.el8 ceph 114 k 2023-11-08T04:48:47.256 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-11-08T04:48:47.256 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction Summary 2023-11-08T04:48:47.256 INFO:teuthology.orchestra.run.smithi077.stdout:================================================================================ 2023-11-08T04:48:47.256 INFO:teuthology.orchestra.run.smithi077.stdout:Install 21 Packages 2023-11-08T04:48:47.256 INFO:teuthology.orchestra.run.smithi077.stdout:Upgrade 2 Packages 2023-11-08T04:48:47.256 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-11-08T04:48:47.257 INFO:teuthology.orchestra.run.smithi077.stdout:Total download size: 57 M 2023-11-08T04:48:47.257 INFO:teuthology.orchestra.run.smithi077.stdout:Downloading Packages: 2023-11-08T04:48:47.703 INFO:teuthology.orchestra.run.smithi087.stdout:(17/23): ceph-common-16.2.14-129.gcfb6fd52.el8. 8.3 MB/s | 24 MB 00:02 2023-11-08T04:48:48.262 INFO:teuthology.orchestra.run.smithi087.stdout:(18/23): liboath-2.6.2-3.el8.x86_64.rpm 105 kB/s | 59 kB 00:00 2023-11-08T04:48:48.687 INFO:teuthology.orchestra.run.smithi087.stdout:(19/23): libunwind-1.3.1-3.el8.x86_64.rpm 177 kB/s | 75 kB 00:00 2023-11-08T04:48:48.838 INFO:teuthology.orchestra.run.smithi087.stdout:(20/23): leveldb-1.22-1.el8.x86_64.rpm 101 kB/s | 181 kB 00:01 2023-11-08T04:48:49.455 INFO:teuthology.orchestra.run.smithi087.stdout:(21/23): librados2-16.2.14-129.gcfb6fd52.el8.x8 4.9 MB/s | 3.7 MB 00:00 2023-11-08T04:48:50.313 INFO:teuthology.orchestra.run.smithi087.stdout:(22/23): librbd1-16.2.14-129.gcfb6fd52.el8.x86_ 2.8 MB/s | 4.2 MB 00:01 2023-11-08T04:48:50.463 INFO:teuthology.orchestra.run.smithi087.stdout:(23/23): gperftools-libs-2.7-9.el8.x86_64.rpm 89 kB/s | 306 kB 00:03 2023-11-08T04:48:50.466 INFO:teuthology.orchestra.run.smithi087.stdout:-------------------------------------------------------------------------------- 2023-11-08T04:48:50.466 INFO:teuthology.orchestra.run.smithi087.stdout:Total 9.0 MB/s | 57 MB 00:06 2023-11-08T04:48:50.508 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction check 2023-11-08T04:48:50.555 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction check succeeded. 2023-11-08T04:48:50.556 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction test 2023-11-08T04:48:50.855 INFO:teuthology.orchestra.run.smithi077.stdout:(1/23): ceph-base-16.2.14-129.gcfb6fd52.el8.x86 1.8 MB/s | 6.1 MB 00:03 2023-11-08T04:48:50.912 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction test succeeded. 2023-11-08T04:48:50.914 INFO:teuthology.orchestra.run.smithi077.stdout:(2/23): ceph-selinux-16.2.14-129.gcfb6fd52.el8. 442 kB/s | 26 kB 00:00 2023-11-08T04:48:50.915 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction 2023-11-08T04:48:51.281 INFO:teuthology.orchestra.run.smithi077.stdout:(3/23): libcephfs2-16.2.14-129.gcfb6fd52.el8.x8 2.2 MB/s | 824 kB 00:00 2023-11-08T04:48:51.523 INFO:teuthology.orchestra.run.smithi077.stdout:(4/23): libradosstriper1-16.2.14-129.gcfb6fd52. 2.0 MB/s | 493 kB 00:00 2023-11-08T04:48:52.026 INFO:teuthology.orchestra.run.smithi087.stdout: Preparing : 1/1 2023-11-08T04:48:52.217 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : liboath-2.6.2-3.el8.x86_64 1/25 2023-11-08T04:48:52.422 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 2/25 2023-11-08T04:48:52.798 INFO:teuthology.orchestra.run.smithi077.stdout:(5/23): ceph-radosgw-16.2.14-129.gcfb6fd52.el8. 2.2 MB/s | 12 MB 00:05 2023-11-08T04:48:52.821 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 2/25 2023-11-08T04:48:52.840 INFO:teuthology.orchestra.run.smithi077.stdout:(6/23): python3-ceph-argparse-16.2.14-129.gcfb6 1.1 MB/s | 45 kB 00:00 2023-11-08T04:48:52.927 INFO:teuthology.orchestra.run.smithi077.stdout:(7/23): librgw2-16.2.14-129.gcfb6fd52.el8.x86_6 2.7 MB/s | 3.8 MB 00:01 2023-11-08T04:48:52.957 INFO:teuthology.orchestra.run.smithi077.stdout:(8/23): python3-ceph-common-16.2.14-129.gcfb6fd 844 kB/s | 99 kB 00:00 2023-11-08T04:48:53.008 INFO:teuthology.orchestra.run.smithi077.stdout:(9/23): python3-cephfs-16.2.14-129.gcfb6fd52.el 2.6 MB/s | 215 kB 00:00 2023-11-08T04:48:53.031 INFO:teuthology.orchestra.run.smithi087.stdout: Upgrading : librados2-2:16.2.14-129.gcfb6fd52.el8.x86_64 3/25 2023-11-08T04:48:53.058 INFO:teuthology.orchestra.run.smithi077.stdout:(10/23): python3-rados-16.2.14-129.gcfb6fd52.el 3.8 MB/s | 390 kB 00:00 2023-11-08T04:48:53.100 INFO:teuthology.orchestra.run.smithi077.stdout:(11/23): python3-rgw-16.2.14-129.gcfb6fd52.el8. 2.7 MB/s | 114 kB 00:00 2023-11-08T04:48:53.125 INFO:teuthology.orchestra.run.smithi077.stdout:(12/23): python3-rbd-16.2.14-129.gcfb6fd52.el8. 3.1 MB/s | 369 kB 00:00 2023-11-08T04:48:53.129 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: librados2-2:16.2.14-129.gcfb6fd52.el8.x86_64 3/25 2023-11-08T04:48:53.437 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-rados-2:16.2.14-129.gcfb6fd52.el8.x86_64 4/25 2023-11-08T04:48:53.525 INFO:teuthology.orchestra.run.smithi077.stdout:(13/23): librdkafka-0.11.4-3.el8.x86_64.rpm 833 kB/s | 354 kB 00:00 2023-11-08T04:48:53.550 INFO:teuthology.orchestra.run.smithi077.stdout:(14/23): lttng-ust-2.8.1-11.el8.x86_64.rpm 610 kB/s | 259 kB 00:00 2023-11-08T04:48:53.634 INFO:teuthology.orchestra.run.smithi077.stdout:(15/23): python3-prettytable-0.7.2-14.el8.noarc 404 kB/s | 44 kB 00:00 2023-11-08T04:48:53.901 INFO:teuthology.orchestra.run.smithi077.stdout:(16/23): librabbitmq-0.9.0-4.el8.x86_64.rpm 134 kB/s | 47 kB 00:00 2023-11-08T04:48:53.972 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : libcephfs2-2:16.2.14-129.gcfb6fd52.el8.x86_64 5/25 2023-11-08T04:48:54.093 INFO:teuthology.orchestra.run.smithi077.stdout:(17/23): gperftools-libs-2.7-9.el8.x86_64.rpm 668 kB/s | 306 kB 00:00 2023-11-08T04:48:54.193 INFO:teuthology.orchestra.run.smithi077.stdout:(18/23): liboath-2.6.2-3.el8.x86_64.rpm 588 kB/s | 59 kB 00:00 2023-11-08T04:48:54.316 INFO:teuthology.orchestra.run.smithi077.stdout:(19/23): leveldb-1.22-1.el8.x86_64.rpm 436 kB/s | 181 kB 00:00 2023-11-08T04:48:54.344 INFO:teuthology.orchestra.run.smithi077.stdout:(20/23): libunwind-1.3.1-3.el8.x86_64.rpm 499 kB/s | 75 kB 00:00 2023-11-08T04:48:54.412 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: libcephfs2-2:16.2.14-129.gcfb6fd52.el8.x86_64 5/25 2023-11-08T04:48:54.639 INFO:teuthology.orchestra.run.smithi087.stdout: Upgrading : librbd1-2:16.2.14-129.gcfb6fd52.el8.x86_64 6/25 2023-11-08T04:48:54.693 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: librbd1-2:16.2.14-129.gcfb6fd52.el8.x86_64 6/25 2023-11-08T04:48:54.920 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : librabbitmq-0.9.0-4.el8.x86_64 7/25 2023-11-08T04:48:54.986 INFO:teuthology.orchestra.run.smithi077.stdout:(21/23): librados2-16.2.14-129.gcfb6fd52.el8.x8 5.6 MB/s | 3.7 MB 00:00 2023-11-08T04:48:55.078 INFO:teuthology.orchestra.run.smithi077.stdout:(22/23): librbd1-16.2.14-129.gcfb6fd52.el8.x86_ 5.7 MB/s | 4.2 MB 00:00 2023-11-08T04:48:55.088 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 8/25 2023-11-08T04:48:55.469 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 8/25 2023-11-08T04:48:55.663 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : librgw2-2:16.2.14-129.gcfb6fd52.el8.x86_64 9/25 2023-11-08T04:48:55.742 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: librgw2-2:16.2.14-129.gcfb6fd52.el8.x86_64 9/25 2023-11-08T04:48:55.936 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : leveldb-1.22-1.el8.x86_64 10/25 2023-11-08T04:48:56.135 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-ceph-argparse-2:16.2.14-129.gcfb6fd52.el8. 11/25 2023-11-08T04:48:56.308 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-cephfs-2:16.2.14-129.gcfb6fd52.el8.x86_64 12/25 2023-11-08T04:48:56.572 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-rgw-2:16.2.14-129.gcfb6fd52.el8.x86_64 13/25 2023-11-08T04:48:56.796 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-rbd-2:16.2.14-129.gcfb6fd52.el8.x86_64 14/25 2023-11-08T04:48:56.980 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : libradosstriper1-2:16.2.14-129.gcfb6fd52.el8.x86_6 15/25 2023-11-08T04:48:57.045 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: libradosstriper1-2:16.2.14-129.gcfb6fd52.el8.x86_6 15/25 2023-11-08T04:48:57.272 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 16/25 2023-11-08T04:48:57.455 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 17/25 2023-11-08T04:48:57.682 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 18/25 2023-11-08T04:48:57.855 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-ceph-common-2:16.2.14-129.gcfb6fd52.el8.x8 19/25 2023-11-08T04:48:58.336 INFO:teuthology.orchestra.run.smithi077.stdout:(23/23): ceph-common-16.2.14-129.gcfb6fd52.el8. 2.2 MB/s | 24 MB 00:10 2023-11-08T04:48:58.339 INFO:teuthology.orchestra.run.smithi077.stdout:-------------------------------------------------------------------------------- 2023-11-08T04:48:58.339 INFO:teuthology.orchestra.run.smithi077.stdout:Total 5.2 MB/s | 57 MB 00:11 2023-11-08T04:48:58.380 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction check 2023-11-08T04:48:58.427 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction check succeeded. 2023-11-08T04:48:58.427 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction test 2023-11-08T04:48:58.756 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction test succeeded. 2023-11-08T04:48:58.759 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction 2023-11-08T04:48:59.819 INFO:teuthology.orchestra.run.smithi077.stdout: Preparing : 1/1 2023-11-08T04:49:00.052 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : liboath-2.6.2-3.el8.x86_64 1/25 2023-11-08T04:49:00.218 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 2/25 2023-11-08T04:49:00.339 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: ceph-common-2:16.2.14-129.gcfb6fd52.el8.x86_64 20/25 2023-11-08T04:49:00.563 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-common-2:16.2.14-129.gcfb6fd52.el8.x86_64 20/25 2023-11-08T04:49:00.622 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 2/25 2023-11-08T04:49:00.835 INFO:teuthology.orchestra.run.smithi077.stdout: Upgrading : librados2-2:16.2.14-129.gcfb6fd52.el8.x86_64 3/25 2023-11-08T04:49:00.919 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: librados2-2:16.2.14-129.gcfb6fd52.el8.x86_64 3/25 2023-11-08T04:49:01.182 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-rados-2:16.2.14-129.gcfb6fd52.el8.x86_64 4/25 2023-11-08T04:49:01.268 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: ceph-common-2:16.2.14-129.gcfb6fd52.el8.x86_64 20/25 2023-11-08T04:49:01.393 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : libcephfs2-2:16.2.14-129.gcfb6fd52.el8.x86_64 5/25 2023-11-08T04:49:01.570 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-base-2:16.2.14-129.gcfb6fd52.el8.x86_64 21/25 2023-11-08T04:49:01.677 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: ceph-base-2:16.2.14-129.gcfb6fd52.el8.x86_64 21/25 2023-11-08T04:49:01.800 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: libcephfs2-2:16.2.14-129.gcfb6fd52.el8.x86_64 5/25 2023-11-08T04:49:01.813 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-selinux-2:16.2.14-129.gcfb6fd52.el8.x86_64 22/25 2023-11-08T04:49:01.993 INFO:teuthology.orchestra.run.smithi077.stdout: Upgrading : librbd1-2:16.2.14-129.gcfb6fd52.el8.x86_64 6/25 2023-11-08T04:49:02.091 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: librbd1-2:16.2.14-129.gcfb6fd52.el8.x86_64 6/25 2023-11-08T04:49:02.297 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : librabbitmq-0.9.0-4.el8.x86_64 7/25 2023-11-08T04:49:02.443 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 8/25 2023-11-08T04:49:02.816 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 8/25 2023-11-08T04:49:02.985 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : librgw2-2:16.2.14-129.gcfb6fd52.el8.x86_64 9/25 2023-11-08T04:49:03.065 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: librgw2-2:16.2.14-129.gcfb6fd52.el8.x86_64 9/25 2023-11-08T04:49:03.240 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : leveldb-1.22-1.el8.x86_64 10/25 2023-11-08T04:49:03.807 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-ceph-argparse-2:16.2.14-129.gcfb6fd52.el8. 11/25 2023-11-08T04:49:03.971 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-cephfs-2:16.2.14-129.gcfb6fd52.el8.x86_64 12/25 2023-11-08T04:49:04.170 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-rgw-2:16.2.14-129.gcfb6fd52.el8.x86_64 13/25 2023-11-08T04:49:04.410 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-rbd-2:16.2.14-129.gcfb6fd52.el8.x86_64 14/25 2023-11-08T04:49:04.576 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : libradosstriper1-2:16.2.14-129.gcfb6fd52.el8.x86_6 15/25 2023-11-08T04:49:04.645 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: libradosstriper1-2:16.2.14-129.gcfb6fd52.el8.x86_6 15/25 2023-11-08T04:49:04.869 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 16/25 2023-11-08T04:49:05.093 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 17/25 2023-11-08T04:49:05.274 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 18/25 2023-11-08T04:49:05.443 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-ceph-common-2:16.2.14-129.gcfb6fd52.el8.x8 19/25 2023-11-08T04:49:07.949 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-common-2:16.2.14-129.gcfb6fd52.el8.x86_64 20/25 2023-11-08T04:49:08.159 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-common-2:16.2.14-129.gcfb6fd52.el8.x86_64 20/25 2023-11-08T04:49:08.862 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-common-2:16.2.14-129.gcfb6fd52.el8.x86_64 20/25 2023-11-08T04:49:09.174 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-base-2:16.2.14-129.gcfb6fd52.el8.x86_64 21/25 2023-11-08T04:49:09.274 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-base-2:16.2.14-129.gcfb6fd52.el8.x86_64 21/25 2023-11-08T04:49:09.392 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-selinux-2:16.2.14-129.gcfb6fd52.el8.x86_64 22/25 2023-11-08T04:49:20.261 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: ceph-selinux-2:16.2.14-129.gcfb6fd52.el8.x86_64 22/25 2023-11-08T04:49:20.262 INFO:teuthology.orchestra.run.smithi087.stdout:skipping the directory /sys 2023-11-08T04:49:20.262 INFO:teuthology.orchestra.run.smithi087.stdout:skipping the directory /proc 2023-11-08T04:49:20.262 INFO:teuthology.orchestra.run.smithi087.stdout:skipping the directory /dev 2023-11-08T04:49:20.263 INFO:teuthology.orchestra.run.smithi087.stdout:skipping the directory /run 2023-11-08T04:49:20.263 INFO:teuthology.orchestra.run.smithi087.stdout:skipping the directory /mnt 2023-11-08T04:49:20.263 INFO:teuthology.orchestra.run.smithi087.stdout:skipping the directory /var/tmp 2023-11-08T04:49:20.263 INFO:teuthology.orchestra.run.smithi087.stdout:skipping the directory /home 2023-11-08T04:49:20.263 INFO:teuthology.orchestra.run.smithi087.stdout:skipping the directory /tmp 2023-11-08T04:49:20.264 INFO:teuthology.orchestra.run.smithi087.stdout:skipping the directory /dev 2023-11-08T04:49:20.264 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T04:49:21.409 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-radosgw-2:16.2.14-129.gcfb6fd52.el8.x86_64 23/25 2023-11-08T04:49:21.476 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: ceph-radosgw-2:16.2.14-129.gcfb6fd52.el8.x86_64 23/25 2023-11-08T04:49:21.476 INFO:teuthology.orchestra.run.smithi087.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 24/25 2023-11-08T04:49:21.712 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 24/25 2023-11-08T04:49:21.713 INFO:teuthology.orchestra.run.smithi087.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 25/25 2023-11-08T04:49:22.418 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 25/25 2023-11-08T04:49:22.418 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-base-2:16.2.14-129.gcfb6fd52.el8.x86_64 1/25 2023-11-08T04:49:22.418 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-common-2:16.2.14-129.gcfb6fd52.el8.x86_64 2/25 2023-11-08T04:49:22.419 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-radosgw-2:16.2.14-129.gcfb6fd52.el8.x86_64 3/25 2023-11-08T04:49:22.419 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-selinux-2:16.2.14-129.gcfb6fd52.el8.x86_64 4/25 2023-11-08T04:49:22.419 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : libcephfs2-2:16.2.14-129.gcfb6fd52.el8.x86_64 5/25 2023-11-08T04:49:22.419 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : libradosstriper1-2:16.2.14-129.gcfb6fd52.el8.x86_6 6/25 2023-11-08T04:49:22.419 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : librgw2-2:16.2.14-129.gcfb6fd52.el8.x86_64 7/25 2023-11-08T04:49:22.420 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-ceph-argparse-2:16.2.14-129.gcfb6fd52.el8. 8/25 2023-11-08T04:49:22.420 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-ceph-common-2:16.2.14-129.gcfb6fd52.el8.x8 9/25 2023-11-08T04:49:22.420 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-cephfs-2:16.2.14-129.gcfb6fd52.el8.x86_64 10/25 2023-11-08T04:49:22.420 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-rados-2:16.2.14-129.gcfb6fd52.el8.x86_64 11/25 2023-11-08T04:49:22.420 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-rbd-2:16.2.14-129.gcfb6fd52.el8.x86_64 12/25 2023-11-08T04:49:22.421 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-rgw-2:16.2.14-129.gcfb6fd52.el8.x86_64 13/25 2023-11-08T04:49:22.421 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 14/25 2023-11-08T04:49:22.421 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 15/25 2023-11-08T04:49:22.421 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 16/25 2023-11-08T04:49:22.421 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : librabbitmq-0.9.0-4.el8.x86_64 17/25 2023-11-08T04:49:22.422 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 18/25 2023-11-08T04:49:22.422 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : leveldb-1.22-1.el8.x86_64 19/25 2023-11-08T04:49:22.422 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 20/25 2023-11-08T04:49:22.422 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 21/25 2023-11-08T04:49:22.422 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : librados2-2:16.2.14-129.gcfb6fd52.el8.x86_64 22/25 2023-11-08T04:49:22.423 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 23/25 2023-11-08T04:49:22.423 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : librbd1-2:16.2.14-129.gcfb6fd52.el8.x86_64 24/25 2023-11-08T04:49:23.614 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 25/25 2023-11-08T04:49:23.615 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T04:49:23.615 INFO:teuthology.orchestra.run.smithi087.stdout:Upgraded: 2023-11-08T04:49:23.615 INFO:teuthology.orchestra.run.smithi087.stdout: librados2-2:16.2.14-129.gcfb6fd52.el8.x86_64 2023-11-08T04:49:23.615 INFO:teuthology.orchestra.run.smithi087.stdout: librbd1-2:16.2.14-129.gcfb6fd52.el8.x86_64 2023-11-08T04:49:23.616 INFO:teuthology.orchestra.run.smithi087.stdout:Installed: 2023-11-08T04:49:23.616 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-base-2:16.2.14-129.gcfb6fd52.el8.x86_64 2023-11-08T04:49:23.616 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-common-2:16.2.14-129.gcfb6fd52.el8.x86_64 2023-11-08T04:49:23.616 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-radosgw-2:16.2.14-129.gcfb6fd52.el8.x86_64 2023-11-08T04:49:23.616 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-selinux-2:16.2.14-129.gcfb6fd52.el8.x86_64 2023-11-08T04:49:23.617 INFO:teuthology.orchestra.run.smithi087.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2023-11-08T04:49:23.617 INFO:teuthology.orchestra.run.smithi087.stdout: leveldb-1.22-1.el8.x86_64 2023-11-08T04:49:23.617 INFO:teuthology.orchestra.run.smithi087.stdout: libcephfs2-2:16.2.14-129.gcfb6fd52.el8.x86_64 2023-11-08T04:49:23.617 INFO:teuthology.orchestra.run.smithi087.stdout: liboath-2.6.2-3.el8.x86_64 2023-11-08T04:49:23.617 INFO:teuthology.orchestra.run.smithi087.stdout: librabbitmq-0.9.0-4.el8.x86_64 2023-11-08T04:49:23.618 INFO:teuthology.orchestra.run.smithi087.stdout: libradosstriper1-2:16.2.14-129.gcfb6fd52.el8.x86_64 2023-11-08T04:49:23.618 INFO:teuthology.orchestra.run.smithi087.stdout: librdkafka-0.11.4-3.el8.x86_64 2023-11-08T04:49:23.618 INFO:teuthology.orchestra.run.smithi087.stdout: librgw2-2:16.2.14-129.gcfb6fd52.el8.x86_64 2023-11-08T04:49:23.618 INFO:teuthology.orchestra.run.smithi087.stdout: libunwind-1.3.1-3.el8.x86_64 2023-11-08T04:49:23.618 INFO:teuthology.orchestra.run.smithi087.stdout: lttng-ust-2.8.1-11.el8.x86_64 2023-11-08T04:49:23.619 INFO:teuthology.orchestra.run.smithi087.stdout: python3-ceph-argparse-2:16.2.14-129.gcfb6fd52.el8.x86_64 2023-11-08T04:49:23.619 INFO:teuthology.orchestra.run.smithi087.stdout: python3-ceph-common-2:16.2.14-129.gcfb6fd52.el8.x86_64 2023-11-08T04:49:23.619 INFO:teuthology.orchestra.run.smithi087.stdout: python3-cephfs-2:16.2.14-129.gcfb6fd52.el8.x86_64 2023-11-08T04:49:23.619 INFO:teuthology.orchestra.run.smithi087.stdout: python3-prettytable-0.7.2-14.el8.noarch 2023-11-08T04:49:23.619 INFO:teuthology.orchestra.run.smithi087.stdout: python3-rados-2:16.2.14-129.gcfb6fd52.el8.x86_64 2023-11-08T04:49:23.620 INFO:teuthology.orchestra.run.smithi087.stdout: python3-rbd-2:16.2.14-129.gcfb6fd52.el8.x86_64 2023-11-08T04:49:23.620 INFO:teuthology.orchestra.run.smithi087.stdout: python3-rgw-2:16.2.14-129.gcfb6fd52.el8.x86_64 2023-11-08T04:49:23.620 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T04:49:23.620 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2023-11-08T04:49:23.893 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install ceph-test 2023-11-08T04:49:24.737 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:00:45 ago on Wed 08 Nov 2023 04:48:39 AM UTC. 2023-11-08T04:49:26.389 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2023-11-08T04:49:26.390 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2023-11-08T04:49:26.390 INFO:teuthology.orchestra.run.smithi087.stdout: Package Arch Version Repository Size 2023-11-08T04:49:26.391 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2023-11-08T04:49:26.391 INFO:teuthology.orchestra.run.smithi087.stdout:Installing: 2023-11-08T04:49:26.391 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-test x86_64 2:16.2.14-129.gcfb6fd52.el8 ceph 52 M 2023-11-08T04:49:26.391 INFO:teuthology.orchestra.run.smithi087.stdout:Installing dependencies: 2023-11-08T04:49:26.391 INFO:teuthology.orchestra.run.smithi087.stdout: jq x86_64 1.6-8.el8 appstream 203 k 2023-11-08T04:49:26.392 INFO:teuthology.orchestra.run.smithi087.stdout: libcephsqlite x86_64 2:16.2.14-129.gcfb6fd52.el8 ceph 174 k 2023-11-08T04:49:26.392 INFO:teuthology.orchestra.run.smithi087.stdout: oniguruma x86_64 6.8.2-2.el8 appstream 187 k 2023-11-08T04:49:26.392 INFO:teuthology.orchestra.run.smithi087.stdout: socat x86_64 1.7.4.1-1.el8 appstream 323 k 2023-11-08T04:49:26.392 INFO:teuthology.orchestra.run.smithi087.stdout: xmlstarlet x86_64 1.6.1-20.el8 appstream 71 k 2023-11-08T04:49:26.392 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T04:49:26.393 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction Summary 2023-11-08T04:49:26.393 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2023-11-08T04:49:26.393 INFO:teuthology.orchestra.run.smithi087.stdout:Install 6 Packages 2023-11-08T04:49:26.393 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T04:49:26.394 INFO:teuthology.orchestra.run.smithi087.stdout:Total download size: 53 M 2023-11-08T04:49:26.394 INFO:teuthology.orchestra.run.smithi087.stdout:Installed size: 196 M 2023-11-08T04:49:26.394 INFO:teuthology.orchestra.run.smithi087.stdout:Downloading Packages: 2023-11-08T04:49:26.949 INFO:teuthology.orchestra.run.smithi087.stdout:(1/6): jq-1.6-8.el8.x86_64.rpm 546 kB/s | 203 kB 00:00 2023-11-08T04:49:27.049 INFO:teuthology.orchestra.run.smithi087.stdout:(2/6): oniguruma-6.8.2-2.el8.x86_64.rpm 1.8 MB/s | 187 kB 00:00 2023-11-08T04:49:27.183 INFO:teuthology.orchestra.run.smithi087.stdout:(3/6): libcephsqlite-16.2.14-129.gcfb6fd52.el8. 287 kB/s | 174 kB 00:00 2023-11-08T04:49:27.217 INFO:teuthology.orchestra.run.smithi087.stdout:(4/6): socat-1.7.4.1-1.el8.x86_64.rpm 1.9 MB/s | 323 kB 00:00 2023-11-08T04:49:27.484 INFO:teuthology.orchestra.run.smithi087.stdout:(5/6): xmlstarlet-1.6.1-20.el8.x86_64.rpm 235 kB/s | 71 kB 00:00 2023-11-08T04:49:27.706 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-selinux-2:16.2.14-129.gcfb6fd52.el8.x86_64 22/25 2023-11-08T04:49:27.706 INFO:teuthology.orchestra.run.smithi077.stdout:skipping the directory /sys 2023-11-08T04:49:27.707 INFO:teuthology.orchestra.run.smithi077.stdout:skipping the directory /proc 2023-11-08T04:49:27.707 INFO:teuthology.orchestra.run.smithi077.stdout:skipping the directory /dev 2023-11-08T04:49:27.707 INFO:teuthology.orchestra.run.smithi077.stdout:skipping the directory /run 2023-11-08T04:49:27.707 INFO:teuthology.orchestra.run.smithi077.stdout:skipping the directory /mnt 2023-11-08T04:49:27.707 INFO:teuthology.orchestra.run.smithi077.stdout:skipping the directory /var/tmp 2023-11-08T04:49:27.708 INFO:teuthology.orchestra.run.smithi077.stdout:skipping the directory /home 2023-11-08T04:49:27.708 INFO:teuthology.orchestra.run.smithi077.stdout:skipping the directory /tmp 2023-11-08T04:49:27.708 INFO:teuthology.orchestra.run.smithi077.stdout:skipping the directory /dev 2023-11-08T04:49:27.708 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-11-08T04:49:28.881 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-radosgw-2:16.2.14-129.gcfb6fd52.el8.x86_64 23/25 2023-11-08T04:49:28.960 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-radosgw-2:16.2.14-129.gcfb6fd52.el8.x86_64 23/25 2023-11-08T04:49:28.960 INFO:teuthology.orchestra.run.smithi077.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 24/25 2023-11-08T04:49:29.184 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 24/25 2023-11-08T04:49:29.184 INFO:teuthology.orchestra.run.smithi077.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 25/25 2023-11-08T04:49:29.863 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 25/25 2023-11-08T04:49:29.863 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-base-2:16.2.14-129.gcfb6fd52.el8.x86_64 1/25 2023-11-08T04:49:29.863 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-common-2:16.2.14-129.gcfb6fd52.el8.x86_64 2/25 2023-11-08T04:49:29.863 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-radosgw-2:16.2.14-129.gcfb6fd52.el8.x86_64 3/25 2023-11-08T04:49:29.864 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-selinux-2:16.2.14-129.gcfb6fd52.el8.x86_64 4/25 2023-11-08T04:49:29.864 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libcephfs2-2:16.2.14-129.gcfb6fd52.el8.x86_64 5/25 2023-11-08T04:49:29.864 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libradosstriper1-2:16.2.14-129.gcfb6fd52.el8.x86_6 6/25 2023-11-08T04:49:29.864 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : librgw2-2:16.2.14-129.gcfb6fd52.el8.x86_64 7/25 2023-11-08T04:49:29.864 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-ceph-argparse-2:16.2.14-129.gcfb6fd52.el8. 8/25 2023-11-08T04:49:29.865 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-ceph-common-2:16.2.14-129.gcfb6fd52.el8.x8 9/25 2023-11-08T04:49:29.865 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-cephfs-2:16.2.14-129.gcfb6fd52.el8.x86_64 10/25 2023-11-08T04:49:29.865 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-rados-2:16.2.14-129.gcfb6fd52.el8.x86_64 11/25 2023-11-08T04:49:29.865 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-rbd-2:16.2.14-129.gcfb6fd52.el8.x86_64 12/25 2023-11-08T04:49:29.865 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-rgw-2:16.2.14-129.gcfb6fd52.el8.x86_64 13/25 2023-11-08T04:49:29.866 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 14/25 2023-11-08T04:49:29.866 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 15/25 2023-11-08T04:49:29.866 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 16/25 2023-11-08T04:49:29.866 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : librabbitmq-0.9.0-4.el8.x86_64 17/25 2023-11-08T04:49:29.866 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 18/25 2023-11-08T04:49:29.867 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : leveldb-1.22-1.el8.x86_64 19/25 2023-11-08T04:49:29.867 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 20/25 2023-11-08T04:49:29.867 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 21/25 2023-11-08T04:49:29.867 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : librados2-2:16.2.14-129.gcfb6fd52.el8.x86_64 22/25 2023-11-08T04:49:29.867 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 23/25 2023-11-08T04:49:29.868 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : librbd1-2:16.2.14-129.gcfb6fd52.el8.x86_64 24/25 2023-11-08T04:49:29.942 INFO:teuthology.orchestra.run.smithi087.stdout:(6/6): ceph-test-16.2.14-129.gcfb6fd52.el8.x86_ 15 MB/s | 52 MB 00:03 2023-11-08T04:49:29.944 INFO:teuthology.orchestra.run.smithi087.stdout:-------------------------------------------------------------------------------- 2023-11-08T04:49:29.944 INFO:teuthology.orchestra.run.smithi087.stdout:Total 15 MB/s | 53 MB 00:03 2023-11-08T04:49:29.986 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction check 2023-11-08T04:49:30.006 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction check succeeded. 2023-11-08T04:49:30.007 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction test 2023-11-08T04:49:30.310 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction test succeeded. 2023-11-08T04:49:30.313 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction 2023-11-08T04:49:30.802 INFO:teuthology.orchestra.run.smithi087.stdout: Preparing : 1/1 2023-11-08T04:49:31.003 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 25/25 2023-11-08T04:49:31.003 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-11-08T04:49:31.003 INFO:teuthology.orchestra.run.smithi077.stdout:Upgraded: 2023-11-08T04:49:31.003 INFO:teuthology.orchestra.run.smithi077.stdout: librados2-2:16.2.14-129.gcfb6fd52.el8.x86_64 2023-11-08T04:49:31.004 INFO:teuthology.orchestra.run.smithi077.stdout: librbd1-2:16.2.14-129.gcfb6fd52.el8.x86_64 2023-11-08T04:49:31.004 INFO:teuthology.orchestra.run.smithi077.stdout:Installed: 2023-11-08T04:49:31.004 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-base-2:16.2.14-129.gcfb6fd52.el8.x86_64 2023-11-08T04:49:31.004 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-common-2:16.2.14-129.gcfb6fd52.el8.x86_64 2023-11-08T04:49:31.004 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-radosgw-2:16.2.14-129.gcfb6fd52.el8.x86_64 2023-11-08T04:49:31.005 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-selinux-2:16.2.14-129.gcfb6fd52.el8.x86_64 2023-11-08T04:49:31.005 INFO:teuthology.orchestra.run.smithi077.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2023-11-08T04:49:31.005 INFO:teuthology.orchestra.run.smithi077.stdout: leveldb-1.22-1.el8.x86_64 2023-11-08T04:49:31.005 INFO:teuthology.orchestra.run.smithi077.stdout: libcephfs2-2:16.2.14-129.gcfb6fd52.el8.x86_64 2023-11-08T04:49:31.006 INFO:teuthology.orchestra.run.smithi077.stdout: liboath-2.6.2-3.el8.x86_64 2023-11-08T04:49:31.006 INFO:teuthology.orchestra.run.smithi077.stdout: librabbitmq-0.9.0-4.el8.x86_64 2023-11-08T04:49:31.006 INFO:teuthology.orchestra.run.smithi077.stdout: libradosstriper1-2:16.2.14-129.gcfb6fd52.el8.x86_64 2023-11-08T04:49:31.006 INFO:teuthology.orchestra.run.smithi077.stdout: librdkafka-0.11.4-3.el8.x86_64 2023-11-08T04:49:31.006 INFO:teuthology.orchestra.run.smithi077.stdout: librgw2-2:16.2.14-129.gcfb6fd52.el8.x86_64 2023-11-08T04:49:31.007 INFO:teuthology.orchestra.run.smithi077.stdout: libunwind-1.3.1-3.el8.x86_64 2023-11-08T04:49:31.007 INFO:teuthology.orchestra.run.smithi077.stdout: lttng-ust-2.8.1-11.el8.x86_64 2023-11-08T04:49:31.007 INFO:teuthology.orchestra.run.smithi077.stdout: python3-ceph-argparse-2:16.2.14-129.gcfb6fd52.el8.x86_64 2023-11-08T04:49:31.007 INFO:teuthology.orchestra.run.smithi077.stdout: python3-ceph-common-2:16.2.14-129.gcfb6fd52.el8.x86_64 2023-11-08T04:49:31.007 INFO:teuthology.orchestra.run.smithi077.stdout: python3-cephfs-2:16.2.14-129.gcfb6fd52.el8.x86_64 2023-11-08T04:49:31.008 INFO:teuthology.orchestra.run.smithi077.stdout: python3-prettytable-0.7.2-14.el8.noarch 2023-11-08T04:49:31.008 INFO:teuthology.orchestra.run.smithi077.stdout: python3-rados-2:16.2.14-129.gcfb6fd52.el8.x86_64 2023-11-08T04:49:31.008 INFO:teuthology.orchestra.run.smithi077.stdout: python3-rbd-2:16.2.14-129.gcfb6fd52.el8.x86_64 2023-11-08T04:49:31.008 INFO:teuthology.orchestra.run.smithi077.stdout: python3-rgw-2:16.2.14-129.gcfb6fd52.el8.x86_64 2023-11-08T04:49:31.008 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-11-08T04:49:31.008 INFO:teuthology.orchestra.run.smithi077.stdout:Complete! 2023-11-08T04:49:31.034 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 1/6 2023-11-08T04:49:31.261 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 2/6 2023-11-08T04:49:31.300 DEBUG:teuthology.orchestra.run.smithi077:> sudo yum -y install ceph-test 2023-11-08T04:49:31.401 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 3/6 2023-11-08T04:49:31.505 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 3/6 2023-11-08T04:49:31.681 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : jq-1.6-8.el8.x86_64 4/6 2023-11-08T04:49:31.851 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : libcephsqlite-2:16.2.14-129.gcfb6fd52.el8.x86_64 5/6 2023-11-08T04:49:32.151 INFO:teuthology.orchestra.run.smithi077.stdout:Last metadata expiration check: 0:00:50 ago on Wed 08 Nov 2023 04:48:42 AM UTC. 2023-11-08T04:49:33.817 INFO:teuthology.orchestra.run.smithi077.stdout:Dependencies resolved. 2023-11-08T04:49:33.818 INFO:teuthology.orchestra.run.smithi077.stdout:================================================================================ 2023-11-08T04:49:33.818 INFO:teuthology.orchestra.run.smithi077.stdout: Package Arch Version Repository Size 2023-11-08T04:49:33.819 INFO:teuthology.orchestra.run.smithi077.stdout:================================================================================ 2023-11-08T04:49:33.819 INFO:teuthology.orchestra.run.smithi077.stdout:Installing: 2023-11-08T04:49:33.819 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-test x86_64 2:16.2.14-129.gcfb6fd52.el8 ceph 52 M 2023-11-08T04:49:33.819 INFO:teuthology.orchestra.run.smithi077.stdout:Installing dependencies: 2023-11-08T04:49:33.819 INFO:teuthology.orchestra.run.smithi077.stdout: jq x86_64 1.6-8.el8 appstream 203 k 2023-11-08T04:49:33.820 INFO:teuthology.orchestra.run.smithi077.stdout: libcephsqlite x86_64 2:16.2.14-129.gcfb6fd52.el8 ceph 174 k 2023-11-08T04:49:33.820 INFO:teuthology.orchestra.run.smithi077.stdout: oniguruma x86_64 6.8.2-2.el8 appstream 187 k 2023-11-08T04:49:33.820 INFO:teuthology.orchestra.run.smithi077.stdout: socat x86_64 1.7.4.1-1.el8 appstream 323 k 2023-11-08T04:49:33.820 INFO:teuthology.orchestra.run.smithi077.stdout: xmlstarlet x86_64 1.6.1-20.el8 appstream 71 k 2023-11-08T04:49:33.820 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-11-08T04:49:33.821 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction Summary 2023-11-08T04:49:33.821 INFO:teuthology.orchestra.run.smithi077.stdout:================================================================================ 2023-11-08T04:49:33.821 INFO:teuthology.orchestra.run.smithi077.stdout:Install 6 Packages 2023-11-08T04:49:33.821 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-11-08T04:49:33.822 INFO:teuthology.orchestra.run.smithi077.stdout:Total download size: 53 M 2023-11-08T04:49:33.822 INFO:teuthology.orchestra.run.smithi077.stdout:Installed size: 196 M 2023-11-08T04:49:33.822 INFO:teuthology.orchestra.run.smithi077.stdout:Downloading Packages: 2023-11-08T04:49:34.083 INFO:teuthology.orchestra.run.smithi077.stdout:(1/6): libcephsqlite-16.2.14-129.gcfb6fd52.el8. 962 kB/s | 174 kB 00:00 2023-11-08T04:49:34.275 INFO:teuthology.orchestra.run.smithi077.stdout:(2/6): jq-1.6-8.el8.x86_64.rpm 545 kB/s | 203 kB 00:00 2023-11-08T04:49:34.442 INFO:teuthology.orchestra.run.smithi077.stdout:(3/6): socat-1.7.4.1-1.el8.x86_64.rpm 1.9 MB/s | 323 kB 00:00 2023-11-08T04:49:34.467 INFO:teuthology.orchestra.run.smithi077.stdout:(4/6): oniguruma-6.8.2-2.el8.x86_64.rpm 488 kB/s | 187 kB 00:00 2023-11-08T04:49:34.543 INFO:teuthology.orchestra.run.smithi077.stdout:(5/6): xmlstarlet-1.6.1-20.el8.x86_64.rpm 704 kB/s | 71 kB 00:00 2023-11-08T04:49:36.447 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: libcephsqlite-2:16.2.14-129.gcfb6fd52.el8.x86_64 5/6 2023-11-08T04:49:36.526 INFO:teuthology.orchestra.run.smithi077.stdout:(6/6): ceph-test-16.2.14-129.gcfb6fd52.el8.x86_ 20 MB/s | 52 MB 00:02 2023-11-08T04:49:36.528 INFO:teuthology.orchestra.run.smithi077.stdout:-------------------------------------------------------------------------------- 2023-11-08T04:49:36.528 INFO:teuthology.orchestra.run.smithi077.stdout:Total 19 MB/s | 53 MB 00:02 2023-11-08T04:49:36.572 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction check 2023-11-08T04:49:36.592 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction check succeeded. 2023-11-08T04:49:36.593 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction test 2023-11-08T04:49:36.698 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-test-2:16.2.14-129.gcfb6fd52.el8.x86_64 6/6 2023-11-08T04:49:36.909 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction test succeeded. 2023-11-08T04:49:36.912 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction 2023-11-08T04:49:36.970 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: ceph-test-2:16.2.14-129.gcfb6fd52.el8.x86_64 6/6 2023-11-08T04:49:36.971 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-test-2:16.2.14-129.gcfb6fd52.el8.x86_64 1/6 2023-11-08T04:49:36.971 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : libcephsqlite-2:16.2.14-129.gcfb6fd52.el8.x86_64 2/6 2023-11-08T04:49:36.971 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : jq-1.6-8.el8.x86_64 3/6 2023-11-08T04:49:36.971 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 4/6 2023-11-08T04:49:36.972 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 5/6 2023-11-08T04:49:37.456 INFO:teuthology.orchestra.run.smithi077.stdout: Preparing : 1/1 2023-11-08T04:49:37.669 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 1/6 2023-11-08T04:49:37.695 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 6/6 2023-11-08T04:49:37.695 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T04:49:37.695 INFO:teuthology.orchestra.run.smithi087.stdout:Installed: 2023-11-08T04:49:37.695 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-test-2:16.2.14-129.gcfb6fd52.el8.x86_64 2023-11-08T04:49:37.695 INFO:teuthology.orchestra.run.smithi087.stdout: jq-1.6-8.el8.x86_64 2023-11-08T04:49:37.696 INFO:teuthology.orchestra.run.smithi087.stdout: libcephsqlite-2:16.2.14-129.gcfb6fd52.el8.x86_64 2023-11-08T04:49:37.696 INFO:teuthology.orchestra.run.smithi087.stdout: oniguruma-6.8.2-2.el8.x86_64 2023-11-08T04:49:37.696 INFO:teuthology.orchestra.run.smithi087.stdout: socat-1.7.4.1-1.el8.x86_64 2023-11-08T04:49:37.697 INFO:teuthology.orchestra.run.smithi087.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2023-11-08T04:49:37.697 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T04:49:37.697 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2023-11-08T04:49:37.858 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 2/6 2023-11-08T04:49:37.900 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install ceph 2023-11-08T04:49:37.980 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 3/6 2023-11-08T04:49:38.097 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 3/6 2023-11-08T04:49:38.286 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : jq-1.6-8.el8.x86_64 4/6 2023-11-08T04:49:38.438 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : libcephsqlite-2:16.2.14-129.gcfb6fd52.el8.x86_64 5/6 2023-11-08T04:49:38.757 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:00:59 ago on Wed 08 Nov 2023 04:48:39 AM UTC. 2023-11-08T04:49:40.414 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2023-11-08T04:49:40.418 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2023-11-08T04:49:40.419 INFO:teuthology.orchestra.run.smithi087.stdout: Package Arch Version Repository Size 2023-11-08T04:49:40.419 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2023-11-08T04:49:40.419 INFO:teuthology.orchestra.run.smithi087.stdout:Installing: 2023-11-08T04:49:40.419 INFO:teuthology.orchestra.run.smithi087.stdout: ceph x86_64 2:16.2.14-129.gcfb6fd52.el8 ceph 6.4 k 2023-11-08T04:49:40.419 INFO:teuthology.orchestra.run.smithi087.stdout:Installing dependencies: 2023-11-08T04:49:40.420 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-mds x86_64 2:16.2.14-129.gcfb6fd52.el8 ceph 2.4 M 2023-11-08T04:49:40.420 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-mgr x86_64 2:16.2.14-129.gcfb6fd52.el8 ceph 1.6 M 2023-11-08T04:49:40.420 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-mgr-modules-core noarch 2:16.2.14-129.gcfb6fd52.el8 ceph-noarch 245 k 2023-11-08T04:49:40.420 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-mon x86_64 2:16.2.14-129.gcfb6fd52.el8 ceph 5.0 M 2023-11-08T04:49:40.421 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-osd x86_64 2:16.2.14-129.gcfb6fd52.el8 ceph 18 M 2023-11-08T04:49:40.421 INFO:teuthology.orchestra.run.smithi087.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2023-11-08T04:49:40.421 INFO:teuthology.orchestra.run.smithi087.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2023-11-08T04:49:40.421 INFO:teuthology.orchestra.run.smithi087.stdout: python3-cffi x86_64 1.11.5-6.el8 baseos 249 k 2023-11-08T04:49:40.421 INFO:teuthology.orchestra.run.smithi087.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2023-11-08T04:49:40.422 INFO:teuthology.orchestra.run.smithi087.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2023-11-08T04:49:40.422 INFO:teuthology.orchestra.run.smithi087.stdout: python3-cryptography x86_64 3.2.1-6.el8 baseos 641 k 2023-11-08T04:49:40.422 INFO:teuthology.orchestra.run.smithi087.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2023-11-08T04:49:40.422 INFO:teuthology.orchestra.run.smithi087.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2023-11-08T04:49:40.422 INFO:teuthology.orchestra.run.smithi087.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2023-11-08T04:49:40.423 INFO:teuthology.orchestra.run.smithi087.stdout: python3-mako noarch 1.0.6-14.el8 appstream 157 k 2023-11-08T04:49:40.423 INFO:teuthology.orchestra.run.smithi087.stdout: python3-markupsafe x86_64 0.23-19.el8 appstream 39 k 2023-11-08T04:49:40.423 INFO:teuthology.orchestra.run.smithi087.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2023-11-08T04:49:40.423 INFO:teuthology.orchestra.run.smithi087.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2023-11-08T04:49:40.423 INFO:teuthology.orchestra.run.smithi087.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2023-11-08T04:49:40.424 INFO:teuthology.orchestra.run.smithi087.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 appstream 103 k 2023-11-08T04:49:40.424 INFO:teuthology.orchestra.run.smithi087.stdout: python3-pycparser noarch 2.14-14.el8 baseos 109 k 2023-11-08T04:49:40.424 INFO:teuthology.orchestra.run.smithi087.stdout: python3-pytz noarch 2017.2-11.el8 appstream 57 k 2023-11-08T04:49:40.424 INFO:teuthology.orchestra.run.smithi087.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2023-11-08T04:49:40.424 INFO:teuthology.orchestra.run.smithi087.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2023-11-08T04:49:40.425 INFO:teuthology.orchestra.run.smithi087.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2023-11-08T04:49:40.425 INFO:teuthology.orchestra.run.smithi087.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2023-11-08T04:49:40.425 INFO:teuthology.orchestra.run.smithi087.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2023-11-08T04:49:40.425 INFO:teuthology.orchestra.run.smithi087.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2023-11-08T04:49:40.425 INFO:teuthology.orchestra.run.smithi087.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2023-11-08T04:49:40.426 INFO:teuthology.orchestra.run.smithi087.stdout: python3-werkzeug noarch 0.12.2-4.el8 appstream 457 k 2023-11-08T04:49:40.426 INFO:teuthology.orchestra.run.smithi087.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2023-11-08T04:49:40.426 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T04:49:40.426 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction Summary 2023-11-08T04:49:40.426 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2023-11-08T04:49:40.427 INFO:teuthology.orchestra.run.smithi087.stdout:Install 32 Packages 2023-11-08T04:49:40.427 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T04:49:40.428 INFO:teuthology.orchestra.run.smithi087.stdout:Total download size: 31 M 2023-11-08T04:49:40.428 INFO:teuthology.orchestra.run.smithi087.stdout:Installed size: 101 M 2023-11-08T04:49:40.428 INFO:teuthology.orchestra.run.smithi087.stdout:Downloading Packages: 2023-11-08T04:49:41.170 INFO:teuthology.orchestra.run.smithi087.stdout:(1/32): ceph-16.2.14-129.gcfb6fd52.el8.x86_64.r 49 kB/s | 6.4 kB 00:00 2023-11-08T04:49:41.445 INFO:teuthology.orchestra.run.smithi087.stdout:(2/32): ceph-mds-16.2.14-129.gcfb6fd52.el8.x86_ 6.0 MB/s | 2.4 MB 00:00 2023-11-08T04:49:41.612 INFO:teuthology.orchestra.run.smithi087.stdout:(3/32): ceph-mgr-16.2.14-129.gcfb6fd52.el8.x86_ 2.8 MB/s | 1.6 MB 00:00 2023-11-08T04:49:41.712 INFO:teuthology.orchestra.run.smithi087.stdout:(4/32): ceph-mgr-modules-core-16.2.14-129.gcfb6 2.4 MB/s | 245 kB 00:00 2023-11-08T04:49:41.778 INFO:teuthology.orchestra.run.smithi087.stdout:(5/32): ceph-mon-16.2.14-129.gcfb6fd52.el8.x86_ 8.3 MB/s | 5.0 MB 00:00 2023-11-08T04:49:42.004 INFO:teuthology.orchestra.run.smithi087.stdout:(6/32): python3-markupsafe-0.23-19.el8.x86_64.r 172 kB/s | 39 kB 00:00 2023-11-08T04:49:42.088 INFO:teuthology.orchestra.run.smithi087.stdout:(7/32): python3-mako-1.0.6-14.el8.noarch.rpm 418 kB/s | 157 kB 00:00 2023-11-08T04:49:42.163 INFO:teuthology.orchestra.run.smithi087.stdout:(8/32): python3-pyOpenSSL-19.0.0-1.el8.noarch.r 653 kB/s | 103 kB 00:00 2023-11-08T04:49:42.190 INFO:teuthology.orchestra.run.smithi087.stdout:(9/32): python3-pytz-2017.2-11.el8.noarch.rpm 569 kB/s | 57 kB 00:00 2023-11-08T04:49:42.330 INFO:teuthology.orchestra.run.smithi087.stdout:(10/32): python3-werkzeug-0.12.2-4.el8.noarch.r 2.7 MB/s | 457 kB 00:00 2023-11-08T04:49:42.564 INFO:teuthology.orchestra.run.smithi087.stdout:(11/32): ceph-osd-16.2.14-129.gcfb6fd52.el8.x86 16 MB/s | 18 MB 00:01 2023-11-08T04:49:42.664 INFO:teuthology.orchestra.run.smithi087.stdout:(12/32): python3-cffi-1.11.5-6.el8.x86_64.rpm 523 kB/s | 249 kB 00:00 2023-11-08T04:49:42.756 INFO:teuthology.orchestra.run.smithi087.stdout:(13/32): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 476 kB/s | 44 kB 00:00 2023-11-08T04:49:42.881 INFO:teuthology.orchestra.run.smithi087.stdout:(14/32): python3-beautifulsoup4-4.6.3-2.el8.1.n 1.4 MB/s | 185 kB 00:00 2023-11-08T04:49:42.948 INFO:teuthology.orchestra.run.smithi087.stdout:(15/32): python3-pycparser-2.14-14.el8.noarch.r 284 kB/s | 109 kB 00:00 2023-11-08T04:49:42.990 INFO:teuthology.orchestra.run.smithi087.stdout:(16/32): python3-cheroot-8.5.2-1.el8.noarch.rpm 1.6 MB/s | 173 kB 00:00 2023-11-08T04:49:43.022 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: libcephsqlite-2:16.2.14-129.gcfb6fd52.el8.x86_64 5/6 2023-11-08T04:49:43.065 INFO:teuthology.orchestra.run.smithi087.stdout:(17/32): python3-cryptography-3.2.1-6.el8.x86_6 873 kB/s | 641 kB 00:00 2023-11-08T04:49:43.082 INFO:teuthology.orchestra.run.smithi087.stdout:(18/32): python3-jaraco-6.2-6.el8.noarch.rpm 121 kB/s | 11 kB 00:00 2023-11-08T04:49:43.133 INFO:teuthology.orchestra.run.smithi087.stdout:(19/32): python3-jaraco-functools-2.0-4.el8.noa 263 kB/s | 18 kB 00:00 2023-11-08T04:49:43.208 INFO:teuthology.orchestra.run.smithi087.stdout:(20/32): python3-more-itertools-7.2.0-3.el8.noa 784 kB/s | 59 kB 00:00 2023-11-08T04:49:43.236 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-test-2:16.2.14-129.gcfb6fd52.el8.x86_64 6/6 2023-11-08T04:49:43.325 INFO:teuthology.orchestra.run.smithi087.stdout:(21/32): python3-logutils-0.3.5-11.el8.noarch.r 201 kB/s | 49 kB 00:00 2023-11-08T04:49:43.383 INFO:teuthology.orchestra.run.smithi087.stdout:(22/32): python3-cherrypy-18.4.0-1.el8.noarch.r 884 kB/s | 384 kB 00:00 2023-11-08T04:49:43.409 INFO:teuthology.orchestra.run.smithi087.stdout:(23/32): python3-portend-2.6-1.el8.noarch.rpm 186 kB/s | 16 kB 00:00 2023-11-08T04:49:43.459 INFO:teuthology.orchestra.run.smithi087.stdout:(24/32): python3-pecan-1.3.2-9.el8.noarch.rpm 1.1 MB/s | 283 kB 00:00 2023-11-08T04:49:43.484 INFO:teuthology.orchestra.run.smithi087.stdout:(25/32): python3-simplegeneric-0.8.1-17.el8.noa 186 kB/s | 19 kB 00:00 2023-11-08T04:49:43.509 INFO:teuthology.orchestra.run.smithi087.stdout:(26/32): python3-singledispatch-3.4.0.3-18.el8. 238 kB/s | 24 kB 00:00 2023-11-08T04:49:43.518 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-test-2:16.2.14-129.gcfb6fd52.el8.x86_64 6/6 2023-11-08T04:49:43.519 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-test-2:16.2.14-129.gcfb6fd52.el8.x86_64 1/6 2023-11-08T04:49:43.519 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libcephsqlite-2:16.2.14-129.gcfb6fd52.el8.x86_64 2/6 2023-11-08T04:49:43.519 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : jq-1.6-8.el8.x86_64 3/6 2023-11-08T04:49:43.519 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 4/6 2023-11-08T04:49:43.520 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 5/6 2023-11-08T04:49:43.543 INFO:teuthology.orchestra.run.smithi087.stdout:(27/32): python3-tempora-1.14.1-5.el8.noarch.rp 348 kB/s | 29 kB 00:00 2023-11-08T04:49:43.594 INFO:teuthology.orchestra.run.smithi087.stdout:(28/32): python3-trustme-0.6.0-4.el8.noarch.rpm 252 kB/s | 27 kB 00:00 2023-11-08T04:49:43.677 INFO:teuthology.orchestra.run.smithi087.stdout:(29/32): python3-webtest-2.0.33-1.el8.noarch.rp 1.0 MB/s | 85 kB 00:00 2023-11-08T04:49:43.719 INFO:teuthology.orchestra.run.smithi087.stdout:(30/32): python3-waitress-1.4.3-1.el8.noarch.rp 1.2 MB/s | 255 kB 00:00 2023-11-08T04:49:43.753 INFO:teuthology.orchestra.run.smithi087.stdout:(31/32): python3-zc-lockfile-2.0-2.el8.noarch.r 305 kB/s | 23 kB 00:00 2023-11-08T04:49:43.778 INFO:teuthology.orchestra.run.smithi087.stdout:(32/32): python3-webob-1.8.5-1.el8.1.noarch.rpm 1.0 MB/s | 251 kB 00:00 2023-11-08T04:49:43.779 INFO:teuthology.orchestra.run.smithi087.stdout:-------------------------------------------------------------------------------- 2023-11-08T04:49:43.779 INFO:teuthology.orchestra.run.smithi087.stdout:Total 9.3 MB/s | 31 MB 00:03 2023-11-08T04:49:43.864 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction check 2023-11-08T04:49:43.902 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction check succeeded. 2023-11-08T04:49:43.902 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction test 2023-11-08T04:49:44.216 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 6/6 2023-11-08T04:49:44.217 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-11-08T04:49:44.217 INFO:teuthology.orchestra.run.smithi077.stdout:Installed: 2023-11-08T04:49:44.217 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-test-2:16.2.14-129.gcfb6fd52.el8.x86_64 2023-11-08T04:49:44.217 INFO:teuthology.orchestra.run.smithi077.stdout: jq-1.6-8.el8.x86_64 2023-11-08T04:49:44.218 INFO:teuthology.orchestra.run.smithi077.stdout: libcephsqlite-2:16.2.14-129.gcfb6fd52.el8.x86_64 2023-11-08T04:49:44.218 INFO:teuthology.orchestra.run.smithi077.stdout: oniguruma-6.8.2-2.el8.x86_64 2023-11-08T04:49:44.218 INFO:teuthology.orchestra.run.smithi077.stdout: socat-1.7.4.1-1.el8.x86_64 2023-11-08T04:49:44.218 INFO:teuthology.orchestra.run.smithi077.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2023-11-08T04:49:44.218 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-11-08T04:49:44.219 INFO:teuthology.orchestra.run.smithi077.stdout:Complete! 2023-11-08T04:49:44.219 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction test succeeded. 2023-11-08T04:49:44.223 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction 2023-11-08T04:49:44.509 DEBUG:teuthology.orchestra.run.smithi077:> sudo yum -y install ceph 2023-11-08T04:49:45.359 INFO:teuthology.orchestra.run.smithi077.stdout:Last metadata expiration check: 0:01:03 ago on Wed 08 Nov 2023 04:48:42 AM UTC. 2023-11-08T04:49:45.395 INFO:teuthology.orchestra.run.smithi087.stdout: Preparing : 1/1 2023-11-08T04:49:45.629 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/32 2023-11-08T04:49:45.865 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/32 2023-11-08T04:49:46.124 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 3/32 2023-11-08T04:49:46.406 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 4/32 2023-11-08T04:49:46.620 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 5/32 2023-11-08T04:49:46.834 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 6/32 2023-11-08T04:49:46.992 INFO:teuthology.orchestra.run.smithi077.stdout:Dependencies resolved. 2023-11-08T04:49:46.995 INFO:teuthology.orchestra.run.smithi077.stdout:================================================================================ 2023-11-08T04:49:46.995 INFO:teuthology.orchestra.run.smithi077.stdout: Package Arch Version Repository Size 2023-11-08T04:49:46.996 INFO:teuthology.orchestra.run.smithi077.stdout:================================================================================ 2023-11-08T04:49:46.996 INFO:teuthology.orchestra.run.smithi077.stdout:Installing: 2023-11-08T04:49:46.996 INFO:teuthology.orchestra.run.smithi077.stdout: ceph x86_64 2:16.2.14-129.gcfb6fd52.el8 ceph 6.4 k 2023-11-08T04:49:46.996 INFO:teuthology.orchestra.run.smithi077.stdout:Installing dependencies: 2023-11-08T04:49:46.996 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mds x86_64 2:16.2.14-129.gcfb6fd52.el8 ceph 2.4 M 2023-11-08T04:49:46.997 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mgr x86_64 2:16.2.14-129.gcfb6fd52.el8 ceph 1.6 M 2023-11-08T04:49:46.997 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mgr-modules-core noarch 2:16.2.14-129.gcfb6fd52.el8 ceph-noarch 245 k 2023-11-08T04:49:46.997 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mon x86_64 2:16.2.14-129.gcfb6fd52.el8 ceph 5.0 M 2023-11-08T04:49:46.997 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-osd x86_64 2:16.2.14-129.gcfb6fd52.el8 ceph 18 M 2023-11-08T04:49:46.998 INFO:teuthology.orchestra.run.smithi077.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2023-11-08T04:49:46.998 INFO:teuthology.orchestra.run.smithi077.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2023-11-08T04:49:46.998 INFO:teuthology.orchestra.run.smithi077.stdout: python3-cffi x86_64 1.11.5-6.el8 baseos 249 k 2023-11-08T04:49:46.998 INFO:teuthology.orchestra.run.smithi077.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2023-11-08T04:49:46.998 INFO:teuthology.orchestra.run.smithi077.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2023-11-08T04:49:46.999 INFO:teuthology.orchestra.run.smithi077.stdout: python3-cryptography x86_64 3.2.1-6.el8 baseos 641 k 2023-11-08T04:49:46.999 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2023-11-08T04:49:46.999 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2023-11-08T04:49:46.999 INFO:teuthology.orchestra.run.smithi077.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2023-11-08T04:49:46.999 INFO:teuthology.orchestra.run.smithi077.stdout: python3-mako noarch 1.0.6-14.el8 appstream 157 k 2023-11-08T04:49:47.000 INFO:teuthology.orchestra.run.smithi077.stdout: python3-markupsafe x86_64 0.23-19.el8 appstream 39 k 2023-11-08T04:49:47.000 INFO:teuthology.orchestra.run.smithi077.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2023-11-08T04:49:47.000 INFO:teuthology.orchestra.run.smithi077.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2023-11-08T04:49:47.000 INFO:teuthology.orchestra.run.smithi077.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2023-11-08T04:49:47.000 INFO:teuthology.orchestra.run.smithi077.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 appstream 103 k 2023-11-08T04:49:47.001 INFO:teuthology.orchestra.run.smithi077.stdout: python3-pycparser noarch 2.14-14.el8 baseos 109 k 2023-11-08T04:49:47.001 INFO:teuthology.orchestra.run.smithi077.stdout: python3-pytz noarch 2017.2-11.el8 appstream 57 k 2023-11-08T04:49:47.001 INFO:teuthology.orchestra.run.smithi077.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2023-11-08T04:49:47.001 INFO:teuthology.orchestra.run.smithi077.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2023-11-08T04:49:47.001 INFO:teuthology.orchestra.run.smithi077.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2023-11-08T04:49:47.002 INFO:teuthology.orchestra.run.smithi077.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2023-11-08T04:49:47.002 INFO:teuthology.orchestra.run.smithi077.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2023-11-08T04:49:47.002 INFO:teuthology.orchestra.run.smithi077.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2023-11-08T04:49:47.002 INFO:teuthology.orchestra.run.smithi077.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2023-11-08T04:49:47.002 INFO:teuthology.orchestra.run.smithi077.stdout: python3-werkzeug noarch 0.12.2-4.el8 appstream 457 k 2023-11-08T04:49:47.003 INFO:teuthology.orchestra.run.smithi077.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2023-11-08T04:49:47.003 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-11-08T04:49:47.003 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction Summary 2023-11-08T04:49:47.003 INFO:teuthology.orchestra.run.smithi077.stdout:================================================================================ 2023-11-08T04:49:47.003 INFO:teuthology.orchestra.run.smithi077.stdout:Install 32 Packages 2023-11-08T04:49:47.004 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-11-08T04:49:47.004 INFO:teuthology.orchestra.run.smithi077.stdout:Total download size: 31 M 2023-11-08T04:49:47.004 INFO:teuthology.orchestra.run.smithi077.stdout:Installed size: 101 M 2023-11-08T04:49:47.005 INFO:teuthology.orchestra.run.smithi077.stdout:Downloading Packages: 2023-11-08T04:49:47.052 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 7/32 2023-11-08T04:49:47.271 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 8/32 2023-11-08T04:49:47.427 INFO:teuthology.orchestra.run.smithi077.stdout:(1/32): ceph-16.2.14-129.gcfb6fd52.el8.x86_64.r 49 kB/s | 6.4 kB 00:00 2023-11-08T04:49:47.488 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 9/32 2023-11-08T04:49:47.758 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 10/32 2023-11-08T04:49:47.927 INFO:teuthology.orchestra.run.smithi077.stdout:(2/32): ceph-mgr-16.2.14-129.gcfb6fd52.el8.x86_ 2.6 MB/s | 1.6 MB 00:00 2023-11-08T04:49:48.046 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 11/32 2023-11-08T04:49:48.061 INFO:teuthology.orchestra.run.smithi077.stdout:(3/32): ceph-mds-16.2.14-129.gcfb6fd52.el8.x86_ 3.2 MB/s | 2.4 MB 00:00 2023-11-08T04:49:48.137 INFO:teuthology.orchestra.run.smithi077.stdout:(4/32): ceph-mgr-modules-core-16.2.14-129.gcfb6 3.2 MB/s | 245 kB 00:00 2023-11-08T04:49:48.287 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 12/32 2023-11-08T04:49:48.478 INFO:teuthology.orchestra.run.smithi077.stdout:(5/32): ceph-mon-16.2.14-129.gcfb6fd52.el8.x86_ 4.8 MB/s | 5.0 MB 00:01 2023-11-08T04:49:48.504 INFO:teuthology.orchestra.run.smithi077.stdout:(6/32): python3-mako-1.0.6-14.el8.noarch.rpm 428 kB/s | 157 kB 00:00 2023-11-08T04:49:48.604 INFO:teuthology.orchestra.run.smithi077.stdout:(7/32): python3-pyOpenSSL-19.0.0-1.el8.noarch.r 1.0 MB/s | 103 kB 00:00 2023-11-08T04:49:48.698 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 13/32 2023-11-08T04:49:48.704 INFO:teuthology.orchestra.run.smithi077.stdout:(8/32): python3-pytz-2017.2-11.el8.noarch.rpm 570 kB/s | 57 kB 00:00 2023-11-08T04:49:48.729 INFO:teuthology.orchestra.run.smithi077.stdout:(9/32): python3-markupsafe-0.23-19.el8.x86_64.r 155 kB/s | 39 kB 00:00 2023-11-08T04:49:48.872 INFO:teuthology.orchestra.run.smithi077.stdout:(10/32): python3-werkzeug-0.12.2-4.el8.noarch.r 2.7 MB/s | 457 kB 00:00 2023-11-08T04:49:48.951 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-cryptography-3.2.1-6.el8.x86_64 14/32 2023-11-08T04:49:49.165 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 15/32 2023-11-08T04:49:49.255 INFO:teuthology.orchestra.run.smithi077.stdout:(11/32): ceph-osd-16.2.14-129.gcfb6fd52.el8.x86 14 MB/s | 18 MB 00:01 2023-11-08T04:49:49.314 INFO:teuthology.orchestra.run.smithi077.stdout:(12/32): python3-cffi-1.11.5-6.el8.x86_64.rpm 426 kB/s | 249 kB 00:00 2023-11-08T04:49:49.431 INFO:teuthology.orchestra.run.smithi077.stdout:(13/32): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 374 kB/s | 44 kB 00:00 2023-11-08T04:49:49.431 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 16/32 2023-11-08T04:49:49.464 INFO:teuthology.orchestra.run.smithi077.stdout:(14/32): python3-cryptography-3.2.1-6.el8.x86_6 1.1 MB/s | 641 kB 00:00 2023-11-08T04:49:49.556 INFO:teuthology.orchestra.run.smithi077.stdout:(15/32): python3-beautifulsoup4-4.6.3-2.el8.1.n 1.4 MB/s | 185 kB 00:00 2023-11-08T04:49:49.615 INFO:teuthology.orchestra.run.smithi077.stdout:(16/32): python3-cheroot-8.5.2-1.el8.noarch.rpm 1.1 MB/s | 173 kB 00:00 2023-11-08T04:49:49.640 INFO:teuthology.orchestra.run.smithi077.stdout:(17/32): python3-pycparser-2.14-14.el8.noarch.r 284 kB/s | 109 kB 00:00 2023-11-08T04:49:49.665 INFO:teuthology.orchestra.run.smithi077.stdout:(18/32): python3-jaraco-6.2-6.el8.noarch.rpm 221 kB/s | 11 kB 00:00 2023-11-08T04:49:49.683 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 17/32 2023-11-08T04:49:49.690 INFO:teuthology.orchestra.run.smithi077.stdout:(19/32): python3-jaraco-functools-2.0-4.el8.noa 351 kB/s | 18 kB 00:00 2023-11-08T04:49:49.716 INFO:teuthology.orchestra.run.smithi077.stdout:(20/32): python3-cherrypy-18.4.0-1.el8.noarch.r 2.4 MB/s | 384 kB 00:00 2023-11-08T04:49:49.741 INFO:teuthology.orchestra.run.smithi077.stdout:(21/32): python3-more-itertools-7.2.0-3.el8.noa 1.1 MB/s | 59 kB 00:00 2023-11-08T04:49:49.766 INFO:teuthology.orchestra.run.smithi077.stdout:(22/32): python3-logutils-0.3.5-11.el8.noarch.r 484 kB/s | 49 kB 00:00 2023-11-08T04:49:49.792 INFO:teuthology.orchestra.run.smithi077.stdout:(23/32): python3-portend-2.6-1.el8.noarch.rpm 311 kB/s | 16 kB 00:00 2023-11-08T04:49:49.817 INFO:teuthology.orchestra.run.smithi077.stdout:(24/32): python3-simplegeneric-0.8.1-17.el8.noa 374 kB/s | 19 kB 00:00 2023-11-08T04:49:49.842 INFO:teuthology.orchestra.run.smithi077.stdout:(25/32): python3-singledispatch-3.4.0.3-18.el8. 476 kB/s | 24 kB 00:00 2023-11-08T04:49:49.868 INFO:teuthology.orchestra.run.smithi077.stdout:(26/32): python3-tempora-1.14.1-5.el8.noarch.rp 583 kB/s | 29 kB 00:00 2023-11-08T04:49:49.893 INFO:teuthology.orchestra.run.smithi077.stdout:(27/32): python3-trustme-0.6.0-4.el8.noarch.rpm 548 kB/s | 27 kB 00:00 2023-11-08T04:49:49.918 INFO:teuthology.orchestra.run.smithi077.stdout:(28/32): python3-pecan-1.3.2-9.el8.noarch.rpm 1.4 MB/s | 283 kB 00:00 2023-11-08T04:49:49.960 INFO:teuthology.orchestra.run.smithi077.stdout:(29/32): python3-waitress-1.4.3-1.el8.noarch.rp 2.7 MB/s | 255 kB 00:00 2023-11-08T04:49:50.010 INFO:teuthology.orchestra.run.smithi077.stdout:(30/32): python3-webtest-2.0.33-1.el8.noarch.rp 930 kB/s | 85 kB 00:00 2023-11-08T04:49:50.035 INFO:teuthology.orchestra.run.smithi077.stdout:(31/32): python3-zc-lockfile-2.0-2.el8.noarch.r 306 kB/s | 23 kB 00:00 2023-11-08T04:49:50.051 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 18/32 2023-11-08T04:49:50.060 INFO:teuthology.orchestra.run.smithi077.stdout:(32/32): python3-webob-1.8.5-1.el8.1.noarch.rpm 1.5 MB/s | 251 kB 00:00 2023-11-08T04:49:50.062 INFO:teuthology.orchestra.run.smithi077.stdout:-------------------------------------------------------------------------------- 2023-11-08T04:49:50.062 INFO:teuthology.orchestra.run.smithi077.stdout:Total 10 MB/s | 31 MB 00:03 2023-11-08T04:49:50.145 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction check 2023-11-08T04:49:50.183 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction check succeeded. 2023-11-08T04:49:50.184 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction test 2023-11-08T04:49:50.262 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 19/32 2023-11-08T04:49:50.491 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction test succeeded. 2023-11-08T04:49:50.493 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction 2023-11-08T04:49:50.525 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 20/32 2023-11-08T04:49:50.761 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/32 2023-11-08T04:49:51.042 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/32 2023-11-08T04:49:51.277 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/32 2023-11-08T04:49:51.536 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 24/32 2023-11-08T04:49:51.593 INFO:teuthology.orchestra.run.smithi077.stdout: Preparing : 1/1 2023-11-08T04:49:51.809 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/32 2023-11-08T04:49:51.874 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 25/32 2023-11-08T04:49:52.020 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/32 2023-11-08T04:49:52.167 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/32 2023-11-08T04:49:52.260 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 3/32 2023-11-08T04:49:52.527 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 4/32 2023-11-08T04:49:52.573 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-mgr-modules-core-2:16.2.14-129.gcfb6fd52.el8. 27/32 2023-11-08T04:49:52.708 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 5/32 2023-11-08T04:49:52.755 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-mgr-2:16.2.14-129.gcfb6fd52.el8.x86_64 28/32 2023-11-08T04:49:52.896 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 6/32 2023-11-08T04:49:53.081 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 7/32 2023-11-08T04:49:53.284 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 8/32 2023-11-08T04:49:53.510 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 9/32 2023-11-08T04:49:53.740 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 10/32 2023-11-08T04:49:53.968 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 11/32 2023-11-08T04:49:54.192 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 12/32 2023-11-08T04:49:54.242 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: ceph-mgr-2:16.2.14-129.gcfb6fd52.el8.x86_64 28/32 2023-11-08T04:49:54.439 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-osd-2:16.2.14-129.gcfb6fd52.el8.x86_64 29/32 2023-11-08T04:49:54.618 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 13/32 2023-11-08T04:49:54.864 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-cryptography-3.2.1-6.el8.x86_64 14/32 2023-11-08T04:49:54.880 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: ceph-osd-2:16.2.14-129.gcfb6fd52.el8.x86_64 29/32 2023-11-08T04:49:55.078 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 15/32 2023-11-08T04:49:55.294 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 16/32 2023-11-08T04:49:55.504 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 17/32 2023-11-08T04:49:55.548 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-mon-2:16.2.14-129.gcfb6fd52.el8.x86_64 30/32 2023-11-08T04:49:55.771 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: ceph-mon-2:16.2.14-129.gcfb6fd52.el8.x86_64 30/32 2023-11-08T04:49:55.853 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 18/32 2023-11-08T04:49:56.006 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-mds-2:16.2.14-129.gcfb6fd52.el8.x86_64 31/32 2023-11-08T04:49:56.025 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: ceph-mds-2:16.2.14-129.gcfb6fd52.el8.x86_64 31/32 2023-11-08T04:49:56.050 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 19/32 2023-11-08T04:49:56.163 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-2:16.2.14-129.gcfb6fd52.el8.x86_64 32/32 2023-11-08T04:49:56.264 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 20/32 2023-11-08T04:49:56.475 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/32 2023-11-08T04:49:56.577 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: ceph-2:16.2.14-129.gcfb6fd52.el8.x86_64 32/32 2023-11-08T04:49:56.577 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-2:16.2.14-129.gcfb6fd52.el8.x86_64 1/32 2023-11-08T04:49:56.577 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-mds-2:16.2.14-129.gcfb6fd52.el8.x86_64 2/32 2023-11-08T04:49:56.577 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-mgr-2:16.2.14-129.gcfb6fd52.el8.x86_64 3/32 2023-11-08T04:49:56.578 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-mon-2:16.2.14-129.gcfb6fd52.el8.x86_64 4/32 2023-11-08T04:49:56.578 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-osd-2:16.2.14-129.gcfb6fd52.el8.x86_64 5/32 2023-11-08T04:49:56.578 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-mgr-modules-core-2:16.2.14-129.gcfb6fd52.el8. 6/32 2023-11-08T04:49:56.578 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 7/32 2023-11-08T04:49:56.579 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 8/32 2023-11-08T04:49:56.579 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 9/32 2023-11-08T04:49:56.579 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 10/32 2023-11-08T04:49:56.579 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 11/32 2023-11-08T04:49:56.579 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 12/32 2023-11-08T04:49:56.579 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-cryptography-3.2.1-6.el8.x86_64 13/32 2023-11-08T04:49:56.580 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 14/32 2023-11-08T04:49:56.580 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 15/32 2023-11-08T04:49:56.580 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 16/32 2023-11-08T04:49:56.580 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 17/32 2023-11-08T04:49:56.580 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 18/32 2023-11-08T04:49:56.580 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 19/32 2023-11-08T04:49:56.581 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 20/32 2023-11-08T04:49:56.581 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 21/32 2023-11-08T04:49:56.581 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 22/32 2023-11-08T04:49:56.581 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 23/32 2023-11-08T04:49:56.581 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-portend-2.6-1.el8.noarch 24/32 2023-11-08T04:49:56.582 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 25/32 2023-11-08T04:49:56.582 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 26/32 2023-11-08T04:49:56.582 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 27/32 2023-11-08T04:49:56.582 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 28/32 2023-11-08T04:49:56.582 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 29/32 2023-11-08T04:49:56.583 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 30/32 2023-11-08T04:49:56.583 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 31/32 2023-11-08T04:49:56.716 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/32 2023-11-08T04:49:56.956 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/32 2023-11-08T04:49:57.222 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 24/32 2023-11-08T04:49:57.510 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 25/32 2023-11-08T04:49:57.658 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 32/32 2023-11-08T04:49:57.658 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T04:49:57.659 INFO:teuthology.orchestra.run.smithi087.stdout:Installed: 2023-11-08T04:49:57.659 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-2:16.2.14-129.gcfb6fd52.el8.x86_64 2023-11-08T04:49:57.659 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-mds-2:16.2.14-129.gcfb6fd52.el8.x86_64 2023-11-08T04:49:57.659 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-mgr-2:16.2.14-129.gcfb6fd52.el8.x86_64 2023-11-08T04:49:57.660 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-mgr-modules-core-2:16.2.14-129.gcfb6fd52.el8.noarch 2023-11-08T04:49:57.660 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-mon-2:16.2.14-129.gcfb6fd52.el8.x86_64 2023-11-08T04:49:57.660 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-osd-2:16.2.14-129.gcfb6fd52.el8.x86_64 2023-11-08T04:49:57.660 INFO:teuthology.orchestra.run.smithi087.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2023-11-08T04:49:57.660 INFO:teuthology.orchestra.run.smithi087.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2023-11-08T04:49:57.661 INFO:teuthology.orchestra.run.smithi087.stdout: python3-cffi-1.11.5-6.el8.x86_64 2023-11-08T04:49:57.661 INFO:teuthology.orchestra.run.smithi087.stdout: python3-cheroot-8.5.2-1.el8.noarch 2023-11-08T04:49:57.661 INFO:teuthology.orchestra.run.smithi087.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2023-11-08T04:49:57.661 INFO:teuthology.orchestra.run.smithi087.stdout: python3-cryptography-3.2.1-6.el8.x86_64 2023-11-08T04:49:57.661 INFO:teuthology.orchestra.run.smithi087.stdout: python3-jaraco-6.2-6.el8.noarch 2023-11-08T04:49:57.662 INFO:teuthology.orchestra.run.smithi087.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2023-11-08T04:49:57.662 INFO:teuthology.orchestra.run.smithi087.stdout: python3-logutils-0.3.5-11.el8.noarch 2023-11-08T04:49:57.662 INFO:teuthology.orchestra.run.smithi087.stdout: python3-mako-1.0.6-14.el8.noarch 2023-11-08T04:49:57.662 INFO:teuthology.orchestra.run.smithi087.stdout: python3-markupsafe-0.23-19.el8.x86_64 2023-11-08T04:49:57.662 INFO:teuthology.orchestra.run.smithi087.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2023-11-08T04:49:57.663 INFO:teuthology.orchestra.run.smithi087.stdout: python3-pecan-1.3.2-9.el8.noarch 2023-11-08T04:49:57.663 INFO:teuthology.orchestra.run.smithi087.stdout: python3-portend-2.6-1.el8.noarch 2023-11-08T04:49:57.663 INFO:teuthology.orchestra.run.smithi087.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2023-11-08T04:49:57.663 INFO:teuthology.orchestra.run.smithi087.stdout: python3-pycparser-2.14-14.el8.noarch 2023-11-08T04:49:57.663 INFO:teuthology.orchestra.run.smithi087.stdout: python3-pytz-2017.2-11.el8.noarch 2023-11-08T04:49:57.664 INFO:teuthology.orchestra.run.smithi087.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2023-11-08T04:49:57.664 INFO:teuthology.orchestra.run.smithi087.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2023-11-08T04:49:57.664 INFO:teuthology.orchestra.run.smithi087.stdout: python3-tempora-1.14.1-5.el8.noarch 2023-11-08T04:49:57.664 INFO:teuthology.orchestra.run.smithi087.stdout: python3-trustme-0.6.0-4.el8.noarch 2023-11-08T04:49:57.664 INFO:teuthology.orchestra.run.smithi087.stdout: python3-waitress-1.4.3-1.el8.noarch 2023-11-08T04:49:57.665 INFO:teuthology.orchestra.run.smithi087.stdout: python3-webob-1.8.5-1.el8.1.noarch 2023-11-08T04:49:57.665 INFO:teuthology.orchestra.run.smithi087.stdout: python3-webtest-2.0.33-1.el8.noarch 2023-11-08T04:49:57.665 INFO:teuthology.orchestra.run.smithi087.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2023-11-08T04:49:57.665 INFO:teuthology.orchestra.run.smithi087.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2023-11-08T04:49:57.665 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T04:49:57.665 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2023-11-08T04:49:57.864 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/32 2023-11-08T04:49:57.926 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install ceph-base 2023-11-08T04:49:58.245 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-mgr-modules-core-2:16.2.14-129.gcfb6fd52.el8. 27/32 2023-11-08T04:49:58.452 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-mgr-2:16.2.14-129.gcfb6fd52.el8.x86_64 28/32 2023-11-08T04:49:58.776 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:01:19 ago on Wed 08 Nov 2023 04:48:39 AM UTC. 2023-11-08T04:49:59.934 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-mgr-2:16.2.14-129.gcfb6fd52.el8.x86_64 28/32 2023-11-08T04:50:00.111 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-osd-2:16.2.14-129.gcfb6fd52.el8.x86_64 29/32 2023-11-08T04:50:00.218 INFO:teuthology.orchestra.run.smithi087.stdout:Package ceph-base-2:16.2.14-129.gcfb6fd52.el8.x86_64 is already installed. 2023-11-08T04:50:00.424 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2023-11-08T04:50:00.425 INFO:teuthology.orchestra.run.smithi087.stdout:Nothing to do. 2023-11-08T04:50:00.425 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2023-11-08T04:50:00.529 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install cephadm 2023-11-08T04:50:00.563 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-osd-2:16.2.14-129.gcfb6fd52.el8.x86_64 29/32 2023-11-08T04:50:00.744 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-mon-2:16.2.14-129.gcfb6fd52.el8.x86_64 30/32 2023-11-08T04:50:00.961 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-mon-2:16.2.14-129.gcfb6fd52.el8.x86_64 30/32 2023-11-08T04:50:01.128 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-mds-2:16.2.14-129.gcfb6fd52.el8.x86_64 31/32 2023-11-08T04:50:01.145 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-mds-2:16.2.14-129.gcfb6fd52.el8.x86_64 31/32 2023-11-08T04:50:01.314 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-2:16.2.14-129.gcfb6fd52.el8.x86_64 32/32 2023-11-08T04:50:01.393 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:01:22 ago on Wed 08 Nov 2023 04:48:39 AM UTC. 2023-11-08T04:50:01.719 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-2:16.2.14-129.gcfb6fd52.el8.x86_64 32/32 2023-11-08T04:50:01.720 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-2:16.2.14-129.gcfb6fd52.el8.x86_64 1/32 2023-11-08T04:50:01.720 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-mds-2:16.2.14-129.gcfb6fd52.el8.x86_64 2/32 2023-11-08T04:50:01.720 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-mgr-2:16.2.14-129.gcfb6fd52.el8.x86_64 3/32 2023-11-08T04:50:01.720 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-mon-2:16.2.14-129.gcfb6fd52.el8.x86_64 4/32 2023-11-08T04:50:01.720 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-osd-2:16.2.14-129.gcfb6fd52.el8.x86_64 5/32 2023-11-08T04:50:01.721 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-mgr-modules-core-2:16.2.14-129.gcfb6fd52.el8. 6/32 2023-11-08T04:50:01.721 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 7/32 2023-11-08T04:50:01.721 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 8/32 2023-11-08T04:50:01.721 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 9/32 2023-11-08T04:50:01.721 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 10/32 2023-11-08T04:50:01.722 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 11/32 2023-11-08T04:50:01.722 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 12/32 2023-11-08T04:50:01.722 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-cryptography-3.2.1-6.el8.x86_64 13/32 2023-11-08T04:50:01.722 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 14/32 2023-11-08T04:50:01.722 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 15/32 2023-11-08T04:50:01.723 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 16/32 2023-11-08T04:50:01.723 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 17/32 2023-11-08T04:50:01.723 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 18/32 2023-11-08T04:50:01.723 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 19/32 2023-11-08T04:50:01.723 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 20/32 2023-11-08T04:50:01.724 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 21/32 2023-11-08T04:50:01.724 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 22/32 2023-11-08T04:50:01.724 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 23/32 2023-11-08T04:50:01.724 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-portend-2.6-1.el8.noarch 24/32 2023-11-08T04:50:01.724 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 25/32 2023-11-08T04:50:01.724 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 26/32 2023-11-08T04:50:01.725 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 27/32 2023-11-08T04:50:01.725 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 28/32 2023-11-08T04:50:01.725 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 29/32 2023-11-08T04:50:01.725 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 30/32 2023-11-08T04:50:01.725 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 31/32 2023-11-08T04:50:02.797 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 32/32 2023-11-08T04:50:02.797 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-11-08T04:50:02.798 INFO:teuthology.orchestra.run.smithi077.stdout:Installed: 2023-11-08T04:50:02.798 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-2:16.2.14-129.gcfb6fd52.el8.x86_64 2023-11-08T04:50:02.798 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mds-2:16.2.14-129.gcfb6fd52.el8.x86_64 2023-11-08T04:50:02.798 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mgr-2:16.2.14-129.gcfb6fd52.el8.x86_64 2023-11-08T04:50:02.799 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mgr-modules-core-2:16.2.14-129.gcfb6fd52.el8.noarch 2023-11-08T04:50:02.799 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mon-2:16.2.14-129.gcfb6fd52.el8.x86_64 2023-11-08T04:50:02.799 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-osd-2:16.2.14-129.gcfb6fd52.el8.x86_64 2023-11-08T04:50:02.799 INFO:teuthology.orchestra.run.smithi077.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2023-11-08T04:50:02.799 INFO:teuthology.orchestra.run.smithi077.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2023-11-08T04:50:02.800 INFO:teuthology.orchestra.run.smithi077.stdout: python3-cffi-1.11.5-6.el8.x86_64 2023-11-08T04:50:02.800 INFO:teuthology.orchestra.run.smithi077.stdout: python3-cheroot-8.5.2-1.el8.noarch 2023-11-08T04:50:02.800 INFO:teuthology.orchestra.run.smithi077.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2023-11-08T04:50:02.800 INFO:teuthology.orchestra.run.smithi077.stdout: python3-cryptography-3.2.1-6.el8.x86_64 2023-11-08T04:50:02.800 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jaraco-6.2-6.el8.noarch 2023-11-08T04:50:02.801 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2023-11-08T04:50:02.801 INFO:teuthology.orchestra.run.smithi077.stdout: python3-logutils-0.3.5-11.el8.noarch 2023-11-08T04:50:02.801 INFO:teuthology.orchestra.run.smithi077.stdout: python3-mako-1.0.6-14.el8.noarch 2023-11-08T04:50:02.801 INFO:teuthology.orchestra.run.smithi077.stdout: python3-markupsafe-0.23-19.el8.x86_64 2023-11-08T04:50:02.801 INFO:teuthology.orchestra.run.smithi077.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2023-11-08T04:50:02.802 INFO:teuthology.orchestra.run.smithi077.stdout: python3-pecan-1.3.2-9.el8.noarch 2023-11-08T04:50:02.802 INFO:teuthology.orchestra.run.smithi077.stdout: python3-portend-2.6-1.el8.noarch 2023-11-08T04:50:02.802 INFO:teuthology.orchestra.run.smithi077.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2023-11-08T04:50:02.802 INFO:teuthology.orchestra.run.smithi077.stdout: python3-pycparser-2.14-14.el8.noarch 2023-11-08T04:50:02.802 INFO:teuthology.orchestra.run.smithi077.stdout: python3-pytz-2017.2-11.el8.noarch 2023-11-08T04:50:02.803 INFO:teuthology.orchestra.run.smithi077.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2023-11-08T04:50:02.803 INFO:teuthology.orchestra.run.smithi077.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2023-11-08T04:50:02.803 INFO:teuthology.orchestra.run.smithi077.stdout: python3-tempora-1.14.1-5.el8.noarch 2023-11-08T04:50:02.803 INFO:teuthology.orchestra.run.smithi077.stdout: python3-trustme-0.6.0-4.el8.noarch 2023-11-08T04:50:02.803 INFO:teuthology.orchestra.run.smithi077.stdout: python3-waitress-1.4.3-1.el8.noarch 2023-11-08T04:50:02.804 INFO:teuthology.orchestra.run.smithi077.stdout: python3-webob-1.8.5-1.el8.1.noarch 2023-11-08T04:50:02.804 INFO:teuthology.orchestra.run.smithi077.stdout: python3-webtest-2.0.33-1.el8.noarch 2023-11-08T04:50:02.804 INFO:teuthology.orchestra.run.smithi077.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2023-11-08T04:50:02.805 INFO:teuthology.orchestra.run.smithi077.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2023-11-08T04:50:02.805 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-11-08T04:50:02.805 INFO:teuthology.orchestra.run.smithi077.stdout:Complete! 2023-11-08T04:50:03.067 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2023-11-08T04:50:03.068 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2023-11-08T04:50:03.068 INFO:teuthology.orchestra.run.smithi087.stdout: Package Arch Version Repository Size 2023-11-08T04:50:03.069 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2023-11-08T04:50:03.069 INFO:teuthology.orchestra.run.smithi087.stdout:Installing: 2023-11-08T04:50:03.069 INFO:teuthology.orchestra.run.smithi087.stdout: cephadm noarch 2:16.2.14-129.gcfb6fd52.el8 ceph-noarch 85 k 2023-11-08T04:50:03.069 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T04:50:03.069 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction Summary 2023-11-08T04:50:03.070 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2023-11-08T04:50:03.070 INFO:teuthology.orchestra.run.smithi087.stdout:Install 1 Package 2023-11-08T04:50:03.070 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T04:50:03.070 INFO:teuthology.orchestra.run.smithi087.stdout:Total download size: 85 k 2023-11-08T04:50:03.070 INFO:teuthology.orchestra.run.smithi087.stdout:Installed size: 352 k 2023-11-08T04:50:03.071 INFO:teuthology.orchestra.run.smithi087.stdout:Downloading Packages: 2023-11-08T04:50:03.267 INFO:teuthology.orchestra.run.smithi087.stdout:cephadm-16.2.14-129.gcfb6fd52.el8.noarch.rpm 432 kB/s | 85 kB 00:00 2023-11-08T04:50:03.268 INFO:teuthology.orchestra.run.smithi087.stdout:-------------------------------------------------------------------------------- 2023-11-08T04:50:03.268 INFO:teuthology.orchestra.run.smithi087.stdout:Total 429 kB/s | 85 kB 00:00 2023-11-08T04:50:03.268 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction check 2023-11-08T04:50:03.275 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction check succeeded. 2023-11-08T04:50:03.275 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction test 2023-11-08T04:50:03.276 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction test succeeded. 2023-11-08T04:50:03.280 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction 2023-11-08T04:50:03.306 DEBUG:teuthology.orchestra.run.smithi077:> sudo yum -y install ceph-base 2023-11-08T04:50:03.329 INFO:teuthology.orchestra.run.smithi087.stdout: Preparing : 1/1 2023-11-08T04:50:03.823 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: cephadm-2:16.2.14-129.gcfb6fd52.el8.noarch 1/1 2023-11-08T04:50:03.983 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : cephadm-2:16.2.14-129.gcfb6fd52.el8.noarch 1/1 2023-11-08T04:50:04.161 INFO:teuthology.orchestra.run.smithi077.stdout:Last metadata expiration check: 0:01:22 ago on Wed 08 Nov 2023 04:48:42 AM UTC. 2023-11-08T04:50:04.204 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: cephadm-2:16.2.14-129.gcfb6fd52.el8.noarch 1/1 2023-11-08T04:50:04.867 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : cephadm-2:16.2.14-129.gcfb6fd52.el8.noarch 1/1 2023-11-08T04:50:04.868 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T04:50:04.868 INFO:teuthology.orchestra.run.smithi087.stdout:Installed: 2023-11-08T04:50:04.868 INFO:teuthology.orchestra.run.smithi087.stdout: cephadm-2:16.2.14-129.gcfb6fd52.el8.noarch 2023-11-08T04:50:04.868 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T04:50:04.869 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2023-11-08T04:50:05.104 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install ceph-immutable-object-cache 2023-11-08T04:50:05.597 INFO:teuthology.orchestra.run.smithi077.stdout:Package ceph-base-2:16.2.14-129.gcfb6fd52.el8.x86_64 is already installed. 2023-11-08T04:50:05.805 INFO:teuthology.orchestra.run.smithi077.stdout:Dependencies resolved. 2023-11-08T04:50:05.806 INFO:teuthology.orchestra.run.smithi077.stdout:Nothing to do. 2023-11-08T04:50:05.806 INFO:teuthology.orchestra.run.smithi077.stdout:Complete! 2023-11-08T04:50:05.919 DEBUG:teuthology.orchestra.run.smithi077:> sudo yum -y install cephadm 2023-11-08T04:50:05.954 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:01:26 ago on Wed 08 Nov 2023 04:48:39 AM UTC. 2023-11-08T04:50:06.767 INFO:teuthology.orchestra.run.smithi077.stdout:Last metadata expiration check: 0:01:24 ago on Wed 08 Nov 2023 04:48:42 AM UTC. 2023-11-08T04:50:07.616 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2023-11-08T04:50:07.617 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2023-11-08T04:50:07.618 INFO:teuthology.orchestra.run.smithi087.stdout: Package Arch Version Repo Size 2023-11-08T04:50:07.618 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2023-11-08T04:50:07.618 INFO:teuthology.orchestra.run.smithi087.stdout:Installing: 2023-11-08T04:50:07.618 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-immutable-object-cache x86_64 2:16.2.14-129.gcfb6fd52.el8 ceph 161 k 2023-11-08T04:50:07.619 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T04:50:07.619 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction Summary 2023-11-08T04:50:07.619 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2023-11-08T04:50:07.619 INFO:teuthology.orchestra.run.smithi087.stdout:Install 1 Package 2023-11-08T04:50:07.619 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T04:50:07.620 INFO:teuthology.orchestra.run.smithi087.stdout:Total download size: 161 k 2023-11-08T04:50:07.620 INFO:teuthology.orchestra.run.smithi087.stdout:Installed size: 470 k 2023-11-08T04:50:07.620 INFO:teuthology.orchestra.run.smithi087.stdout:Downloading Packages: 2023-11-08T04:50:07.995 INFO:teuthology.orchestra.run.smithi087.stdout:ceph-immutable-object-cache-16.2.14-129.gcfb6fd 428 kB/s | 161 kB 00:00 2023-11-08T04:50:07.996 INFO:teuthology.orchestra.run.smithi087.stdout:-------------------------------------------------------------------------------- 2023-11-08T04:50:07.996 INFO:teuthology.orchestra.run.smithi087.stdout:Total 427 kB/s | 161 kB 00:00 2023-11-08T04:50:07.996 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction check 2023-11-08T04:50:08.007 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction check succeeded. 2023-11-08T04:50:08.007 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction test 2023-11-08T04:50:08.055 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction test succeeded. 2023-11-08T04:50:08.058 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction 2023-11-08T04:50:08.176 INFO:teuthology.orchestra.run.smithi087.stdout: Preparing : 1/1 2023-11-08T04:50:08.338 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-immutable-object-cache-2:16.2.14-129.gcfb6fd52. 1/1 2023-11-08T04:50:08.402 INFO:teuthology.orchestra.run.smithi077.stdout:Dependencies resolved. 2023-11-08T04:50:08.403 INFO:teuthology.orchestra.run.smithi077.stdout:================================================================================ 2023-11-08T04:50:08.404 INFO:teuthology.orchestra.run.smithi077.stdout: Package Arch Version Repository Size 2023-11-08T04:50:08.404 INFO:teuthology.orchestra.run.smithi077.stdout:================================================================================ 2023-11-08T04:50:08.404 INFO:teuthology.orchestra.run.smithi077.stdout:Installing: 2023-11-08T04:50:08.404 INFO:teuthology.orchestra.run.smithi077.stdout: cephadm noarch 2:16.2.14-129.gcfb6fd52.el8 ceph-noarch 85 k 2023-11-08T04:50:08.404 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-11-08T04:50:08.405 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction Summary 2023-11-08T04:50:08.405 INFO:teuthology.orchestra.run.smithi077.stdout:================================================================================ 2023-11-08T04:50:08.405 INFO:teuthology.orchestra.run.smithi077.stdout:Install 1 Package 2023-11-08T04:50:08.405 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-11-08T04:50:08.405 INFO:teuthology.orchestra.run.smithi077.stdout:Total download size: 85 k 2023-11-08T04:50:08.405 INFO:teuthology.orchestra.run.smithi077.stdout:Installed size: 352 k 2023-11-08T04:50:08.406 INFO:teuthology.orchestra.run.smithi077.stdout:Downloading Packages: 2023-11-08T04:50:08.762 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: ceph-immutable-object-cache-2:16.2.14-129.gcfb6fd52. 1/1 2023-11-08T04:50:08.800 INFO:teuthology.orchestra.run.smithi077.stdout:cephadm-16.2.14-129.gcfb6fd52.el8.noarch.rpm 216 kB/s | 85 kB 00:00 2023-11-08T04:50:08.801 INFO:teuthology.orchestra.run.smithi077.stdout:-------------------------------------------------------------------------------- 2023-11-08T04:50:08.801 INFO:teuthology.orchestra.run.smithi077.stdout:Total 215 kB/s | 85 kB 00:00 2023-11-08T04:50:08.801 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction check 2023-11-08T04:50:08.808 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction check succeeded. 2023-11-08T04:50:08.808 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction test 2023-11-08T04:50:08.810 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction test succeeded. 2023-11-08T04:50:08.813 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction 2023-11-08T04:50:08.860 INFO:teuthology.orchestra.run.smithi077.stdout: Preparing : 1/1 2023-11-08T04:50:09.348 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: cephadm-2:16.2.14-129.gcfb6fd52.el8.noarch 1/1 2023-11-08T04:50:09.424 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-immutable-object-cache-2:16.2.14-129.gcfb6fd52. 1/1 2023-11-08T04:50:09.424 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T04:50:09.425 INFO:teuthology.orchestra.run.smithi087.stdout:Installed: 2023-11-08T04:50:09.425 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-immutable-object-cache-2:16.2.14-129.gcfb6fd52.el8.x86_64 2023-11-08T04:50:09.425 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T04:50:09.425 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2023-11-08T04:50:09.497 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : cephadm-2:16.2.14-129.gcfb6fd52.el8.noarch 1/1 2023-11-08T04:50:09.606 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install ceph-mgr 2023-11-08T04:50:09.730 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: cephadm-2:16.2.14-129.gcfb6fd52.el8.noarch 1/1 2023-11-08T04:50:10.389 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : cephadm-2:16.2.14-129.gcfb6fd52.el8.noarch 1/1 2023-11-08T04:50:10.389 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-11-08T04:50:10.389 INFO:teuthology.orchestra.run.smithi077.stdout:Installed: 2023-11-08T04:50:10.389 INFO:teuthology.orchestra.run.smithi077.stdout: cephadm-2:16.2.14-129.gcfb6fd52.el8.noarch 2023-11-08T04:50:10.390 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-11-08T04:50:10.390 INFO:teuthology.orchestra.run.smithi077.stdout:Complete! 2023-11-08T04:50:10.460 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:01:31 ago on Wed 08 Nov 2023 04:48:39 AM UTC. 2023-11-08T04:50:10.622 DEBUG:teuthology.orchestra.run.smithi077:> sudo yum -y install ceph-immutable-object-cache 2023-11-08T04:50:11.475 INFO:teuthology.orchestra.run.smithi077.stdout:Last metadata expiration check: 0:01:29 ago on Wed 08 Nov 2023 04:48:42 AM UTC. 2023-11-08T04:50:11.917 INFO:teuthology.orchestra.run.smithi087.stdout:Package ceph-mgr-2:16.2.14-129.gcfb6fd52.el8.x86_64 is already installed. 2023-11-08T04:50:12.131 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2023-11-08T04:50:12.132 INFO:teuthology.orchestra.run.smithi087.stdout:Nothing to do. 2023-11-08T04:50:12.132 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2023-11-08T04:50:12.218 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install ceph-mgr-dashboard 2023-11-08T04:50:13.065 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:01:34 ago on Wed 08 Nov 2023 04:48:39 AM UTC. 2023-11-08T04:50:13.142 INFO:teuthology.orchestra.run.smithi077.stdout:Dependencies resolved. 2023-11-08T04:50:13.143 INFO:teuthology.orchestra.run.smithi077.stdout:================================================================================ 2023-11-08T04:50:13.143 INFO:teuthology.orchestra.run.smithi077.stdout: Package Arch Version Repo Size 2023-11-08T04:50:13.143 INFO:teuthology.orchestra.run.smithi077.stdout:================================================================================ 2023-11-08T04:50:13.144 INFO:teuthology.orchestra.run.smithi077.stdout:Installing: 2023-11-08T04:50:13.144 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-immutable-object-cache x86_64 2:16.2.14-129.gcfb6fd52.el8 ceph 161 k 2023-11-08T04:50:13.144 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-11-08T04:50:13.144 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction Summary 2023-11-08T04:50:13.144 INFO:teuthology.orchestra.run.smithi077.stdout:================================================================================ 2023-11-08T04:50:13.145 INFO:teuthology.orchestra.run.smithi077.stdout:Install 1 Package 2023-11-08T04:50:13.145 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-11-08T04:50:13.145 INFO:teuthology.orchestra.run.smithi077.stdout:Total download size: 161 k 2023-11-08T04:50:13.146 INFO:teuthology.orchestra.run.smithi077.stdout:Installed size: 470 k 2023-11-08T04:50:13.146 INFO:teuthology.orchestra.run.smithi077.stdout:Downloading Packages: 2023-11-08T04:50:13.620 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-immutable-object-cache-16.2.14-129.gcfb6fd 339 kB/s | 161 kB 00:00 2023-11-08T04:50:13.621 INFO:teuthology.orchestra.run.smithi077.stdout:-------------------------------------------------------------------------------- 2023-11-08T04:50:13.621 INFO:teuthology.orchestra.run.smithi077.stdout:Total 338 kB/s | 161 kB 00:00 2023-11-08T04:50:13.622 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction check 2023-11-08T04:50:13.632 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction check succeeded. 2023-11-08T04:50:13.633 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction test 2023-11-08T04:50:13.681 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction test succeeded. 2023-11-08T04:50:13.684 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction 2023-11-08T04:50:13.798 INFO:teuthology.orchestra.run.smithi077.stdout: Preparing : 1/1 2023-11-08T04:50:14.718 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2023-11-08T04:50:14.719 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2023-11-08T04:50:14.719 INFO:teuthology.orchestra.run.smithi087.stdout: Package Arch Version Repository Size 2023-11-08T04:50:14.720 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2023-11-08T04:50:14.720 INFO:teuthology.orchestra.run.smithi087.stdout:Installing: 2023-11-08T04:50:14.720 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-mgr-dashboard noarch 2:16.2.14-129.gcfb6fd52.el8 ceph-noarch 3.1 M 2023-11-08T04:50:14.720 INFO:teuthology.orchestra.run.smithi087.stdout:Installing dependencies: 2023-11-08T04:50:14.720 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-grafana-dashboards noarch 2:16.2.14-129.gcfb6fd52.el8 ceph-noarch 24 k 2023-11-08T04:50:14.721 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-prometheus-alerts noarch 2:16.2.14-129.gcfb6fd52.el8 ceph-noarch 15 k 2023-11-08T04:50:14.721 INFO:teuthology.orchestra.run.smithi087.stdout: python3-jwt noarch 1.6.1-2.el8 baseos 43 k 2023-11-08T04:50:14.721 INFO:teuthology.orchestra.run.smithi087.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2023-11-08T04:50:14.721 INFO:teuthology.orchestra.run.smithi087.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2023-11-08T04:50:14.721 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T04:50:14.722 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction Summary 2023-11-08T04:50:14.722 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2023-11-08T04:50:14.722 INFO:teuthology.orchestra.run.smithi087.stdout:Install 6 Packages 2023-11-08T04:50:14.722 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T04:50:14.723 INFO:teuthology.orchestra.run.smithi087.stdout:Total download size: 3.4 M 2023-11-08T04:50:14.723 INFO:teuthology.orchestra.run.smithi087.stdout:Installed size: 22 M 2023-11-08T04:50:14.723 INFO:teuthology.orchestra.run.smithi087.stdout:Downloading Packages: 2023-11-08T04:50:14.772 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-immutable-object-cache-2:16.2.14-129.gcfb6fd52. 1/1 2023-11-08T04:50:15.163 INFO:teuthology.orchestra.run.smithi087.stdout:(1/6): ceph-prometheus-alerts-16.2.14-129.gcfb6 85 kB/s | 15 kB 00:00 2023-11-08T04:50:15.198 INFO:teuthology.orchestra.run.smithi087.stdout:(2/6): ceph-grafana-dashboards-16.2.14-129.gcfb 116 kB/s | 24 kB 00:00 2023-11-08T04:50:15.365 INFO:teuthology.orchestra.run.smithi087.stdout:(3/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 202 kB/s | 34 kB 00:00 2023-11-08T04:50:15.515 INFO:teuthology.orchestra.run.smithi087.stdout:(4/6): python3-routes-2.4.1-12.el8.noarch.rpm 1.3 MB/s | 196 kB 00:00 2023-11-08T04:50:15.565 INFO:teuthology.orchestra.run.smithi087.stdout:(5/6): python3-jwt-1.6.1-2.el8.noarch.rpm 106 kB/s | 43 kB 00:00 2023-11-08T04:50:15.615 INFO:teuthology.orchestra.run.smithi087.stdout:(6/6): ceph-mgr-dashboard-16.2.14-129.gcfb6fd52 5.0 MB/s | 3.1 MB 00:00 2023-11-08T04:50:15.617 INFO:teuthology.orchestra.run.smithi087.stdout:-------------------------------------------------------------------------------- 2023-11-08T04:50:15.618 INFO:teuthology.orchestra.run.smithi087.stdout:Total 3.8 MB/s | 3.4 MB 00:00 2023-11-08T04:50:15.628 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction check 2023-11-08T04:50:15.638 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction check succeeded. 2023-11-08T04:50:15.638 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction test 2023-11-08T04:50:15.792 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-immutable-object-cache-2:16.2.14-129.gcfb6fd52. 1/1 2023-11-08T04:50:15.799 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction test succeeded. 2023-11-08T04:50:15.802 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction 2023-11-08T04:50:16.104 INFO:teuthology.orchestra.run.smithi087.stdout: Preparing : 1/1 2023-11-08T04:50:16.320 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/6 2023-11-08T04:50:16.459 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-immutable-object-cache-2:16.2.14-129.gcfb6fd52. 1/1 2023-11-08T04:50:16.460 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-11-08T04:50:16.460 INFO:teuthology.orchestra.run.smithi077.stdout:Installed: 2023-11-08T04:50:16.460 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-immutable-object-cache-2:16.2.14-129.gcfb6fd52.el8.x86_64 2023-11-08T04:50:16.461 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-11-08T04:50:16.461 INFO:teuthology.orchestra.run.smithi077.stdout:Complete! 2023-11-08T04:50:16.581 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/6 2023-11-08T04:50:16.670 DEBUG:teuthology.orchestra.run.smithi077:> sudo yum -y install ceph-mgr 2023-11-08T04:50:16.797 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 3/6 2023-11-08T04:50:16.926 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-prometheus-alerts-2:16.2.14-129.gcfb6fd52.el8.n 4/6 2023-11-08T04:50:17.523 INFO:teuthology.orchestra.run.smithi077.stdout:Last metadata expiration check: 0:01:35 ago on Wed 08 Nov 2023 04:48:42 AM UTC. 2023-11-08T04:50:18.056 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-grafana-dashboards-2:16.2.14-129.gcfb6fd52.el8. 5/6 2023-11-08T04:50:18.385 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-mgr-dashboard-2:16.2.14-129.gcfb6fd52.el8.noarc 6/6 2023-11-08T04:50:18.653 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: ceph-mgr-dashboard-2:16.2.14-129.gcfb6fd52.el8.noarc 6/6 2023-11-08T04:50:18.653 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-grafana-dashboards-2:16.2.14-129.gcfb6fd52.el8. 1/6 2023-11-08T04:50:18.653 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-mgr-dashboard-2:16.2.14-129.gcfb6fd52.el8.noarc 2/6 2023-11-08T04:50:18.654 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-prometheus-alerts-2:16.2.14-129.gcfb6fd52.el8.n 3/6 2023-11-08T04:50:18.654 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 4/6 2023-11-08T04:50:18.654 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 5/6 2023-11-08T04:50:18.958 INFO:teuthology.orchestra.run.smithi077.stdout:Package ceph-mgr-2:16.2.14-129.gcfb6fd52.el8.x86_64 is already installed. 2023-11-08T04:50:19.174 INFO:teuthology.orchestra.run.smithi077.stdout:Dependencies resolved. 2023-11-08T04:50:19.175 INFO:teuthology.orchestra.run.smithi077.stdout:Nothing to do. 2023-11-08T04:50:19.175 INFO:teuthology.orchestra.run.smithi077.stdout:Complete! 2023-11-08T04:50:19.284 DEBUG:teuthology.orchestra.run.smithi077:> sudo yum -y install ceph-mgr-dashboard 2023-11-08T04:50:19.356 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 6/6 2023-11-08T04:50:19.356 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T04:50:19.357 INFO:teuthology.orchestra.run.smithi087.stdout:Installed: 2023-11-08T04:50:19.357 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-grafana-dashboards-2:16.2.14-129.gcfb6fd52.el8.noarch 2023-11-08T04:50:19.357 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-mgr-dashboard-2:16.2.14-129.gcfb6fd52.el8.noarch 2023-11-08T04:50:19.357 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-prometheus-alerts-2:16.2.14-129.gcfb6fd52.el8.noarch 2023-11-08T04:50:19.357 INFO:teuthology.orchestra.run.smithi087.stdout: python3-jwt-1.6.1-2.el8.noarch 2023-11-08T04:50:19.358 INFO:teuthology.orchestra.run.smithi087.stdout: python3-repoze-lru-0.7-6.el8.noarch 2023-11-08T04:50:19.358 INFO:teuthology.orchestra.run.smithi087.stdout: python3-routes-2.4.1-12.el8.noarch 2023-11-08T04:50:19.358 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T04:50:19.358 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2023-11-08T04:50:19.477 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install ceph-mgr-diskprediction-local 2023-11-08T04:50:20.131 INFO:teuthology.orchestra.run.smithi077.stdout:Last metadata expiration check: 0:01:38 ago on Wed 08 Nov 2023 04:48:42 AM UTC. 2023-11-08T04:50:20.322 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:01:41 ago on Wed 08 Nov 2023 04:48:39 AM UTC. 2023-11-08T04:50:21.778 INFO:teuthology.orchestra.run.smithi077.stdout:Dependencies resolved. 2023-11-08T04:50:21.779 INFO:teuthology.orchestra.run.smithi077.stdout:================================================================================ 2023-11-08T04:50:21.779 INFO:teuthology.orchestra.run.smithi077.stdout: Package Arch Version Repository Size 2023-11-08T04:50:21.779 INFO:teuthology.orchestra.run.smithi077.stdout:================================================================================ 2023-11-08T04:50:21.779 INFO:teuthology.orchestra.run.smithi077.stdout:Installing: 2023-11-08T04:50:21.780 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mgr-dashboard noarch 2:16.2.14-129.gcfb6fd52.el8 ceph-noarch 3.1 M 2023-11-08T04:50:21.780 INFO:teuthology.orchestra.run.smithi077.stdout:Installing dependencies: 2023-11-08T04:50:21.780 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-grafana-dashboards noarch 2:16.2.14-129.gcfb6fd52.el8 ceph-noarch 24 k 2023-11-08T04:50:21.780 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-prometheus-alerts noarch 2:16.2.14-129.gcfb6fd52.el8 ceph-noarch 15 k 2023-11-08T04:50:21.780 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jwt noarch 1.6.1-2.el8 baseos 43 k 2023-11-08T04:50:21.781 INFO:teuthology.orchestra.run.smithi077.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2023-11-08T04:50:21.781 INFO:teuthology.orchestra.run.smithi077.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2023-11-08T04:50:21.781 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-11-08T04:50:21.781 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction Summary 2023-11-08T04:50:21.781 INFO:teuthology.orchestra.run.smithi077.stdout:================================================================================ 2023-11-08T04:50:21.782 INFO:teuthology.orchestra.run.smithi077.stdout:Install 6 Packages 2023-11-08T04:50:21.782 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-11-08T04:50:21.782 INFO:teuthology.orchestra.run.smithi077.stdout:Total download size: 3.4 M 2023-11-08T04:50:21.783 INFO:teuthology.orchestra.run.smithi077.stdout:Installed size: 22 M 2023-11-08T04:50:21.783 INFO:teuthology.orchestra.run.smithi077.stdout:Downloading Packages: 2023-11-08T04:50:21.979 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2023-11-08T04:50:21.980 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================================ 2023-11-08T04:50:21.980 INFO:teuthology.orchestra.run.smithi087.stdout: Package Arch Version Repository Size 2023-11-08T04:50:21.980 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================================ 2023-11-08T04:50:21.981 INFO:teuthology.orchestra.run.smithi087.stdout:Installing: 2023-11-08T04:50:21.981 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-mgr-diskprediction-local noarch 2:16.2.14-129.gcfb6fd52.el8 ceph-noarch 8.6 M 2023-11-08T04:50:21.981 INFO:teuthology.orchestra.run.smithi087.stdout:Installing dependencies: 2023-11-08T04:50:21.981 INFO:teuthology.orchestra.run.smithi087.stdout: libgfortran x86_64 8.5.0-21.el8 baseos 645 k 2023-11-08T04:50:21.981 INFO:teuthology.orchestra.run.smithi087.stdout: libquadmath x86_64 8.5.0-21.el8 baseos 172 k 2023-11-08T04:50:21.982 INFO:teuthology.orchestra.run.smithi087.stdout: openblas x86_64 0.3.15-4.el8 appstream 4.7 M 2023-11-08T04:50:21.982 INFO:teuthology.orchestra.run.smithi087.stdout: openblas-threads x86_64 0.3.15-4.el8 appstream 4.9 M 2023-11-08T04:50:21.982 INFO:teuthology.orchestra.run.smithi087.stdout: python3-numpy x86_64 1:1.14.3-10.el8 appstream 3.7 M 2023-11-08T04:50:21.982 INFO:teuthology.orchestra.run.smithi087.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 appstream 225 k 2023-11-08T04:50:21.982 INFO:teuthology.orchestra.run.smithi087.stdout: python3-scipy x86_64 1.0.0-21.module_el8.5.0+771+e5d9a225 appstream 14 M 2023-11-08T04:50:21.983 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T04:50:21.983 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction Summary 2023-11-08T04:50:21.983 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================================ 2023-11-08T04:50:21.983 INFO:teuthology.orchestra.run.smithi087.stdout:Install 8 Packages 2023-11-08T04:50:21.983 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T04:50:21.984 INFO:teuthology.orchestra.run.smithi087.stdout:Total download size: 37 M 2023-11-08T04:50:21.984 INFO:teuthology.orchestra.run.smithi087.stdout:Installed size: 210 M 2023-11-08T04:50:21.985 INFO:teuthology.orchestra.run.smithi087.stdout:Downloading Packages: 2023-11-08T04:50:22.257 INFO:teuthology.orchestra.run.smithi077.stdout:(1/6): ceph-prometheus-alerts-16.2.14-129.gcfb6 101 kB/s | 15 kB 00:00 2023-11-08T04:50:22.282 INFO:teuthology.orchestra.run.smithi077.stdout:(2/6): ceph-grafana-dashboards-16.2.14-129.gcfb 142 kB/s | 24 kB 00:00 2023-11-08T04:50:22.599 INFO:teuthology.orchestra.run.smithi077.stdout:(3/6): python3-jwt-1.6.1-2.el8.noarch.rpm 125 kB/s | 43 kB 00:00 2023-11-08T04:50:23.033 INFO:teuthology.orchestra.run.smithi077.stdout:(4/6): ceph-mgr-dashboard-16.2.14-129.gcfb6fd52 3.4 MB/s | 3.1 MB 00:00 2023-11-08T04:50:23.472 INFO:teuthology.orchestra.run.smithi087.stdout:(1/8): openblas-0.3.15-4.el8.x86_64.rpm 4.6 MB/s | 4.7 MB 00:01 2023-11-08T04:50:23.746 INFO:teuthology.orchestra.run.smithi087.stdout:(2/8): openblas-threads-0.3.15-4.el8.x86_64.rpm 3.7 MB/s | 4.9 MB 00:01 2023-11-08T04:50:23.984 INFO:teuthology.orchestra.run.smithi087.stdout:(3/8): python3-numpy-f2py-1.14.3-10.el8.x86_64. 946 kB/s | 225 kB 00:00 2023-11-08T04:50:24.038 INFO:teuthology.orchestra.run.smithi087.stdout:(4/8): python3-numpy-1.14.3-10.el8.x86_64.rpm 6.6 MB/s | 3.7 MB 00:00 2023-11-08T04:50:24.108 INFO:teuthology.orchestra.run.smithi077.stdout:(5/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 18 kB/s | 34 kB 00:01 2023-11-08T04:50:24.175 INFO:teuthology.orchestra.run.smithi077.stdout:(6/6): python3-routes-2.4.1-12.el8.noarch.rpm 125 kB/s | 196 kB 00:01 2023-11-08T04:50:24.177 INFO:teuthology.orchestra.run.smithi077.stdout:-------------------------------------------------------------------------------- 2023-11-08T04:50:24.177 INFO:teuthology.orchestra.run.smithi077.stdout:Total 1.4 MB/s | 3.4 MB 00:02 2023-11-08T04:50:24.189 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction check 2023-11-08T04:50:24.200 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction check succeeded. 2023-11-08T04:50:24.200 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction test 2023-11-08T04:50:24.357 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction test succeeded. 2023-11-08T04:50:24.360 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction 2023-11-08T04:50:24.588 INFO:teuthology.orchestra.run.smithi087.stdout:(5/8): libgfortran-8.5.0-21.el8.x86_64.rpm 1.1 MB/s | 645 kB 00:00 2023-11-08T04:50:24.661 INFO:teuthology.orchestra.run.smithi077.stdout: Preparing : 1/1 2023-11-08T04:50:24.721 INFO:teuthology.orchestra.run.smithi087.stdout:(6/8): libquadmath-8.5.0-21.el8.x86_64.rpm 1.3 MB/s | 172 kB 00:00 2023-11-08T04:50:24.838 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/6 2023-11-08T04:50:25.042 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/6 2023-11-08T04:50:25.047 INFO:teuthology.orchestra.run.smithi087.stdout:(7/8): python3-scipy-1.0.0-21.module_el8.5.0+77 13 MB/s | 14 MB 00:01 2023-11-08T04:50:25.252 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 3/6 2023-11-08T04:50:25.373 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-prometheus-alerts-2:16.2.14-129.gcfb6fd52.el8.n 4/6 2023-11-08T04:50:25.514 INFO:teuthology.orchestra.run.smithi087.stdout:(8/8): ceph-mgr-diskprediction-local-16.2.14-12 2.8 MB/s | 8.6 MB 00:03 2023-11-08T04:50:25.515 INFO:teuthology.orchestra.run.smithi087.stdout:-------------------------------------------------------------------------------- 2023-11-08T04:50:25.516 INFO:teuthology.orchestra.run.smithi087.stdout:Total 10 MB/s | 37 MB 00:03 2023-11-08T04:50:25.750 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction check 2023-11-08T04:50:25.768 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction check succeeded. 2023-11-08T04:50:25.768 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction test 2023-11-08T04:50:26.094 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction test succeeded. 2023-11-08T04:50:26.097 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction 2023-11-08T04:50:26.385 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-grafana-dashboards-2:16.2.14-129.gcfb6fd52.el8. 5/6 2023-11-08T04:50:26.648 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-mgr-dashboard-2:16.2.14-129.gcfb6fd52.el8.noarc 6/6 2023-11-08T04:50:26.677 INFO:teuthology.orchestra.run.smithi087.stdout: Preparing : 1/1 2023-11-08T04:50:26.835 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : libquadmath-8.5.0-21.el8.x86_64 1/8 2023-11-08T04:50:26.916 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-mgr-dashboard-2:16.2.14-129.gcfb6fd52.el8.noarc 6/6 2023-11-08T04:50:26.916 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-grafana-dashboards-2:16.2.14-129.gcfb6fd52.el8. 1/6 2023-11-08T04:50:26.916 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-mgr-dashboard-2:16.2.14-129.gcfb6fd52.el8.noarc 2/6 2023-11-08T04:50:26.917 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-prometheus-alerts-2:16.2.14-129.gcfb6fd52.el8.n 3/6 2023-11-08T04:50:26.917 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 4/6 2023-11-08T04:50:26.917 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 5/6 2023-11-08T04:50:26.986 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: libquadmath-8.5.0-21.el8.x86_64 1/8 2023-11-08T04:50:27.178 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : libgfortran-8.5.0-21.el8.x86_64 2/8 2023-11-08T04:50:27.566 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 6/6 2023-11-08T04:50:27.567 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-11-08T04:50:27.567 INFO:teuthology.orchestra.run.smithi077.stdout:Installed: 2023-11-08T04:50:27.567 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-grafana-dashboards-2:16.2.14-129.gcfb6fd52.el8.noarch 2023-11-08T04:50:27.567 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mgr-dashboard-2:16.2.14-129.gcfb6fd52.el8.noarch 2023-11-08T04:50:27.568 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-prometheus-alerts-2:16.2.14-129.gcfb6fd52.el8.noarch 2023-11-08T04:50:27.568 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jwt-1.6.1-2.el8.noarch 2023-11-08T04:50:27.568 INFO:teuthology.orchestra.run.smithi077.stdout: python3-repoze-lru-0.7-6.el8.noarch 2023-11-08T04:50:27.568 INFO:teuthology.orchestra.run.smithi077.stdout: python3-routes-2.4.1-12.el8.noarch 2023-11-08T04:50:27.568 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-11-08T04:50:27.568 INFO:teuthology.orchestra.run.smithi077.stdout:Complete! 2023-11-08T04:50:27.701 DEBUG:teuthology.orchestra.run.smithi077:> sudo yum -y install ceph-mgr-diskprediction-local 2023-11-08T04:50:27.777 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: libgfortran-8.5.0-21.el8.x86_64 2/8 2023-11-08T04:50:28.001 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : openblas-0.3.15-4.el8.x86_64 3/8 2023-11-08T04:50:28.558 INFO:teuthology.orchestra.run.smithi077.stdout:Last metadata expiration check: 0:01:46 ago on Wed 08 Nov 2023 04:48:42 AM UTC. 2023-11-08T04:50:28.620 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: openblas-0.3.15-4.el8.x86_64 3/8 2023-11-08T04:50:28.770 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : openblas-threads-0.3.15-4.el8.x86_64 4/8 2023-11-08T04:50:29.449 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: openblas-threads-0.3.15-4.el8.x86_64 4/8 2023-11-08T04:50:30.167 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 5/8 2023-11-08T04:50:30.212 INFO:teuthology.orchestra.run.smithi077.stdout:Dependencies resolved. 2023-11-08T04:50:30.214 INFO:teuthology.orchestra.run.smithi077.stdout:================================================================================================ 2023-11-08T04:50:30.214 INFO:teuthology.orchestra.run.smithi077.stdout: Package Arch Version Repository Size 2023-11-08T04:50:30.214 INFO:teuthology.orchestra.run.smithi077.stdout:================================================================================================ 2023-11-08T04:50:30.215 INFO:teuthology.orchestra.run.smithi077.stdout:Installing: 2023-11-08T04:50:30.215 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mgr-diskprediction-local noarch 2:16.2.14-129.gcfb6fd52.el8 ceph-noarch 8.6 M 2023-11-08T04:50:30.215 INFO:teuthology.orchestra.run.smithi077.stdout:Installing dependencies: 2023-11-08T04:50:30.215 INFO:teuthology.orchestra.run.smithi077.stdout: libgfortran x86_64 8.5.0-21.el8 baseos 645 k 2023-11-08T04:50:30.216 INFO:teuthology.orchestra.run.smithi077.stdout: libquadmath x86_64 8.5.0-21.el8 baseos 172 k 2023-11-08T04:50:30.216 INFO:teuthology.orchestra.run.smithi077.stdout: openblas x86_64 0.3.15-4.el8 appstream 4.7 M 2023-11-08T04:50:30.216 INFO:teuthology.orchestra.run.smithi077.stdout: openblas-threads x86_64 0.3.15-4.el8 appstream 4.9 M 2023-11-08T04:50:30.216 INFO:teuthology.orchestra.run.smithi077.stdout: python3-numpy x86_64 1:1.14.3-10.el8 appstream 3.7 M 2023-11-08T04:50:30.217 INFO:teuthology.orchestra.run.smithi077.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 appstream 225 k 2023-11-08T04:50:30.217 INFO:teuthology.orchestra.run.smithi077.stdout: python3-scipy x86_64 1.0.0-21.module_el8.5.0+771+e5d9a225 appstream 14 M 2023-11-08T04:50:30.217 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-11-08T04:50:30.217 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction Summary 2023-11-08T04:50:30.217 INFO:teuthology.orchestra.run.smithi077.stdout:================================================================================================ 2023-11-08T04:50:30.217 INFO:teuthology.orchestra.run.smithi077.stdout:Install 8 Packages 2023-11-08T04:50:30.218 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-11-08T04:50:30.218 INFO:teuthology.orchestra.run.smithi077.stdout:Total download size: 37 M 2023-11-08T04:50:30.219 INFO:teuthology.orchestra.run.smithi077.stdout:Installed size: 210 M 2023-11-08T04:50:30.219 INFO:teuthology.orchestra.run.smithi077.stdout:Downloading Packages: 2023-11-08T04:50:30.434 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2023-11-08T04:50:31.456 INFO:teuthology.orchestra.run.smithi077.stdout:(1/8): ceph-mgr-diskprediction-local-16.2.14-12 8.0 MB/s | 8.6 MB 00:01 2023-11-08T04:50:31.798 INFO:teuthology.orchestra.run.smithi077.stdout:(2/8): openblas-0.3.15-4.el8.x86_64.rpm 3.3 MB/s | 4.7 MB 00:01 2023-11-08T04:50:31.865 INFO:teuthology.orchestra.run.smithi077.stdout:(3/8): openblas-threads-0.3.15-4.el8.x86_64.rpm 3.3 MB/s | 4.9 MB 00:01 2023-11-08T04:50:31.890 INFO:teuthology.orchestra.run.smithi077.stdout:(4/8): python3-numpy-f2py-1.14.3-10.el8.x86_64. 2.4 MB/s | 225 kB 00:00 2023-11-08T04:50:32.207 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2023-11-08T04:50:32.440 INFO:teuthology.orchestra.run.smithi077.stdout:(5/8): libgfortran-8.5.0-21.el8.x86_64.rpm 1.1 MB/s | 645 kB 00:00 2023-11-08T04:50:32.507 INFO:teuthology.orchestra.run.smithi077.stdout:(6/8): python3-numpy-1.14.3-10.el8.x86_64.rpm 3.6 MB/s | 3.7 MB 00:01 2023-11-08T04:50:32.541 INFO:teuthology.orchestra.run.smithi077.stdout:(7/8): libquadmath-8.5.0-21.el8.x86_64.rpm 1.7 MB/s | 172 kB 00:00 2023-11-08T04:50:32.944 INFO:teuthology.orchestra.run.smithi077.stdout:(8/8): python3-scipy-1.0.0-21.module_el8.5.0+77 13 MB/s | 14 MB 00:01 2023-11-08T04:50:32.945 INFO:teuthology.orchestra.run.smithi077.stdout:-------------------------------------------------------------------------------- 2023-11-08T04:50:32.945 INFO:teuthology.orchestra.run.smithi077.stdout:Total 13 MB/s | 37 MB 00:02 2023-11-08T04:50:33.179 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction check 2023-11-08T04:50:33.197 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction check succeeded. 2023-11-08T04:50:33.197 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction test 2023-11-08T04:50:33.505 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x 7/8 2023-11-08T04:50:33.516 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction test succeeded. 2023-11-08T04:50:33.520 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction 2023-11-08T04:50:33.700 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-mgr-diskprediction-local-2:16.2.14-129.gcfb6fd5 8/8 2023-11-08T04:50:34.046 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:16.2.14-129.gcfb6fd5 8/8 2023-11-08T04:50:34.046 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-mgr-diskprediction-local-2:16.2.14-129.gcfb6fd5 1/8 2023-11-08T04:50:34.047 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : openblas-0.3.15-4.el8.x86_64 2/8 2023-11-08T04:50:34.047 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : openblas-threads-0.3.15-4.el8.x86_64 3/8 2023-11-08T04:50:34.047 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 4/8 2023-11-08T04:50:34.047 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 5/8 2023-11-08T04:50:34.047 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x 6/8 2023-11-08T04:50:34.047 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : libgfortran-8.5.0-21.el8.x86_64 7/8 2023-11-08T04:50:34.090 INFO:teuthology.orchestra.run.smithi077.stdout: Preparing : 1/1 2023-11-08T04:50:34.257 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : libquadmath-8.5.0-21.el8.x86_64 1/8 2023-11-08T04:50:34.389 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: libquadmath-8.5.0-21.el8.x86_64 1/8 2023-11-08T04:50:34.517 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : libgfortran-8.5.0-21.el8.x86_64 2/8 2023-11-08T04:50:34.752 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : libquadmath-8.5.0-21.el8.x86_64 8/8 2023-11-08T04:50:34.752 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T04:50:34.752 INFO:teuthology.orchestra.run.smithi087.stdout:Installed: 2023-11-08T04:50:34.752 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-mgr-diskprediction-local-2:16.2.14-129.gcfb6fd52.el8.noarch 2023-11-08T04:50:34.753 INFO:teuthology.orchestra.run.smithi087.stdout: libgfortran-8.5.0-21.el8.x86_64 2023-11-08T04:50:34.753 INFO:teuthology.orchestra.run.smithi087.stdout: libquadmath-8.5.0-21.el8.x86_64 2023-11-08T04:50:34.753 INFO:teuthology.orchestra.run.smithi087.stdout: openblas-0.3.15-4.el8.x86_64 2023-11-08T04:50:34.753 INFO:teuthology.orchestra.run.smithi087.stdout: openblas-threads-0.3.15-4.el8.x86_64 2023-11-08T04:50:34.753 INFO:teuthology.orchestra.run.smithi087.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2023-11-08T04:50:34.754 INFO:teuthology.orchestra.run.smithi087.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2023-11-08T04:50:34.754 INFO:teuthology.orchestra.run.smithi087.stdout: python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x86_64 2023-11-08T04:50:34.754 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T04:50:34.754 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2023-11-08T04:50:34.867 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install ceph-mgr-rook 2023-11-08T04:50:35.113 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: libgfortran-8.5.0-21.el8.x86_64 2/8 2023-11-08T04:50:35.298 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : openblas-0.3.15-4.el8.x86_64 3/8 2023-11-08T04:50:35.718 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:01:56 ago on Wed 08 Nov 2023 04:48:39 AM UTC. 2023-11-08T04:50:35.924 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: openblas-0.3.15-4.el8.x86_64 3/8 2023-11-08T04:50:36.101 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : openblas-threads-0.3.15-4.el8.x86_64 4/8 2023-11-08T04:50:36.779 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: openblas-threads-0.3.15-4.el8.x86_64 4/8 2023-11-08T04:50:37.177 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 5/8 2023-11-08T04:50:37.383 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2023-11-08T04:50:37.385 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2023-11-08T04:50:37.385 INFO:teuthology.orchestra.run.smithi087.stdout: Package Arch Version Repository Size 2023-11-08T04:50:37.385 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2023-11-08T04:50:37.386 INFO:teuthology.orchestra.run.smithi087.stdout:Installing: 2023-11-08T04:50:37.386 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-mgr-rook noarch 2:16.2.14-129.gcfb6fd52.el8 ceph-noarch 27 k 2023-11-08T04:50:37.386 INFO:teuthology.orchestra.run.smithi087.stdout:Installing dependencies: 2023-11-08T04:50:37.386 INFO:teuthology.orchestra.run.smithi087.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2023-11-08T04:50:37.387 INFO:teuthology.orchestra.run.smithi087.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2023-11-08T04:50:37.387 INFO:teuthology.orchestra.run.smithi087.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2023-11-08T04:50:37.387 INFO:teuthology.orchestra.run.smithi087.stdout: python3-jsonpatch noarch 1.21-2.el8 appstream 27 k 2023-11-08T04:50:37.387 INFO:teuthology.orchestra.run.smithi087.stdout: python3-jsonpointer noarch 1.10-11.el8 appstream 20 k 2023-11-08T04:50:37.387 INFO:teuthology.orchestra.run.smithi087.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2023-11-08T04:50:37.388 INFO:teuthology.orchestra.run.smithi087.stdout: python3-oauthlib noarch 2.1.0-1.el8 baseos 155 k 2023-11-08T04:50:37.388 INFO:teuthology.orchestra.run.smithi087.stdout: python3-pyasn1 noarch 0.3.7-6.el8 appstream 126 k 2023-11-08T04:50:37.388 INFO:teuthology.orchestra.run.smithi087.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 appstream 110 k 2023-11-08T04:50:37.388 INFO:teuthology.orchestra.run.smithi087.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 baseos 43 k 2023-11-08T04:50:37.388 INFO:teuthology.orchestra.run.smithi087.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2023-11-08T04:50:37.388 INFO:teuthology.orchestra.run.smithi087.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2023-11-08T04:50:37.389 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T04:50:37.389 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction Summary 2023-11-08T04:50:37.389 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2023-11-08T04:50:37.389 INFO:teuthology.orchestra.run.smithi087.stdout:Install 13 Packages 2023-11-08T04:50:37.389 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T04:50:37.390 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2023-11-08T04:50:37.391 INFO:teuthology.orchestra.run.smithi087.stdout:Total download size: 1.9 M 2023-11-08T04:50:37.391 INFO:teuthology.orchestra.run.smithi087.stdout:Installed size: 24 M 2023-11-08T04:50:37.391 INFO:teuthology.orchestra.run.smithi087.stdout:Downloading Packages: 2023-11-08T04:50:38.210 INFO:teuthology.orchestra.run.smithi087.stdout:(1/13): ceph-mgr-rook-16.2.14-129.gcfb6fd52.el8 182 kB/s | 27 kB 00:00 2023-11-08T04:50:38.294 INFO:teuthology.orchestra.run.smithi087.stdout:(2/13): python3-jsonpointer-1.10-11.el8.noarch. 85 kB/s | 20 kB 00:00 2023-11-08T04:50:38.319 INFO:teuthology.orchestra.run.smithi087.stdout:(3/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 102 kB/s | 27 kB 00:00 2023-11-08T04:50:38.528 INFO:teuthology.orchestra.run.smithi087.stdout:(4/13): python3-pyasn1-modules-0.3.7-6.el8.noar 471 kB/s | 110 kB 00:00 2023-11-08T04:50:38.587 INFO:teuthology.orchestra.run.smithi087.stdout:(5/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 336 kB/s | 126 kB 00:00 2023-11-08T04:50:38.712 INFO:teuthology.orchestra.run.smithi087.stdout:(6/13): python3-cachetools-3.1.1-4.el8.noarch.r 264 kB/s | 33 kB 00:00 2023-11-08T04:50:38.737 INFO:teuthology.orchestra.run.smithi087.stdout:(7/13): python3-oauthlib-2.1.0-1.el8.noarch.rpm 371 kB/s | 155 kB 00:00 2023-11-08T04:50:38.771 INFO:teuthology.orchestra.run.smithi087.stdout:(8/13): python3-certifi-2018.10.15-7.el8.noarch 278 kB/s | 16 kB 00:00 2023-11-08T04:50:38.871 INFO:teuthology.orchestra.run.smithi087.stdout:(9/13): python3-requests-oauthlib-1.0.0-1.el8.n 124 kB/s | 43 kB 00:00 2023-11-08T04:50:38.913 INFO:teuthology.orchestra.run.smithi087.stdout:(10/13): python3-google-auth-1.1.1-10.el8.noarc 463 kB/s | 81 kB 00:00 2023-11-08T04:50:38.988 INFO:teuthology.orchestra.run.smithi087.stdout:(11/13): python3-websocket-client-0.56.0-5.el8. 819 kB/s | 61 kB 00:00 2023-11-08T04:50:39.063 INFO:teuthology.orchestra.run.smithi087.stdout:(12/13): python3-rsa-4.9-2.el8.noarch.rpm 334 kB/s | 64 kB 00:00 2023-11-08T04:50:39.150 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2023-11-08T04:50:39.256 INFO:teuthology.orchestra.run.smithi087.stdout:(13/13): python3-kubernetes-11.0.0-6.el8.noarch 2.4 MB/s | 1.2 MB 00:00 2023-11-08T04:50:39.256 INFO:teuthology.orchestra.run.smithi087.stdout:-------------------------------------------------------------------------------- 2023-11-08T04:50:39.256 INFO:teuthology.orchestra.run.smithi087.stdout:Total 1.0 MB/s | 1.9 MB 00:01 2023-11-08T04:50:39.316 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction check 2023-11-08T04:50:39.332 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction check succeeded. 2023-11-08T04:50:39.332 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction test 2023-11-08T04:50:39.419 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction test succeeded. 2023-11-08T04:50:39.422 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction 2023-11-08T04:50:39.922 INFO:teuthology.orchestra.run.smithi087.stdout: Preparing : 1/1 2023-11-08T04:50:40.235 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2023-11-08T04:50:40.451 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x 7/8 2023-11-08T04:50:40.458 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/13 2023-11-08T04:50:40.639 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-mgr-diskprediction-local-2:16.2.14-129.gcfb6fd5 8/8 2023-11-08T04:50:40.702 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-rsa-4.9-2.el8.noarch 3/13 2023-11-08T04:50:40.957 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/13 2023-11-08T04:50:40.971 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:16.2.14-129.gcfb6fd5 8/8 2023-11-08T04:50:40.971 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-mgr-diskprediction-local-2:16.2.14-129.gcfb6fd5 1/8 2023-11-08T04:50:40.971 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : openblas-0.3.15-4.el8.x86_64 2/8 2023-11-08T04:50:40.972 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : openblas-threads-0.3.15-4.el8.x86_64 3/8 2023-11-08T04:50:40.972 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 4/8 2023-11-08T04:50:40.972 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 5/8 2023-11-08T04:50:40.972 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x 6/8 2023-11-08T04:50:40.972 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libgfortran-8.5.0-21.el8.x86_64 7/8 2023-11-08T04:50:41.182 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/13 2023-11-08T04:50:41.457 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/13 2023-11-08T04:50:41.647 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libquadmath-8.5.0-21.el8.x86_64 8/8 2023-11-08T04:50:41.647 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-11-08T04:50:41.647 INFO:teuthology.orchestra.run.smithi077.stdout:Installed: 2023-11-08T04:50:41.647 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mgr-diskprediction-local-2:16.2.14-129.gcfb6fd52.el8.noarch 2023-11-08T04:50:41.648 INFO:teuthology.orchestra.run.smithi077.stdout: libgfortran-8.5.0-21.el8.x86_64 2023-11-08T04:50:41.648 INFO:teuthology.orchestra.run.smithi077.stdout: libquadmath-8.5.0-21.el8.x86_64 2023-11-08T04:50:41.648 INFO:teuthology.orchestra.run.smithi077.stdout: openblas-0.3.15-4.el8.x86_64 2023-11-08T04:50:41.648 INFO:teuthology.orchestra.run.smithi077.stdout: openblas-threads-0.3.15-4.el8.x86_64 2023-11-08T04:50:41.648 INFO:teuthology.orchestra.run.smithi077.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2023-11-08T04:50:41.649 INFO:teuthology.orchestra.run.smithi077.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2023-11-08T04:50:41.649 INFO:teuthology.orchestra.run.smithi077.stdout: python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x86_64 2023-11-08T04:50:41.649 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-11-08T04:50:41.649 INFO:teuthology.orchestra.run.smithi077.stdout:Complete! 2023-11-08T04:50:41.774 DEBUG:teuthology.orchestra.run.smithi077:> sudo yum -y install ceph-mgr-rook 2023-11-08T04:50:41.792 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/13 2023-11-08T04:50:42.073 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 8/13 2023-11-08T04:50:42.629 INFO:teuthology.orchestra.run.smithi077.stdout:Last metadata expiration check: 0:02:00 ago on Wed 08 Nov 2023 04:48:42 AM UTC. 2023-11-08T04:50:43.120 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/13 2023-11-08T04:50:43.431 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 10/13 2023-11-08T04:50:43.675 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 11/13 2023-11-08T04:50:43.866 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 12/13 2023-11-08T04:50:44.066 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-mgr-rook-2:16.2.14-129.gcfb6fd52.el8.noarch 13/13 2023-11-08T04:50:44.276 INFO:teuthology.orchestra.run.smithi077.stdout:Dependencies resolved. 2023-11-08T04:50:44.278 INFO:teuthology.orchestra.run.smithi077.stdout:================================================================================ 2023-11-08T04:50:44.278 INFO:teuthology.orchestra.run.smithi077.stdout: Package Arch Version Repository Size 2023-11-08T04:50:44.278 INFO:teuthology.orchestra.run.smithi077.stdout:================================================================================ 2023-11-08T04:50:44.279 INFO:teuthology.orchestra.run.smithi077.stdout:Installing: 2023-11-08T04:50:44.279 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mgr-rook noarch 2:16.2.14-129.gcfb6fd52.el8 ceph-noarch 27 k 2023-11-08T04:50:44.279 INFO:teuthology.orchestra.run.smithi077.stdout:Installing dependencies: 2023-11-08T04:50:44.279 INFO:teuthology.orchestra.run.smithi077.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2023-11-08T04:50:44.279 INFO:teuthology.orchestra.run.smithi077.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2023-11-08T04:50:44.280 INFO:teuthology.orchestra.run.smithi077.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2023-11-08T04:50:44.280 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jsonpatch noarch 1.21-2.el8 appstream 27 k 2023-11-08T04:50:44.280 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jsonpointer noarch 1.10-11.el8 appstream 20 k 2023-11-08T04:50:44.280 INFO:teuthology.orchestra.run.smithi077.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2023-11-08T04:50:44.280 INFO:teuthology.orchestra.run.smithi077.stdout: python3-oauthlib noarch 2.1.0-1.el8 baseos 155 k 2023-11-08T04:50:44.280 INFO:teuthology.orchestra.run.smithi077.stdout: python3-pyasn1 noarch 0.3.7-6.el8 appstream 126 k 2023-11-08T04:50:44.281 INFO:teuthology.orchestra.run.smithi077.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 appstream 110 k 2023-11-08T04:50:44.281 INFO:teuthology.orchestra.run.smithi077.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 baseos 43 k 2023-11-08T04:50:44.281 INFO:teuthology.orchestra.run.smithi077.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2023-11-08T04:50:44.281 INFO:teuthology.orchestra.run.smithi077.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2023-11-08T04:50:44.281 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-11-08T04:50:44.282 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction Summary 2023-11-08T04:50:44.282 INFO:teuthology.orchestra.run.smithi077.stdout:================================================================================ 2023-11-08T04:50:44.282 INFO:teuthology.orchestra.run.smithi077.stdout:Install 13 Packages 2023-11-08T04:50:44.282 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-11-08T04:50:44.283 INFO:teuthology.orchestra.run.smithi077.stdout:Total download size: 1.9 M 2023-11-08T04:50:44.283 INFO:teuthology.orchestra.run.smithi077.stdout:Installed size: 24 M 2023-11-08T04:50:44.283 INFO:teuthology.orchestra.run.smithi077.stdout:Downloading Packages: 2023-11-08T04:50:44.370 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: ceph-mgr-rook-2:16.2.14-129.gcfb6fd52.el8.noarch 13/13 2023-11-08T04:50:44.370 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-mgr-rook-2:16.2.14-129.gcfb6fd52.el8.noarch 1/13 2023-11-08T04:50:44.371 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 2/13 2023-11-08T04:50:44.371 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 3/13 2023-11-08T04:50:44.371 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 4/13 2023-11-08T04:50:44.371 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 5/13 2023-11-08T04:50:44.372 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2023-11-08T04:50:44.372 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2023-11-08T04:50:44.372 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 8/13 2023-11-08T04:50:44.372 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 9/13 2023-11-08T04:50:44.372 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 10/13 2023-11-08T04:50:44.373 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 11/13 2023-11-08T04:50:44.373 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 12/13 2023-11-08T04:50:44.939 INFO:teuthology.orchestra.run.smithi077.stdout:(1/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 104 kB/s | 27 kB 00:00 2023-11-08T04:50:44.963 INFO:teuthology.orchestra.run.smithi077.stdout:(2/13): python3-jsonpointer-1.10-11.el8.noarch. 71 kB/s | 20 kB 00:00 2023-11-08T04:50:45.030 INFO:teuthology.orchestra.run.smithi077.stdout:(3/13): ceph-mgr-rook-16.2.14-129.gcfb6fd52.el8 79 kB/s | 27 kB 00:00 2023-11-08T04:50:45.154 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 13/13 2023-11-08T04:50:45.155 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T04:50:45.155 INFO:teuthology.orchestra.run.smithi087.stdout:Installed: 2023-11-08T04:50:45.155 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-mgr-rook-2:16.2.14-129.gcfb6fd52.el8.noarch 2023-11-08T04:50:45.155 INFO:teuthology.orchestra.run.smithi087.stdout: python3-cachetools-3.1.1-4.el8.noarch 2023-11-08T04:50:45.155 INFO:teuthology.orchestra.run.smithi087.stdout: python3-certifi-2018.10.15-7.el8.noarch 2023-11-08T04:50:45.156 INFO:teuthology.orchestra.run.smithi087.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2023-11-08T04:50:45.156 INFO:teuthology.orchestra.run.smithi087.stdout: python3-jsonpatch-1.21-2.el8.noarch 2023-11-08T04:50:45.156 INFO:teuthology.orchestra.run.smithi087.stdout: python3-jsonpointer-1.10-11.el8.noarch 2023-11-08T04:50:45.156 INFO:teuthology.orchestra.run.smithi087.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2023-11-08T04:50:45.156 INFO:teuthology.orchestra.run.smithi087.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2023-11-08T04:50:45.157 INFO:teuthology.orchestra.run.smithi087.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2023-11-08T04:50:45.157 INFO:teuthology.orchestra.run.smithi087.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2023-11-08T04:50:45.157 INFO:teuthology.orchestra.run.smithi087.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2023-11-08T04:50:45.157 INFO:teuthology.orchestra.run.smithi087.stdout: python3-rsa-4.9-2.el8.noarch 2023-11-08T04:50:45.157 INFO:teuthology.orchestra.run.smithi087.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2023-11-08T04:50:45.158 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T04:50:45.158 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2023-11-08T04:50:45.197 INFO:teuthology.orchestra.run.smithi077.stdout:(4/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 541 kB/s | 126 kB 00:00 2023-11-08T04:50:45.214 INFO:teuthology.orchestra.run.smithi077.stdout:(5/13): python3-pyasn1-modules-0.3.7-6.el8.noar 439 kB/s | 110 kB 00:00 2023-11-08T04:50:45.303 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install ceph-mgr-cephadm 2023-11-08T04:50:45.365 INFO:teuthology.orchestra.run.smithi077.stdout:(6/13): python3-cachetools-3.1.1-4.el8.noarch.r 220 kB/s | 33 kB 00:00 2023-11-08T04:50:45.423 INFO:teuthology.orchestra.run.smithi077.stdout:(7/13): python3-oauthlib-2.1.0-1.el8.noarch.rpm 394 kB/s | 155 kB 00:00 2023-11-08T04:50:45.449 INFO:teuthology.orchestra.run.smithi077.stdout:(8/13): python3-certifi-2018.10.15-7.el8.noarch 195 kB/s | 16 kB 00:00 2023-11-08T04:50:45.507 INFO:teuthology.orchestra.run.smithi077.stdout:(9/13): python3-requests-oauthlib-1.0.0-1.el8.n 138 kB/s | 43 kB 00:00 2023-11-08T04:50:45.624 INFO:teuthology.orchestra.run.smithi077.stdout:(10/13): python3-google-auth-1.1.1-10.el8.noarc 405 kB/s | 81 kB 00:00 2023-11-08T04:50:45.691 INFO:teuthology.orchestra.run.smithi077.stdout:(11/13): python3-websocket-client-0.56.0-5.el8. 921 kB/s | 61 kB 00:00 2023-11-08T04:50:45.716 INFO:teuthology.orchestra.run.smithi077.stdout:(12/13): python3-rsa-4.9-2.el8.noarch.rpm 307 kB/s | 64 kB 00:00 2023-11-08T04:50:45.750 INFO:teuthology.orchestra.run.smithi077.stdout:(13/13): python3-kubernetes-11.0.0-6.el8.noarch 3.9 MB/s | 1.2 MB 00:00 2023-11-08T04:50:45.751 INFO:teuthology.orchestra.run.smithi077.stdout:-------------------------------------------------------------------------------- 2023-11-08T04:50:45.751 INFO:teuthology.orchestra.run.smithi077.stdout:Total 1.3 MB/s | 1.9 MB 00:01 2023-11-08T04:50:45.810 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction check 2023-11-08T04:50:45.826 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction check succeeded. 2023-11-08T04:50:45.826 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction test 2023-11-08T04:50:45.912 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction test succeeded. 2023-11-08T04:50:45.915 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction 2023-11-08T04:50:46.153 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:02:07 ago on Wed 08 Nov 2023 04:48:39 AM UTC. 2023-11-08T04:50:46.405 INFO:teuthology.orchestra.run.smithi077.stdout: Preparing : 1/1 2023-11-08T04:50:46.670 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2023-11-08T04:50:46.871 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/13 2023-11-08T04:50:47.107 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-rsa-4.9-2.el8.noarch 3/13 2023-11-08T04:50:47.354 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/13 2023-11-08T04:50:47.570 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/13 2023-11-08T04:50:47.811 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2023-11-08T04:50:47.812 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2023-11-08T04:50:47.812 INFO:teuthology.orchestra.run.smithi087.stdout: Package Arch Version Repository Size 2023-11-08T04:50:47.812 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2023-11-08T04:50:47.813 INFO:teuthology.orchestra.run.smithi087.stdout:Installing: 2023-11-08T04:50:47.813 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-mgr-cephadm noarch 2:16.2.14-129.gcfb6fd52.el8 ceph-noarch 116 k 2023-11-08T04:50:47.813 INFO:teuthology.orchestra.run.smithi087.stdout:Installing dependencies: 2023-11-08T04:50:47.813 INFO:teuthology.orchestra.run.smithi087.stdout: python3-apipkg noarch 1.5-6.el8 epel 21 k 2023-11-08T04:50:47.813 INFO:teuthology.orchestra.run.smithi087.stdout: python3-babel noarch 2.5.1-7.el8 appstream 4.8 M 2023-11-08T04:50:47.814 INFO:teuthology.orchestra.run.smithi087.stdout: python3-execnet noarch 1.7.1-1.el8 epel 355 k 2023-11-08T04:50:47.814 INFO:teuthology.orchestra.run.smithi087.stdout: python3-jinja2 noarch 2.10.1-3.el8 appstream 538 k 2023-11-08T04:50:47.814 INFO:teuthology.orchestra.run.smithi087.stdout: python3-remoto noarch 1.2.1-1.el8 epel 49 k 2023-11-08T04:50:47.814 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T04:50:47.815 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction Summary 2023-11-08T04:50:47.815 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2023-11-08T04:50:47.815 INFO:teuthology.orchestra.run.smithi087.stdout:Install 6 Packages 2023-11-08T04:50:47.815 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T04:50:47.816 INFO:teuthology.orchestra.run.smithi087.stdout:Total download size: 5.8 M 2023-11-08T04:50:47.816 INFO:teuthology.orchestra.run.smithi087.stdout:Installed size: 24 M 2023-11-08T04:50:47.816 INFO:teuthology.orchestra.run.smithi087.stdout:Downloading Packages: 2023-11-08T04:50:47.836 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/13 2023-11-08T04:50:48.141 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/13 2023-11-08T04:50:48.262 INFO:teuthology.orchestra.run.smithi087.stdout:(1/6): ceph-mgr-cephadm-16.2.14-129.gcfb6fd52.e 626 kB/s | 116 kB 00:00 2023-11-08T04:50:48.385 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 8/13 2023-11-08T04:50:48.587 INFO:teuthology.orchestra.run.smithi087.stdout:(2/6): python3-jinja2-2.10.1-3.el8.noarch.rpm 1.0 MB/s | 538 kB 00:00 2023-11-08T04:50:48.712 INFO:teuthology.orchestra.run.smithi087.stdout:(3/6): python3-apipkg-1.5-6.el8.noarch.rpm 47 kB/s | 21 kB 00:00 2023-11-08T04:50:48.921 INFO:teuthology.orchestra.run.smithi087.stdout:(4/6): python3-babel-2.5.1-7.el8.noarch.rpm 5.7 MB/s | 4.8 MB 00:00 2023-11-08T04:50:48.946 INFO:teuthology.orchestra.run.smithi087.stdout:(5/6): python3-remoto-1.2.1-1.el8.noarch.rpm 209 kB/s | 49 kB 00:00 2023-11-08T04:50:49.421 INFO:teuthology.orchestra.run.smithi087.stdout:(6/6): python3-execnet-1.7.1-1.el8.noarch.rpm 426 kB/s | 355 kB 00:00 2023-11-08T04:50:49.422 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/13 2023-11-08T04:50:49.424 INFO:teuthology.orchestra.run.smithi087.stdout:-------------------------------------------------------------------------------- 2023-11-08T04:50:49.425 INFO:teuthology.orchestra.run.smithi087.stdout:Total 3.6 MB/s | 5.8 MB 00:01 2023-11-08T04:50:49.479 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction check 2023-11-08T04:50:49.489 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction check succeeded. 2023-11-08T04:50:49.490 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction test 2023-11-08T04:50:49.600 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction test succeeded. 2023-11-08T04:50:49.603 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction 2023-11-08T04:50:49.694 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 10/13 2023-11-08T04:50:49.923 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 11/13 2023-11-08T04:50:49.933 INFO:teuthology.orchestra.run.smithi087.stdout: Preparing : 1/1 2023-11-08T04:50:50.137 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 12/13 2023-11-08T04:50:50.206 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-apipkg-1.5-6.el8.noarch 1/6 2023-11-08T04:50:50.304 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-mgr-rook-2:16.2.14-129.gcfb6fd52.el8.noarch 13/13 2023-11-08T04:50:50.609 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-mgr-rook-2:16.2.14-129.gcfb6fd52.el8.noarch 13/13 2023-11-08T04:50:50.609 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-mgr-rook-2:16.2.14-129.gcfb6fd52.el8.noarch 1/13 2023-11-08T04:50:50.610 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 2/13 2023-11-08T04:50:50.610 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 3/13 2023-11-08T04:50:50.610 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 4/13 2023-11-08T04:50:50.611 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 5/13 2023-11-08T04:50:50.611 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2023-11-08T04:50:50.611 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2023-11-08T04:50:50.611 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 8/13 2023-11-08T04:50:50.612 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 9/13 2023-11-08T04:50:50.612 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 10/13 2023-11-08T04:50:50.612 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 11/13 2023-11-08T04:50:50.612 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 12/13 2023-11-08T04:50:50.854 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-execnet-1.7.1-1.el8.noarch 2/6 2023-11-08T04:50:51.338 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 13/13 2023-11-08T04:50:51.339 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-11-08T04:50:51.339 INFO:teuthology.orchestra.run.smithi077.stdout:Installed: 2023-11-08T04:50:51.339 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mgr-rook-2:16.2.14-129.gcfb6fd52.el8.noarch 2023-11-08T04:50:51.340 INFO:teuthology.orchestra.run.smithi077.stdout: python3-cachetools-3.1.1-4.el8.noarch 2023-11-08T04:50:51.340 INFO:teuthology.orchestra.run.smithi077.stdout: python3-certifi-2018.10.15-7.el8.noarch 2023-11-08T04:50:51.340 INFO:teuthology.orchestra.run.smithi077.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2023-11-08T04:50:51.340 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jsonpatch-1.21-2.el8.noarch 2023-11-08T04:50:51.341 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jsonpointer-1.10-11.el8.noarch 2023-11-08T04:50:51.341 INFO:teuthology.orchestra.run.smithi077.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2023-11-08T04:50:51.341 INFO:teuthology.orchestra.run.smithi077.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2023-11-08T04:50:51.341 INFO:teuthology.orchestra.run.smithi077.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2023-11-08T04:50:51.341 INFO:teuthology.orchestra.run.smithi077.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2023-11-08T04:50:51.342 INFO:teuthology.orchestra.run.smithi077.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2023-11-08T04:50:51.342 INFO:teuthology.orchestra.run.smithi077.stdout: python3-rsa-4.9-2.el8.noarch 2023-11-08T04:50:51.342 INFO:teuthology.orchestra.run.smithi077.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2023-11-08T04:50:51.342 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-11-08T04:50:51.342 INFO:teuthology.orchestra.run.smithi077.stdout:Complete! 2023-11-08T04:50:51.479 DEBUG:teuthology.orchestra.run.smithi077:> sudo yum -y install ceph-mgr-cephadm 2023-11-08T04:50:51.837 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-remoto-1.2.1-1.el8.noarch 3/6 2023-11-08T04:50:52.210 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 4/6 2023-11-08T04:50:52.337 INFO:teuthology.orchestra.run.smithi077.stdout:Last metadata expiration check: 0:02:10 ago on Wed 08 Nov 2023 04:48:42 AM UTC. 2023-11-08T04:50:52.481 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 5/6 2023-11-08T04:50:52.710 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-mgr-cephadm-2:16.2.14-129.gcfb6fd52.el8.noarch 6/6 2023-11-08T04:50:52.995 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: ceph-mgr-cephadm-2:16.2.14-129.gcfb6fd52.el8.noarch 6/6 2023-11-08T04:50:52.996 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-mgr-cephadm-2:16.2.14-129.gcfb6fd52.el8.noarch 1/6 2023-11-08T04:50:52.996 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 2/6 2023-11-08T04:50:52.996 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 3/6 2023-11-08T04:50:52.996 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-apipkg-1.5-6.el8.noarch 4/6 2023-11-08T04:50:52.996 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-execnet-1.7.1-1.el8.noarch 5/6 2023-11-08T04:50:53.707 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : python3-remoto-1.2.1-1.el8.noarch 6/6 2023-11-08T04:50:53.707 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T04:50:53.707 INFO:teuthology.orchestra.run.smithi087.stdout:Installed: 2023-11-08T04:50:53.708 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-mgr-cephadm-2:16.2.14-129.gcfb6fd52.el8.noarch 2023-11-08T04:50:53.708 INFO:teuthology.orchestra.run.smithi087.stdout: python3-apipkg-1.5-6.el8.noarch 2023-11-08T04:50:53.708 INFO:teuthology.orchestra.run.smithi087.stdout: python3-babel-2.5.1-7.el8.noarch 2023-11-08T04:50:53.708 INFO:teuthology.orchestra.run.smithi087.stdout: python3-execnet-1.7.1-1.el8.noarch 2023-11-08T04:50:53.709 INFO:teuthology.orchestra.run.smithi087.stdout: python3-jinja2-2.10.1-3.el8.noarch 2023-11-08T04:50:53.709 INFO:teuthology.orchestra.run.smithi087.stdout: python3-remoto-1.2.1-1.el8.noarch 2023-11-08T04:50:53.709 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T04:50:53.709 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2023-11-08T04:50:53.817 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install ceph-fuse 2023-11-08T04:50:53.997 INFO:teuthology.orchestra.run.smithi077.stdout:Dependencies resolved. 2023-11-08T04:50:53.998 INFO:teuthology.orchestra.run.smithi077.stdout:================================================================================ 2023-11-08T04:50:53.998 INFO:teuthology.orchestra.run.smithi077.stdout: Package Arch Version Repository Size 2023-11-08T04:50:53.998 INFO:teuthology.orchestra.run.smithi077.stdout:================================================================================ 2023-11-08T04:50:53.999 INFO:teuthology.orchestra.run.smithi077.stdout:Installing: 2023-11-08T04:50:53.999 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mgr-cephadm noarch 2:16.2.14-129.gcfb6fd52.el8 ceph-noarch 116 k 2023-11-08T04:50:53.999 INFO:teuthology.orchestra.run.smithi077.stdout:Installing dependencies: 2023-11-08T04:50:53.999 INFO:teuthology.orchestra.run.smithi077.stdout: python3-apipkg noarch 1.5-6.el8 epel 21 k 2023-11-08T04:50:53.999 INFO:teuthology.orchestra.run.smithi077.stdout: python3-babel noarch 2.5.1-7.el8 appstream 4.8 M 2023-11-08T04:50:54.000 INFO:teuthology.orchestra.run.smithi077.stdout: python3-execnet noarch 1.7.1-1.el8 epel 355 k 2023-11-08T04:50:54.000 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jinja2 noarch 2.10.1-3.el8 appstream 538 k 2023-11-08T04:50:54.000 INFO:teuthology.orchestra.run.smithi077.stdout: python3-remoto noarch 1.2.1-1.el8 epel 49 k 2023-11-08T04:50:54.000 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-11-08T04:50:54.000 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction Summary 2023-11-08T04:50:54.000 INFO:teuthology.orchestra.run.smithi077.stdout:================================================================================ 2023-11-08T04:50:54.001 INFO:teuthology.orchestra.run.smithi077.stdout:Install 6 Packages 2023-11-08T04:50:54.001 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-11-08T04:50:54.001 INFO:teuthology.orchestra.run.smithi077.stdout:Total download size: 5.8 M 2023-11-08T04:50:54.002 INFO:teuthology.orchestra.run.smithi077.stdout:Installed size: 24 M 2023-11-08T04:50:54.002 INFO:teuthology.orchestra.run.smithi077.stdout:Downloading Packages: 2023-11-08T04:50:54.680 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:02:15 ago on Wed 08 Nov 2023 04:48:39 AM UTC. 2023-11-08T04:50:54.733 INFO:teuthology.orchestra.run.smithi077.stdout:(1/6): ceph-mgr-cephadm-16.2.14-129.gcfb6fd52.e 579 kB/s | 116 kB 00:00 2023-11-08T04:50:54.933 INFO:teuthology.orchestra.run.smithi077.stdout:(2/6): python3-apipkg-1.5-6.el8.noarch.rpm 106 kB/s | 21 kB 00:00 2023-11-08T04:50:55.042 INFO:teuthology.orchestra.run.smithi077.stdout:(3/6): python3-jinja2-2.10.1-3.el8.noarch.rpm 1.0 MB/s | 538 kB 00:00 2023-11-08T04:50:55.100 INFO:teuthology.orchestra.run.smithi077.stdout:(4/6): python3-execnet-1.7.1-1.el8.noarch.rpm 2.1 MB/s | 355 kB 00:00 2023-11-08T04:50:55.292 INFO:teuthology.orchestra.run.smithi077.stdout:(5/6): python3-remoto-1.2.1-1.el8.noarch.rpm 195 kB/s | 49 kB 00:00 2023-11-08T04:50:55.376 INFO:teuthology.orchestra.run.smithi077.stdout:(6/6): python3-babel-2.5.1-7.el8.noarch.rpm 5.7 MB/s | 4.8 MB 00:00 2023-11-08T04:50:55.379 INFO:teuthology.orchestra.run.smithi077.stdout:-------------------------------------------------------------------------------- 2023-11-08T04:50:55.379 INFO:teuthology.orchestra.run.smithi077.stdout:Total 4.2 MB/s | 5.8 MB 00:01 2023-11-08T04:50:55.433 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction check 2023-11-08T04:50:55.444 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction check succeeded. 2023-11-08T04:50:55.444 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction test 2023-11-08T04:50:55.546 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction test succeeded. 2023-11-08T04:50:55.549 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction 2023-11-08T04:50:55.872 INFO:teuthology.orchestra.run.smithi077.stdout: Preparing : 1/1 2023-11-08T04:50:56.137 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-apipkg-1.5-6.el8.noarch 1/6 2023-11-08T04:50:56.330 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2023-11-08T04:50:56.331 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2023-11-08T04:50:56.331 INFO:teuthology.orchestra.run.smithi087.stdout: Package Arch Version Repository Size 2023-11-08T04:50:56.331 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2023-11-08T04:50:56.331 INFO:teuthology.orchestra.run.smithi087.stdout:Installing: 2023-11-08T04:50:56.332 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-fuse x86_64 2:16.2.14-129.gcfb6fd52.el8 ceph 910 k 2023-11-08T04:50:56.332 INFO:teuthology.orchestra.run.smithi087.stdout:Installing dependencies: 2023-11-08T04:50:56.332 INFO:teuthology.orchestra.run.smithi087.stdout: fuse x86_64 2.9.7-17.el8 baseos 83 k 2023-11-08T04:50:56.332 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T04:50:56.333 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction Summary 2023-11-08T04:50:56.333 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2023-11-08T04:50:56.333 INFO:teuthology.orchestra.run.smithi087.stdout:Install 2 Packages 2023-11-08T04:50:56.333 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T04:50:56.334 INFO:teuthology.orchestra.run.smithi087.stdout:Total download size: 993 k 2023-11-08T04:50:56.334 INFO:teuthology.orchestra.run.smithi087.stdout:Installed size: 2.8 M 2023-11-08T04:50:56.334 INFO:teuthology.orchestra.run.smithi087.stdout:Downloading Packages: 2023-11-08T04:50:56.448 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-execnet-1.7.1-1.el8.noarch 2/6 2023-11-08T04:50:56.848 INFO:teuthology.orchestra.run.smithi087.stdout:(1/2): fuse-2.9.7-17.el8.x86_64.rpm 249 kB/s | 83 kB 00:00 2023-11-08T04:50:56.948 INFO:teuthology.orchestra.run.smithi087.stdout:(2/2): ceph-fuse-16.2.14-129.gcfb6fd52.el8.x86_ 2.1 MB/s | 910 kB 00:00 2023-11-08T04:50:56.949 INFO:teuthology.orchestra.run.smithi087.stdout:-------------------------------------------------------------------------------- 2023-11-08T04:50:56.950 INFO:teuthology.orchestra.run.smithi087.stdout:Total 1.6 MB/s | 993 kB 00:00 2023-11-08T04:50:56.961 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction check 2023-11-08T04:50:56.982 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction check succeeded. 2023-11-08T04:50:56.982 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction test 2023-11-08T04:50:57.068 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction test succeeded. 2023-11-08T04:50:57.071 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction 2023-11-08T04:50:57.242 INFO:teuthology.orchestra.run.smithi087.stdout: Preparing : 1/1 2023-11-08T04:50:57.454 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-remoto-1.2.1-1.el8.noarch 3/6 2023-11-08T04:50:57.588 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : fuse-2.9.7-17.el8.x86_64 1/2 2023-11-08T04:50:57.806 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : ceph-fuse-2:16.2.14-129.gcfb6fd52.el8.x86_64 2/2 2023-11-08T04:50:58.083 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 4/6 2023-11-08T04:50:58.199 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: ceph-fuse-2:16.2.14-129.gcfb6fd52.el8.x86_64 2/2 2023-11-08T04:50:58.200 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : ceph-fuse-2:16.2.14-129.gcfb6fd52.el8.x86_64 1/2 2023-11-08T04:50:58.345 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 5/6 2023-11-08T04:50:58.556 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-mgr-cephadm-2:16.2.14-129.gcfb6fd52.el8.noarch 6/6 2023-11-08T04:50:58.838 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-mgr-cephadm-2:16.2.14-129.gcfb6fd52.el8.noarch 6/6 2023-11-08T04:50:58.839 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-mgr-cephadm-2:16.2.14-129.gcfb6fd52.el8.noarch 1/6 2023-11-08T04:50:58.839 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 2/6 2023-11-08T04:50:58.839 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 3/6 2023-11-08T04:50:58.839 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-apipkg-1.5-6.el8.noarch 4/6 2023-11-08T04:50:58.840 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-execnet-1.7.1-1.el8.noarch 5/6 2023-11-08T04:50:58.866 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : fuse-2.9.7-17.el8.x86_64 2/2 2023-11-08T04:50:58.866 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T04:50:58.866 INFO:teuthology.orchestra.run.smithi087.stdout:Installed: 2023-11-08T04:50:58.867 INFO:teuthology.orchestra.run.smithi087.stdout: ceph-fuse-2:16.2.14-129.gcfb6fd52.el8.x86_64 fuse-2.9.7-17.el8.x86_64 2023-11-08T04:50:58.867 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T04:50:58.867 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2023-11-08T04:50:59.123 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install librados-devel 2023-11-08T04:50:59.491 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-remoto-1.2.1-1.el8.noarch 6/6 2023-11-08T04:50:59.491 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-11-08T04:50:59.492 INFO:teuthology.orchestra.run.smithi077.stdout:Installed: 2023-11-08T04:50:59.492 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mgr-cephadm-2:16.2.14-129.gcfb6fd52.el8.noarch 2023-11-08T04:50:59.492 INFO:teuthology.orchestra.run.smithi077.stdout: python3-apipkg-1.5-6.el8.noarch 2023-11-08T04:50:59.492 INFO:teuthology.orchestra.run.smithi077.stdout: python3-babel-2.5.1-7.el8.noarch 2023-11-08T04:50:59.492 INFO:teuthology.orchestra.run.smithi077.stdout: python3-execnet-1.7.1-1.el8.noarch 2023-11-08T04:50:59.493 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jinja2-2.10.1-3.el8.noarch 2023-11-08T04:50:59.493 INFO:teuthology.orchestra.run.smithi077.stdout: python3-remoto-1.2.1-1.el8.noarch 2023-11-08T04:50:59.493 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-11-08T04:50:59.493 INFO:teuthology.orchestra.run.smithi077.stdout:Complete! 2023-11-08T04:50:59.596 DEBUG:teuthology.orchestra.run.smithi077:> sudo yum -y install ceph-fuse 2023-11-08T04:50:59.986 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:02:20 ago on Wed 08 Nov 2023 04:48:39 AM UTC. 2023-11-08T04:51:00.456 INFO:teuthology.orchestra.run.smithi077.stdout:Last metadata expiration check: 0:02:18 ago on Wed 08 Nov 2023 04:48:42 AM UTC. 2023-11-08T04:51:01.651 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2023-11-08T04:51:01.652 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2023-11-08T04:51:01.653 INFO:teuthology.orchestra.run.smithi087.stdout: Package Arch Version Repo Size 2023-11-08T04:51:01.653 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2023-11-08T04:51:01.653 INFO:teuthology.orchestra.run.smithi087.stdout:Installing: 2023-11-08T04:51:01.653 INFO:teuthology.orchestra.run.smithi087.stdout: librados-devel x86_64 2:16.2.14-129.gcfb6fd52.el8 ceph 131 k 2023-11-08T04:51:01.653 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T04:51:01.654 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction Summary 2023-11-08T04:51:01.654 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2023-11-08T04:51:01.654 INFO:teuthology.orchestra.run.smithi087.stdout:Install 1 Package 2023-11-08T04:51:01.654 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T04:51:01.655 INFO:teuthology.orchestra.run.smithi087.stdout:Total download size: 131 k 2023-11-08T04:51:01.655 INFO:teuthology.orchestra.run.smithi087.stdout:Installed size: 432 k 2023-11-08T04:51:01.655 INFO:teuthology.orchestra.run.smithi087.stdout:Downloading Packages: 2023-11-08T04:51:02.113 INFO:teuthology.orchestra.run.smithi077.stdout:Dependencies resolved. 2023-11-08T04:51:02.119 INFO:teuthology.orchestra.run.smithi077.stdout:================================================================================ 2023-11-08T04:51:02.120 INFO:teuthology.orchestra.run.smithi077.stdout: Package Arch Version Repository Size 2023-11-08T04:51:02.120 INFO:teuthology.orchestra.run.smithi077.stdout:================================================================================ 2023-11-08T04:51:02.120 INFO:teuthology.orchestra.run.smithi077.stdout:Installing: 2023-11-08T04:51:02.121 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-fuse x86_64 2:16.2.14-129.gcfb6fd52.el8 ceph 910 k 2023-11-08T04:51:02.121 INFO:teuthology.orchestra.run.smithi077.stdout:Installing dependencies: 2023-11-08T04:51:02.122 INFO:teuthology.orchestra.run.smithi077.stdout: fuse x86_64 2.9.7-17.el8 baseos 83 k 2023-11-08T04:51:02.122 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-11-08T04:51:02.122 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction Summary 2023-11-08T04:51:02.123 INFO:teuthology.orchestra.run.smithi077.stdout:================================================================================ 2023-11-08T04:51:02.123 INFO:teuthology.orchestra.run.smithi077.stdout:Install 2 Packages 2023-11-08T04:51:02.123 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-11-08T04:51:02.124 INFO:teuthology.orchestra.run.smithi077.stdout:Total download size: 993 k 2023-11-08T04:51:02.124 INFO:teuthology.orchestra.run.smithi077.stdout:Installed size: 2.8 M 2023-11-08T04:51:02.124 INFO:teuthology.orchestra.run.smithi077.stdout:Downloading Packages: 2023-11-08T04:51:02.640 INFO:teuthology.orchestra.run.smithi087.stdout:librados-devel-16.2.14-129.gcfb6fd52.el8.x86_64 133 kB/s | 131 kB 00:00 2023-11-08T04:51:02.641 INFO:teuthology.orchestra.run.smithi087.stdout:-------------------------------------------------------------------------------- 2023-11-08T04:51:02.641 INFO:teuthology.orchestra.run.smithi087.stdout:Total 132 kB/s | 131 kB 00:00 2023-11-08T04:51:02.642 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction check 2023-11-08T04:51:02.645 INFO:teuthology.orchestra.run.smithi077.stdout:(1/2): fuse-2.9.7-17.el8.x86_64.rpm 255 kB/s | 83 kB 00:00 2023-11-08T04:51:02.648 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction check succeeded. 2023-11-08T04:51:02.648 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction test 2023-11-08T04:51:02.704 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction test succeeded. 2023-11-08T04:51:02.707 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction 2023-11-08T04:51:02.878 INFO:teuthology.orchestra.run.smithi077.stdout:(2/2): ceph-fuse-16.2.14-129.gcfb6fd52.el8.x86_ 1.6 MB/s | 910 kB 00:00 2023-11-08T04:51:02.879 INFO:teuthology.orchestra.run.smithi077.stdout:-------------------------------------------------------------------------------- 2023-11-08T04:51:02.879 INFO:teuthology.orchestra.run.smithi077.stdout:Total 1.3 MB/s | 993 kB 00:00 2023-11-08T04:51:02.890 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction check 2023-11-08T04:51:02.911 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction check succeeded. 2023-11-08T04:51:02.911 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction test 2023-11-08T04:51:02.997 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction test succeeded. 2023-11-08T04:51:02.999 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction 2023-11-08T04:51:03.156 INFO:teuthology.orchestra.run.smithi077.stdout: Preparing : 1/1 2023-11-08T04:51:03.315 INFO:teuthology.orchestra.run.smithi087.stdout: Preparing : 1/1 2023-11-08T04:51:03.502 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : fuse-2.9.7-17.el8.x86_64 1/2 2023-11-08T04:51:03.588 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : librados-devel-2:16.2.14-129.gcfb6fd52.el8.x86_64 1/1 2023-11-08T04:51:03.736 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-fuse-2:16.2.14-129.gcfb6fd52.el8.x86_64 2/2 2023-11-08T04:51:03.886 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: librados-devel-2:16.2.14-129.gcfb6fd52.el8.x86_64 1/1 2023-11-08T04:51:04.127 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-fuse-2:16.2.14-129.gcfb6fd52.el8.x86_64 2/2 2023-11-08T04:51:04.127 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-fuse-2:16.2.14-129.gcfb6fd52.el8.x86_64 1/2 2023-11-08T04:51:04.564 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : librados-devel-2:16.2.14-129.gcfb6fd52.el8.x86_64 1/1 2023-11-08T04:51:04.564 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T04:51:04.564 INFO:teuthology.orchestra.run.smithi087.stdout:Installed: 2023-11-08T04:51:04.565 INFO:teuthology.orchestra.run.smithi087.stdout: librados-devel-2:16.2.14-129.gcfb6fd52.el8.x86_64 2023-11-08T04:51:04.565 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T04:51:04.565 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2023-11-08T04:51:04.782 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : fuse-2.9.7-17.el8.x86_64 2/2 2023-11-08T04:51:04.782 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-11-08T04:51:04.782 INFO:teuthology.orchestra.run.smithi077.stdout:Installed: 2023-11-08T04:51:04.783 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-fuse-2:16.2.14-129.gcfb6fd52.el8.x86_64 fuse-2.9.7-17.el8.x86_64 2023-11-08T04:51:04.783 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-11-08T04:51:04.783 INFO:teuthology.orchestra.run.smithi077.stdout:Complete! 2023-11-08T04:51:04.818 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install libcephfs2 2023-11-08T04:51:04.968 DEBUG:teuthology.orchestra.run.smithi077:> sudo yum -y install librados-devel 2023-11-08T04:51:05.683 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:02:26 ago on Wed 08 Nov 2023 04:48:39 AM UTC. 2023-11-08T04:51:05.829 INFO:teuthology.orchestra.run.smithi077.stdout:Last metadata expiration check: 0:02:23 ago on Wed 08 Nov 2023 04:48:42 AM UTC. 2023-11-08T04:51:07.116 INFO:teuthology.orchestra.run.smithi087.stdout:Package libcephfs2-2:16.2.14-129.gcfb6fd52.el8.x86_64 is already installed. 2023-11-08T04:51:07.326 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2023-11-08T04:51:07.327 INFO:teuthology.orchestra.run.smithi087.stdout:Nothing to do. 2023-11-08T04:51:07.327 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2023-11-08T04:51:07.412 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install libcephfs-devel 2023-11-08T04:51:07.491 INFO:teuthology.orchestra.run.smithi077.stdout:Dependencies resolved. 2023-11-08T04:51:07.492 INFO:teuthology.orchestra.run.smithi077.stdout:================================================================================ 2023-11-08T04:51:07.492 INFO:teuthology.orchestra.run.smithi077.stdout: Package Arch Version Repo Size 2023-11-08T04:51:07.492 INFO:teuthology.orchestra.run.smithi077.stdout:================================================================================ 2023-11-08T04:51:07.493 INFO:teuthology.orchestra.run.smithi077.stdout:Installing: 2023-11-08T04:51:07.493 INFO:teuthology.orchestra.run.smithi077.stdout: librados-devel x86_64 2:16.2.14-129.gcfb6fd52.el8 ceph 131 k 2023-11-08T04:51:07.493 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-11-08T04:51:07.493 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction Summary 2023-11-08T04:51:07.493 INFO:teuthology.orchestra.run.smithi077.stdout:================================================================================ 2023-11-08T04:51:07.494 INFO:teuthology.orchestra.run.smithi077.stdout:Install 1 Package 2023-11-08T04:51:07.494 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-11-08T04:51:07.494 INFO:teuthology.orchestra.run.smithi077.stdout:Total download size: 131 k 2023-11-08T04:51:07.494 INFO:teuthology.orchestra.run.smithi077.stdout:Installed size: 432 k 2023-11-08T04:51:07.495 INFO:teuthology.orchestra.run.smithi077.stdout:Downloading Packages: 2023-11-08T04:51:07.999 INFO:teuthology.orchestra.run.smithi077.stdout:librados-devel-16.2.14-129.gcfb6fd52.el8.x86_64 259 kB/s | 131 kB 00:00 2023-11-08T04:51:08.000 INFO:teuthology.orchestra.run.smithi077.stdout:-------------------------------------------------------------------------------- 2023-11-08T04:51:08.000 INFO:teuthology.orchestra.run.smithi077.stdout:Total 258 kB/s | 131 kB 00:00 2023-11-08T04:51:08.000 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction check 2023-11-08T04:51:08.005 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction check succeeded. 2023-11-08T04:51:08.005 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction test 2023-11-08T04:51:08.057 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction test succeeded. 2023-11-08T04:51:08.059 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction 2023-11-08T04:51:08.167 INFO:teuthology.orchestra.run.smithi077.stdout: Preparing : 1/1 2023-11-08T04:51:08.261 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:02:29 ago on Wed 08 Nov 2023 04:48:39 AM UTC. 2023-11-08T04:51:09.400 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : librados-devel-2:16.2.14-129.gcfb6fd52.el8.x86_64 1/1 2023-11-08T04:51:09.910 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2023-11-08T04:51:09.911 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2023-11-08T04:51:09.912 INFO:teuthology.orchestra.run.smithi087.stdout: Package Arch Version Repo Size 2023-11-08T04:51:09.912 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2023-11-08T04:51:09.912 INFO:teuthology.orchestra.run.smithi087.stdout:Installing: 2023-11-08T04:51:09.912 INFO:teuthology.orchestra.run.smithi087.stdout: libcephfs-devel x86_64 2:16.2.14-129.gcfb6fd52.el8 ceph 25 k 2023-11-08T04:51:09.912 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T04:51:09.913 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction Summary 2023-11-08T04:51:09.913 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2023-11-08T04:51:09.913 INFO:teuthology.orchestra.run.smithi087.stdout:Install 1 Package 2023-11-08T04:51:09.913 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T04:51:09.913 INFO:teuthology.orchestra.run.smithi087.stdout:Total download size: 25 k 2023-11-08T04:51:09.913 INFO:teuthology.orchestra.run.smithi087.stdout:Installed size: 107 k 2023-11-08T04:51:09.914 INFO:teuthology.orchestra.run.smithi087.stdout:Downloading Packages: 2023-11-08T04:51:10.070 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: librados-devel-2:16.2.14-129.gcfb6fd52.el8.x86_64 1/1 2023-11-08T04:51:10.166 INFO:teuthology.orchestra.run.smithi087.stdout:libcephfs-devel-16.2.14-129.gcfb6fd52.el8.x86_6 100 kB/s | 25 kB 00:00 2023-11-08T04:51:10.167 INFO:teuthology.orchestra.run.smithi087.stdout:-------------------------------------------------------------------------------- 2023-11-08T04:51:10.167 INFO:teuthology.orchestra.run.smithi087.stdout:Total 100 kB/s | 25 kB 00:00 2023-11-08T04:51:10.168 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction check 2023-11-08T04:51:10.170 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction check succeeded. 2023-11-08T04:51:10.171 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction test 2023-11-08T04:51:10.173 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction test succeeded. 2023-11-08T04:51:10.176 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction 2023-11-08T04:51:10.223 INFO:teuthology.orchestra.run.smithi087.stdout: Preparing : 1/1 2023-11-08T04:51:10.411 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : libcephfs-devel-2:16.2.14-129.gcfb6fd52.el8.x86_64 1/1 2023-11-08T04:51:10.678 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: libcephfs-devel-2:16.2.14-129.gcfb6fd52.el8.x86_64 1/1 2023-11-08T04:51:11.104 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : librados-devel-2:16.2.14-129.gcfb6fd52.el8.x86_64 1/1 2023-11-08T04:51:11.105 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-11-08T04:51:11.105 INFO:teuthology.orchestra.run.smithi077.stdout:Installed: 2023-11-08T04:51:11.105 INFO:teuthology.orchestra.run.smithi077.stdout: librados-devel-2:16.2.14-129.gcfb6fd52.el8.x86_64 2023-11-08T04:51:11.105 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-11-08T04:51:11.106 INFO:teuthology.orchestra.run.smithi077.stdout:Complete! 2023-11-08T04:51:11.309 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : libcephfs-devel-2:16.2.14-129.gcfb6fd52.el8.x86_64 1/1 2023-11-08T04:51:11.309 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T04:51:11.309 INFO:teuthology.orchestra.run.smithi087.stdout:Installed: 2023-11-08T04:51:11.310 INFO:teuthology.orchestra.run.smithi087.stdout: libcephfs-devel-2:16.2.14-129.gcfb6fd52.el8.x86_64 2023-11-08T04:51:11.310 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T04:51:11.310 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2023-11-08T04:51:11.311 DEBUG:teuthology.orchestra.run.smithi077:> sudo yum -y install libcephfs2 2023-11-08T04:51:11.415 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install librados2 2023-11-08T04:51:12.182 INFO:teuthology.orchestra.run.smithi077.stdout:Last metadata expiration check: 0:02:30 ago on Wed 08 Nov 2023 04:48:42 AM UTC. 2023-11-08T04:51:12.270 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:02:33 ago on Wed 08 Nov 2023 04:48:39 AM UTC. 2023-11-08T04:51:13.629 INFO:teuthology.orchestra.run.smithi077.stdout:Package libcephfs2-2:16.2.14-129.gcfb6fd52.el8.x86_64 is already installed. 2023-11-08T04:51:13.708 INFO:teuthology.orchestra.run.smithi087.stdout:Package librados2-2:16.2.14-129.gcfb6fd52.el8.x86_64 is already installed. 2023-11-08T04:51:13.841 INFO:teuthology.orchestra.run.smithi077.stdout:Dependencies resolved. 2023-11-08T04:51:13.842 INFO:teuthology.orchestra.run.smithi077.stdout:Nothing to do. 2023-11-08T04:51:13.842 INFO:teuthology.orchestra.run.smithi077.stdout:Complete! 2023-11-08T04:51:13.922 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2023-11-08T04:51:13.923 INFO:teuthology.orchestra.run.smithi087.stdout:Nothing to do. 2023-11-08T04:51:13.924 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2023-11-08T04:51:13.936 DEBUG:teuthology.orchestra.run.smithi077:> sudo yum -y install libcephfs-devel 2023-11-08T04:51:14.212 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install librbd1 2023-11-08T04:51:14.788 INFO:teuthology.orchestra.run.smithi077.stdout:Last metadata expiration check: 0:02:32 ago on Wed 08 Nov 2023 04:48:42 AM UTC. 2023-11-08T04:51:15.073 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:02:36 ago on Wed 08 Nov 2023 04:48:39 AM UTC. 2023-11-08T04:51:16.443 INFO:teuthology.orchestra.run.smithi077.stdout:Dependencies resolved. 2023-11-08T04:51:16.445 INFO:teuthology.orchestra.run.smithi077.stdout:================================================================================ 2023-11-08T04:51:16.445 INFO:teuthology.orchestra.run.smithi077.stdout: Package Arch Version Repo Size 2023-11-08T04:51:16.445 INFO:teuthology.orchestra.run.smithi077.stdout:================================================================================ 2023-11-08T04:51:16.445 INFO:teuthology.orchestra.run.smithi077.stdout:Installing: 2023-11-08T04:51:16.446 INFO:teuthology.orchestra.run.smithi077.stdout: libcephfs-devel x86_64 2:16.2.14-129.gcfb6fd52.el8 ceph 25 k 2023-11-08T04:51:16.446 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-11-08T04:51:16.446 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction Summary 2023-11-08T04:51:16.446 INFO:teuthology.orchestra.run.smithi077.stdout:================================================================================ 2023-11-08T04:51:16.446 INFO:teuthology.orchestra.run.smithi077.stdout:Install 1 Package 2023-11-08T04:51:16.447 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-11-08T04:51:16.447 INFO:teuthology.orchestra.run.smithi077.stdout:Total download size: 25 k 2023-11-08T04:51:16.447 INFO:teuthology.orchestra.run.smithi077.stdout:Installed size: 107 k 2023-11-08T04:51:16.447 INFO:teuthology.orchestra.run.smithi077.stdout:Downloading Packages: 2023-11-08T04:51:16.520 INFO:teuthology.orchestra.run.smithi087.stdout:Package librbd1-2:16.2.14-129.gcfb6fd52.el8.x86_64 is already installed. 2023-11-08T04:51:16.732 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2023-11-08T04:51:16.733 INFO:teuthology.orchestra.run.smithi087.stdout:Nothing to do. 2023-11-08T04:51:16.734 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2023-11-08T04:51:16.756 INFO:teuthology.orchestra.run.smithi077.stdout:libcephfs-devel-16.2.14-129.gcfb6fd52.el8.x86_6 82 kB/s | 25 kB 00:00 2023-11-08T04:51:16.757 INFO:teuthology.orchestra.run.smithi077.stdout:-------------------------------------------------------------------------------- 2023-11-08T04:51:16.757 INFO:teuthology.orchestra.run.smithi077.stdout:Total 82 kB/s | 25 kB 00:00 2023-11-08T04:51:16.757 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction check 2023-11-08T04:51:16.760 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction check succeeded. 2023-11-08T04:51:16.760 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction test 2023-11-08T04:51:16.763 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction test succeeded. 2023-11-08T04:51:16.766 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction 2023-11-08T04:51:16.813 INFO:teuthology.orchestra.run.smithi077.stdout: Preparing : 1/1 2023-11-08T04:51:16.825 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install python3-rados 2023-11-08T04:51:16.991 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : libcephfs-devel-2:16.2.14-129.gcfb6fd52.el8.x86_64 1/1 2023-11-08T04:51:17.249 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: libcephfs-devel-2:16.2.14-129.gcfb6fd52.el8.x86_64 1/1 2023-11-08T04:51:17.686 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:02:38 ago on Wed 08 Nov 2023 04:48:39 AM UTC. 2023-11-08T04:51:17.855 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libcephfs-devel-2:16.2.14-129.gcfb6fd52.el8.x86_64 1/1 2023-11-08T04:51:17.856 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-11-08T04:51:17.856 INFO:teuthology.orchestra.run.smithi077.stdout:Installed: 2023-11-08T04:51:17.856 INFO:teuthology.orchestra.run.smithi077.stdout: libcephfs-devel-2:16.2.14-129.gcfb6fd52.el8.x86_64 2023-11-08T04:51:17.856 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-11-08T04:51:17.856 INFO:teuthology.orchestra.run.smithi077.stdout:Complete! 2023-11-08T04:51:17.963 DEBUG:teuthology.orchestra.run.smithi077:> sudo yum -y install librados2 2023-11-08T04:51:18.814 INFO:teuthology.orchestra.run.smithi077.stdout:Last metadata expiration check: 0:02:36 ago on Wed 08 Nov 2023 04:48:42 AM UTC. 2023-11-08T04:51:19.130 INFO:teuthology.orchestra.run.smithi087.stdout:Package python3-rados-2:16.2.14-129.gcfb6fd52.el8.x86_64 is already installed. 2023-11-08T04:51:19.338 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2023-11-08T04:51:19.339 INFO:teuthology.orchestra.run.smithi087.stdout:Nothing to do. 2023-11-08T04:51:19.339 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2023-11-08T04:51:19.441 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install python3-rgw 2023-11-08T04:51:20.264 INFO:teuthology.orchestra.run.smithi077.stdout:Package librados2-2:16.2.14-129.gcfb6fd52.el8.x86_64 is already installed. 2023-11-08T04:51:20.299 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:02:41 ago on Wed 08 Nov 2023 04:48:39 AM UTC. 2023-11-08T04:51:20.474 INFO:teuthology.orchestra.run.smithi077.stdout:Dependencies resolved. 2023-11-08T04:51:20.475 INFO:teuthology.orchestra.run.smithi077.stdout:Nothing to do. 2023-11-08T04:51:20.476 INFO:teuthology.orchestra.run.smithi077.stdout:Complete! 2023-11-08T04:51:20.565 DEBUG:teuthology.orchestra.run.smithi077:> sudo yum -y install librbd1 2023-11-08T04:51:21.429 INFO:teuthology.orchestra.run.smithi077.stdout:Last metadata expiration check: 0:02:39 ago on Wed 08 Nov 2023 04:48:42 AM UTC. 2023-11-08T04:51:21.760 INFO:teuthology.orchestra.run.smithi087.stdout:Package python3-rgw-2:16.2.14-129.gcfb6fd52.el8.x86_64 is already installed. 2023-11-08T04:51:21.970 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2023-11-08T04:51:21.971 INFO:teuthology.orchestra.run.smithi087.stdout:Nothing to do. 2023-11-08T04:51:21.972 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2023-11-08T04:51:22.058 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install python3-cephfs 2023-11-08T04:51:22.875 INFO:teuthology.orchestra.run.smithi077.stdout:Package librbd1-2:16.2.14-129.gcfb6fd52.el8.x86_64 is already installed. 2023-11-08T04:51:22.914 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:02:43 ago on Wed 08 Nov 2023 04:48:39 AM UTC. 2023-11-08T04:51:23.082 INFO:teuthology.orchestra.run.smithi077.stdout:Dependencies resolved. 2023-11-08T04:51:23.083 INFO:teuthology.orchestra.run.smithi077.stdout:Nothing to do. 2023-11-08T04:51:23.083 INFO:teuthology.orchestra.run.smithi077.stdout:Complete! 2023-11-08T04:51:23.173 DEBUG:teuthology.orchestra.run.smithi077:> sudo yum -y install python3-rados 2023-11-08T04:51:24.031 INFO:teuthology.orchestra.run.smithi077.stdout:Last metadata expiration check: 0:02:42 ago on Wed 08 Nov 2023 04:48:42 AM UTC. 2023-11-08T04:51:24.349 INFO:teuthology.orchestra.run.smithi087.stdout:Package python3-cephfs-2:16.2.14-129.gcfb6fd52.el8.x86_64 is already installed. 2023-11-08T04:51:24.558 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2023-11-08T04:51:24.559 INFO:teuthology.orchestra.run.smithi087.stdout:Nothing to do. 2023-11-08T04:51:24.559 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2023-11-08T04:51:24.703 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install python3-rbd 2023-11-08T04:51:25.447 INFO:teuthology.orchestra.run.smithi077.stdout:Package python3-rados-2:16.2.14-129.gcfb6fd52.el8.x86_64 is already installed. 2023-11-08T04:51:25.559 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:02:46 ago on Wed 08 Nov 2023 04:48:39 AM UTC. 2023-11-08T04:51:25.656 INFO:teuthology.orchestra.run.smithi077.stdout:Dependencies resolved. 2023-11-08T04:51:25.657 INFO:teuthology.orchestra.run.smithi077.stdout:Nothing to do. 2023-11-08T04:51:25.658 INFO:teuthology.orchestra.run.smithi077.stdout:Complete! 2023-11-08T04:51:25.754 DEBUG:teuthology.orchestra.run.smithi077:> sudo yum -y install python3-rgw 2023-11-08T04:51:26.615 INFO:teuthology.orchestra.run.smithi077.stdout:Last metadata expiration check: 0:02:44 ago on Wed 08 Nov 2023 04:48:42 AM UTC. 2023-11-08T04:51:26.993 INFO:teuthology.orchestra.run.smithi087.stdout:Package python3-rbd-2:16.2.14-129.gcfb6fd52.el8.x86_64 is already installed. 2023-11-08T04:51:27.199 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2023-11-08T04:51:27.200 INFO:teuthology.orchestra.run.smithi087.stdout:Nothing to do. 2023-11-08T04:51:27.200 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2023-11-08T04:51:27.299 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install rbd-fuse 2023-11-08T04:51:28.072 INFO:teuthology.orchestra.run.smithi077.stdout:Package python3-rgw-2:16.2.14-129.gcfb6fd52.el8.x86_64 is already installed. 2023-11-08T04:51:28.151 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:02:49 ago on Wed 08 Nov 2023 04:48:39 AM UTC. 2023-11-08T04:51:28.304 INFO:teuthology.orchestra.run.smithi077.stdout:Dependencies resolved. 2023-11-08T04:51:28.305 INFO:teuthology.orchestra.run.smithi077.stdout:Nothing to do. 2023-11-08T04:51:28.305 INFO:teuthology.orchestra.run.smithi077.stdout:Complete! 2023-11-08T04:51:28.395 DEBUG:teuthology.orchestra.run.smithi077:> sudo yum -y install python3-cephfs 2023-11-08T04:51:29.254 INFO:teuthology.orchestra.run.smithi077.stdout:Last metadata expiration check: 0:02:47 ago on Wed 08 Nov 2023 04:48:42 AM UTC. 2023-11-08T04:51:29.810 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2023-11-08T04:51:29.811 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2023-11-08T04:51:29.811 INFO:teuthology.orchestra.run.smithi087.stdout: Package Architecture Version Repository Size 2023-11-08T04:51:29.811 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2023-11-08T04:51:29.812 INFO:teuthology.orchestra.run.smithi087.stdout:Installing: 2023-11-08T04:51:29.812 INFO:teuthology.orchestra.run.smithi087.stdout: rbd-fuse x86_64 2:16.2.14-129.gcfb6fd52.el8 ceph 88 k 2023-11-08T04:51:29.812 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T04:51:29.812 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction Summary 2023-11-08T04:51:29.812 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2023-11-08T04:51:29.812 INFO:teuthology.orchestra.run.smithi087.stdout:Install 1 Package 2023-11-08T04:51:29.813 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T04:51:29.813 INFO:teuthology.orchestra.run.smithi087.stdout:Total download size: 88 k 2023-11-08T04:51:29.813 INFO:teuthology.orchestra.run.smithi087.stdout:Installed size: 240 k 2023-11-08T04:51:29.813 INFO:teuthology.orchestra.run.smithi087.stdout:Downloading Packages: 2023-11-08T04:51:29.989 INFO:teuthology.orchestra.run.smithi087.stdout:rbd-fuse-16.2.14-129.gcfb6fd52.el8.x86_64.rpm 498 kB/s | 88 kB 00:00 2023-11-08T04:51:29.990 INFO:teuthology.orchestra.run.smithi087.stdout:-------------------------------------------------------------------------------- 2023-11-08T04:51:29.990 INFO:teuthology.orchestra.run.smithi087.stdout:Total 495 kB/s | 88 kB 00:00 2023-11-08T04:51:29.990 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction check 2023-11-08T04:51:29.999 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction check succeeded. 2023-11-08T04:51:29.999 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction test 2023-11-08T04:51:30.047 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction test succeeded. 2023-11-08T04:51:30.050 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction 2023-11-08T04:51:30.190 INFO:teuthology.orchestra.run.smithi087.stdout: Preparing : 1/1 2023-11-08T04:51:30.459 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : rbd-fuse-2:16.2.14-129.gcfb6fd52.el8.x86_64 1/1 2023-11-08T04:51:30.710 INFO:teuthology.orchestra.run.smithi077.stdout:Package python3-cephfs-2:16.2.14-129.gcfb6fd52.el8.x86_64 is already installed. 2023-11-08T04:51:30.732 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: rbd-fuse-2:16.2.14-129.gcfb6fd52.el8.x86_64 1/1 2023-11-08T04:51:30.918 INFO:teuthology.orchestra.run.smithi077.stdout:Dependencies resolved. 2023-11-08T04:51:30.919 INFO:teuthology.orchestra.run.smithi077.stdout:Nothing to do. 2023-11-08T04:51:30.919 INFO:teuthology.orchestra.run.smithi077.stdout:Complete! 2023-11-08T04:51:31.007 DEBUG:teuthology.orchestra.run.smithi077:> sudo yum -y install python3-rbd 2023-11-08T04:51:31.419 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : rbd-fuse-2:16.2.14-129.gcfb6fd52.el8.x86_64 1/1 2023-11-08T04:51:31.419 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T04:51:31.420 INFO:teuthology.orchestra.run.smithi087.stdout:Installed: 2023-11-08T04:51:31.420 INFO:teuthology.orchestra.run.smithi087.stdout: rbd-fuse-2:16.2.14-129.gcfb6fd52.el8.x86_64 2023-11-08T04:51:31.420 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T04:51:31.420 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2023-11-08T04:51:31.593 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install rbd-mirror 2023-11-08T04:51:31.871 INFO:teuthology.orchestra.run.smithi077.stdout:Last metadata expiration check: 0:02:49 ago on Wed 08 Nov 2023 04:48:42 AM UTC. 2023-11-08T04:51:32.450 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:02:53 ago on Wed 08 Nov 2023 04:48:39 AM UTC. 2023-11-08T04:51:33.293 INFO:teuthology.orchestra.run.smithi077.stdout:Package python3-rbd-2:16.2.14-129.gcfb6fd52.el8.x86_64 is already installed. 2023-11-08T04:51:33.500 INFO:teuthology.orchestra.run.smithi077.stdout:Dependencies resolved. 2023-11-08T04:51:33.501 INFO:teuthology.orchestra.run.smithi077.stdout:Nothing to do. 2023-11-08T04:51:33.502 INFO:teuthology.orchestra.run.smithi077.stdout:Complete! 2023-11-08T04:51:33.595 DEBUG:teuthology.orchestra.run.smithi077:> sudo yum -y install rbd-fuse 2023-11-08T04:51:34.086 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2023-11-08T04:51:34.087 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2023-11-08T04:51:34.087 INFO:teuthology.orchestra.run.smithi087.stdout: Package Arch Version Repo Size 2023-11-08T04:51:34.087 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2023-11-08T04:51:34.087 INFO:teuthology.orchestra.run.smithi087.stdout:Installing: 2023-11-08T04:51:34.088 INFO:teuthology.orchestra.run.smithi087.stdout: rbd-mirror x86_64 2:16.2.14-129.gcfb6fd52.el8 ceph 3.7 M 2023-11-08T04:51:34.088 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T04:51:34.088 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction Summary 2023-11-08T04:51:34.088 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2023-11-08T04:51:34.088 INFO:teuthology.orchestra.run.smithi087.stdout:Install 1 Package 2023-11-08T04:51:34.089 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T04:51:34.089 INFO:teuthology.orchestra.run.smithi087.stdout:Total download size: 3.7 M 2023-11-08T04:51:34.089 INFO:teuthology.orchestra.run.smithi087.stdout:Installed size: 15 M 2023-11-08T04:51:34.089 INFO:teuthology.orchestra.run.smithi087.stdout:Downloading Packages: 2023-11-08T04:51:34.450 INFO:teuthology.orchestra.run.smithi077.stdout:Last metadata expiration check: 0:02:52 ago on Wed 08 Nov 2023 04:48:42 AM UTC. 2023-11-08T04:51:34.641 INFO:teuthology.orchestra.run.smithi087.stdout:rbd-mirror-16.2.14-129.gcfb6fd52.el8.x86_64.rpm 6.6 MB/s | 3.7 MB 00:00 2023-11-08T04:51:34.642 INFO:teuthology.orchestra.run.smithi087.stdout:-------------------------------------------------------------------------------- 2023-11-08T04:51:34.643 INFO:teuthology.orchestra.run.smithi087.stdout:Total 6.6 MB/s | 3.7 MB 00:00 2023-11-08T04:51:34.643 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction check 2023-11-08T04:51:34.653 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction check succeeded. 2023-11-08T04:51:34.653 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction test 2023-11-08T04:51:34.715 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction test succeeded. 2023-11-08T04:51:34.718 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction 2023-11-08T04:51:35.152 INFO:teuthology.orchestra.run.smithi087.stdout: Preparing : 1/1 2023-11-08T04:51:35.630 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : rbd-mirror-2:16.2.14-129.gcfb6fd52.el8.x86_64 1/1 2023-11-08T04:51:36.054 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: rbd-mirror-2:16.2.14-129.gcfb6fd52.el8.x86_64 1/1 2023-11-08T04:51:36.116 INFO:teuthology.orchestra.run.smithi077.stdout:Dependencies resolved. 2023-11-08T04:51:36.117 INFO:teuthology.orchestra.run.smithi077.stdout:================================================================================ 2023-11-08T04:51:36.118 INFO:teuthology.orchestra.run.smithi077.stdout: Package Architecture Version Repository Size 2023-11-08T04:51:36.118 INFO:teuthology.orchestra.run.smithi077.stdout:================================================================================ 2023-11-08T04:51:36.118 INFO:teuthology.orchestra.run.smithi077.stdout:Installing: 2023-11-08T04:51:36.118 INFO:teuthology.orchestra.run.smithi077.stdout: rbd-fuse x86_64 2:16.2.14-129.gcfb6fd52.el8 ceph 88 k 2023-11-08T04:51:36.119 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-11-08T04:51:36.119 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction Summary 2023-11-08T04:51:36.119 INFO:teuthology.orchestra.run.smithi077.stdout:================================================================================ 2023-11-08T04:51:36.119 INFO:teuthology.orchestra.run.smithi077.stdout:Install 1 Package 2023-11-08T04:51:36.119 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-11-08T04:51:36.120 INFO:teuthology.orchestra.run.smithi077.stdout:Total download size: 88 k 2023-11-08T04:51:36.120 INFO:teuthology.orchestra.run.smithi077.stdout:Installed size: 240 k 2023-11-08T04:51:36.120 INFO:teuthology.orchestra.run.smithi077.stdout:Downloading Packages: 2023-11-08T04:51:36.287 INFO:teuthology.orchestra.run.smithi077.stdout:rbd-fuse-16.2.14-129.gcfb6fd52.el8.x86_64.rpm 523 kB/s | 88 kB 00:00 2023-11-08T04:51:36.288 INFO:teuthology.orchestra.run.smithi077.stdout:-------------------------------------------------------------------------------- 2023-11-08T04:51:36.288 INFO:teuthology.orchestra.run.smithi077.stdout:Total 519 kB/s | 88 kB 00:00 2023-11-08T04:51:36.289 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction check 2023-11-08T04:51:36.298 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction check succeeded. 2023-11-08T04:51:36.298 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction test 2023-11-08T04:51:36.347 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction test succeeded. 2023-11-08T04:51:36.350 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction 2023-11-08T04:51:36.467 INFO:teuthology.orchestra.run.smithi077.stdout: Preparing : 1/1 2023-11-08T04:51:36.681 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : rbd-fuse-2:16.2.14-129.gcfb6fd52.el8.x86_64 1/1 2023-11-08T04:51:36.713 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : rbd-mirror-2:16.2.14-129.gcfb6fd52.el8.x86_64 1/1 2023-11-08T04:51:36.713 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T04:51:36.713 INFO:teuthology.orchestra.run.smithi087.stdout:Installed: 2023-11-08T04:51:36.713 INFO:teuthology.orchestra.run.smithi087.stdout: rbd-mirror-2:16.2.14-129.gcfb6fd52.el8.x86_64 2023-11-08T04:51:36.714 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T04:51:36.714 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2023-11-08T04:51:36.943 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install rbd-nbd 2023-11-08T04:51:36.963 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: rbd-fuse-2:16.2.14-129.gcfb6fd52.el8.x86_64 1/1 2023-11-08T04:51:37.638 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : rbd-fuse-2:16.2.14-129.gcfb6fd52.el8.x86_64 1/1 2023-11-08T04:51:37.638 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-11-08T04:51:37.638 INFO:teuthology.orchestra.run.smithi077.stdout:Installed: 2023-11-08T04:51:37.638 INFO:teuthology.orchestra.run.smithi077.stdout: rbd-fuse-2:16.2.14-129.gcfb6fd52.el8.x86_64 2023-11-08T04:51:37.639 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-11-08T04:51:37.639 INFO:teuthology.orchestra.run.smithi077.stdout:Complete! 2023-11-08T04:51:37.803 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:02:58 ago on Wed 08 Nov 2023 04:48:39 AM UTC. 2023-11-08T04:51:37.837 DEBUG:teuthology.orchestra.run.smithi077:> sudo yum -y install rbd-mirror 2023-11-08T04:51:38.703 INFO:teuthology.orchestra.run.smithi077.stdout:Last metadata expiration check: 0:02:56 ago on Wed 08 Nov 2023 04:48:42 AM UTC. 2023-11-08T04:51:39.467 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2023-11-08T04:51:39.468 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2023-11-08T04:51:39.469 INFO:teuthology.orchestra.run.smithi087.stdout: Package Architecture Version Repository Size 2023-11-08T04:51:39.469 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2023-11-08T04:51:39.469 INFO:teuthology.orchestra.run.smithi087.stdout:Installing: 2023-11-08T04:51:39.469 INFO:teuthology.orchestra.run.smithi087.stdout: rbd-nbd x86_64 2:16.2.14-129.gcfb6fd52.el8 ceph 175 k 2023-11-08T04:51:39.470 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T04:51:39.470 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction Summary 2023-11-08T04:51:39.470 INFO:teuthology.orchestra.run.smithi087.stdout:================================================================================ 2023-11-08T04:51:39.470 INFO:teuthology.orchestra.run.smithi087.stdout:Install 1 Package 2023-11-08T04:51:39.470 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T04:51:39.471 INFO:teuthology.orchestra.run.smithi087.stdout:Total download size: 175 k 2023-11-08T04:51:39.471 INFO:teuthology.orchestra.run.smithi087.stdout:Installed size: 523 k 2023-11-08T04:51:39.471 INFO:teuthology.orchestra.run.smithi087.stdout:Downloading Packages: 2023-11-08T04:51:39.659 INFO:teuthology.orchestra.run.smithi087.stdout:rbd-nbd-16.2.14-129.gcfb6fd52.el8.x86_64.rpm 923 kB/s | 175 kB 00:00 2023-11-08T04:51:39.660 INFO:teuthology.orchestra.run.smithi087.stdout:-------------------------------------------------------------------------------- 2023-11-08T04:51:39.661 INFO:teuthology.orchestra.run.smithi087.stdout:Total 916 kB/s | 175 kB 00:00 2023-11-08T04:51:39.661 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction check 2023-11-08T04:51:39.671 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction check succeeded. 2023-11-08T04:51:39.671 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction test 2023-11-08T04:51:39.720 INFO:teuthology.orchestra.run.smithi087.stdout:Transaction test succeeded. 2023-11-08T04:51:39.723 INFO:teuthology.orchestra.run.smithi087.stdout:Running transaction 2023-11-08T04:51:39.830 INFO:teuthology.orchestra.run.smithi087.stdout: Preparing : 1/1 2023-11-08T04:51:40.045 INFO:teuthology.orchestra.run.smithi087.stdout: Installing : rbd-nbd-2:16.2.14-129.gcfb6fd52.el8.x86_64 1/1 2023-11-08T04:51:40.335 INFO:teuthology.orchestra.run.smithi087.stdout: Running scriptlet: rbd-nbd-2:16.2.14-129.gcfb6fd52.el8.x86_64 1/1 2023-11-08T04:51:40.360 INFO:teuthology.orchestra.run.smithi077.stdout:Dependencies resolved. 2023-11-08T04:51:40.361 INFO:teuthology.orchestra.run.smithi077.stdout:================================================================================ 2023-11-08T04:51:40.361 INFO:teuthology.orchestra.run.smithi077.stdout: Package Arch Version Repo Size 2023-11-08T04:51:40.361 INFO:teuthology.orchestra.run.smithi077.stdout:================================================================================ 2023-11-08T04:51:40.361 INFO:teuthology.orchestra.run.smithi077.stdout:Installing: 2023-11-08T04:51:40.362 INFO:teuthology.orchestra.run.smithi077.stdout: rbd-mirror x86_64 2:16.2.14-129.gcfb6fd52.el8 ceph 3.7 M 2023-11-08T04:51:40.362 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-11-08T04:51:40.362 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction Summary 2023-11-08T04:51:40.362 INFO:teuthology.orchestra.run.smithi077.stdout:================================================================================ 2023-11-08T04:51:40.362 INFO:teuthology.orchestra.run.smithi077.stdout:Install 1 Package 2023-11-08T04:51:40.363 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-11-08T04:51:40.363 INFO:teuthology.orchestra.run.smithi077.stdout:Total download size: 3.7 M 2023-11-08T04:51:40.363 INFO:teuthology.orchestra.run.smithi077.stdout:Installed size: 15 M 2023-11-08T04:51:40.364 INFO:teuthology.orchestra.run.smithi077.stdout:Downloading Packages: 2023-11-08T04:51:40.907 INFO:teuthology.orchestra.run.smithi077.stdout:rbd-mirror-16.2.14-129.gcfb6fd52.el8.x86_64.rpm 6.7 MB/s | 3.7 MB 00:00 2023-11-08T04:51:40.907 INFO:teuthology.orchestra.run.smithi077.stdout:-------------------------------------------------------------------------------- 2023-11-08T04:51:40.908 INFO:teuthology.orchestra.run.smithi077.stdout:Total 6.7 MB/s | 3.7 MB 00:00 2023-11-08T04:51:40.908 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction check 2023-11-08T04:51:40.919 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction check succeeded. 2023-11-08T04:51:40.920 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction test 2023-11-08T04:51:40.976 INFO:teuthology.orchestra.run.smithi087.stdout: Verifying : rbd-nbd-2:16.2.14-129.gcfb6fd52.el8.x86_64 1/1 2023-11-08T04:51:40.977 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T04:51:40.977 INFO:teuthology.orchestra.run.smithi087.stdout:Installed: 2023-11-08T04:51:40.977 INFO:teuthology.orchestra.run.smithi087.stdout: rbd-nbd-2:16.2.14-129.gcfb6fd52.el8.x86_64 2023-11-08T04:51:40.977 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T04:51:40.977 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2023-11-08T04:51:40.981 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction test succeeded. 2023-11-08T04:51:40.984 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction 2023-11-08T04:51:41.264 DEBUG:teuthology.orchestra.run.smithi087:> sudo yum -y install rbd-mirror 2023-11-08T04:51:41.418 INFO:teuthology.orchestra.run.smithi077.stdout: Preparing : 1/1 2023-11-08T04:51:41.640 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : rbd-mirror-2:16.2.14-129.gcfb6fd52.el8.x86_64 1/1 2023-11-08T04:51:42.051 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: rbd-mirror-2:16.2.14-129.gcfb6fd52.el8.x86_64 1/1 2023-11-08T04:51:42.130 INFO:teuthology.orchestra.run.smithi087.stdout:Last metadata expiration check: 0:03:03 ago on Wed 08 Nov 2023 04:48:39 AM UTC. 2023-11-08T04:51:42.723 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : rbd-mirror-2:16.2.14-129.gcfb6fd52.el8.x86_64 1/1 2023-11-08T04:51:42.723 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-11-08T04:51:42.723 INFO:teuthology.orchestra.run.smithi077.stdout:Installed: 2023-11-08T04:51:42.724 INFO:teuthology.orchestra.run.smithi077.stdout: rbd-mirror-2:16.2.14-129.gcfb6fd52.el8.x86_64 2023-11-08T04:51:42.724 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-11-08T04:51:42.724 INFO:teuthology.orchestra.run.smithi077.stdout:Complete! 2023-11-08T04:51:42.994 DEBUG:teuthology.orchestra.run.smithi077:> sudo yum -y install rbd-nbd 2023-11-08T04:51:43.585 INFO:teuthology.orchestra.run.smithi087.stdout:Package rbd-mirror-2:16.2.14-129.gcfb6fd52.el8.x86_64 is already installed. 2023-11-08T04:51:43.799 INFO:teuthology.orchestra.run.smithi087.stdout:Dependencies resolved. 2023-11-08T04:51:43.800 INFO:teuthology.orchestra.run.smithi087.stdout:Nothing to do. 2023-11-08T04:51:43.800 INFO:teuthology.orchestra.run.smithi087.stdout:Complete! 2023-11-08T04:51:43.858 INFO:teuthology.orchestra.run.smithi077.stdout:Last metadata expiration check: 0:03:01 ago on Wed 08 Nov 2023 04:48:42 AM UTC. 2023-11-08T04:51:43.882 DEBUG:teuthology.parallel:result is None 2023-11-08T04:51:45.499 INFO:teuthology.orchestra.run.smithi077.stdout:Dependencies resolved. 2023-11-08T04:51:45.500 INFO:teuthology.orchestra.run.smithi077.stdout:================================================================================ 2023-11-08T04:51:45.500 INFO:teuthology.orchestra.run.smithi077.stdout: Package Architecture Version Repository Size 2023-11-08T04:51:45.500 INFO:teuthology.orchestra.run.smithi077.stdout:================================================================================ 2023-11-08T04:51:45.500 INFO:teuthology.orchestra.run.smithi077.stdout:Installing: 2023-11-08T04:51:45.500 INFO:teuthology.orchestra.run.smithi077.stdout: rbd-nbd x86_64 2:16.2.14-129.gcfb6fd52.el8 ceph 175 k 2023-11-08T04:51:45.501 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-11-08T04:51:45.501 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction Summary 2023-11-08T04:51:45.501 INFO:teuthology.orchestra.run.smithi077.stdout:================================================================================ 2023-11-08T04:51:45.501 INFO:teuthology.orchestra.run.smithi077.stdout:Install 1 Package 2023-11-08T04:51:45.501 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-11-08T04:51:45.502 INFO:teuthology.orchestra.run.smithi077.stdout:Total download size: 175 k 2023-11-08T04:51:45.502 INFO:teuthology.orchestra.run.smithi077.stdout:Installed size: 523 k 2023-11-08T04:51:45.502 INFO:teuthology.orchestra.run.smithi077.stdout:Downloading Packages: 2023-11-08T04:51:45.818 INFO:teuthology.orchestra.run.smithi077.stdout:rbd-nbd-16.2.14-129.gcfb6fd52.el8.x86_64.rpm 552 kB/s | 175 kB 00:00 2023-11-08T04:51:45.819 INFO:teuthology.orchestra.run.smithi077.stdout:-------------------------------------------------------------------------------- 2023-11-08T04:51:45.819 INFO:teuthology.orchestra.run.smithi077.stdout:Total 550 kB/s | 175 kB 00:00 2023-11-08T04:51:45.819 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction check 2023-11-08T04:51:45.829 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction check succeeded. 2023-11-08T04:51:45.829 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction test 2023-11-08T04:51:45.878 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction test succeeded. 2023-11-08T04:51:45.881 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction 2023-11-08T04:51:46.002 INFO:teuthology.orchestra.run.smithi077.stdout: Preparing : 1/1 2023-11-08T04:51:46.174 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : rbd-nbd-2:16.2.14-129.gcfb6fd52.el8.x86_64 1/1 2023-11-08T04:51:46.461 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: rbd-nbd-2:16.2.14-129.gcfb6fd52.el8.x86_64 1/1 2023-11-08T04:51:47.124 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : rbd-nbd-2:16.2.14-129.gcfb6fd52.el8.x86_64 1/1 2023-11-08T04:51:47.124 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-11-08T04:51:47.124 INFO:teuthology.orchestra.run.smithi077.stdout:Installed: 2023-11-08T04:51:47.125 INFO:teuthology.orchestra.run.smithi077.stdout: rbd-nbd-2:16.2.14-129.gcfb6fd52.el8.x86_64 2023-11-08T04:51:47.125 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-11-08T04:51:47.125 INFO:teuthology.orchestra.run.smithi077.stdout:Complete! 2023-11-08T04:51:47.331 DEBUG:teuthology.orchestra.run.smithi077:> sudo yum -y install rbd-mirror 2023-11-08T04:51:48.200 INFO:teuthology.orchestra.run.smithi077.stdout:Last metadata expiration check: 0:03:06 ago on Wed 08 Nov 2023 04:48:42 AM UTC. 2023-11-08T04:51:49.635 INFO:teuthology.orchestra.run.smithi077.stdout:Package rbd-mirror-2:16.2.14-129.gcfb6fd52.el8.x86_64 is already installed. 2023-11-08T04:51:49.842 INFO:teuthology.orchestra.run.smithi077.stdout:Dependencies resolved. 2023-11-08T04:51:49.843 INFO:teuthology.orchestra.run.smithi077.stdout:Nothing to do. 2023-11-08T04:51:49.844 INFO:teuthology.orchestra.run.smithi077.stdout:Complete! 2023-11-08T04:51:49.944 DEBUG:teuthology.parallel:result is None 2023-11-08T04:51:49.945 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=cfb6fd52113be6a0af3fe7fb89b2e08474eadc0f 2023-11-08T04:51:50.231 DEBUG:teuthology.orchestra.run.smithi077:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-11-08T04:51:50.254 INFO:teuthology.orchestra.run.smithi077.stdout:16.2.14-129.gcfb6fd52.el8 2023-11-08T04:51:50.255 INFO:teuthology.packaging:The installed version of ceph is 16.2.14-129.gcfb6fd52.el8 2023-11-08T04:51:50.255 INFO:teuthology.task.install:The correct ceph version 16.2.14-129.gcfb6fd52 is installed. 2023-11-08T04:51:50.256 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=cfb6fd52113be6a0af3fe7fb89b2e08474eadc0f 2023-11-08T04:51:50.478 DEBUG:teuthology.orchestra.run.smithi087:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-11-08T04:51:50.505 INFO:teuthology.orchestra.run.smithi087.stdout:16.2.14-129.gcfb6fd52.el8 2023-11-08T04:51:50.506 INFO:teuthology.packaging:The installed version of ceph is 16.2.14-129.gcfb6fd52.el8 2023-11-08T04:51:50.506 INFO:teuthology.task.install:The correct ceph version 16.2.14-129.gcfb6fd52 is installed. 2023-11-08T04:51:50.507 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2023-11-08T04:51:50.507 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2023-11-08T04:51:50.508 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-11-08T04:51:50.536 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2023-11-08T04:51:50.536 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-11-08T04:51:50.576 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2023-11-08T04:51:50.577 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2023-11-08T04:51:50.577 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd of=/usr/bin/daemon-helper 2023-11-08T04:51:50.605 DEBUG:teuthology.orchestra.run.smithi077:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-11-08T04:51:50.672 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2023-11-08T04:51:50.672 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd of=/usr/bin/daemon-helper 2023-11-08T04:51:50.703 DEBUG:teuthology.orchestra.run.smithi087:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-11-08T04:51:50.771 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2023-11-08T04:51:50.772 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2023-11-08T04:51:50.772 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd of=/usr/bin/adjust-ulimits 2023-11-08T04:51:50.799 DEBUG:teuthology.orchestra.run.smithi077:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-11-08T04:51:50.866 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2023-11-08T04:51:50.866 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd of=/usr/bin/adjust-ulimits 2023-11-08T04:51:50.895 DEBUG:teuthology.orchestra.run.smithi087:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-11-08T04:51:50.964 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2023-11-08T04:51:50.965 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2023-11-08T04:51:50.965 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd of=/usr/bin/stdin-killer 2023-11-08T04:51:50.991 DEBUG:teuthology.orchestra.run.smithi077:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-11-08T04:51:51.057 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2023-11-08T04:51:51.057 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd of=/usr/bin/stdin-killer 2023-11-08T04:51:51.085 DEBUG:teuthology.orchestra.run.smithi087:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-11-08T04:51:51.153 INFO:teuthology.run_tasks:Running task ceph... 2023-11-08T04:51:51.219 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2023-11-08T04:51:51.220 DEBUG:teuthology.orchestra.run.smithi077:> sudo chmod 777 /var/log/ceph 2023-11-08T04:51:51.223 DEBUG:teuthology.orchestra.run.smithi087:> sudo chmod 777 /var/log/ceph 2023-11-08T04:51:51.249 INFO:tasks.ceph:Disabling ceph logrotate... 2023-11-08T04:51:51.250 DEBUG:teuthology.orchestra.run.smithi077:> sudo rm -f -- /etc/logrotate.d/ceph 2023-11-08T04:51:51.289 DEBUG:teuthology.orchestra.run.smithi087:> sudo rm -f -- /etc/logrotate.d/ceph 2023-11-08T04:51:51.316 INFO:tasks.ceph:Creating extra log directories... 2023-11-08T04:51:51.316 DEBUG:teuthology.orchestra.run.smithi077:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-11-08T04:51:51.355 DEBUG:teuthology.orchestra.run.smithi087:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-11-08T04:51:51.397 INFO:tasks.ceph:Creating ceph cluster cluster1... 2023-11-08T04:51:51.397 INFO:tasks.ceph:config {'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'simple'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd memory target': 1610612736, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'cpu_profile': set(), 'cluster': 'cluster1', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2023-11-08T04:51:51.398 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/yuriw-2023-11-07_19:42:25-rbd-wip-yuri3-testing-2023-11-07-0801-pacific-distro-default-smithi/7451368', 'branch': 'wip-yuri3-testing-2023-11-07-0801-pacific', 'description': 'rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-low-osd-mem-target policy/simple rbd-mirror/four-per-cluster supported-random-distro$/{centos_8} workloads/rbd-mirror-snapshot-stress-workunit-exclusive-lock}', 'email': 'yweinste@redhat.com', 'first_in_suite': False, 'job_id': '7451368', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'meta': [{'desc': 'run two ceph clusters and install rbd-mirror'}, {'desc': 'configure the permissions for client.mirror'}, {'desc': '2 ceph clusters with 1 mon and 3 osds each'}, {'desc': 'run four rbd-mirror daemons per cluster'}, {'desc': 'run the rbd_mirror_stress.sh workunit to test the rbd-mirror daemon'}], 'name': 'yuriw-2023-11-07_19:42:25-rbd-wip-yuri3-testing-2023-11-07-0801-pacific-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 3, 'size': 30}}], 'os_type': 'centos', 'os_version': '8.stream', 'overrides': {'admin_socket': {'branch': 'wip-yuri3-testing-2023-11-07-0801-pacific'}, 'ceph': {'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'simple'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd memory target': 1610612736, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': 'cfb6fd52113be6a0af3fe7fb89b2e08474eadc0f'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': 'cfb6fd52113be6a0af3fe7fb89b2e08474eadc0f'}}, 'selinux': {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-yuri3-testing-2023-11-07-0801-pacific', 'sha1': 'cfb6fd52113be6a0af3fe7fb89b2e08474eadc0f'}}, 'owner': 'scheduled_yuriw@teuthology', 'priority': 75, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['cluster1.mon.a', 'cluster1.mgr.x', 'cluster2.mgr.x', 'cluster1.osd.0', 'cluster1.osd.1', 'cluster1.osd.2', 'cluster1.client.0', 'cluster2.client.0'], ['cluster2.mon.a', 'cluster2.osd.0', 'cluster2.osd.1', 'cluster2.osd.2', 'cluster1.client.mirror', 'cluster1.client.mirror.0', 'cluster1.client.mirror.1', 'cluster1.client.mirror.2', 'cluster1.client.mirror.3', 'cluster1.client.mirror.4', 'cluster1.client.mirror.5', 'cluster1.client.mirror.6', 'cluster2.client.mirror', 'cluster2.client.mirror.0', 'cluster2.client.mirror.1', 'cluster2.client.mirror.2', 'cluster2.client.mirror.3', 'cluster2.client.mirror.4', 'cluster2.client.mirror.5', 'cluster2.client.mirror.6']], 'seed': 9776, 'sha1': 'cfb6fd52113be6a0af3fe7fb89b2e08474eadc0f', 'sleep_before_teardown': 0, 'subset': '111/120000', 'suite': 'rbd', 'suite_branch': 'wip-yuri3-testing-2023-11-07-0801-pacific', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_cfb6fd52113be6a0af3fe7fb89b2e08474eadc0f/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': 'cfb6fd52113be6a0af3fe7fb89b2e08474eadc0f', 'targets': {'smithi077.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGSQ/WLcJdpJ0ClJVyBsFrcYIQ2QRbAwp2zUvz0ifkJpShDNVGje+V/Jdm6AZOafWAIrxu2EWTj6axjmMfZQozM=', 'smithi087.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBD+XKU8wa6pBVq4mhdEwRNPaK1Sl9uWyjisTURFaLby1G+ogJ5JiOLEZ7VuzQt8PXKmmvgJJqHhmd6n+97zE5BE='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': ['rbd-mirror'], 'flavor': 'default', 'sha1': 'cfb6fd52113be6a0af3fe7fb89b2e08474eadc0f'}}, {'ceph': {'cluster': 'cluster1', 'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'simple'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd memory target': 1610612736, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': 'cfb6fd52113be6a0af3fe7fb89b2e08474eadc0f'}}, {'ceph': {'cluster': 'cluster2'}}, {'exec': {'cluster1.client.mirror.0': ["sudo ceph --cluster cluster1 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd'"]}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.0', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.1', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.2', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.3', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.0', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.1', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.2', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.3', 'thrash': True}}, {'rbd-mirror-thrash': {'cluster': 'cluster1'}}, {'rbd-mirror-thrash': {'cluster': 'cluster2'}}, {'workunit': {'clients': {'cluster1.client.mirror': ['rbd/rbd_mirror_stress.sh']}, 'env': {'CEPH_ARGS': '', 'MIRROR_IMAGE_MODE': 'snapshot', 'MIRROR_POOL_MODE': 'image', 'RBD_IMAGE_FEATURES': 'layering,exclusive-lock', 'RBD_MIRROR_INSTANCES': '4', 'RBD_MIRROR_USE_EXISTING_CLUSTER': '1', 'RBD_MIRROR_USE_RBD_MIRROR': '1'}, 'timeout': '6h'}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '6899cd26fceddb2fec83dc1a1349394b28c8998e', 'timestamp': '2023-11-07_19:42:25', 'tube': 'smithi', 'user': 'yuriw', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.3590714'} 2023-11-08T04:51:51.399 DEBUG:teuthology.orchestra.run.smithi077:> install -d -m0755 -- /home/ubuntu/cephtest/cluster1.data 2023-11-08T04:51:51.422 DEBUG:teuthology.orchestra.run.smithi087:> install -d -m0755 -- /home/ubuntu/cephtest/cluster1.data 2023-11-08T04:51:51.441 DEBUG:teuthology.orchestra.run.smithi077:> sudo install -d -m0777 -- /var/run/ceph 2023-11-08T04:51:51.480 DEBUG:teuthology.orchestra.run.smithi087:> sudo install -d -m0777 -- /var/run/ceph 2023-11-08T04:51:51.507 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2023-11-08T04:51:51.508 DEBUG:teuthology.orchestra.run.smithi077:> dd if=/scratch_devs of=/dev/stdout 2023-11-08T04:51:51.563 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-11-08T04:51:51.564 DEBUG:teuthology.orchestra.run.smithi077:> stat /dev/vg_nvme/lv_1 2023-11-08T04:51:51.621 INFO:teuthology.orchestra.run.smithi077.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-11-08T04:51:51.621 INFO:teuthology.orchestra.run.smithi077.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-11-08T04:51:51.622 INFO:teuthology.orchestra.run.smithi077.stdout:Device: 6h/6d Inode: 398339 Links: 1 2023-11-08T04:51:51.622 INFO:teuthology.orchestra.run.smithi077.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-08T04:51:51.622 INFO:teuthology.orchestra.run.smithi077.stdout:Context: system_u:object_r:device_t:s0 2023-11-08T04:51:51.623 INFO:teuthology.orchestra.run.smithi077.stdout:Access: 2023-11-08 04:51:41.801928412 +0000 2023-11-08T04:51:51.623 INFO:teuthology.orchestra.run.smithi077.stdout:Modify: 2023-11-08 04:47:32.676921911 +0000 2023-11-08T04:51:51.623 INFO:teuthology.orchestra.run.smithi077.stdout:Change: 2023-11-08 04:47:32.676921911 +0000 2023-11-08T04:51:51.623 INFO:teuthology.orchestra.run.smithi077.stdout: Birth: - 2023-11-08T04:51:51.624 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-11-08T04:51:51.686 INFO:teuthology.orchestra.run.smithi077.stderr:1+0 records in 2023-11-08T04:51:51.686 INFO:teuthology.orchestra.run.smithi077.stderr:1+0 records out 2023-11-08T04:51:51.686 INFO:teuthology.orchestra.run.smithi077.stderr:512 bytes copied, 0.000236841 s, 2.2 MB/s 2023-11-08T04:51:51.688 DEBUG:teuthology.orchestra.run.smithi077:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-11-08T04:51:51.747 DEBUG:teuthology.orchestra.run.smithi077:> stat /dev/vg_nvme/lv_2 2023-11-08T04:51:51.805 INFO:teuthology.orchestra.run.smithi077.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-11-08T04:51:51.805 INFO:teuthology.orchestra.run.smithi077.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-11-08T04:51:51.806 INFO:teuthology.orchestra.run.smithi077.stdout:Device: 6h/6d Inode: 395833 Links: 1 2023-11-08T04:51:51.806 INFO:teuthology.orchestra.run.smithi077.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-08T04:51:51.806 INFO:teuthology.orchestra.run.smithi077.stdout:Context: system_u:object_r:device_t:s0 2023-11-08T04:51:51.806 INFO:teuthology.orchestra.run.smithi077.stdout:Access: 2023-11-08 04:51:41.801928412 +0000 2023-11-08T04:51:51.806 INFO:teuthology.orchestra.run.smithi077.stdout:Modify: 2023-11-08 04:47:33.152911162 +0000 2023-11-08T04:51:51.807 INFO:teuthology.orchestra.run.smithi077.stdout:Change: 2023-11-08 04:47:33.152911162 +0000 2023-11-08T04:51:51.807 INFO:teuthology.orchestra.run.smithi077.stdout: Birth: - 2023-11-08T04:51:51.807 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-11-08T04:51:51.870 INFO:teuthology.orchestra.run.smithi077.stderr:1+0 records in 2023-11-08T04:51:51.870 INFO:teuthology.orchestra.run.smithi077.stderr:1+0 records out 2023-11-08T04:51:51.870 INFO:teuthology.orchestra.run.smithi077.stderr:512 bytes copied, 0.000213641 s, 2.4 MB/s 2023-11-08T04:51:51.871 DEBUG:teuthology.orchestra.run.smithi077:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-11-08T04:51:51.931 DEBUG:teuthology.orchestra.run.smithi077:> stat /dev/vg_nvme/lv_3 2023-11-08T04:51:51.988 INFO:teuthology.orchestra.run.smithi077.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-11-08T04:51:51.988 INFO:teuthology.orchestra.run.smithi077.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-11-08T04:51:51.989 INFO:teuthology.orchestra.run.smithi077.stdout:Device: 6h/6d Inode: 399538 Links: 1 2023-11-08T04:51:51.989 INFO:teuthology.orchestra.run.smithi077.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-08T04:51:51.989 INFO:teuthology.orchestra.run.smithi077.stdout:Context: system_u:object_r:device_t:s0 2023-11-08T04:51:51.989 INFO:teuthology.orchestra.run.smithi077.stdout:Access: 2023-11-08 04:51:41.801928412 +0000 2023-11-08T04:51:51.989 INFO:teuthology.orchestra.run.smithi077.stdout:Modify: 2023-11-08 04:47:33.655899803 +0000 2023-11-08T04:51:51.990 INFO:teuthology.orchestra.run.smithi077.stdout:Change: 2023-11-08 04:47:33.655899803 +0000 2023-11-08T04:51:51.990 INFO:teuthology.orchestra.run.smithi077.stdout: Birth: - 2023-11-08T04:51:51.990 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-11-08T04:51:52.053 INFO:teuthology.orchestra.run.smithi077.stderr:1+0 records in 2023-11-08T04:51:52.053 INFO:teuthology.orchestra.run.smithi077.stderr:1+0 records out 2023-11-08T04:51:52.053 INFO:teuthology.orchestra.run.smithi077.stderr:512 bytes copied, 0.000212834 s, 2.4 MB/s 2023-11-08T04:51:52.054 DEBUG:teuthology.orchestra.run.smithi077:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-11-08T04:51:52.112 DEBUG:teuthology.orchestra.run.smithi077:> stat /dev/vg_nvme/lv_4 2023-11-08T04:51:52.171 INFO:teuthology.orchestra.run.smithi077.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-11-08T04:51:52.171 INFO:teuthology.orchestra.run.smithi077.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-11-08T04:51:52.171 INFO:teuthology.orchestra.run.smithi077.stdout:Device: 6h/6d Inode: 398402 Links: 1 2023-11-08T04:51:52.171 INFO:teuthology.orchestra.run.smithi077.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-08T04:51:52.172 INFO:teuthology.orchestra.run.smithi077.stdout:Context: system_u:object_r:device_t:s0 2023-11-08T04:51:52.172 INFO:teuthology.orchestra.run.smithi077.stdout:Access: 2023-11-08 04:51:41.801928412 +0000 2023-11-08T04:51:52.172 INFO:teuthology.orchestra.run.smithi077.stdout:Modify: 2023-11-08 04:47:34.138888896 +0000 2023-11-08T04:51:52.172 INFO:teuthology.orchestra.run.smithi077.stdout:Change: 2023-11-08 04:47:34.138888896 +0000 2023-11-08T04:51:52.172 INFO:teuthology.orchestra.run.smithi077.stdout: Birth: - 2023-11-08T04:51:52.173 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-11-08T04:51:52.236 INFO:teuthology.orchestra.run.smithi077.stderr:1+0 records in 2023-11-08T04:51:52.236 INFO:teuthology.orchestra.run.smithi077.stderr:1+0 records out 2023-11-08T04:51:52.236 INFO:teuthology.orchestra.run.smithi077.stderr:512 bytes copied, 0.000205372 s, 2.5 MB/s 2023-11-08T04:51:52.237 DEBUG:teuthology.orchestra.run.smithi077:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-11-08T04:51:52.295 INFO:tasks.ceph:osd dev map: {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'} 2023-11-08T04:51:52.295 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi077.front.sepia.ceph.com'): {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'}} 2023-11-08T04:51:52.296 INFO:tasks.ceph:Generating config... 2023-11-08T04:51:52.297 INFO:tasks.ceph:[client] debug rbd = 20 2023-11-08T04:51:52.297 INFO:tasks.ceph:[client] debug rbd_mirror = 15 2023-11-08T04:51:52.298 INFO:tasks.ceph:[client] log to stderr = False 2023-11-08T04:51:52.298 INFO:tasks.ceph:[client] rbd default features = 125 2023-11-08T04:51:52.298 INFO:tasks.ceph:[client] rbd mirror image policy type = simple 2023-11-08T04:51:52.298 INFO:tasks.ceph:[client.mirror.0] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-11-08T04:51:52.299 INFO:tasks.ceph:[client.mirror.0] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-11-08T04:51:52.299 INFO:tasks.ceph:[client.mirror.1] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-11-08T04:51:52.299 INFO:tasks.ceph:[client.mirror.1] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-11-08T04:51:52.299 INFO:tasks.ceph:[client.mirror.2] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-11-08T04:51:52.300 INFO:tasks.ceph:[client.mirror.2] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-11-08T04:51:52.300 INFO:tasks.ceph:[client.mirror.3] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-11-08T04:51:52.300 INFO:tasks.ceph:[client.mirror.3] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-11-08T04:51:52.300 INFO:tasks.ceph:[global] mon client directed command retry = 5 2023-11-08T04:51:52.301 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2023-11-08T04:51:52.301 INFO:tasks.ceph:[mgr] debug mgr = 20 2023-11-08T04:51:52.301 INFO:tasks.ceph:[mgr] debug ms = 1 2023-11-08T04:51:52.301 INFO:tasks.ceph:[mon] debug mon = 20 2023-11-08T04:51:52.302 INFO:tasks.ceph:[mon] debug ms = 1 2023-11-08T04:51:52.302 INFO:tasks.ceph:[mon] debug paxos = 20 2023-11-08T04:51:52.302 INFO:tasks.ceph:[osd] bdev async discard = True 2023-11-08T04:51:52.302 INFO:tasks.ceph:[osd] bdev enable discard = True 2023-11-08T04:51:52.302 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2023-11-08T04:51:52.303 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2023-11-08T04:51:52.303 INFO:tasks.ceph:[osd] debug bluefs = 1/20 2023-11-08T04:51:52.303 INFO:tasks.ceph:[osd] debug bluestore = 1/20 2023-11-08T04:51:52.304 INFO:tasks.ceph:[osd] debug ms = 1 2023-11-08T04:51:52.304 INFO:tasks.ceph:[osd] debug osd = 20 2023-11-08T04:51:52.304 INFO:tasks.ceph:[osd] debug rocksdb = 4/10 2023-11-08T04:51:52.304 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2023-11-08T04:51:52.305 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2023-11-08T04:51:52.305 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2023-11-08T04:51:52.305 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2023-11-08T04:51:52.305 INFO:tasks.ceph:[osd] osd memory target = 1610612736 2023-11-08T04:51:52.305 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2023-11-08T04:51:52.306 INFO:tasks.ceph:Setting up cluster1.mon.a... 2023-11-08T04:51:52.306 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/cluster1.keyring 2023-11-08T04:51:52.444 INFO:teuthology.orchestra.run.smithi077.stdout:creating /etc/ceph/cluster1.keyring 2023-11-08T04:51:52.446 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/cluster1.keyring 2023-11-08T04:51:52.487 DEBUG:teuthology.orchestra.run.smithi077:> sudo chmod 0644 /etc/ceph/cluster1.keyring 2023-11-08T04:51:52.554 DEBUG:tasks.ceph:Ceph mon addresses: [('cluster1.mon.a', '172.21.15.77')] 2023-11-08T04:51:52.554 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.77', 'mon client directed command retry': 5, 'ms inject socket failures': 5000}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd memory target': 1610612736, 'osd objectstore': 'bluestore'}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false'}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'simple'}, 'mon.a': {}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}} 2023-11-08T04:51:52.555 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2023-11-08T04:51:52.556 DEBUG:teuthology.orchestra.run.smithi077:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2023-11-08T04:51:52.611 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.77 --print /home/ubuntu/cephtest/cluster1.monmap 2023-11-08T04:51:52.688 INFO:teuthology.orchestra.run.smithi077.stdout:monmaptool: monmap file /home/ubuntu/cephtest/cluster1.monmap 2023-11-08T04:51:52.689 INFO:teuthology.orchestra.run.smithi077.stdout:monmaptool: generated fsid 1244e29e-ccb6-44ee-9f2f-7bbf2125940c 2023-11-08T04:51:52.689 INFO:teuthology.orchestra.run.smithi077.stdout:epoch 0 2023-11-08T04:51:52.689 INFO:teuthology.orchestra.run.smithi077.stdout:fsid 1244e29e-ccb6-44ee-9f2f-7bbf2125940c 2023-11-08T04:51:52.690 INFO:teuthology.orchestra.run.smithi077.stdout:last_changed 2023-11-08T04:51:52.687600+0000 2023-11-08T04:51:52.690 INFO:teuthology.orchestra.run.smithi077.stdout:created 2023-11-08T04:51:52.687600+0000 2023-11-08T04:51:52.690 INFO:teuthology.orchestra.run.smithi077.stdout:min_mon_release 0 (unknown) 2023-11-08T04:51:52.690 INFO:teuthology.orchestra.run.smithi077.stdout:election_strategy: 1 2023-11-08T04:51:52.690 INFO:teuthology.orchestra.run.smithi077.stdout:0: [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] mon.a 2023-11-08T04:51:52.691 INFO:teuthology.orchestra.run.smithi077.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/cluster1.monmap (1 monitors) 2023-11-08T04:51:52.692 DEBUG:teuthology.orchestra.run.smithi077:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2023-11-08T04:51:52.749 INFO:tasks.ceph:Writing /etc/ceph/cluster1.conf for FSID 1244e29e-ccb6-44ee-9f2f-7bbf2125940c... 2023-11-08T04:51:52.750 DEBUG:teuthology.orchestra.run.smithi077:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/cluster1.conf && sudo chmod 0644 /etc/ceph/cluster1.conf > /dev/null 2023-11-08T04:51:52.792 DEBUG:teuthology.orchestra.run.smithi087:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/cluster1.conf && sudo chmod 0644 /etc/ceph/cluster1.conf > /dev/null 2023-11-08T04:51:52.834 INFO:teuthology.orchestra.run.smithi077.stdout:[global] 2023-11-08T04:51:52.834 INFO:teuthology.orchestra.run.smithi077.stdout: chdir = "" 2023-11-08T04:51:52.834 INFO:teuthology.orchestra.run.smithi077.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-11-08T04:51:52.834 INFO:teuthology.orchestra.run.smithi077.stdout: auth supported = cephx 2023-11-08T04:51:52.834 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-11-08T04:51:52.835 INFO:teuthology.orchestra.run.smithi077.stdout: filestore xattr use omap = true 2023-11-08T04:51:52.835 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-11-08T04:51:52.835 INFO:teuthology.orchestra.run.smithi077.stdout: mon clock drift allowed = 1.000 2023-11-08T04:51:52.835 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-11-08T04:51:52.835 INFO:teuthology.orchestra.run.smithi077.stdout: osd crush chooseleaf type = 0 2023-11-08T04:51:52.836 INFO:teuthology.orchestra.run.smithi077.stdout: auth debug = true 2023-11-08T04:51:52.836 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-11-08T04:51:52.836 INFO:teuthology.orchestra.run.smithi077.stdout: ms die on old message = true 2023-11-08T04:51:52.836 INFO:teuthology.orchestra.run.smithi077.stdout: ms die on bug = true 2023-11-08T04:51:52.836 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-11-08T04:51:52.837 INFO:teuthology.orchestra.run.smithi077.stdout: mon max pg per osd = 10000 # >= luminous 2023-11-08T04:51:52.837 INFO:teuthology.orchestra.run.smithi077.stdout: mon pg warn max object skew = 0 2023-11-08T04:51:52.837 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-11-08T04:51:52.837 INFO:teuthology.orchestra.run.smithi077.stdout: # disable pg_autoscaler by default for new pools 2023-11-08T04:51:52.837 INFO:teuthology.orchestra.run.smithi077.stdout: osd_pool_default_pg_autoscale_mode = off 2023-11-08T04:51:52.838 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-11-08T04:51:52.838 INFO:teuthology.orchestra.run.smithi077.stdout: osd pool default size = 2 2023-11-08T04:51:52.838 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-11-08T04:51:52.838 INFO:teuthology.orchestra.run.smithi077.stdout: mon osd allow primary affinity = true 2023-11-08T04:51:52.839 INFO:teuthology.orchestra.run.smithi077.stdout: mon osd allow pg remap = true 2023-11-08T04:51:52.839 INFO:teuthology.orchestra.run.smithi077.stdout: mon warn on legacy crush tunables = false 2023-11-08T04:51:52.839 INFO:teuthology.orchestra.run.smithi077.stdout: mon warn on crush straw calc version zero = false 2023-11-08T04:51:52.839 INFO:teuthology.orchestra.run.smithi077.stdout: mon warn on no sortbitwise = false 2023-11-08T04:51:52.839 INFO:teuthology.orchestra.run.smithi077.stdout: mon warn on osd down out interval zero = false 2023-11-08T04:51:52.839 INFO:teuthology.orchestra.run.smithi077.stdout: mon warn on too few osds = false 2023-11-08T04:51:52.840 INFO:teuthology.orchestra.run.smithi077.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-11-08T04:51:52.840 INFO:teuthology.orchestra.run.smithi077.stdout: mon_warn_on_pool_no_redundancy = false 2023-11-08T04:51:52.840 INFO:teuthology.orchestra.run.smithi077.stdout: mon_allow_pool_size_one = true 2023-11-08T04:51:52.840 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-11-08T04:51:52.840 INFO:teuthology.orchestra.run.smithi077.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2023-11-08T04:51:52.841 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-11-08T04:51:52.841 INFO:teuthology.orchestra.run.smithi077.stdout: osd default data pool replay window = 5 2023-11-08T04:51:52.841 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-11-08T04:51:52.841 INFO:teuthology.orchestra.run.smithi077.stdout: mon allow pool delete = true 2023-11-08T04:51:52.841 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-11-08T04:51:52.841 INFO:teuthology.orchestra.run.smithi077.stdout: mon cluster log file level = debug 2023-11-08T04:51:52.842 INFO:teuthology.orchestra.run.smithi077.stdout: debug asserts on shutdown = true 2023-11-08T04:51:52.842 INFO:teuthology.orchestra.run.smithi077.stdout: mon health detail to clog = false 2023-11-08T04:51:52.842 INFO:teuthology.orchestra.run.smithi077.stdout: mon host = 172.21.15.77 2023-11-08T04:51:52.842 INFO:teuthology.orchestra.run.smithi077.stdout: mon client directed command retry = 5 2023-11-08T04:51:52.842 INFO:teuthology.orchestra.run.smithi077.stdout: ms inject socket failures = 5000 2023-11-08T04:51:52.843 INFO:teuthology.orchestra.run.smithi077.stdout: fsid = 1244e29e-ccb6-44ee-9f2f-7bbf2125940c 2023-11-08T04:51:52.843 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-11-08T04:51:52.843 INFO:teuthology.orchestra.run.smithi077.stdout:[osd] 2023-11-08T04:51:52.843 INFO:teuthology.orchestra.run.smithi077.stdout: osd journal size = 100 2023-11-08T04:51:52.843 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-11-08T04:51:52.844 INFO:teuthology.orchestra.run.smithi077.stdout: osd scrub load threshold = 5.0 2023-11-08T04:51:52.844 INFO:teuthology.orchestra.run.smithi077.stdout: osd scrub max interval = 600 2023-11-08T04:51:52.844 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-11-08T04:51:52.844 INFO:teuthology.orchestra.run.smithi077.stdout: osd recover clone overlap = true 2023-11-08T04:51:52.844 INFO:teuthology.orchestra.run.smithi077.stdout: osd recovery max chunk = 1048576 2023-11-08T04:51:52.845 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-11-08T04:51:52.845 INFO:teuthology.orchestra.run.smithi077.stdout: osd debug shutdown = true 2023-11-08T04:51:52.845 INFO:teuthology.orchestra.run.smithi077.stdout: osd debug op order = true 2023-11-08T04:51:52.845 INFO:teuthology.orchestra.run.smithi077.stdout: osd debug verify stray on activate = true 2023-11-08T04:51:52.845 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-11-08T04:51:52.846 INFO:teuthology.orchestra.run.smithi077.stdout: osd open classes on start = true 2023-11-08T04:51:52.846 INFO:teuthology.orchestra.run.smithi077.stdout: osd debug pg log writeout = true 2023-11-08T04:51:52.846 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-11-08T04:51:52.846 INFO:teuthology.orchestra.run.smithi077.stdout: osd deep scrub update digest min age = 30 2023-11-08T04:51:52.846 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-11-08T04:51:52.847 INFO:teuthology.orchestra.run.smithi077.stdout: osd map max advance = 10 2023-11-08T04:51:52.847 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-11-08T04:51:52.847 INFO:teuthology.orchestra.run.smithi077.stdout: journal zero on create = true 2023-11-08T04:51:52.847 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-11-08T04:51:52.847 INFO:teuthology.orchestra.run.smithi077.stdout: filestore ondisk finisher threads = 3 2023-11-08T04:51:52.847 INFO:teuthology.orchestra.run.smithi077.stdout: filestore apply finisher threads = 3 2023-11-08T04:51:52.848 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-11-08T04:51:52.848 INFO:teuthology.orchestra.run.smithi077.stdout: bdev debug aio = true 2023-11-08T04:51:52.848 INFO:teuthology.orchestra.run.smithi077.stdout: osd debug misdirected ops = true 2023-11-08T04:51:52.848 INFO:teuthology.orchestra.run.smithi077.stdout: bdev async discard = True 2023-11-08T04:51:52.848 INFO:teuthology.orchestra.run.smithi077.stdout: bdev enable discard = True 2023-11-08T04:51:52.849 INFO:teuthology.orchestra.run.smithi077.stdout: bluestore block size = 96636764160 2023-11-08T04:51:52.849 INFO:teuthology.orchestra.run.smithi077.stdout: bluestore fsck on mount = True 2023-11-08T04:51:52.849 INFO:teuthology.orchestra.run.smithi077.stdout: debug bluefs = 1/20 2023-11-08T04:51:52.849 INFO:teuthology.orchestra.run.smithi077.stdout: debug bluestore = 1/20 2023-11-08T04:51:52.849 INFO:teuthology.orchestra.run.smithi077.stdout: debug ms = 1 2023-11-08T04:51:52.850 INFO:teuthology.orchestra.run.smithi077.stdout: debug osd = 20 2023-11-08T04:51:52.850 INFO:teuthology.orchestra.run.smithi077.stdout: debug rocksdb = 4/10 2023-11-08T04:51:52.850 INFO:teuthology.orchestra.run.smithi077.stdout: mon osd backfillfull_ratio = 0.85 2023-11-08T04:51:52.850 INFO:teuthology.orchestra.run.smithi077.stdout: mon osd full ratio = 0.9 2023-11-08T04:51:52.851 INFO:teuthology.orchestra.run.smithi077.stdout: mon osd nearfull ratio = 0.8 2023-11-08T04:51:52.851 INFO:teuthology.orchestra.run.smithi077.stdout: osd failsafe full ratio = 0.95 2023-11-08T04:51:52.851 INFO:teuthology.orchestra.run.smithi077.stdout: osd memory target = 1610612736 2023-11-08T04:51:52.851 INFO:teuthology.orchestra.run.smithi077.stdout: osd objectstore = bluestore 2023-11-08T04:51:52.851 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-11-08T04:51:52.851 INFO:teuthology.orchestra.run.smithi077.stdout:[mgr] 2023-11-08T04:51:52.852 INFO:teuthology.orchestra.run.smithi077.stdout: debug ms = 1 2023-11-08T04:51:52.852 INFO:teuthology.orchestra.run.smithi077.stdout: debug mgr = 20 2023-11-08T04:51:52.852 INFO:teuthology.orchestra.run.smithi077.stdout: debug mon = 20 2023-11-08T04:51:52.852 INFO:teuthology.orchestra.run.smithi077.stdout: debug auth = 20 2023-11-08T04:51:52.852 INFO:teuthology.orchestra.run.smithi077.stdout: mon reweight min pgs per osd = 4 2023-11-08T04:51:52.853 INFO:teuthology.orchestra.run.smithi077.stdout: mon reweight min bytes per osd = 10 2023-11-08T04:51:52.853 INFO:teuthology.orchestra.run.smithi077.stdout: mgr/telemetry/nag = false 2023-11-08T04:51:52.853 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-11-08T04:51:52.853 INFO:teuthology.orchestra.run.smithi077.stdout:[mon] 2023-11-08T04:51:52.853 INFO:teuthology.orchestra.run.smithi077.stdout: debug ms = 1 2023-11-08T04:51:52.854 INFO:teuthology.orchestra.run.smithi077.stdout: debug mon = 20 2023-11-08T04:51:52.854 INFO:teuthology.orchestra.run.smithi077.stdout: debug paxos = 20 2023-11-08T04:51:52.854 INFO:teuthology.orchestra.run.smithi077.stdout: debug auth = 20 2023-11-08T04:51:52.854 INFO:teuthology.orchestra.run.smithi077.stdout: mon data avail warn = 5 2023-11-08T04:51:52.854 INFO:teuthology.orchestra.run.smithi077.stdout: mon mgr mkfs grace = 240 2023-11-08T04:51:52.855 INFO:teuthology.orchestra.run.smithi077.stdout: mon reweight min pgs per osd = 4 2023-11-08T04:51:52.855 INFO:teuthology.orchestra.run.smithi077.stdout: mon osd reporter subtree level = osd 2023-11-08T04:51:52.855 INFO:teuthology.orchestra.run.smithi077.stdout: mon osd prime pg temp = true 2023-11-08T04:51:52.855 INFO:teuthology.orchestra.run.smithi077.stdout: mon reweight min bytes per osd = 10 2023-11-08T04:51:52.855 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-11-08T04:51:52.855 INFO:teuthology.orchestra.run.smithi077.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-11-08T04:51:52.856 INFO:teuthology.orchestra.run.smithi077.stdout: auth mon ticket ttl = 660 # 11m 2023-11-08T04:51:52.856 INFO:teuthology.orchestra.run.smithi077.stdout: auth service ticket ttl = 240 # 4m 2023-11-08T04:51:52.856 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-11-08T04:51:52.856 INFO:teuthology.orchestra.run.smithi077.stdout: # don't complain about insecure global_id in the test suite 2023-11-08T04:51:52.856 INFO:teuthology.orchestra.run.smithi077.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-11-08T04:51:52.857 INFO:teuthology.orchestra.run.smithi077.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-11-08T04:51:52.857 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-11-08T04:51:52.857 INFO:teuthology.orchestra.run.smithi077.stdout:[client] 2023-11-08T04:51:52.857 INFO:teuthology.orchestra.run.smithi077.stdout: rgw cache enabled = true 2023-11-08T04:51:52.857 INFO:teuthology.orchestra.run.smithi077.stdout: rgw enable ops log = true 2023-11-08T04:51:52.857 INFO:teuthology.orchestra.run.smithi077.stdout: rgw enable usage log = true 2023-11-08T04:51:52.858 INFO:teuthology.orchestra.run.smithi077.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-11-08T04:51:52.858 INFO:teuthology.orchestra.run.smithi077.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-11-08T04:51:52.858 INFO:teuthology.orchestra.run.smithi077.stdout: debug rbd = 20 2023-11-08T04:51:52.858 INFO:teuthology.orchestra.run.smithi077.stdout: debug rbd_mirror = 15 2023-11-08T04:51:52.859 INFO:teuthology.orchestra.run.smithi077.stdout: log to stderr = False 2023-11-08T04:51:52.859 INFO:teuthology.orchestra.run.smithi077.stdout: rbd default features = 125 2023-11-08T04:51:52.859 INFO:teuthology.orchestra.run.smithi077.stdout: rbd mirror image policy type = simple 2023-11-08T04:51:52.859 INFO:teuthology.orchestra.run.smithi077.stdout:[mon.a] 2023-11-08T04:51:52.859 INFO:teuthology.orchestra.run.smithi077.stdout:[client.mirror.0] 2023-11-08T04:51:52.860 INFO:teuthology.orchestra.run.smithi077.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-11-08T04:51:52.860 INFO:teuthology.orchestra.run.smithi077.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-11-08T04:51:52.860 INFO:teuthology.orchestra.run.smithi077.stdout:[client.mirror.1] 2023-11-08T04:51:52.860 INFO:teuthology.orchestra.run.smithi077.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-11-08T04:51:52.860 INFO:teuthology.orchestra.run.smithi077.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-11-08T04:51:52.860 INFO:teuthology.orchestra.run.smithi077.stdout:[client.mirror.2] 2023-11-08T04:51:52.861 INFO:teuthology.orchestra.run.smithi077.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-11-08T04:51:52.861 INFO:teuthology.orchestra.run.smithi077.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-11-08T04:51:52.861 INFO:teuthology.orchestra.run.smithi077.stdout:[client.mirror.3] 2023-11-08T04:51:52.861 INFO:teuthology.orchestra.run.smithi077.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-11-08T04:51:52.861 INFO:teuthology.orchestra.run.smithi077.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-11-08T04:51:52.863 INFO:teuthology.orchestra.run.smithi087.stdout:[global] 2023-11-08T04:51:52.863 INFO:teuthology.orchestra.run.smithi087.stdout: chdir = "" 2023-11-08T04:51:52.864 INFO:teuthology.orchestra.run.smithi087.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-11-08T04:51:52.864 INFO:teuthology.orchestra.run.smithi087.stdout: auth supported = cephx 2023-11-08T04:51:52.864 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T04:51:52.864 INFO:teuthology.orchestra.run.smithi087.stdout: filestore xattr use omap = true 2023-11-08T04:51:52.864 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T04:51:52.865 INFO:teuthology.orchestra.run.smithi087.stdout: mon clock drift allowed = 1.000 2023-11-08T04:51:52.865 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T04:51:52.865 INFO:teuthology.orchestra.run.smithi087.stdout: osd crush chooseleaf type = 0 2023-11-08T04:51:52.865 INFO:teuthology.orchestra.run.smithi087.stdout: auth debug = true 2023-11-08T04:51:52.865 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T04:51:52.866 INFO:teuthology.orchestra.run.smithi087.stdout: ms die on old message = true 2023-11-08T04:51:52.866 INFO:teuthology.orchestra.run.smithi087.stdout: ms die on bug = true 2023-11-08T04:51:52.866 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T04:51:52.866 INFO:teuthology.orchestra.run.smithi087.stdout: mon max pg per osd = 10000 # >= luminous 2023-11-08T04:51:52.867 INFO:teuthology.orchestra.run.smithi087.stdout: mon pg warn max object skew = 0 2023-11-08T04:51:52.867 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T04:51:52.867 INFO:teuthology.orchestra.run.smithi087.stdout: # disable pg_autoscaler by default for new pools 2023-11-08T04:51:52.867 INFO:teuthology.orchestra.run.smithi087.stdout: osd_pool_default_pg_autoscale_mode = off 2023-11-08T04:51:52.868 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T04:51:52.868 INFO:teuthology.orchestra.run.smithi087.stdout: osd pool default size = 2 2023-11-08T04:51:52.868 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T04:51:52.868 INFO:teuthology.orchestra.run.smithi087.stdout: mon osd allow primary affinity = true 2023-11-08T04:51:52.868 INFO:teuthology.orchestra.run.smithi087.stdout: mon osd allow pg remap = true 2023-11-08T04:51:52.869 INFO:teuthology.orchestra.run.smithi087.stdout: mon warn on legacy crush tunables = false 2023-11-08T04:51:52.869 INFO:teuthology.orchestra.run.smithi087.stdout: mon warn on crush straw calc version zero = false 2023-11-08T04:51:52.869 INFO:teuthology.orchestra.run.smithi087.stdout: mon warn on no sortbitwise = false 2023-11-08T04:51:52.869 INFO:teuthology.orchestra.run.smithi087.stdout: mon warn on osd down out interval zero = false 2023-11-08T04:51:52.870 INFO:teuthology.orchestra.run.smithi087.stdout: mon warn on too few osds = false 2023-11-08T04:51:52.870 INFO:teuthology.orchestra.run.smithi087.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-11-08T04:51:52.870 INFO:teuthology.orchestra.run.smithi087.stdout: mon_warn_on_pool_no_redundancy = false 2023-11-08T04:51:52.870 INFO:teuthology.orchestra.run.smithi087.stdout: mon_allow_pool_size_one = true 2023-11-08T04:51:52.870 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T04:51:52.871 INFO:teuthology.orchestra.run.smithi087.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2023-11-08T04:51:52.871 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T04:51:52.871 INFO:teuthology.orchestra.run.smithi087.stdout: osd default data pool replay window = 5 2023-11-08T04:51:52.871 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T04:51:52.871 INFO:teuthology.orchestra.run.smithi087.stdout: mon allow pool delete = true 2023-11-08T04:51:52.871 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T04:51:52.872 INFO:teuthology.orchestra.run.smithi087.stdout: mon cluster log file level = debug 2023-11-08T04:51:52.872 INFO:teuthology.orchestra.run.smithi087.stdout: debug asserts on shutdown = true 2023-11-08T04:51:52.872 INFO:teuthology.orchestra.run.smithi087.stdout: mon health detail to clog = false 2023-11-08T04:51:52.872 INFO:teuthology.orchestra.run.smithi087.stdout: mon host = 172.21.15.77 2023-11-08T04:51:52.872 INFO:teuthology.orchestra.run.smithi087.stdout: mon client directed command retry = 5 2023-11-08T04:51:52.873 INFO:teuthology.orchestra.run.smithi087.stdout: ms inject socket failures = 5000 2023-11-08T04:51:52.873 INFO:teuthology.orchestra.run.smithi087.stdout: fsid = 1244e29e-ccb6-44ee-9f2f-7bbf2125940c 2023-11-08T04:51:52.873 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T04:51:52.873 INFO:teuthology.orchestra.run.smithi087.stdout:[osd] 2023-11-08T04:51:52.873 INFO:teuthology.orchestra.run.smithi087.stdout: osd journal size = 100 2023-11-08T04:51:52.874 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T04:51:52.874 INFO:teuthology.orchestra.run.smithi087.stdout: osd scrub load threshold = 5.0 2023-11-08T04:51:52.874 INFO:teuthology.orchestra.run.smithi087.stdout: osd scrub max interval = 600 2023-11-08T04:51:52.874 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T04:51:52.875 INFO:teuthology.orchestra.run.smithi087.stdout: osd recover clone overlap = true 2023-11-08T04:51:52.875 INFO:teuthology.orchestra.run.smithi087.stdout: osd recovery max chunk = 1048576 2023-11-08T04:51:52.875 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T04:51:52.875 INFO:teuthology.orchestra.run.smithi087.stdout: osd debug shutdown = true 2023-11-08T04:51:52.875 INFO:teuthology.orchestra.run.smithi087.stdout: osd debug op order = true 2023-11-08T04:51:52.875 INFO:teuthology.orchestra.run.smithi087.stdout: osd debug verify stray on activate = true 2023-11-08T04:51:52.876 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T04:51:52.876 INFO:teuthology.orchestra.run.smithi087.stdout: osd open classes on start = true 2023-11-08T04:51:52.876 INFO:teuthology.orchestra.run.smithi087.stdout: osd debug pg log writeout = true 2023-11-08T04:51:52.876 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T04:51:52.877 INFO:teuthology.orchestra.run.smithi087.stdout: osd deep scrub update digest min age = 30 2023-11-08T04:51:52.877 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T04:51:52.877 INFO:teuthology.orchestra.run.smithi087.stdout: osd map max advance = 10 2023-11-08T04:51:52.877 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T04:51:52.877 INFO:teuthology.orchestra.run.smithi087.stdout: journal zero on create = true 2023-11-08T04:51:52.877 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T04:51:52.878 INFO:teuthology.orchestra.run.smithi087.stdout: filestore ondisk finisher threads = 3 2023-11-08T04:51:52.878 INFO:teuthology.orchestra.run.smithi087.stdout: filestore apply finisher threads = 3 2023-11-08T04:51:52.878 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T04:51:52.878 INFO:teuthology.orchestra.run.smithi087.stdout: bdev debug aio = true 2023-11-08T04:51:52.879 INFO:teuthology.orchestra.run.smithi087.stdout: osd debug misdirected ops = true 2023-11-08T04:51:52.879 INFO:teuthology.orchestra.run.smithi087.stdout: bdev async discard = True 2023-11-08T04:51:52.879 INFO:teuthology.orchestra.run.smithi087.stdout: bdev enable discard = True 2023-11-08T04:51:52.879 INFO:teuthology.orchestra.run.smithi087.stdout: bluestore block size = 96636764160 2023-11-08T04:51:52.879 INFO:teuthology.orchestra.run.smithi087.stdout: bluestore fsck on mount = True 2023-11-08T04:51:52.880 INFO:teuthology.orchestra.run.smithi087.stdout: debug bluefs = 1/20 2023-11-08T04:51:52.880 INFO:teuthology.orchestra.run.smithi087.stdout: debug bluestore = 1/20 2023-11-08T04:51:52.880 INFO:teuthology.orchestra.run.smithi087.stdout: debug ms = 1 2023-11-08T04:51:52.880 INFO:teuthology.orchestra.run.smithi087.stdout: debug osd = 20 2023-11-08T04:51:52.880 INFO:teuthology.orchestra.run.smithi087.stdout: debug rocksdb = 4/10 2023-11-08T04:51:52.880 INFO:teuthology.orchestra.run.smithi087.stdout: mon osd backfillfull_ratio = 0.85 2023-11-08T04:51:52.881 INFO:teuthology.orchestra.run.smithi087.stdout: mon osd full ratio = 0.9 2023-11-08T04:51:52.881 INFO:teuthology.orchestra.run.smithi087.stdout: mon osd nearfull ratio = 0.8 2023-11-08T04:51:52.881 INFO:teuthology.orchestra.run.smithi087.stdout: osd failsafe full ratio = 0.95 2023-11-08T04:51:52.881 INFO:teuthology.orchestra.run.smithi087.stdout: osd memory target = 1610612736 2023-11-08T04:51:52.881 INFO:teuthology.orchestra.run.smithi087.stdout: osd objectstore = bluestore 2023-11-08T04:51:52.882 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T04:51:52.882 INFO:teuthology.orchestra.run.smithi087.stdout:[mgr] 2023-11-08T04:51:52.882 INFO:teuthology.orchestra.run.smithi087.stdout: debug ms = 1 2023-11-08T04:51:52.882 INFO:teuthology.orchestra.run.smithi087.stdout: debug mgr = 20 2023-11-08T04:51:52.882 INFO:teuthology.orchestra.run.smithi087.stdout: debug mon = 20 2023-11-08T04:51:52.883 INFO:teuthology.orchestra.run.smithi087.stdout: debug auth = 20 2023-11-08T04:51:52.883 INFO:teuthology.orchestra.run.smithi087.stdout: mon reweight min pgs per osd = 4 2023-11-08T04:51:52.883 INFO:teuthology.orchestra.run.smithi087.stdout: mon reweight min bytes per osd = 10 2023-11-08T04:51:52.883 INFO:teuthology.orchestra.run.smithi087.stdout: mgr/telemetry/nag = false 2023-11-08T04:51:52.883 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T04:51:52.884 INFO:teuthology.orchestra.run.smithi087.stdout:[mon] 2023-11-08T04:51:52.884 INFO:teuthology.orchestra.run.smithi087.stdout: debug ms = 1 2023-11-08T04:51:52.884 INFO:teuthology.orchestra.run.smithi087.stdout: debug mon = 20 2023-11-08T04:51:52.884 INFO:teuthology.orchestra.run.smithi087.stdout: debug paxos = 20 2023-11-08T04:51:52.884 INFO:teuthology.orchestra.run.smithi087.stdout: debug auth = 20 2023-11-08T04:51:52.884 INFO:teuthology.orchestra.run.smithi087.stdout: mon data avail warn = 5 2023-11-08T04:51:52.885 INFO:teuthology.orchestra.run.smithi087.stdout: mon mgr mkfs grace = 240 2023-11-08T04:51:52.885 INFO:teuthology.orchestra.run.smithi087.stdout: mon reweight min pgs per osd = 4 2023-11-08T04:51:52.885 INFO:teuthology.orchestra.run.smithi087.stdout: mon osd reporter subtree level = osd 2023-11-08T04:51:52.885 INFO:teuthology.orchestra.run.smithi087.stdout: mon osd prime pg temp = true 2023-11-08T04:51:52.885 INFO:teuthology.orchestra.run.smithi087.stdout: mon reweight min bytes per osd = 10 2023-11-08T04:51:52.885 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T04:51:52.886 INFO:teuthology.orchestra.run.smithi087.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-11-08T04:51:52.886 INFO:teuthology.orchestra.run.smithi087.stdout: auth mon ticket ttl = 660 # 11m 2023-11-08T04:51:52.886 INFO:teuthology.orchestra.run.smithi087.stdout: auth service ticket ttl = 240 # 4m 2023-11-08T04:51:52.886 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T04:51:52.886 INFO:teuthology.orchestra.run.smithi087.stdout: # don't complain about insecure global_id in the test suite 2023-11-08T04:51:52.887 INFO:teuthology.orchestra.run.smithi087.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-11-08T04:51:52.887 INFO:teuthology.orchestra.run.smithi087.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-11-08T04:51:52.887 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T04:51:52.887 INFO:teuthology.orchestra.run.smithi087.stdout:[client] 2023-11-08T04:51:52.887 INFO:teuthology.orchestra.run.smithi087.stdout: rgw cache enabled = true 2023-11-08T04:51:52.887 INFO:teuthology.orchestra.run.smithi087.stdout: rgw enable ops log = true 2023-11-08T04:51:52.888 INFO:teuthology.orchestra.run.smithi087.stdout: rgw enable usage log = true 2023-11-08T04:51:52.888 INFO:teuthology.orchestra.run.smithi087.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-11-08T04:51:52.888 INFO:teuthology.orchestra.run.smithi087.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-11-08T04:51:52.888 INFO:teuthology.orchestra.run.smithi087.stdout: debug rbd = 20 2023-11-08T04:51:52.888 INFO:teuthology.orchestra.run.smithi087.stdout: debug rbd_mirror = 15 2023-11-08T04:51:52.888 INFO:teuthology.orchestra.run.smithi087.stdout: log to stderr = False 2023-11-08T04:51:52.889 INFO:teuthology.orchestra.run.smithi087.stdout: rbd default features = 125 2023-11-08T04:51:52.889 INFO:teuthology.orchestra.run.smithi087.stdout: rbd mirror image policy type = simple 2023-11-08T04:51:52.889 INFO:teuthology.orchestra.run.smithi087.stdout:[mon.a] 2023-11-08T04:51:52.889 INFO:teuthology.orchestra.run.smithi087.stdout:[client.mirror.0] 2023-11-08T04:51:52.889 INFO:teuthology.orchestra.run.smithi087.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-11-08T04:51:52.890 INFO:teuthology.orchestra.run.smithi087.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-11-08T04:51:52.890 INFO:teuthology.orchestra.run.smithi087.stdout:[client.mirror.1] 2023-11-08T04:51:52.890 INFO:teuthology.orchestra.run.smithi087.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-11-08T04:51:52.890 INFO:teuthology.orchestra.run.smithi087.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-11-08T04:51:52.890 INFO:teuthology.orchestra.run.smithi087.stdout:[client.mirror.2] 2023-11-08T04:51:52.891 INFO:teuthology.orchestra.run.smithi087.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-11-08T04:51:52.891 INFO:teuthology.orchestra.run.smithi087.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-11-08T04:51:52.891 INFO:teuthology.orchestra.run.smithi087.stdout:[client.mirror.3] 2023-11-08T04:51:52.891 INFO:teuthology.orchestra.run.smithi087.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-11-08T04:51:52.891 INFO:teuthology.orchestra.run.smithi087.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-11-08T04:51:52.892 INFO:tasks.ceph:Creating admin key on cluster1.mon.a... 2023-11-08T04:51:52.892 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/cluster1.keyring 2023-11-08T04:51:52.954 INFO:tasks.ceph:Copying monmap to all nodes... 2023-11-08T04:51:52.955 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2023-11-08T04:51:52.955 DEBUG:teuthology.orchestra.run.smithi077:> dd if=/etc/ceph/cluster1.keyring of=/dev/stdout 2023-11-08T04:51:52.973 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2023-11-08T04:51:52.973 DEBUG:teuthology.orchestra.run.smithi077:> dd if=/home/ubuntu/cephtest/cluster1.monmap of=/dev/stdout 2023-11-08T04:51:53.030 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi077.front.sepia.ceph.com 2023-11-08T04:51:53.031 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2023-11-08T04:51:53.031 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd of=/etc/ceph/cluster1.keyring 2023-11-08T04:51:53.031 DEBUG:teuthology.orchestra.run.smithi077:> sudo chmod 0644 /etc/ceph/cluster1.keyring 2023-11-08T04:51:53.105 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2023-11-08T04:51:53.105 DEBUG:teuthology.orchestra.run.smithi077:> dd of=/home/ubuntu/cephtest/cluster1.monmap 2023-11-08T04:51:53.163 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi087.front.sepia.ceph.com 2023-11-08T04:51:53.163 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2023-11-08T04:51:53.163 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd of=/etc/ceph/cluster1.keyring 2023-11-08T04:51:53.164 DEBUG:teuthology.orchestra.run.smithi087:> sudo chmod 0644 /etc/ceph/cluster1.keyring 2023-11-08T04:51:53.207 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2023-11-08T04:51:53.208 DEBUG:teuthology.orchestra.run.smithi087:> dd of=/home/ubuntu/cephtest/cluster1.monmap 2023-11-08T04:51:53.268 INFO:tasks.ceph:Setting up mon nodes... 2023-11-08T04:51:53.268 INFO:tasks.ceph:Setting up mgr nodes... 2023-11-08T04:51:53.269 DEBUG:teuthology.orchestra.run.smithi077:> sudo mkdir -p /var/lib/ceph/mgr/cluster1-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/cluster1-x/keyring 2023-11-08T04:51:53.318 INFO:teuthology.orchestra.run.smithi077.stdout:creating /var/lib/ceph/mgr/cluster1-x/keyring 2023-11-08T04:51:53.320 INFO:tasks.ceph:Setting up mds nodes... 2023-11-08T04:51:53.321 INFO:tasks.ceph_client:Setting up client nodes... 2023-11-08T04:51:53.321 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/cluster1.client.0.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.0.keyring 2023-11-08T04:51:53.360 INFO:teuthology.orchestra.run.smithi077.stdout:creating /etc/ceph/cluster1.client.0.keyring 2023-11-08T04:51:53.372 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror /etc/ceph/cluster1.client.mirror.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.keyring 2023-11-08T04:51:53.495 INFO:teuthology.orchestra.run.smithi087.stdout:creating /etc/ceph/cluster1.client.mirror.keyring 2023-11-08T04:51:53.509 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.0 /etc/ceph/cluster1.client.mirror.0.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.0.keyring 2023-11-08T04:51:53.551 INFO:teuthology.orchestra.run.smithi087.stdout:creating /etc/ceph/cluster1.client.mirror.0.keyring 2023-11-08T04:51:53.565 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.1 /etc/ceph/cluster1.client.mirror.1.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.1.keyring 2023-11-08T04:51:53.646 INFO:teuthology.orchestra.run.smithi087.stdout:creating /etc/ceph/cluster1.client.mirror.1.keyring 2023-11-08T04:51:53.658 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.2 /etc/ceph/cluster1.client.mirror.2.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.2.keyring 2023-11-08T04:51:53.738 INFO:teuthology.orchestra.run.smithi087.stdout:creating /etc/ceph/cluster1.client.mirror.2.keyring 2023-11-08T04:51:53.750 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.3 /etc/ceph/cluster1.client.mirror.3.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.3.keyring 2023-11-08T04:51:53.830 INFO:teuthology.orchestra.run.smithi087.stdout:creating /etc/ceph/cluster1.client.mirror.3.keyring 2023-11-08T04:51:53.841 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.4 /etc/ceph/cluster1.client.mirror.4.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.4.keyring 2023-11-08T04:51:53.921 INFO:teuthology.orchestra.run.smithi087.stdout:creating /etc/ceph/cluster1.client.mirror.4.keyring 2023-11-08T04:51:53.932 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.5 /etc/ceph/cluster1.client.mirror.5.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.5.keyring 2023-11-08T04:51:54.012 INFO:teuthology.orchestra.run.smithi087.stdout:creating /etc/ceph/cluster1.client.mirror.5.keyring 2023-11-08T04:51:54.023 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.6 /etc/ceph/cluster1.client.mirror.6.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.6.keyring 2023-11-08T04:51:54.102 INFO:teuthology.orchestra.run.smithi087.stdout:creating /etc/ceph/cluster1.client.mirror.6.keyring 2023-11-08T04:51:54.114 INFO:tasks.ceph:Running mkfs on osd nodes... 2023-11-08T04:51:54.114 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi077.front.sepia.ceph.com'): {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'}} 2023-11-08T04:51:54.115 DEBUG:teuthology.orchestra.run.smithi077:> sudo mkdir -p /var/lib/ceph/osd/cluster1-0 2023-11-08T04:51:54.147 INFO:tasks.ceph:roles_to_devs: {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'} 2023-11-08T04:51:54.148 INFO:tasks.ceph:role: cluster1.osd.0 2023-11-08T04:51:54.148 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi077.front.sepia.ceph.com 2023-11-08T04:51:54.148 DEBUG:teuthology.orchestra.run.smithi077:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-11-08T04:51:54.226 INFO:teuthology.orchestra.run.smithi077.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-11-08T04:51:54.226 INFO:teuthology.orchestra.run.smithi077.stdout: = sectsz=512 attr=2, projid32bit=1 2023-11-08T04:51:54.226 INFO:teuthology.orchestra.run.smithi077.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-11-08T04:51:54.227 INFO:teuthology.orchestra.run.smithi077.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-11-08T04:51:54.227 INFO:teuthology.orchestra.run.smithi077.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-11-08T04:51:54.227 INFO:teuthology.orchestra.run.smithi077.stdout: = sunit=0 swidth=0 blks 2023-11-08T04:51:54.227 INFO:teuthology.orchestra.run.smithi077.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-11-08T04:51:54.227 INFO:teuthology.orchestra.run.smithi077.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-11-08T04:51:54.228 INFO:teuthology.orchestra.run.smithi077.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-11-08T04:51:54.228 INFO:teuthology.orchestra.run.smithi077.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-11-08T04:51:54.658 INFO:teuthology.orchestra.run.smithi077.stdout:Discarding blocks...Done. 2023-11-08T04:51:54.660 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi077.front.sepia.ceph.com -o noatime 2023-11-08T04:51:54.661 DEBUG:teuthology.orchestra.run.smithi077:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/cluster1-0 2023-11-08T04:51:54.849 DEBUG:teuthology.orchestra.run.smithi077:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster1-0 2023-11-08T04:51:54.887 DEBUG:teuthology.orchestra.run.smithi077:> sudo mkdir -p /var/lib/ceph/osd/cluster1-1 2023-11-08T04:51:54.958 INFO:tasks.ceph:roles_to_devs: {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'} 2023-11-08T04:51:54.958 INFO:tasks.ceph:role: cluster1.osd.1 2023-11-08T04:51:54.959 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi077.front.sepia.ceph.com 2023-11-08T04:51:54.959 DEBUG:teuthology.orchestra.run.smithi077:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-11-08T04:51:55.036 INFO:teuthology.orchestra.run.smithi077.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-11-08T04:51:55.036 INFO:teuthology.orchestra.run.smithi077.stdout: = sectsz=512 attr=2, projid32bit=1 2023-11-08T04:51:55.036 INFO:teuthology.orchestra.run.smithi077.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-11-08T04:51:55.037 INFO:teuthology.orchestra.run.smithi077.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-11-08T04:51:55.037 INFO:teuthology.orchestra.run.smithi077.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-11-08T04:51:55.037 INFO:teuthology.orchestra.run.smithi077.stdout: = sunit=0 swidth=0 blks 2023-11-08T04:51:55.037 INFO:teuthology.orchestra.run.smithi077.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-11-08T04:51:55.037 INFO:teuthology.orchestra.run.smithi077.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-11-08T04:51:55.038 INFO:teuthology.orchestra.run.smithi077.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-11-08T04:51:55.038 INFO:teuthology.orchestra.run.smithi077.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-11-08T04:51:55.455 INFO:teuthology.orchestra.run.smithi077.stdout:Discarding blocks...Done. 2023-11-08T04:51:55.457 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi077.front.sepia.ceph.com -o noatime 2023-11-08T04:51:55.457 DEBUG:teuthology.orchestra.run.smithi077:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/cluster1-1 2023-11-08T04:51:55.566 DEBUG:teuthology.orchestra.run.smithi077:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster1-1 2023-11-08T04:51:55.600 DEBUG:teuthology.orchestra.run.smithi077:> sudo mkdir -p /var/lib/ceph/osd/cluster1-2 2023-11-08T04:51:55.669 INFO:tasks.ceph:roles_to_devs: {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'} 2023-11-08T04:51:55.670 INFO:tasks.ceph:role: cluster1.osd.2 2023-11-08T04:51:55.670 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi077.front.sepia.ceph.com 2023-11-08T04:51:55.671 DEBUG:teuthology.orchestra.run.smithi077:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-11-08T04:51:55.745 INFO:teuthology.orchestra.run.smithi077.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-11-08T04:51:55.745 INFO:teuthology.orchestra.run.smithi077.stdout: = sectsz=512 attr=2, projid32bit=1 2023-11-08T04:51:55.745 INFO:teuthology.orchestra.run.smithi077.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-11-08T04:51:55.746 INFO:teuthology.orchestra.run.smithi077.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-11-08T04:51:55.746 INFO:teuthology.orchestra.run.smithi077.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-11-08T04:51:55.746 INFO:teuthology.orchestra.run.smithi077.stdout: = sunit=0 swidth=0 blks 2023-11-08T04:51:55.747 INFO:teuthology.orchestra.run.smithi077.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-11-08T04:51:55.747 INFO:teuthology.orchestra.run.smithi077.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-11-08T04:51:55.747 INFO:teuthology.orchestra.run.smithi077.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-11-08T04:51:55.747 INFO:teuthology.orchestra.run.smithi077.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-11-08T04:51:56.187 INFO:teuthology.orchestra.run.smithi077.stdout:Discarding blocks...Done. 2023-11-08T04:51:56.189 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi077.front.sepia.ceph.com -o noatime 2023-11-08T04:51:56.190 DEBUG:teuthology.orchestra.run.smithi077:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/cluster1-2 2023-11-08T04:51:56.358 DEBUG:teuthology.orchestra.run.smithi077:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster1-2 2023-11-08T04:51:56.395 DEBUG:teuthology.orchestra.run.smithi077:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster1 --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/cluster1.monmap 2023-11-08T04:51:56.526 INFO:teuthology.orchestra.run.smithi077.stderr:2023-11-08T04:51:56.523+0000 7f4962e4f380 -1 auth: error reading file: /var/lib/ceph/osd/cluster1-0/keyring: can't open /var/lib/ceph/osd/cluster1-0/keyring: (2) No such file or directory 2023-11-08T04:51:56.527 INFO:teuthology.orchestra.run.smithi077.stderr:2023-11-08T04:51:56.524+0000 7f4962e4f380 -1 created new key in keyring /var/lib/ceph/osd/cluster1-0/keyring 2023-11-08T04:51:56.527 INFO:teuthology.orchestra.run.smithi077.stderr:2023-11-08T04:51:56.524+0000 7f4962e4f380 -1 bluestore(/var/lib/ceph/osd/cluster1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-0/block: (2) No such file or directory 2023-11-08T04:51:56.527 INFO:teuthology.orchestra.run.smithi077.stderr:2023-11-08T04:51:56.524+0000 7f4962e4f380 -1 bluestore(/var/lib/ceph/osd/cluster1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-0/block: (2) No such file or directory 2023-11-08T04:51:56.527 INFO:teuthology.orchestra.run.smithi077.stderr:2023-11-08T04:51:56.524+0000 7f4962e4f380 -1 bluestore(/var/lib/ceph/osd/cluster1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-0/block: (2) No such file or directory 2023-11-08T04:51:56.534 INFO:teuthology.orchestra.run.smithi077.stderr:2023-11-08T04:51:56.532+0000 7f4962e4f380 -1 bluestore(/var/lib/ceph/osd/cluster1-0) _read_fsid unparsable uuid 2023-11-08T04:51:59.374 DEBUG:teuthology.orchestra.run.smithi077:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster1-0 2023-11-08T04:51:59.407 DEBUG:teuthology.orchestra.run.smithi077:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster1 --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/cluster1.monmap 2023-11-08T04:51:59.495 INFO:teuthology.orchestra.run.smithi077.stderr:2023-11-08T04:51:59.492+0000 7fd3f5c1c380 -1 auth: error reading file: /var/lib/ceph/osd/cluster1-1/keyring: can't open /var/lib/ceph/osd/cluster1-1/keyring: (2) No such file or directory 2023-11-08T04:51:59.495 INFO:teuthology.orchestra.run.smithi077.stderr:2023-11-08T04:51:59.492+0000 7fd3f5c1c380 -1 created new key in keyring /var/lib/ceph/osd/cluster1-1/keyring 2023-11-08T04:51:59.495 INFO:teuthology.orchestra.run.smithi077.stderr:2023-11-08T04:51:59.492+0000 7fd3f5c1c380 -1 bluestore(/var/lib/ceph/osd/cluster1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-1/block: (2) No such file or directory 2023-11-08T04:51:59.496 INFO:teuthology.orchestra.run.smithi077.stderr:2023-11-08T04:51:59.492+0000 7fd3f5c1c380 -1 bluestore(/var/lib/ceph/osd/cluster1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-1/block: (2) No such file or directory 2023-11-08T04:51:59.496 INFO:teuthology.orchestra.run.smithi077.stderr:2023-11-08T04:51:59.492+0000 7fd3f5c1c380 -1 bluestore(/var/lib/ceph/osd/cluster1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-1/block: (2) No such file or directory 2023-11-08T04:51:59.496 INFO:teuthology.orchestra.run.smithi077.stderr:2023-11-08T04:51:59.493+0000 7fd3f5c1c380 -1 bluestore(/var/lib/ceph/osd/cluster1-1) _read_fsid unparsable uuid 2023-11-08T04:52:02.345 DEBUG:teuthology.orchestra.run.smithi077:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster1-1 2023-11-08T04:52:02.376 DEBUG:teuthology.orchestra.run.smithi077:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster1 --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/cluster1.monmap 2023-11-08T04:52:02.462 INFO:teuthology.orchestra.run.smithi077.stderr:2023-11-08T04:52:02.459+0000 7fdbab5ae380 -1 auth: error reading file: /var/lib/ceph/osd/cluster1-2/keyring: can't open /var/lib/ceph/osd/cluster1-2/keyring: (2) No such file or directory 2023-11-08T04:52:02.462 INFO:teuthology.orchestra.run.smithi077.stderr:2023-11-08T04:52:02.459+0000 7fdbab5ae380 -1 created new key in keyring /var/lib/ceph/osd/cluster1-2/keyring 2023-11-08T04:52:02.462 INFO:teuthology.orchestra.run.smithi077.stderr:2023-11-08T04:52:02.459+0000 7fdbab5ae380 -1 bluestore(/var/lib/ceph/osd/cluster1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-2/block: (2) No such file or directory 2023-11-08T04:52:02.462 INFO:teuthology.orchestra.run.smithi077.stderr:2023-11-08T04:52:02.459+0000 7fdbab5ae380 -1 bluestore(/var/lib/ceph/osd/cluster1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-2/block: (2) No such file or directory 2023-11-08T04:52:02.463 INFO:teuthology.orchestra.run.smithi077.stderr:2023-11-08T04:52:02.459+0000 7fdbab5ae380 -1 bluestore(/var/lib/ceph/osd/cluster1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-2/block: (2) No such file or directory 2023-11-08T04:52:02.463 INFO:teuthology.orchestra.run.smithi077.stderr:2023-11-08T04:52:02.460+0000 7fdbab5ae380 -1 bluestore(/var/lib/ceph/osd/cluster1-2) _read_fsid unparsable uuid 2023-11-08T04:52:05.301 DEBUG:teuthology.orchestra.run.smithi077:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster1-2 2023-11-08T04:52:05.333 INFO:tasks.ceph:Reading keys from all nodes... 2023-11-08T04:52:05.334 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2023-11-08T04:52:05.334 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd if=/var/lib/ceph/mgr/cluster1-x/keyring of=/dev/stdout 2023-11-08T04:52:05.404 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2023-11-08T04:52:05.405 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd if=/var/lib/ceph/osd/cluster1-0/keyring of=/dev/stdout 2023-11-08T04:52:05.473 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2023-11-08T04:52:05.473 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd if=/var/lib/ceph/osd/cluster1-1/keyring of=/dev/stdout 2023-11-08T04:52:05.541 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2023-11-08T04:52:05.542 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd if=/var/lib/ceph/osd/cluster1-2/keyring of=/dev/stdout 2023-11-08T04:52:05.609 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2023-11-08T04:52:05.610 DEBUG:teuthology.orchestra.run.smithi077:> dd if=/etc/ceph/cluster1.client.0.keyring of=/dev/stdout 2023-11-08T04:52:05.667 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2023-11-08T04:52:05.668 DEBUG:teuthology.orchestra.run.smithi087:> dd if=/etc/ceph/cluster1.client.mirror.keyring of=/dev/stdout 2023-11-08T04:52:05.689 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2023-11-08T04:52:05.690 DEBUG:teuthology.orchestra.run.smithi087:> dd if=/etc/ceph/cluster1.client.mirror.0.keyring of=/dev/stdout 2023-11-08T04:52:05.751 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2023-11-08T04:52:05.751 DEBUG:teuthology.orchestra.run.smithi087:> dd if=/etc/ceph/cluster1.client.mirror.1.keyring of=/dev/stdout 2023-11-08T04:52:05.811 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2023-11-08T04:52:05.811 DEBUG:teuthology.orchestra.run.smithi087:> dd if=/etc/ceph/cluster1.client.mirror.2.keyring of=/dev/stdout 2023-11-08T04:52:05.870 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2023-11-08T04:52:05.870 DEBUG:teuthology.orchestra.run.smithi087:> dd if=/etc/ceph/cluster1.client.mirror.3.keyring of=/dev/stdout 2023-11-08T04:52:05.929 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2023-11-08T04:52:05.929 DEBUG:teuthology.orchestra.run.smithi087:> dd if=/etc/ceph/cluster1.client.mirror.4.keyring of=/dev/stdout 2023-11-08T04:52:05.988 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2023-11-08T04:52:05.988 DEBUG:teuthology.orchestra.run.smithi087:> dd if=/etc/ceph/cluster1.client.mirror.5.keyring of=/dev/stdout 2023-11-08T04:52:06.046 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2023-11-08T04:52:06.047 DEBUG:teuthology.orchestra.run.smithi087:> dd if=/etc/ceph/cluster1.client.mirror.6.keyring of=/dev/stdout 2023-11-08T04:52:06.105 INFO:tasks.ceph:Adding keys to all mons... 2023-11-08T04:52:06.105 DEBUG:teuthology.orchestra.run.smithi077:> sudo tee -a /etc/ceph/cluster1.keyring 2023-11-08T04:52:06.132 INFO:teuthology.orchestra.run.smithi077.stdout:[mgr.x] 2023-11-08T04:52:06.132 INFO:teuthology.orchestra.run.smithi077.stdout: key = AQDpE0tlXPnwEhAAQQ06LpV0DNGJMEDKJX0Zhw== 2023-11-08T04:52:06.132 INFO:teuthology.orchestra.run.smithi077.stdout:[osd.0] 2023-11-08T04:52:06.133 INFO:teuthology.orchestra.run.smithi077.stdout: key = AQDsE0tlaoRLHxAAuzKgaAcafEF3ubmDtHPo9g== 2023-11-08T04:52:06.133 INFO:teuthology.orchestra.run.smithi077.stdout:[osd.1] 2023-11-08T04:52:06.133 INFO:teuthology.orchestra.run.smithi077.stdout: key = AQDvE0tl+QhqHRAAqr0AWiCHP27QKAXNVV+O+g== 2023-11-08T04:52:06.133 INFO:teuthology.orchestra.run.smithi077.stdout:[osd.2] 2023-11-08T04:52:06.133 INFO:teuthology.orchestra.run.smithi077.stdout: key = AQDyE0tlt491GxAAnr2u9m9efpeneak54FXdZw== 2023-11-08T04:52:06.134 INFO:teuthology.orchestra.run.smithi077.stdout:[client.0] 2023-11-08T04:52:06.134 INFO:teuthology.orchestra.run.smithi077.stdout: key = AQDpE0tlkSNxFRAAKYLsIcv9PIwWOTlyeVoSgw== 2023-11-08T04:52:06.134 INFO:teuthology.orchestra.run.smithi077.stdout:[client.mirror] 2023-11-08T04:52:06.134 INFO:teuthology.orchestra.run.smithi077.stdout: key = AQDpE0tlXVl3HRAAxk0N4Dm5RctcgTAPlYgeWQ== 2023-11-08T04:52:06.134 INFO:teuthology.orchestra.run.smithi077.stdout:[client.mirror.0] 2023-11-08T04:52:06.135 INFO:teuthology.orchestra.run.smithi077.stdout: key = AQDpE0tlULDSIBAAvUE9gDSx7apSEHuHUvlM/A== 2023-11-08T04:52:06.135 INFO:teuthology.orchestra.run.smithi077.stdout:[client.mirror.1] 2023-11-08T04:52:06.135 INFO:teuthology.orchestra.run.smithi077.stdout: key = AQDpE0tlZ294JhAA0fj3w0CiqkGpRCAjzPUoAA== 2023-11-08T04:52:06.135 INFO:teuthology.orchestra.run.smithi077.stdout:[client.mirror.2] 2023-11-08T04:52:06.135 INFO:teuthology.orchestra.run.smithi077.stdout: key = AQDpE0tlKh3yKxAAWHq+PCcu4jRYskbvwTU02A== 2023-11-08T04:52:06.135 INFO:teuthology.orchestra.run.smithi077.stdout:[client.mirror.3] 2023-11-08T04:52:06.136 INFO:teuthology.orchestra.run.smithi077.stdout: key = AQDpE0tlQh5sMRAAtwxB+kTUHXCghsXjT/PyWw== 2023-11-08T04:52:06.136 INFO:teuthology.orchestra.run.smithi077.stdout:[client.mirror.4] 2023-11-08T04:52:06.136 INFO:teuthology.orchestra.run.smithi077.stdout: key = AQDpE0tlU63XNhAAIKPptsfus0KEsYMilrS4og== 2023-11-08T04:52:06.136 INFO:teuthology.orchestra.run.smithi077.stdout:[client.mirror.5] 2023-11-08T04:52:06.136 INFO:teuthology.orchestra.run.smithi077.stdout: key = AQDqE0tloaCpABAAXLvk005Lu2PEQv1UxaAWcA== 2023-11-08T04:52:06.136 INFO:teuthology.orchestra.run.smithi077.stdout:[client.mirror.6] 2023-11-08T04:52:06.137 INFO:teuthology.orchestra.run.smithi077.stdout: key = AQDqE0tlcwUUBhAAMCPQYkQYjeHxT9xTiM+brQ== 2023-11-08T04:52:06.138 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-11-08T04:52:06.228 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-08T04:52:06.276 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-08T04:52:06.324 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-08T04:52:06.373 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-11-08T04:52:06.423 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-11-08T04:52:06.474 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-11-08T04:52:06.525 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-11-08T04:52:06.578 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-11-08T04:52:06.629 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-11-08T04:52:06.679 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.4 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-11-08T04:52:06.730 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.5 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-11-08T04:52:06.781 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.6 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-11-08T04:52:06.832 INFO:tasks.ceph:Running mkfs on mon nodes... 2023-11-08T04:52:06.833 DEBUG:teuthology.orchestra.run.smithi077:> sudo mkdir -p /var/lib/ceph/mon/cluster1-a 2023-11-08T04:52:06.859 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster cluster1 --mkfs -i a --monmap /home/ubuntu/cephtest/cluster1.monmap --keyring /etc/ceph/cluster1.keyring 2023-11-08T04:52:06.953 DEBUG:teuthology.orchestra.run.smithi077:> sudo chown -R ceph:ceph /var/lib/ceph/mon/cluster1-a 2023-11-08T04:52:06.979 DEBUG:teuthology.orchestra.run.smithi077:> rm -- /home/ubuntu/cephtest/cluster1.monmap 2023-11-08T04:52:07.038 INFO:tasks.ceph:Starting mon daemons in cluster cluster1... 2023-11-08T04:52:07.038 INFO:tasks.ceph.cluster1.mon.a:Restarting daemon 2023-11-08T04:52:07.039 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster cluster1 -i a 2023-11-08T04:52:07.080 INFO:tasks.ceph.cluster1.mon.a:Started 2023-11-08T04:52:07.081 INFO:tasks.ceph:Starting mgr daemons in cluster cluster1... 2023-11-08T04:52:07.081 INFO:tasks.ceph.cluster1.mgr.x:Restarting daemon 2023-11-08T04:52:07.081 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster cluster1 -i x 2023-11-08T04:52:07.083 INFO:tasks.ceph.cluster1.mgr.x:Started 2023-11-08T04:52:07.084 INFO:tasks.ceph:Setting crush tunables to default 2023-11-08T04:52:07.084 DEBUG:teuthology.orchestra.run.smithi077:> sudo ceph --cluster cluster1 osd crush tunables default 2023-11-08T04:52:07.297 INFO:teuthology.orchestra.run.smithi077.stderr:adjusted tunables profile to default 2023-11-08T04:52:07.308 INFO:tasks.ceph:Starting osd daemons in cluster cluster1... 2023-11-08T04:52:07.309 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2023-11-08T04:52:07.309 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd if=/var/lib/ceph/osd/cluster1-0/fsid of=/dev/stdout 2023-11-08T04:52:07.335 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2023-11-08T04:52:07.336 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd if=/var/lib/ceph/osd/cluster1-1/fsid of=/dev/stdout 2023-11-08T04:52:07.401 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2023-11-08T04:52:07.402 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd if=/var/lib/ceph/osd/cluster1-2/fsid of=/dev/stdout 2023-11-08T04:52:07.469 DEBUG:teuthology.orchestra.run.smithi077:> sudo ceph --cluster cluster1 osd new ce6698f1-7ce9-42d0-9385-7c2e92eb500a 0 2023-11-08T04:52:07.486 INFO:tasks.ceph.cluster1.mgr.x.smithi077.stderr:2023-11-08T04:52:07.483+0000 7f5df7955680 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-11-08T04:52:07.558 INFO:tasks.ceph.cluster1.mgr.x.smithi077.stderr:2023-11-08T04:52:07.555+0000 7f5df7955680 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-11-08T04:52:07.690 INFO:teuthology.orchestra.run.smithi077.stdout:0 2023-11-08T04:52:07.702 DEBUG:teuthology.orchestra.run.smithi077:> sudo ceph --cluster cluster1 osd new 84aa8223-3bd2-440d-ba7b-5f1675e4d25d 1 2023-11-08T04:52:07.906 INFO:teuthology.orchestra.run.smithi077.stdout:1 2023-11-08T04:52:07.918 DEBUG:teuthology.orchestra.run.smithi077:> sudo ceph --cluster cluster1 osd new 0f508d08-7d10-451b-9213-4f36d0c2e9f1 2 2023-11-08T04:52:08.006 INFO:tasks.ceph.cluster1.mgr.x.smithi077.stderr:2023-11-08T04:52:08.004+0000 7f5df7955680 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-11-08T04:52:08.122 INFO:teuthology.orchestra.run.smithi077.stdout:2 2023-11-08T04:52:08.135 INFO:tasks.ceph.cluster1.osd.0:Restarting daemon 2023-11-08T04:52:08.135 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster1 -i 0 2023-11-08T04:52:08.138 INFO:tasks.ceph.cluster1.osd.0:Started 2023-11-08T04:52:08.140 INFO:tasks.ceph.cluster1.osd.1:Restarting daemon 2023-11-08T04:52:08.140 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster1 -i 1 2023-11-08T04:52:08.142 INFO:tasks.ceph.cluster1.osd.1:Started 2023-11-08T04:52:08.142 INFO:tasks.ceph.cluster1.osd.2:Restarting daemon 2023-11-08T04:52:08.143 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster1 -i 2 2023-11-08T04:52:08.145 INFO:tasks.ceph.cluster1.osd.2:Started 2023-11-08T04:52:08.145 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd dump --format=json 2023-11-08T04:52:08.344 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-11-08T04:52:08.344 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":5,"fsid":"1244e29e-ccb6-44ee-9f2f-7bbf2125940c","created":"2023-11-08T04:52:07.196830+0000","modified":"2023-11-08T04:52:08.120187+0000","last_up_change":"0.000000","last_in_change":"2023-11-08T04:52:08.120187+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[],"osds":[{"osd":0,"uuid":"ce6698f1-7ce9-42d0-9385-7c2e92eb500a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"84aa8223-3bd2-440d-ba7b-5f1675e4d25d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"0f508d08-7d10-451b-9213-4f36d0c2e9f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-08T04:52:08.356 INFO:tasks.ceph.ceph_manager.cluster1:[] 2023-11-08T04:52:08.356 INFO:tasks.ceph:Waiting for OSDs to come up 2023-11-08T04:52:08.659 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster1 osd dump --format=json 2023-11-08T04:52:08.745 INFO:tasks.ceph.cluster1.osd.2.smithi077.stderr:2023-11-08T04:52:08.743+0000 7f991826e380 -1 Falling back to public interface 2023-11-08T04:52:08.754 INFO:tasks.ceph.cluster1.osd.1.smithi077.stderr:2023-11-08T04:52:08.747+0000 7f63f30af380 -1 Falling back to public interface 2023-11-08T04:52:08.774 INFO:tasks.ceph.cluster1.osd.0.smithi077.stderr:2023-11-08T04:52:08.772+0000 7f399eac3380 -1 Falling back to public interface 2023-11-08T04:52:08.779 INFO:tasks.ceph.cluster1.mgr.x.smithi077.stderr:2023-11-08T04:52:08.777+0000 7f5df7955680 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-11-08T04:52:08.849 INFO:tasks.ceph.cluster1.mgr.x.smithi077.stderr:2023-11-08T04:52:08.846+0000 7f5df7955680 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-11-08T04:52:09.038 INFO:tasks.ceph.cluster1.mgr.x.smithi077.stderr:2023-11-08T04:52:09.035+0000 7f5df7955680 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-11-08T04:52:09.073 INFO:teuthology.misc.health.smithi077.stdout: 2023-11-08T04:52:09.073 INFO:teuthology.misc.health.smithi077.stdout:{"epoch":5,"fsid":"1244e29e-ccb6-44ee-9f2f-7bbf2125940c","created":"2023-11-08T04:52:07.196830+0000","modified":"2023-11-08T04:52:08.120187+0000","last_up_change":"0.000000","last_in_change":"2023-11-08T04:52:08.120187+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[],"osds":[{"osd":0,"uuid":"ce6698f1-7ce9-42d0-9385-7c2e92eb500a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"84aa8223-3bd2-440d-ba7b-5f1675e4d25d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"0f508d08-7d10-451b-9213-4f36d0c2e9f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-08T04:52:09.084 DEBUG:teuthology.misc:0 of 3 OSDs are up 2023-11-08T04:52:09.478 INFO:tasks.ceph.cluster1.mgr.x.smithi077.stderr:2023-11-08T04:52:09.476+0000 7f5df7955680 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-11-08T04:52:09.580 INFO:tasks.ceph.cluster1.mgr.x.smithi077.stderr:2023-11-08T04:52:09.578+0000 7f5df7955680 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-11-08T04:52:09.732 INFO:tasks.ceph.cluster1.mgr.x.smithi077.stderr:2023-11-08T04:52:09.730+0000 7f5df7955680 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-11-08T04:52:09.808 INFO:tasks.ceph.cluster1.mgr.x.smithi077.stderr:2023-11-08T04:52:09.806+0000 7f5df7955680 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-11-08T04:52:09.879 INFO:tasks.ceph.cluster1.mgr.x.smithi077.stderr:2023-11-08T04:52:09.876+0000 7f5df7955680 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-11-08T04:52:10.135 INFO:tasks.ceph.cluster1.mgr.x.smithi077.stderr:2023-11-08T04:52:10.133+0000 7f5df7955680 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-11-08T04:52:10.328 INFO:tasks.ceph.cluster1.mgr.x.smithi077.stderr:2023-11-08T04:52:10.326+0000 7f5df7955680 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-11-08T04:52:11.001 INFO:tasks.ceph.cluster1.mgr.x.smithi077.stderr:2023-11-08T04:52:10.997+0000 7f5df7955680 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-11-08T04:52:11.068 INFO:tasks.ceph.cluster1.mgr.x.smithi077.stderr:2023-11-08T04:52:11.065+0000 7f5df7955680 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-11-08T04:52:11.146 INFO:tasks.ceph.cluster1.mgr.x.smithi077.stderr:2023-11-08T04:52:11.144+0000 7f5df7955680 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-11-08T04:52:11.296 INFO:tasks.ceph.cluster1.mgr.x.smithi077.stderr:2023-11-08T04:52:11.294+0000 7f5df7955680 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-11-08T04:52:11.367 INFO:tasks.ceph.cluster1.mgr.x.smithi077.stderr:2023-11-08T04:52:11.365+0000 7f5df7955680 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-11-08T04:52:11.489 INFO:tasks.ceph.cluster1.mgr.x.smithi077.stderr:2023-11-08T04:52:11.487+0000 7f5df7955680 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-11-08T04:52:11.607 INFO:tasks.ceph.cluster1.mgr.x.smithi077.stderr:2023-11-08T04:52:11.605+0000 7f5df7955680 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-11-08T04:52:11.929 INFO:tasks.ceph.cluster1.mgr.x.smithi077.stderr:2023-11-08T04:52:11.926+0000 7f5df7955680 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-11-08T04:52:12.002 INFO:tasks.ceph.cluster1.mgr.x.smithi077.stderr:2023-11-08T04:52:12.000+0000 7f5df7955680 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-11-08T04:52:12.235 INFO:tasks.ceph.cluster1.osd.2.smithi077.stderr:2023-11-08T04:52:12.233+0000 7f991826e380 -1 osd.2 0 log_to_monitors {default=true} 2023-11-08T04:52:12.258 INFO:tasks.ceph.cluster1.osd.1.smithi077.stderr:2023-11-08T04:52:12.255+0000 7f63f30af380 -1 osd.1 0 log_to_monitors {default=true} 2023-11-08T04:52:12.271 INFO:tasks.ceph.cluster1.osd.0.smithi077.stderr:2023-11-08T04:52:12.269+0000 7f399eac3380 -1 osd.0 0 log_to_monitors {default=true} 2023-11-08T04:52:13.021 INFO:tasks.ceph.cluster1.mgr.x.smithi077.stderr:2023-11-08T04:52:13.019+0000 7f5dace4d700 -1 mgr.server handle_report got status from non-daemon mon.a 2023-11-08T04:52:14.176 INFO:tasks.ceph.cluster1.osd.2.smithi077.stderr:2023-11-08T04:52:14.175+0000 7f990ec0f700 -1 osd.2 0 waiting for initial osdmap 2023-11-08T04:52:14.177 INFO:tasks.ceph.cluster1.osd.1.smithi077.stderr:2023-11-08T04:52:14.175+0000 7f63e9a50700 -1 osd.1 0 waiting for initial osdmap 2023-11-08T04:52:14.177 INFO:tasks.ceph.cluster1.osd.0.smithi077.stderr:2023-11-08T04:52:14.175+0000 7f3995464700 -1 osd.0 0 waiting for initial osdmap 2023-11-08T04:52:14.181 INFO:tasks.ceph.cluster1.osd.0.smithi077.stderr:2023-11-08T04:52:14.179+0000 7f3991be0700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-11-08T04:52:14.181 INFO:tasks.ceph.cluster1.osd.2.smithi077.stderr:2023-11-08T04:52:14.180+0000 7f990b38b700 -1 osd.2 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-11-08T04:52:14.181 INFO:tasks.ceph.cluster1.osd.1.smithi077.stderr:2023-11-08T04:52:14.180+0000 7f63e61cc700 -1 osd.1 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-11-08T04:52:15.388 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster1 osd dump --format=json 2023-11-08T04:52:15.676 INFO:teuthology.misc.health.smithi077.stdout: 2023-11-08T04:52:15.676 INFO:teuthology.misc.health.smithi077.stdout:{"epoch":8,"fsid":"1244e29e-ccb6-44ee-9f2f-7bbf2125940c","created":"2023-11-08T04:52:07.196830+0000","modified":"2023-11-08T04:52:15.170916+0000","last_up_change":"2023-11-08T04:52:15.170916+0000","last_in_change":"2023-11-08T04:52:08.120187+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[],"osds":[{"osd":0,"uuid":"ce6698f1-7ce9-42d0-9385-7c2e92eb500a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":115421},{"type":"v1","addr":"172.21.15.77:6817","nonce":115421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":115421},{"type":"v1","addr":"172.21.15.77:6819","nonce":115421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":115421},{"type":"v1","addr":"172.21.15.77:6823","nonce":115421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":115421},{"type":"v1","addr":"172.21.15.77:6821","nonce":115421}]},"public_addr":"172.21.15.77:6817/115421","cluster_addr":"172.21.15.77:6819/115421","heartbeat_back_addr":"172.21.15.77:6823/115421","heartbeat_front_addr":"172.21.15.77:6821/115421","state":["exists","up"]},{"osd":1,"uuid":"84aa8223-3bd2-440d-ba7b-5f1675e4d25d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":115411},{"type":"v1","addr":"172.21.15.77:6809","nonce":115411}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":115411},{"type":"v1","addr":"172.21.15.77:6811","nonce":115411}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":115411},{"type":"v1","addr":"172.21.15.77:6815","nonce":115411}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":115411},{"type":"v1","addr":"172.21.15.77:6813","nonce":115411}]},"public_addr":"172.21.15.77:6809/115411","cluster_addr":"172.21.15.77:6811/115411","heartbeat_back_addr":"172.21.15.77:6815/115411","heartbeat_front_addr":"172.21.15.77:6813/115411","state":["exists","up"]},{"osd":2,"uuid":"0f508d08-7d10-451b-9213-4f36d0c2e9f1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":115412},{"type":"v1","addr":"172.21.15.77:6801","nonce":115412}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":115412},{"type":"v1","addr":"172.21.15.77:6803","nonce":115412}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":115412},{"type":"v1","addr":"172.21.15.77:6807","nonce":115412}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":115412},{"type":"v1","addr":"172.21.15.77:6805","nonce":115412}]},"public_addr":"172.21.15.77:6801/115412","cluster_addr":"172.21.15.77:6803/115412","heartbeat_back_addr":"172.21.15.77:6807/115412","heartbeat_front_addr":"172.21.15.77:6805/115412","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-08T04:52:15.688 DEBUG:teuthology.misc:3 of 3 OSDs are up 2023-11-08T04:52:15.688 INFO:tasks.ceph:Creating RBD pool 2023-11-08T04:52:15.688 DEBUG:teuthology.orchestra.run.smithi077:> sudo ceph --cluster cluster1 osd pool create rbd 8 2023-11-08T04:52:16.184 INFO:teuthology.orchestra.run.smithi077.stderr:pool 'rbd' created 2023-11-08T04:52:16.195 DEBUG:teuthology.orchestra.run.smithi077:> sudo ceph --cluster cluster1 osd pool application enable rbd rbd --yes-i-really-mean-it 2023-11-08T04:52:17.190 INFO:teuthology.orchestra.run.smithi077.stderr:enabled application 'rbd' on pool 'rbd' 2023-11-08T04:52:17.205 INFO:tasks.ceph:Starting mds daemons in cluster cluster1... 2023-11-08T04:52:17.206 INFO:tasks.ceph:Waiting until cluster1 daemons up and pgs clean... 2023-11-08T04:52:17.206 INFO:tasks.ceph.ceph_manager.cluster1:waiting for mgr available 2023-11-08T04:52:17.207 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 mgr dump --format=json 2023-11-08T04:52:17.207 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2023-11-08T04:52:17.541 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-11-08T04:52:17.553 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":4,"active_gid":4101,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":115004},{"type":"v1","addr":"172.21.15.77:6825","nonce":115004}]},"active_addr":"172.21.15.77:6825/115004","active_change":"2023-11-08T04:52:12.014221+0000","active_mgr_features":4540138314316775423,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.23.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:8.3.5","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"docker.io/library/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"docker.io/arcts/keepalived","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.3.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.33.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"str","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"str","level":"advanced","flags":0,"default_value":"30","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"str","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"str","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"str","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"nautilus":["balancer","crash","devicehealth","orchestrator_cli","progress","rbd_support","status","volumes"],"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.77:0","nonce":556441448}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.77:0","nonce":815075879}]}]} 2023-11-08T04:52:17.555 INFO:tasks.ceph.ceph_manager.cluster1:mgr available! 2023-11-08T04:52:17.556 INFO:tasks.ceph.ceph_manager.cluster1:waiting for all up 2023-11-08T04:52:17.556 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd dump --format=json 2023-11-08T04:52:17.842 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-11-08T04:52:17.843 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":10,"fsid":"1244e29e-ccb6-44ee-9f2f-7bbf2125940c","created":"2023-11-08T04:52:07.196830+0000","modified":"2023-11-08T04:52:17.177864+0000","last_up_change":"2023-11-08T04:52:15.170916+0000","last_in_change":"2023-11-08T04:52:08.120187+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2023-11-08T04:52:15.210778+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"10","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2023-11-08T04:52:16.003312+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"10","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"ce6698f1-7ce9-42d0-9385-7c2e92eb500a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":115421},{"type":"v1","addr":"172.21.15.77:6817","nonce":115421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":115421},{"type":"v1","addr":"172.21.15.77:6819","nonce":115421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":115421},{"type":"v1","addr":"172.21.15.77:6823","nonce":115421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":115421},{"type":"v1","addr":"172.21.15.77:6821","nonce":115421}]},"public_addr":"172.21.15.77:6817/115421","cluster_addr":"172.21.15.77:6819/115421","heartbeat_back_addr":"172.21.15.77:6823/115421","heartbeat_front_addr":"172.21.15.77:6821/115421","state":["exists","up"]},{"osd":1,"uuid":"84aa8223-3bd2-440d-ba7b-5f1675e4d25d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":115411},{"type":"v1","addr":"172.21.15.77:6809","nonce":115411}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":115411},{"type":"v1","addr":"172.21.15.77:6811","nonce":115411}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":115411},{"type":"v1","addr":"172.21.15.77:6815","nonce":115411}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":115411},{"type":"v1","addr":"172.21.15.77:6813","nonce":115411}]},"public_addr":"172.21.15.77:6809/115411","cluster_addr":"172.21.15.77:6811/115411","heartbeat_back_addr":"172.21.15.77:6815/115411","heartbeat_front_addr":"172.21.15.77:6813/115411","state":["exists","up"]},{"osd":2,"uuid":"0f508d08-7d10-451b-9213-4f36d0c2e9f1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":115412},{"type":"v1","addr":"172.21.15.77:6801","nonce":115412}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":115412},{"type":"v1","addr":"172.21.15.77:6803","nonce":115412}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":115412},{"type":"v1","addr":"172.21.15.77:6807","nonce":115412}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":115412},{"type":"v1","addr":"172.21.15.77:6805","nonce":115412}]},"public_addr":"172.21.15.77:6801/115412","cluster_addr":"172.21.15.77:6803/115412","heartbeat_back_addr":"172.21.15.77:6807/115412","heartbeat_front_addr":"172.21.15.77:6805/115412","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T04:52:13.259518+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T04:52:13.235929+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T04:52:13.212169+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-08T04:52:17.856 INFO:tasks.ceph.ceph_manager.cluster1:all up! 2023-11-08T04:52:17.856 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd dump --format=json 2023-11-08T04:52:18.143 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-11-08T04:52:18.144 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":10,"fsid":"1244e29e-ccb6-44ee-9f2f-7bbf2125940c","created":"2023-11-08T04:52:07.196830+0000","modified":"2023-11-08T04:52:17.177864+0000","last_up_change":"2023-11-08T04:52:15.170916+0000","last_in_change":"2023-11-08T04:52:08.120187+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2023-11-08T04:52:15.210778+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"10","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2023-11-08T04:52:16.003312+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"10","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"ce6698f1-7ce9-42d0-9385-7c2e92eb500a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":115421},{"type":"v1","addr":"172.21.15.77:6817","nonce":115421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":115421},{"type":"v1","addr":"172.21.15.77:6819","nonce":115421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":115421},{"type":"v1","addr":"172.21.15.77:6823","nonce":115421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":115421},{"type":"v1","addr":"172.21.15.77:6821","nonce":115421}]},"public_addr":"172.21.15.77:6817/115421","cluster_addr":"172.21.15.77:6819/115421","heartbeat_back_addr":"172.21.15.77:6823/115421","heartbeat_front_addr":"172.21.15.77:6821/115421","state":["exists","up"]},{"osd":1,"uuid":"84aa8223-3bd2-440d-ba7b-5f1675e4d25d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":115411},{"type":"v1","addr":"172.21.15.77:6809","nonce":115411}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":115411},{"type":"v1","addr":"172.21.15.77:6811","nonce":115411}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":115411},{"type":"v1","addr":"172.21.15.77:6815","nonce":115411}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":115411},{"type":"v1","addr":"172.21.15.77:6813","nonce":115411}]},"public_addr":"172.21.15.77:6809/115411","cluster_addr":"172.21.15.77:6811/115411","heartbeat_back_addr":"172.21.15.77:6815/115411","heartbeat_front_addr":"172.21.15.77:6813/115411","state":["exists","up"]},{"osd":2,"uuid":"0f508d08-7d10-451b-9213-4f36d0c2e9f1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":115412},{"type":"v1","addr":"172.21.15.77:6801","nonce":115412}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":115412},{"type":"v1","addr":"172.21.15.77:6803","nonce":115412}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":115412},{"type":"v1","addr":"172.21.15.77:6807","nonce":115412}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":115412},{"type":"v1","addr":"172.21.15.77:6805","nonce":115412}]},"public_addr":"172.21.15.77:6801/115412","cluster_addr":"172.21.15.77:6803/115412","heartbeat_back_addr":"172.21.15.77:6807/115412","heartbeat_front_addr":"172.21.15.77:6805/115412","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T04:52:13.259518+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T04:52:13.235929+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T04:52:13.212169+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-08T04:52:18.156 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.0 flush_pg_stats 2023-11-08T04:52:18.305 INFO:teuthology.orchestra.run.smithi077.stdout:34359738370 2023-11-08T04:52:18.306 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.1 flush_pg_stats 2023-11-08T04:52:18.452 INFO:teuthology.orchestra.run.smithi077.stdout:34359738371 2023-11-08T04:52:18.453 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.2 flush_pg_stats 2023-11-08T04:52:18.599 INFO:teuthology.orchestra.run.smithi077.stdout:34359738370 2023-11-08T04:52:18.599 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.0 2023-11-08T04:52:18.888 INFO:teuthology.orchestra.run.smithi077.stdout:34359738369 2023-11-08T04:52:18.901 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738370 got 34359738369 for osd.0 2023-11-08T04:52:19.902 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.0 2023-11-08T04:52:20.202 INFO:teuthology.orchestra.run.smithi077.stdout:34359738370 2023-11-08T04:52:20.218 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738370 got 34359738370 for osd.0 2023-11-08T04:52:20.219 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.1 2023-11-08T04:52:20.508 INFO:teuthology.orchestra.run.smithi077.stdout:34359738371 2023-11-08T04:52:20.522 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738371 got 34359738371 for osd.1 2023-11-08T04:52:20.522 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.2 2023-11-08T04:52:20.808 INFO:teuthology.orchestra.run.smithi077.stdout:34359738370 2023-11-08T04:52:20.822 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738370 got 34359738370 for osd.2 2023-11-08T04:52:20.823 INFO:tasks.ceph.ceph_manager.cluster1:waiting for clean 2023-11-08T04:52:20.823 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2023-11-08T04:52:21.106 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-11-08T04:52:21.107 INFO:teuthology.orchestra.run.smithi077.stderr:dumped all 2023-11-08T04:52:21.118 INFO:teuthology.orchestra.run.smithi077.stdout:{"pg_ready":true,"pg_map":{"version":11,"stamp":"2023-11-08T04:52:20.018414+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":891560,"kb_used_data":272,"kb_used_omap":0,"kb_used_meta":891264,"kb_avail":282223960,"statfs":{"total":289910292480,"available":288997335040,"internally_reserved":0,"allocated":278528,"data_stored":72600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":912654336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-11-08T04:52:17.200134+0000","last_change":"2023-11-08T04:52:17.196926+0000","last_active":"2023-11-08T04:52:17.200134+0000","last_peered":"2023-11-08T04:52:17.200134+0000","last_clean":"2023-11-08T04:52:17.200134+0000","last_became_active":"2023-11-08T04:52:17.196648+0000","last_became_peered":"2023-11-08T04:52:17.196648+0000","last_unstale":"2023-11-08T04:52:17.200134+0000","last_undegraded":"2023-11-08T04:52:17.200134+0000","last_fullsized":"2023-11-08T04:52:17.200134+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:16.174854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:16.174854+0000","last_clean_scrub_stamp":"2023-11-08T04:52:16.174854+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-11-08T04:52:17.200118+0000","last_change":"2023-11-08T04:52:17.196842+0000","last_active":"2023-11-08T04:52:17.200118+0000","last_peered":"2023-11-08T04:52:17.200118+0000","last_clean":"2023-11-08T04:52:17.200118+0000","last_became_active":"2023-11-08T04:52:17.196558+0000","last_became_peered":"2023-11-08T04:52:17.196558+0000","last_unstale":"2023-11-08T04:52:17.200118+0000","last_undegraded":"2023-11-08T04:52:17.200118+0000","last_fullsized":"2023-11-08T04:52:17.200118+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:16.174854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:16.174854+0000","last_clean_scrub_stamp":"2023-11-08T04:52:16.174854+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-11-08T04:52:17.197242+0000","last_change":"2023-11-08T04:52:17.195685+0000","last_active":"2023-11-08T04:52:17.197242+0000","last_peered":"2023-11-08T04:52:17.197242+0000","last_clean":"2023-11-08T04:52:17.197242+0000","last_became_active":"2023-11-08T04:52:17.195400+0000","last_became_peered":"2023-11-08T04:52:17.195400+0000","last_unstale":"2023-11-08T04:52:17.197242+0000","last_undegraded":"2023-11-08T04:52:17.197242+0000","last_fullsized":"2023-11-08T04:52:17.197242+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:16.174854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:16.174854+0000","last_clean_scrub_stamp":"2023-11-08T04:52:16.174854+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-11-08T04:52:17.193152+0000","last_change":"2023-11-08T04:52:17.193152+0000","last_active":"2023-11-08T04:52:17.193152+0000","last_peered":"2023-11-08T04:52:17.193152+0000","last_clean":"2023-11-08T04:52:17.193152+0000","last_became_active":"2023-11-08T04:52:17.192859+0000","last_became_peered":"2023-11-08T04:52:17.192859+0000","last_unstale":"2023-11-08T04:52:17.193152+0000","last_undegraded":"2023-11-08T04:52:17.193152+0000","last_fullsized":"2023-11-08T04:52:17.193152+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:16.174854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:16.174854+0000","last_clean_scrub_stamp":"2023-11-08T04:52:16.174854+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-11-08T04:52:17.197216+0000","last_change":"2023-11-08T04:52:17.195090+0000","last_active":"2023-11-08T04:52:17.197216+0000","last_peered":"2023-11-08T04:52:17.197216+0000","last_clean":"2023-11-08T04:52:17.197216+0000","last_became_active":"2023-11-08T04:52:17.194795+0000","last_became_peered":"2023-11-08T04:52:17.194795+0000","last_unstale":"2023-11-08T04:52:17.197216+0000","last_undegraded":"2023-11-08T04:52:17.197216+0000","last_fullsized":"2023-11-08T04:52:17.197216+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:16.174854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:16.174854+0000","last_clean_scrub_stamp":"2023-11-08T04:52:16.174854+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-11-08T04:52:17.195285+0000","last_change":"2023-11-08T04:52:17.193960+0000","last_active":"2023-11-08T04:52:17.195285+0000","last_peered":"2023-11-08T04:52:17.195285+0000","last_clean":"2023-11-08T04:52:17.195285+0000","last_became_active":"2023-11-08T04:52:17.193661+0000","last_became_peered":"2023-11-08T04:52:17.193661+0000","last_unstale":"2023-11-08T04:52:17.195285+0000","last_undegraded":"2023-11-08T04:52:17.195285+0000","last_fullsized":"2023-11-08T04:52:17.195285+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:16.174854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:16.174854+0000","last_clean_scrub_stamp":"2023-11-08T04:52:16.174854+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":8,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-11-08T04:52:17.193579+0000","last_change":"2023-11-08T04:52:17.193579+0000","last_active":"2023-11-08T04:52:17.193579+0000","last_peered":"2023-11-08T04:52:17.193579+0000","last_clean":"2023-11-08T04:52:17.193579+0000","last_became_active":"2023-11-08T04:52:17.193313+0000","last_became_peered":"2023-11-08T04:52:17.193313+0000","last_unstale":"2023-11-08T04:52:17.193579+0000","last_undegraded":"2023-11-08T04:52:17.193579+0000","last_fullsized":"2023-11-08T04:52:17.193579+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:16.174854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:16.174854+0000","last_clean_scrub_stamp":"2023-11-08T04:52:16.174854+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-11-08T04:52:17.194882+0000","last_change":"2023-11-08T04:52:17.194882+0000","last_active":"2023-11-08T04:52:17.194882+0000","last_peered":"2023-11-08T04:52:17.194882+0000","last_clean":"2023-11-08T04:52:17.194882+0000","last_became_active":"2023-11-08T04:52:17.194611+0000","last_became_peered":"2023-11-08T04:52:17.194611+0000","last_unstale":"2023-11-08T04:52:17.194882+0000","last_undegraded":"2023-11-08T04:52:17.194882+0000","last_fullsized":"2023-11-08T04:52:17.194882+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:16.174854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:16.174854+0000","last_clean_scrub_stamp":"2023-11-08T04:52:16.174854+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-11-08T04:52:17.196412+0000","last_change":"2023-11-08T04:52:17.195087+0000","last_active":"2023-11-08T04:52:17.196412+0000","last_peered":"2023-11-08T04:52:17.196412+0000","last_clean":"2023-11-08T04:52:17.196412+0000","last_became_active":"2023-11-08T04:52:17.194821+0000","last_became_peered":"2023-11-08T04:52:17.194821+0000","last_unstale":"2023-11-08T04:52:17.196412+0000","last_undegraded":"2023-11-08T04:52:17.196412+0000","last_fullsized":"2023-11-08T04:52:17.196412+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:16.174854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:16.174854+0000","last_clean_scrub_stamp":"2023-11-08T04:52:16.174854+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738370,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297184,"kb_used_data":88,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94074656,"statfs":{"total":96636764160,"available":96332447744,"internally_reserved":0,"allocated":90112,"data_stored":22896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":8,"seq":34359738370,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297184,"kb_used_data":88,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94074656,"statfs":{"total":96636764160,"available":96332447744,"internally_reserved":0,"allocated":90112,"data_stored":22896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":8,"seq":34359738371,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297192,"kb_used_data":96,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94074648,"statfs":{"total":96636764160,"available":96332439552,"internally_reserved":0,"allocated":98304,"data_stored":26808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-08T04:52:21.120 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2023-11-08T04:52:21.402 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-11-08T04:52:21.404 INFO:teuthology.orchestra.run.smithi077.stderr:dumped all 2023-11-08T04:52:21.414 INFO:teuthology.orchestra.run.smithi077.stdout:{"pg_ready":true,"pg_map":{"version":11,"stamp":"2023-11-08T04:52:20.018414+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":891560,"kb_used_data":272,"kb_used_omap":0,"kb_used_meta":891264,"kb_avail":282223960,"statfs":{"total":289910292480,"available":288997335040,"internally_reserved":0,"allocated":278528,"data_stored":72600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":912654336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-11-08T04:52:17.200134+0000","last_change":"2023-11-08T04:52:17.196926+0000","last_active":"2023-11-08T04:52:17.200134+0000","last_peered":"2023-11-08T04:52:17.200134+0000","last_clean":"2023-11-08T04:52:17.200134+0000","last_became_active":"2023-11-08T04:52:17.196648+0000","last_became_peered":"2023-11-08T04:52:17.196648+0000","last_unstale":"2023-11-08T04:52:17.200134+0000","last_undegraded":"2023-11-08T04:52:17.200134+0000","last_fullsized":"2023-11-08T04:52:17.200134+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:16.174854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:16.174854+0000","last_clean_scrub_stamp":"2023-11-08T04:52:16.174854+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-11-08T04:52:17.200118+0000","last_change":"2023-11-08T04:52:17.196842+0000","last_active":"2023-11-08T04:52:17.200118+0000","last_peered":"2023-11-08T04:52:17.200118+0000","last_clean":"2023-11-08T04:52:17.200118+0000","last_became_active":"2023-11-08T04:52:17.196558+0000","last_became_peered":"2023-11-08T04:52:17.196558+0000","last_unstale":"2023-11-08T04:52:17.200118+0000","last_undegraded":"2023-11-08T04:52:17.200118+0000","last_fullsized":"2023-11-08T04:52:17.200118+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:16.174854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:16.174854+0000","last_clean_scrub_stamp":"2023-11-08T04:52:16.174854+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-11-08T04:52:17.197242+0000","last_change":"2023-11-08T04:52:17.195685+0000","last_active":"2023-11-08T04:52:17.197242+0000","last_peered":"2023-11-08T04:52:17.197242+0000","last_clean":"2023-11-08T04:52:17.197242+0000","last_became_active":"2023-11-08T04:52:17.195400+0000","last_became_peered":"2023-11-08T04:52:17.195400+0000","last_unstale":"2023-11-08T04:52:17.197242+0000","last_undegraded":"2023-11-08T04:52:17.197242+0000","last_fullsized":"2023-11-08T04:52:17.197242+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:16.174854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:16.174854+0000","last_clean_scrub_stamp":"2023-11-08T04:52:16.174854+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-11-08T04:52:17.193152+0000","last_change":"2023-11-08T04:52:17.193152+0000","last_active":"2023-11-08T04:52:17.193152+0000","last_peered":"2023-11-08T04:52:17.193152+0000","last_clean":"2023-11-08T04:52:17.193152+0000","last_became_active":"2023-11-08T04:52:17.192859+0000","last_became_peered":"2023-11-08T04:52:17.192859+0000","last_unstale":"2023-11-08T04:52:17.193152+0000","last_undegraded":"2023-11-08T04:52:17.193152+0000","last_fullsized":"2023-11-08T04:52:17.193152+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:16.174854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:16.174854+0000","last_clean_scrub_stamp":"2023-11-08T04:52:16.174854+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-11-08T04:52:17.197216+0000","last_change":"2023-11-08T04:52:17.195090+0000","last_active":"2023-11-08T04:52:17.197216+0000","last_peered":"2023-11-08T04:52:17.197216+0000","last_clean":"2023-11-08T04:52:17.197216+0000","last_became_active":"2023-11-08T04:52:17.194795+0000","last_became_peered":"2023-11-08T04:52:17.194795+0000","last_unstale":"2023-11-08T04:52:17.197216+0000","last_undegraded":"2023-11-08T04:52:17.197216+0000","last_fullsized":"2023-11-08T04:52:17.197216+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:16.174854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:16.174854+0000","last_clean_scrub_stamp":"2023-11-08T04:52:16.174854+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-11-08T04:52:17.195285+0000","last_change":"2023-11-08T04:52:17.193960+0000","last_active":"2023-11-08T04:52:17.195285+0000","last_peered":"2023-11-08T04:52:17.195285+0000","last_clean":"2023-11-08T04:52:17.195285+0000","last_became_active":"2023-11-08T04:52:17.193661+0000","last_became_peered":"2023-11-08T04:52:17.193661+0000","last_unstale":"2023-11-08T04:52:17.195285+0000","last_undegraded":"2023-11-08T04:52:17.195285+0000","last_fullsized":"2023-11-08T04:52:17.195285+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:16.174854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:16.174854+0000","last_clean_scrub_stamp":"2023-11-08T04:52:16.174854+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":8,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-11-08T04:52:17.193579+0000","last_change":"2023-11-08T04:52:17.193579+0000","last_active":"2023-11-08T04:52:17.193579+0000","last_peered":"2023-11-08T04:52:17.193579+0000","last_clean":"2023-11-08T04:52:17.193579+0000","last_became_active":"2023-11-08T04:52:17.193313+0000","last_became_peered":"2023-11-08T04:52:17.193313+0000","last_unstale":"2023-11-08T04:52:17.193579+0000","last_undegraded":"2023-11-08T04:52:17.193579+0000","last_fullsized":"2023-11-08T04:52:17.193579+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:16.174854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:16.174854+0000","last_clean_scrub_stamp":"2023-11-08T04:52:16.174854+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-11-08T04:52:17.194882+0000","last_change":"2023-11-08T04:52:17.194882+0000","last_active":"2023-11-08T04:52:17.194882+0000","last_peered":"2023-11-08T04:52:17.194882+0000","last_clean":"2023-11-08T04:52:17.194882+0000","last_became_active":"2023-11-08T04:52:17.194611+0000","last_became_peered":"2023-11-08T04:52:17.194611+0000","last_unstale":"2023-11-08T04:52:17.194882+0000","last_undegraded":"2023-11-08T04:52:17.194882+0000","last_fullsized":"2023-11-08T04:52:17.194882+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:16.174854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:16.174854+0000","last_clean_scrub_stamp":"2023-11-08T04:52:16.174854+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":10,"state":"active+clean","last_fresh":"2023-11-08T04:52:17.196412+0000","last_change":"2023-11-08T04:52:17.195087+0000","last_active":"2023-11-08T04:52:17.196412+0000","last_peered":"2023-11-08T04:52:17.196412+0000","last_clean":"2023-11-08T04:52:17.196412+0000","last_became_active":"2023-11-08T04:52:17.194821+0000","last_became_peered":"2023-11-08T04:52:17.194821+0000","last_unstale":"2023-11-08T04:52:17.196412+0000","last_undegraded":"2023-11-08T04:52:17.196412+0000","last_fullsized":"2023-11-08T04:52:17.196412+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:16.174854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:16.174854+0000","last_clean_scrub_stamp":"2023-11-08T04:52:16.174854+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738370,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297184,"kb_used_data":88,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94074656,"statfs":{"total":96636764160,"available":96332447744,"internally_reserved":0,"allocated":90112,"data_stored":22896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":8,"seq":34359738370,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297184,"kb_used_data":88,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94074656,"statfs":{"total":96636764160,"available":96332447744,"internally_reserved":0,"allocated":90112,"data_stored":22896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":8,"seq":34359738371,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297192,"kb_used_data":96,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94074648,"statfs":{"total":96636764160,"available":96332439552,"internally_reserved":0,"allocated":98304,"data_stored":26808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-08T04:52:21.415 INFO:tasks.ceph.ceph_manager.cluster1:clean! 2023-11-08T04:52:21.415 INFO:tasks.ceph:Waiting until ceph cluster cluster1 is healthy... 2023-11-08T04:52:21.416 INFO:tasks.ceph.ceph_manager.cluster1:wait_until_healthy 2023-11-08T04:52:21.416 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 health --format=json 2023-11-08T04:52:21.736 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-11-08T04:52:21.749 INFO:teuthology.orchestra.run.smithi077.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2023-11-08T04:52:21.750 INFO:tasks.ceph.ceph_manager.cluster1:wait_until_healthy done 2023-11-08T04:52:21.751 INFO:teuthology.run_tasks:Running task ceph... 2023-11-08T04:52:21.765 INFO:tasks.ceph:Creating ceph cluster cluster2... 2023-11-08T04:52:21.765 INFO:tasks.ceph:config {'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'simple'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd memory target': 1610612736, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'cpu_profile': set(), 'cluster': 'cluster2', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2023-11-08T04:52:21.765 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/yuriw-2023-11-07_19:42:25-rbd-wip-yuri3-testing-2023-11-07-0801-pacific-distro-default-smithi/7451368', 'branch': 'wip-yuri3-testing-2023-11-07-0801-pacific', 'description': 'rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-low-osd-mem-target policy/simple rbd-mirror/four-per-cluster supported-random-distro$/{centos_8} workloads/rbd-mirror-snapshot-stress-workunit-exclusive-lock}', 'email': 'yweinste@redhat.com', 'first_in_suite': False, 'job_id': '7451368', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'meta': [{'desc': 'run two ceph clusters and install rbd-mirror'}, {'desc': 'configure the permissions for client.mirror'}, {'desc': '2 ceph clusters with 1 mon and 3 osds each'}, {'desc': 'run four rbd-mirror daemons per cluster'}, {'desc': 'run the rbd_mirror_stress.sh workunit to test the rbd-mirror daemon'}], 'name': 'yuriw-2023-11-07_19:42:25-rbd-wip-yuri3-testing-2023-11-07-0801-pacific-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 3, 'size': 30}}], 'os_type': 'centos', 'os_version': '8.stream', 'overrides': {'admin_socket': {'branch': 'wip-yuri3-testing-2023-11-07-0801-pacific'}, 'ceph': {'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'simple'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd memory target': 1610612736, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': 'cfb6fd52113be6a0af3fe7fb89b2e08474eadc0f'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': 'cfb6fd52113be6a0af3fe7fb89b2e08474eadc0f'}}, 'selinux': {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-yuri3-testing-2023-11-07-0801-pacific', 'sha1': 'cfb6fd52113be6a0af3fe7fb89b2e08474eadc0f'}}, 'owner': 'scheduled_yuriw@teuthology', 'priority': 75, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['cluster1.mon.a', 'cluster1.mgr.x', 'cluster2.mgr.x', 'cluster1.osd.0', 'cluster1.osd.1', 'cluster1.osd.2', 'cluster1.client.0', 'cluster2.client.0'], ['cluster2.mon.a', 'cluster2.osd.0', 'cluster2.osd.1', 'cluster2.osd.2', 'cluster1.client.mirror', 'cluster1.client.mirror.0', 'cluster1.client.mirror.1', 'cluster1.client.mirror.2', 'cluster1.client.mirror.3', 'cluster1.client.mirror.4', 'cluster1.client.mirror.5', 'cluster1.client.mirror.6', 'cluster2.client.mirror', 'cluster2.client.mirror.0', 'cluster2.client.mirror.1', 'cluster2.client.mirror.2', 'cluster2.client.mirror.3', 'cluster2.client.mirror.4', 'cluster2.client.mirror.5', 'cluster2.client.mirror.6']], 'seed': 9776, 'sha1': 'cfb6fd52113be6a0af3fe7fb89b2e08474eadc0f', 'sleep_before_teardown': 0, 'subset': '111/120000', 'suite': 'rbd', 'suite_branch': 'wip-yuri3-testing-2023-11-07-0801-pacific', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_cfb6fd52113be6a0af3fe7fb89b2e08474eadc0f/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': 'cfb6fd52113be6a0af3fe7fb89b2e08474eadc0f', 'targets': {'smithi077.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGSQ/WLcJdpJ0ClJVyBsFrcYIQ2QRbAwp2zUvz0ifkJpShDNVGje+V/Jdm6AZOafWAIrxu2EWTj6axjmMfZQozM=', 'smithi087.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBD+XKU8wa6pBVq4mhdEwRNPaK1Sl9uWyjisTURFaLby1G+ogJ5JiOLEZ7VuzQt8PXKmmvgJJqHhmd6n+97zE5BE='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': ['rbd-mirror'], 'flavor': 'default', 'sha1': 'cfb6fd52113be6a0af3fe7fb89b2e08474eadc0f'}}, {'ceph': {'cluster': 'cluster1', 'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'simple'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd memory target': 1610612736, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': 'cfb6fd52113be6a0af3fe7fb89b2e08474eadc0f'}}, {'ceph': {'cluster': 'cluster2', 'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'simple'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd memory target': 1610612736, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': 'cfb6fd52113be6a0af3fe7fb89b2e08474eadc0f'}}, {'exec': {'cluster1.client.mirror.0': ["sudo ceph --cluster cluster1 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd'"]}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.0', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.1', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.2', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.3', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.0', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.1', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.2', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.3', 'thrash': True}}, {'rbd-mirror-thrash': {'cluster': 'cluster1'}}, {'rbd-mirror-thrash': {'cluster': 'cluster2'}}, {'workunit': {'clients': {'cluster1.client.mirror': ['rbd/rbd_mirror_stress.sh']}, 'env': {'CEPH_ARGS': '', 'MIRROR_IMAGE_MODE': 'snapshot', 'MIRROR_POOL_MODE': 'image', 'RBD_IMAGE_FEATURES': 'layering,exclusive-lock', 'RBD_MIRROR_INSTANCES': '4', 'RBD_MIRROR_USE_EXISTING_CLUSTER': '1', 'RBD_MIRROR_USE_RBD_MIRROR': '1'}, 'timeout': '6h'}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '6899cd26fceddb2fec83dc1a1349394b28c8998e', 'timestamp': '2023-11-07_19:42:25', 'tube': 'smithi', 'user': 'yuriw', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.3590714'} 2023-11-08T04:52:21.767 DEBUG:teuthology.orchestra.run.smithi077:> install -d -m0755 -- /home/ubuntu/cephtest/cluster2.data 2023-11-08T04:52:21.769 DEBUG:teuthology.orchestra.run.smithi087:> install -d -m0755 -- /home/ubuntu/cephtest/cluster2.data 2023-11-08T04:52:21.791 DEBUG:teuthology.orchestra.run.smithi077:> sudo install -d -m0777 -- /var/run/ceph 2023-11-08T04:52:21.827 DEBUG:teuthology.orchestra.run.smithi087:> sudo install -d -m0777 -- /var/run/ceph 2023-11-08T04:52:21.863 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2023-11-08T04:52:21.864 DEBUG:teuthology.orchestra.run.smithi087:> dd if=/scratch_devs of=/dev/stdout 2023-11-08T04:52:21.923 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-11-08T04:52:21.924 DEBUG:teuthology.orchestra.run.smithi087:> stat /dev/vg_nvme/lv_1 2023-11-08T04:52:21.983 INFO:teuthology.orchestra.run.smithi087.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-11-08T04:52:21.983 INFO:teuthology.orchestra.run.smithi087.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-11-08T04:52:21.984 INFO:teuthology.orchestra.run.smithi087.stdout:Device: 6h/6d Inode: 389799 Links: 1 2023-11-08T04:52:21.984 INFO:teuthology.orchestra.run.smithi087.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-08T04:52:21.984 INFO:teuthology.orchestra.run.smithi087.stdout:Context: system_u:object_r:device_t:s0 2023-11-08T04:52:21.984 INFO:teuthology.orchestra.run.smithi087.stdout:Access: 2023-11-08 04:51:35.807709959 +0000 2023-11-08T04:52:21.984 INFO:teuthology.orchestra.run.smithi087.stdout:Modify: 2023-11-08 04:47:32.684456984 +0000 2023-11-08T04:52:21.984 INFO:teuthology.orchestra.run.smithi087.stdout:Change: 2023-11-08 04:47:32.684456984 +0000 2023-11-08T04:52:21.985 INFO:teuthology.orchestra.run.smithi087.stdout: Birth: - 2023-11-08T04:52:21.985 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-11-08T04:52:22.051 INFO:teuthology.orchestra.run.smithi087.stderr:1+0 records in 2023-11-08T04:52:22.053 INFO:teuthology.orchestra.run.smithi087.stderr:1+0 records out 2023-11-08T04:52:22.053 INFO:teuthology.orchestra.run.smithi087.stderr:512 bytes copied, 0.00025831 s, 2.0 MB/s 2023-11-08T04:52:22.054 DEBUG:teuthology.orchestra.run.smithi087:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-11-08T04:52:22.114 DEBUG:teuthology.orchestra.run.smithi087:> stat /dev/vg_nvme/lv_2 2023-11-08T04:52:22.173 INFO:teuthology.orchestra.run.smithi087.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-11-08T04:52:22.173 INFO:teuthology.orchestra.run.smithi087.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-11-08T04:52:22.173 INFO:teuthology.orchestra.run.smithi087.stdout:Device: 6h/6d Inode: 387849 Links: 1 2023-11-08T04:52:22.174 INFO:teuthology.orchestra.run.smithi087.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-08T04:52:22.174 INFO:teuthology.orchestra.run.smithi087.stdout:Context: system_u:object_r:device_t:s0 2023-11-08T04:52:22.174 INFO:teuthology.orchestra.run.smithi087.stdout:Access: 2023-11-08 04:51:35.807709959 +0000 2023-11-08T04:52:22.174 INFO:teuthology.orchestra.run.smithi087.stdout:Modify: 2023-11-08 04:47:33.195445577 +0000 2023-11-08T04:52:22.174 INFO:teuthology.orchestra.run.smithi087.stdout:Change: 2023-11-08 04:47:33.195445577 +0000 2023-11-08T04:52:22.175 INFO:teuthology.orchestra.run.smithi087.stdout: Birth: - 2023-11-08T04:52:22.175 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-11-08T04:52:22.239 INFO:teuthology.orchestra.run.smithi087.stderr:1+0 records in 2023-11-08T04:52:22.239 INFO:teuthology.orchestra.run.smithi087.stderr:1+0 records out 2023-11-08T04:52:22.240 INFO:teuthology.orchestra.run.smithi087.stderr:512 bytes copied, 0.000198355 s, 2.6 MB/s 2023-11-08T04:52:22.241 DEBUG:teuthology.orchestra.run.smithi087:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-11-08T04:52:22.301 DEBUG:teuthology.orchestra.run.smithi087:> stat /dev/vg_nvme/lv_3 2023-11-08T04:52:22.360 INFO:teuthology.orchestra.run.smithi087.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-11-08T04:52:22.360 INFO:teuthology.orchestra.run.smithi087.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-11-08T04:52:22.360 INFO:teuthology.orchestra.run.smithi087.stdout:Device: 6h/6d Inode: 389891 Links: 1 2023-11-08T04:52:22.360 INFO:teuthology.orchestra.run.smithi087.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-08T04:52:22.361 INFO:teuthology.orchestra.run.smithi087.stdout:Context: system_u:object_r:device_t:s0 2023-11-08T04:52:22.361 INFO:teuthology.orchestra.run.smithi087.stdout:Access: 2023-11-08 04:51:35.807709959 +0000 2023-11-08T04:52:22.361 INFO:teuthology.orchestra.run.smithi087.stdout:Modify: 2023-11-08 04:47:33.706434170 +0000 2023-11-08T04:52:22.361 INFO:teuthology.orchestra.run.smithi087.stdout:Change: 2023-11-08 04:47:33.706434170 +0000 2023-11-08T04:52:22.361 INFO:teuthology.orchestra.run.smithi087.stdout: Birth: - 2023-11-08T04:52:22.362 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-11-08T04:52:22.425 INFO:teuthology.orchestra.run.smithi087.stderr:1+0 records in 2023-11-08T04:52:22.425 INFO:teuthology.orchestra.run.smithi087.stderr:1+0 records out 2023-11-08T04:52:22.425 INFO:teuthology.orchestra.run.smithi087.stderr:512 bytes copied, 0.00021613 s, 2.4 MB/s 2023-11-08T04:52:22.426 DEBUG:teuthology.orchestra.run.smithi087:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-11-08T04:52:22.485 DEBUG:teuthology.orchestra.run.smithi087:> stat /dev/vg_nvme/lv_4 2023-11-08T04:52:22.543 INFO:teuthology.orchestra.run.smithi087.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-11-08T04:52:22.543 INFO:teuthology.orchestra.run.smithi087.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-11-08T04:52:22.544 INFO:teuthology.orchestra.run.smithi087.stdout:Device: 6h/6d Inode: 389947 Links: 1 2023-11-08T04:52:22.544 INFO:teuthology.orchestra.run.smithi087.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-08T04:52:22.544 INFO:teuthology.orchestra.run.smithi087.stdout:Context: system_u:object_r:device_t:s0 2023-11-08T04:52:22.544 INFO:teuthology.orchestra.run.smithi087.stdout:Access: 2023-11-08 04:51:35.807709959 +0000 2023-11-08T04:52:22.544 INFO:teuthology.orchestra.run.smithi087.stdout:Modify: 2023-11-08 04:47:34.212422875 +0000 2023-11-08T04:52:22.545 INFO:teuthology.orchestra.run.smithi087.stdout:Change: 2023-11-08 04:47:34.212422875 +0000 2023-11-08T04:52:22.545 INFO:teuthology.orchestra.run.smithi087.stdout: Birth: - 2023-11-08T04:52:22.545 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-11-08T04:52:22.609 INFO:teuthology.orchestra.run.smithi087.stderr:1+0 records in 2023-11-08T04:52:22.609 INFO:teuthology.orchestra.run.smithi087.stderr:1+0 records out 2023-11-08T04:52:22.609 INFO:teuthology.orchestra.run.smithi087.stderr:512 bytes copied, 0.000270101 s, 1.9 MB/s 2023-11-08T04:52:22.611 DEBUG:teuthology.orchestra.run.smithi087:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-11-08T04:52:22.670 INFO:tasks.ceph:osd dev map: {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'} 2023-11-08T04:52:22.671 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi087.front.sepia.ceph.com'): {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'}} 2023-11-08T04:52:22.671 INFO:tasks.ceph:Generating config... 2023-11-08T04:52:22.673 INFO:tasks.ceph:[client] debug rbd = 20 2023-11-08T04:52:22.674 INFO:tasks.ceph:[client] debug rbd_mirror = 15 2023-11-08T04:52:22.674 INFO:tasks.ceph:[client] log to stderr = False 2023-11-08T04:52:22.675 INFO:tasks.ceph:[client] rbd default features = 125 2023-11-08T04:52:22.675 INFO:tasks.ceph:[client] rbd mirror image policy type = simple 2023-11-08T04:52:22.676 INFO:tasks.ceph:[client.mirror.0] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-11-08T04:52:22.676 INFO:tasks.ceph:[client.mirror.0] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-11-08T04:52:22.677 INFO:tasks.ceph:[client.mirror.1] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-11-08T04:52:22.677 INFO:tasks.ceph:[client.mirror.1] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-11-08T04:52:22.678 INFO:tasks.ceph:[client.mirror.2] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-11-08T04:52:22.678 INFO:tasks.ceph:[client.mirror.2] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-11-08T04:52:22.678 INFO:tasks.ceph:[client.mirror.3] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-11-08T04:52:22.679 INFO:tasks.ceph:[client.mirror.3] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-11-08T04:52:22.679 INFO:tasks.ceph:[global] mon client directed command retry = 5 2023-11-08T04:52:22.680 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2023-11-08T04:52:22.680 INFO:tasks.ceph:[mgr] debug mgr = 20 2023-11-08T04:52:22.680 INFO:tasks.ceph:[mgr] debug ms = 1 2023-11-08T04:52:22.681 INFO:tasks.ceph:[mon] debug mon = 20 2023-11-08T04:52:22.681 INFO:tasks.ceph:[mon] debug ms = 1 2023-11-08T04:52:22.682 INFO:tasks.ceph:[mon] debug paxos = 20 2023-11-08T04:52:22.682 INFO:tasks.ceph:[osd] bdev async discard = True 2023-11-08T04:52:22.682 INFO:tasks.ceph:[osd] bdev enable discard = True 2023-11-08T04:52:22.683 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2023-11-08T04:52:22.683 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2023-11-08T04:52:22.683 INFO:tasks.ceph:[osd] debug bluefs = 1/20 2023-11-08T04:52:22.683 INFO:tasks.ceph:[osd] debug bluestore = 1/20 2023-11-08T04:52:22.684 INFO:tasks.ceph:[osd] debug ms = 1 2023-11-08T04:52:22.684 INFO:tasks.ceph:[osd] debug osd = 20 2023-11-08T04:52:22.684 INFO:tasks.ceph:[osd] debug rocksdb = 4/10 2023-11-08T04:52:22.685 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2023-11-08T04:52:22.685 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2023-11-08T04:52:22.685 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2023-11-08T04:52:22.686 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2023-11-08T04:52:22.686 INFO:tasks.ceph:[osd] osd memory target = 1610612736 2023-11-08T04:52:22.686 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2023-11-08T04:52:22.686 INFO:tasks.ceph:Setting up cluster2.mon.a... 2023-11-08T04:52:22.687 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/cluster2.keyring 2023-11-08T04:52:22.748 INFO:teuthology.orchestra.run.smithi087.stdout:creating /etc/ceph/cluster2.keyring 2023-11-08T04:52:22.750 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/cluster2.keyring 2023-11-08T04:52:22.830 DEBUG:teuthology.orchestra.run.smithi087:> sudo chmod 0644 /etc/ceph/cluster2.keyring 2023-11-08T04:52:22.899 DEBUG:tasks.ceph:Ceph mon addresses: [('cluster2.mon.a', '172.21.15.87')] 2023-11-08T04:52:22.900 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.87', 'mon client directed command retry': 5, 'ms inject socket failures': 5000}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd memory target': 1610612736, 'osd objectstore': 'bluestore'}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false'}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'simple'}, 'mon.a': {}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}} 2023-11-08T04:52:22.901 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2023-11-08T04:52:22.901 DEBUG:teuthology.orchestra.run.smithi087:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2023-11-08T04:52:22.957 DEBUG:teuthology.orchestra.run.smithi087:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.87 --print /home/ubuntu/cephtest/cluster2.monmap 2023-11-08T04:52:23.036 INFO:teuthology.orchestra.run.smithi087.stdout:monmaptool: monmap file /home/ubuntu/cephtest/cluster2.monmap 2023-11-08T04:52:23.036 INFO:teuthology.orchestra.run.smithi087.stdout:monmaptool: generated fsid ab4cbc06-449d-497d-b513-220d9db2b590 2023-11-08T04:52:23.036 INFO:teuthology.orchestra.run.smithi087.stdout:epoch 0 2023-11-08T04:52:23.037 INFO:teuthology.orchestra.run.smithi087.stdout:fsid ab4cbc06-449d-497d-b513-220d9db2b590 2023-11-08T04:52:23.037 INFO:teuthology.orchestra.run.smithi087.stdout:last_changed 2023-11-08T04:52:23.034448+0000 2023-11-08T04:52:23.038 INFO:teuthology.orchestra.run.smithi087.stdout:created 2023-11-08T04:52:23.034448+0000 2023-11-08T04:52:23.038 INFO:teuthology.orchestra.run.smithi087.stdout:min_mon_release 0 (unknown) 2023-11-08T04:52:23.038 INFO:teuthology.orchestra.run.smithi087.stdout:election_strategy: 1 2023-11-08T04:52:23.039 INFO:teuthology.orchestra.run.smithi087.stdout:0: [v2:172.21.15.87:3300/0,v1:172.21.15.87:6789/0] mon.a 2023-11-08T04:52:23.039 INFO:teuthology.orchestra.run.smithi087.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/cluster2.monmap (1 monitors) 2023-11-08T04:52:23.041 DEBUG:teuthology.orchestra.run.smithi087:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2023-11-08T04:52:23.098 INFO:tasks.ceph:Writing /etc/ceph/cluster2.conf for FSID ab4cbc06-449d-497d-b513-220d9db2b590... 2023-11-08T04:52:23.100 DEBUG:teuthology.orchestra.run.smithi077:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/cluster2.conf && sudo chmod 0644 /etc/ceph/cluster2.conf > /dev/null 2023-11-08T04:52:23.103 DEBUG:teuthology.orchestra.run.smithi087:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/cluster2.conf && sudo chmod 0644 /etc/ceph/cluster2.conf > /dev/null 2023-11-08T04:52:23.153 INFO:teuthology.orchestra.run.smithi077.stdout:[global] 2023-11-08T04:52:23.154 INFO:teuthology.orchestra.run.smithi077.stdout: chdir = "" 2023-11-08T04:52:23.154 INFO:teuthology.orchestra.run.smithi077.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-11-08T04:52:23.154 INFO:teuthology.orchestra.run.smithi077.stdout: auth supported = cephx 2023-11-08T04:52:23.154 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-11-08T04:52:23.155 INFO:teuthology.orchestra.run.smithi077.stdout: filestore xattr use omap = true 2023-11-08T04:52:23.155 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-11-08T04:52:23.155 INFO:teuthology.orchestra.run.smithi077.stdout: mon clock drift allowed = 1.000 2023-11-08T04:52:23.155 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-11-08T04:52:23.155 INFO:teuthology.orchestra.run.smithi077.stdout: osd crush chooseleaf type = 0 2023-11-08T04:52:23.156 INFO:teuthology.orchestra.run.smithi077.stdout: auth debug = true 2023-11-08T04:52:23.156 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-11-08T04:52:23.156 INFO:teuthology.orchestra.run.smithi077.stdout: ms die on old message = true 2023-11-08T04:52:23.156 INFO:teuthology.orchestra.run.smithi077.stdout: ms die on bug = true 2023-11-08T04:52:23.156 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-11-08T04:52:23.156 INFO:teuthology.orchestra.run.smithi077.stdout: mon max pg per osd = 10000 # >= luminous 2023-11-08T04:52:23.157 INFO:teuthology.orchestra.run.smithi077.stdout: mon pg warn max object skew = 0 2023-11-08T04:52:23.157 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-11-08T04:52:23.157 INFO:teuthology.orchestra.run.smithi077.stdout: # disable pg_autoscaler by default for new pools 2023-11-08T04:52:23.157 INFO:teuthology.orchestra.run.smithi077.stdout: osd_pool_default_pg_autoscale_mode = off 2023-11-08T04:52:23.157 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-11-08T04:52:23.158 INFO:teuthology.orchestra.run.smithi077.stdout: osd pool default size = 2 2023-11-08T04:52:23.158 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-11-08T04:52:23.158 INFO:teuthology.orchestra.run.smithi077.stdout: mon osd allow primary affinity = true 2023-11-08T04:52:23.158 INFO:teuthology.orchestra.run.smithi077.stdout: mon osd allow pg remap = true 2023-11-08T04:52:23.158 INFO:teuthology.orchestra.run.smithi077.stdout: mon warn on legacy crush tunables = false 2023-11-08T04:52:23.159 INFO:teuthology.orchestra.run.smithi077.stdout: mon warn on crush straw calc version zero = false 2023-11-08T04:52:23.159 INFO:teuthology.orchestra.run.smithi077.stdout: mon warn on no sortbitwise = false 2023-11-08T04:52:23.159 INFO:teuthology.orchestra.run.smithi077.stdout: mon warn on osd down out interval zero = false 2023-11-08T04:52:23.159 INFO:teuthology.orchestra.run.smithi077.stdout: mon warn on too few osds = false 2023-11-08T04:52:23.159 INFO:teuthology.orchestra.run.smithi077.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-11-08T04:52:23.160 INFO:teuthology.orchestra.run.smithi077.stdout: mon_warn_on_pool_no_redundancy = false 2023-11-08T04:52:23.160 INFO:teuthology.orchestra.run.smithi077.stdout: mon_allow_pool_size_one = true 2023-11-08T04:52:23.160 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-11-08T04:52:23.160 INFO:teuthology.orchestra.run.smithi077.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2023-11-08T04:52:23.160 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-11-08T04:52:23.161 INFO:teuthology.orchestra.run.smithi077.stdout: osd default data pool replay window = 5 2023-11-08T04:52:23.161 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-11-08T04:52:23.161 INFO:teuthology.orchestra.run.smithi077.stdout: mon allow pool delete = true 2023-11-08T04:52:23.161 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-11-08T04:52:23.161 INFO:teuthology.orchestra.run.smithi077.stdout: mon cluster log file level = debug 2023-11-08T04:52:23.161 INFO:teuthology.orchestra.run.smithi077.stdout: debug asserts on shutdown = true 2023-11-08T04:52:23.162 INFO:teuthology.orchestra.run.smithi077.stdout: mon health detail to clog = false 2023-11-08T04:52:23.162 INFO:teuthology.orchestra.run.smithi077.stdout: mon host = 172.21.15.87 2023-11-08T04:52:23.162 INFO:teuthology.orchestra.run.smithi077.stdout: mon client directed command retry = 5 2023-11-08T04:52:23.162 INFO:teuthology.orchestra.run.smithi077.stdout: ms inject socket failures = 5000 2023-11-08T04:52:23.162 INFO:teuthology.orchestra.run.smithi077.stdout: fsid = ab4cbc06-449d-497d-b513-220d9db2b590 2023-11-08T04:52:23.163 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-11-08T04:52:23.163 INFO:teuthology.orchestra.run.smithi077.stdout:[osd] 2023-11-08T04:52:23.163 INFO:teuthology.orchestra.run.smithi077.stdout: osd journal size = 100 2023-11-08T04:52:23.163 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-11-08T04:52:23.163 INFO:teuthology.orchestra.run.smithi077.stdout: osd scrub load threshold = 5.0 2023-11-08T04:52:23.164 INFO:teuthology.orchestra.run.smithi077.stdout: osd scrub max interval = 600 2023-11-08T04:52:23.164 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-11-08T04:52:23.164 INFO:teuthology.orchestra.run.smithi077.stdout: osd recover clone overlap = true 2023-11-08T04:52:23.164 INFO:teuthology.orchestra.run.smithi077.stdout: osd recovery max chunk = 1048576 2023-11-08T04:52:23.164 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-11-08T04:52:23.165 INFO:teuthology.orchestra.run.smithi077.stdout: osd debug shutdown = true 2023-11-08T04:52:23.165 INFO:teuthology.orchestra.run.smithi077.stdout: osd debug op order = true 2023-11-08T04:52:23.165 INFO:teuthology.orchestra.run.smithi077.stdout: osd debug verify stray on activate = true 2023-11-08T04:52:23.165 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-11-08T04:52:23.165 INFO:teuthology.orchestra.run.smithi077.stdout: osd open classes on start = true 2023-11-08T04:52:23.165 INFO:teuthology.orchestra.run.smithi077.stdout: osd debug pg log writeout = true 2023-11-08T04:52:23.166 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-11-08T04:52:23.166 INFO:teuthology.orchestra.run.smithi077.stdout: osd deep scrub update digest min age = 30 2023-11-08T04:52:23.166 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-11-08T04:52:23.166 INFO:teuthology.orchestra.run.smithi077.stdout: osd map max advance = 10 2023-11-08T04:52:23.167 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-11-08T04:52:23.167 INFO:teuthology.orchestra.run.smithi077.stdout: journal zero on create = true 2023-11-08T04:52:23.167 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-11-08T04:52:23.167 INFO:teuthology.orchestra.run.smithi077.stdout: filestore ondisk finisher threads = 3 2023-11-08T04:52:23.167 INFO:teuthology.orchestra.run.smithi077.stdout: filestore apply finisher threads = 3 2023-11-08T04:52:23.168 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-11-08T04:52:23.168 INFO:teuthology.orchestra.run.smithi077.stdout: bdev debug aio = true 2023-11-08T04:52:23.168 INFO:teuthology.orchestra.run.smithi077.stdout: osd debug misdirected ops = true 2023-11-08T04:52:23.168 INFO:teuthology.orchestra.run.smithi077.stdout: bdev async discard = True 2023-11-08T04:52:23.168 INFO:teuthology.orchestra.run.smithi077.stdout: bdev enable discard = True 2023-11-08T04:52:23.169 INFO:teuthology.orchestra.run.smithi077.stdout: bluestore block size = 96636764160 2023-11-08T04:52:23.169 INFO:teuthology.orchestra.run.smithi077.stdout: bluestore fsck on mount = True 2023-11-08T04:52:23.169 INFO:teuthology.orchestra.run.smithi077.stdout: debug bluefs = 1/20 2023-11-08T04:52:23.169 INFO:teuthology.orchestra.run.smithi077.stdout: debug bluestore = 1/20 2023-11-08T04:52:23.169 INFO:teuthology.orchestra.run.smithi077.stdout: debug ms = 1 2023-11-08T04:52:23.170 INFO:teuthology.orchestra.run.smithi077.stdout: debug osd = 20 2023-11-08T04:52:23.170 INFO:teuthology.orchestra.run.smithi077.stdout: debug rocksdb = 4/10 2023-11-08T04:52:23.170 INFO:teuthology.orchestra.run.smithi077.stdout: mon osd backfillfull_ratio = 0.85 2023-11-08T04:52:23.170 INFO:teuthology.orchestra.run.smithi077.stdout: mon osd full ratio = 0.9 2023-11-08T04:52:23.171 INFO:teuthology.orchestra.run.smithi077.stdout: mon osd nearfull ratio = 0.8 2023-11-08T04:52:23.171 INFO:teuthology.orchestra.run.smithi077.stdout: osd failsafe full ratio = 0.95 2023-11-08T04:52:23.171 INFO:teuthology.orchestra.run.smithi077.stdout: osd memory target = 1610612736 2023-11-08T04:52:23.171 INFO:teuthology.orchestra.run.smithi077.stdout: osd objectstore = bluestore 2023-11-08T04:52:23.171 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-11-08T04:52:23.171 INFO:teuthology.orchestra.run.smithi077.stdout:[mgr] 2023-11-08T04:52:23.172 INFO:teuthology.orchestra.run.smithi077.stdout: debug ms = 1 2023-11-08T04:52:23.172 INFO:teuthology.orchestra.run.smithi077.stdout: debug mgr = 20 2023-11-08T04:52:23.172 INFO:teuthology.orchestra.run.smithi077.stdout: debug mon = 20 2023-11-08T04:52:23.172 INFO:teuthology.orchestra.run.smithi077.stdout: debug auth = 20 2023-11-08T04:52:23.172 INFO:teuthology.orchestra.run.smithi077.stdout: mon reweight min pgs per osd = 4 2023-11-08T04:52:23.173 INFO:teuthology.orchestra.run.smithi077.stdout: mon reweight min bytes per osd = 10 2023-11-08T04:52:23.173 INFO:teuthology.orchestra.run.smithi077.stdout: mgr/telemetry/nag = false 2023-11-08T04:52:23.173 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-11-08T04:52:23.173 INFO:teuthology.orchestra.run.smithi077.stdout:[mon] 2023-11-08T04:52:23.173 INFO:teuthology.orchestra.run.smithi077.stdout: debug ms = 1 2023-11-08T04:52:23.174 INFO:teuthology.orchestra.run.smithi077.stdout: debug mon = 20 2023-11-08T04:52:23.174 INFO:teuthology.orchestra.run.smithi077.stdout: debug paxos = 20 2023-11-08T04:52:23.174 INFO:teuthology.orchestra.run.smithi077.stdout: debug auth = 20 2023-11-08T04:52:23.174 INFO:teuthology.orchestra.run.smithi077.stdout: mon data avail warn = 5 2023-11-08T04:52:23.174 INFO:teuthology.orchestra.run.smithi077.stdout: mon mgr mkfs grace = 240 2023-11-08T04:52:23.174 INFO:teuthology.orchestra.run.smithi077.stdout: mon reweight min pgs per osd = 4 2023-11-08T04:52:23.175 INFO:teuthology.orchestra.run.smithi077.stdout: mon osd reporter subtree level = osd 2023-11-08T04:52:23.175 INFO:teuthology.orchestra.run.smithi077.stdout: mon osd prime pg temp = true 2023-11-08T04:52:23.175 INFO:teuthology.orchestra.run.smithi077.stdout: mon reweight min bytes per osd = 10 2023-11-08T04:52:23.175 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-11-08T04:52:23.175 INFO:teuthology.orchestra.run.smithi077.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-11-08T04:52:23.176 INFO:teuthology.orchestra.run.smithi077.stdout: auth mon ticket ttl = 660 # 11m 2023-11-08T04:52:23.176 INFO:teuthology.orchestra.run.smithi077.stdout: auth service ticket ttl = 240 # 4m 2023-11-08T04:52:23.176 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-11-08T04:52:23.176 INFO:teuthology.orchestra.run.smithi077.stdout: # don't complain about insecure global_id in the test suite 2023-11-08T04:52:23.176 INFO:teuthology.orchestra.run.smithi077.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-11-08T04:52:23.177 INFO:teuthology.orchestra.run.smithi077.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-11-08T04:52:23.177 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-11-08T04:52:23.177 INFO:teuthology.orchestra.run.smithi077.stdout:[client] 2023-11-08T04:52:23.177 INFO:teuthology.orchestra.run.smithi077.stdout: rgw cache enabled = true 2023-11-08T04:52:23.177 INFO:teuthology.orchestra.run.smithi077.stdout: rgw enable ops log = true 2023-11-08T04:52:23.178 INFO:teuthology.orchestra.run.smithi077.stdout: rgw enable usage log = true 2023-11-08T04:52:23.178 INFO:teuthology.orchestra.run.smithi077.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-11-08T04:52:23.178 INFO:teuthology.orchestra.run.smithi077.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-11-08T04:52:23.178 INFO:teuthology.orchestra.run.smithi077.stdout: debug rbd = 20 2023-11-08T04:52:23.178 INFO:teuthology.orchestra.run.smithi077.stdout: debug rbd_mirror = 15 2023-11-08T04:52:23.179 INFO:teuthology.orchestra.run.smithi077.stdout: log to stderr = False 2023-11-08T04:52:23.179 INFO:teuthology.orchestra.run.smithi077.stdout: rbd default features = 125 2023-11-08T04:52:23.179 INFO:teuthology.orchestra.run.smithi077.stdout: rbd mirror image policy type = simple 2023-11-08T04:52:23.179 INFO:teuthology.orchestra.run.smithi077.stdout:[mon.a] 2023-11-08T04:52:23.179 INFO:teuthology.orchestra.run.smithi077.stdout:[client.mirror.0] 2023-11-08T04:52:23.180 INFO:teuthology.orchestra.run.smithi077.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-11-08T04:52:23.180 INFO:teuthology.orchestra.run.smithi077.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-11-08T04:52:23.180 INFO:teuthology.orchestra.run.smithi077.stdout:[client.mirror.1] 2023-11-08T04:52:23.180 INFO:teuthology.orchestra.run.smithi077.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-11-08T04:52:23.180 INFO:teuthology.orchestra.run.smithi077.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-11-08T04:52:23.181 INFO:teuthology.orchestra.run.smithi077.stdout:[client.mirror.2] 2023-11-08T04:52:23.181 INFO:teuthology.orchestra.run.smithi077.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-11-08T04:52:23.181 INFO:teuthology.orchestra.run.smithi077.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-11-08T04:52:23.181 INFO:teuthology.orchestra.run.smithi077.stdout:[client.mirror.3] 2023-11-08T04:52:23.181 INFO:teuthology.orchestra.run.smithi077.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-11-08T04:52:23.182 INFO:teuthology.orchestra.run.smithi077.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-11-08T04:52:23.182 INFO:teuthology.orchestra.run.smithi087.stdout:[global] 2023-11-08T04:52:23.183 INFO:teuthology.orchestra.run.smithi087.stdout: chdir = "" 2023-11-08T04:52:23.183 INFO:teuthology.orchestra.run.smithi087.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-11-08T04:52:23.183 INFO:teuthology.orchestra.run.smithi087.stdout: auth supported = cephx 2023-11-08T04:52:23.183 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T04:52:23.183 INFO:teuthology.orchestra.run.smithi087.stdout: filestore xattr use omap = true 2023-11-08T04:52:23.183 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T04:52:23.184 INFO:teuthology.orchestra.run.smithi087.stdout: mon clock drift allowed = 1.000 2023-11-08T04:52:23.184 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T04:52:23.184 INFO:teuthology.orchestra.run.smithi087.stdout: osd crush chooseleaf type = 0 2023-11-08T04:52:23.184 INFO:teuthology.orchestra.run.smithi087.stdout: auth debug = true 2023-11-08T04:52:23.184 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T04:52:23.185 INFO:teuthology.orchestra.run.smithi087.stdout: ms die on old message = true 2023-11-08T04:52:23.185 INFO:teuthology.orchestra.run.smithi087.stdout: ms die on bug = true 2023-11-08T04:52:23.185 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T04:52:23.185 INFO:teuthology.orchestra.run.smithi087.stdout: mon max pg per osd = 10000 # >= luminous 2023-11-08T04:52:23.186 INFO:teuthology.orchestra.run.smithi087.stdout: mon pg warn max object skew = 0 2023-11-08T04:52:23.186 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T04:52:23.186 INFO:teuthology.orchestra.run.smithi087.stdout: # disable pg_autoscaler by default for new pools 2023-11-08T04:52:23.186 INFO:teuthology.orchestra.run.smithi087.stdout: osd_pool_default_pg_autoscale_mode = off 2023-11-08T04:52:23.186 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T04:52:23.187 INFO:teuthology.orchestra.run.smithi087.stdout: osd pool default size = 2 2023-11-08T04:52:23.187 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T04:52:23.187 INFO:teuthology.orchestra.run.smithi087.stdout: mon osd allow primary affinity = true 2023-11-08T04:52:23.187 INFO:teuthology.orchestra.run.smithi087.stdout: mon osd allow pg remap = true 2023-11-08T04:52:23.187 INFO:teuthology.orchestra.run.smithi087.stdout: mon warn on legacy crush tunables = false 2023-11-08T04:52:23.188 INFO:teuthology.orchestra.run.smithi087.stdout: mon warn on crush straw calc version zero = false 2023-11-08T04:52:23.188 INFO:teuthology.orchestra.run.smithi087.stdout: mon warn on no sortbitwise = false 2023-11-08T04:52:23.188 INFO:teuthology.orchestra.run.smithi087.stdout: mon warn on osd down out interval zero = false 2023-11-08T04:52:23.188 INFO:teuthology.orchestra.run.smithi087.stdout: mon warn on too few osds = false 2023-11-08T04:52:23.188 INFO:teuthology.orchestra.run.smithi087.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-11-08T04:52:23.189 INFO:teuthology.orchestra.run.smithi087.stdout: mon_warn_on_pool_no_redundancy = false 2023-11-08T04:52:23.189 INFO:teuthology.orchestra.run.smithi087.stdout: mon_allow_pool_size_one = true 2023-11-08T04:52:23.189 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T04:52:23.189 INFO:teuthology.orchestra.run.smithi087.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2023-11-08T04:52:23.189 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T04:52:23.189 INFO:teuthology.orchestra.run.smithi087.stdout: osd default data pool replay window = 5 2023-11-08T04:52:23.190 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T04:52:23.190 INFO:teuthology.orchestra.run.smithi087.stdout: mon allow pool delete = true 2023-11-08T04:52:23.190 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T04:52:23.190 INFO:teuthology.orchestra.run.smithi087.stdout: mon cluster log file level = debug 2023-11-08T04:52:23.191 INFO:teuthology.orchestra.run.smithi087.stdout: debug asserts on shutdown = true 2023-11-08T04:52:23.191 INFO:teuthology.orchestra.run.smithi087.stdout: mon health detail to clog = false 2023-11-08T04:52:23.191 INFO:teuthology.orchestra.run.smithi087.stdout: mon host = 172.21.15.87 2023-11-08T04:52:23.191 INFO:teuthology.orchestra.run.smithi087.stdout: mon client directed command retry = 5 2023-11-08T04:52:23.191 INFO:teuthology.orchestra.run.smithi087.stdout: ms inject socket failures = 5000 2023-11-08T04:52:23.191 INFO:teuthology.orchestra.run.smithi087.stdout: fsid = ab4cbc06-449d-497d-b513-220d9db2b590 2023-11-08T04:52:23.192 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T04:52:23.192 INFO:teuthology.orchestra.run.smithi087.stdout:[osd] 2023-11-08T04:52:23.192 INFO:teuthology.orchestra.run.smithi087.stdout: osd journal size = 100 2023-11-08T04:52:23.192 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T04:52:23.192 INFO:teuthology.orchestra.run.smithi087.stdout: osd scrub load threshold = 5.0 2023-11-08T04:52:23.193 INFO:teuthology.orchestra.run.smithi087.stdout: osd scrub max interval = 600 2023-11-08T04:52:23.193 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T04:52:23.193 INFO:teuthology.orchestra.run.smithi087.stdout: osd recover clone overlap = true 2023-11-08T04:52:23.193 INFO:teuthology.orchestra.run.smithi087.stdout: osd recovery max chunk = 1048576 2023-11-08T04:52:23.193 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T04:52:23.194 INFO:teuthology.orchestra.run.smithi087.stdout: osd debug shutdown = true 2023-11-08T04:52:23.194 INFO:teuthology.orchestra.run.smithi087.stdout: osd debug op order = true 2023-11-08T04:52:23.194 INFO:teuthology.orchestra.run.smithi087.stdout: osd debug verify stray on activate = true 2023-11-08T04:52:23.194 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T04:52:23.194 INFO:teuthology.orchestra.run.smithi087.stdout: osd open classes on start = true 2023-11-08T04:52:23.195 INFO:teuthology.orchestra.run.smithi087.stdout: osd debug pg log writeout = true 2023-11-08T04:52:23.195 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T04:52:23.195 INFO:teuthology.orchestra.run.smithi087.stdout: osd deep scrub update digest min age = 30 2023-11-08T04:52:23.195 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T04:52:23.195 INFO:teuthology.orchestra.run.smithi087.stdout: osd map max advance = 10 2023-11-08T04:52:23.196 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T04:52:23.196 INFO:teuthology.orchestra.run.smithi087.stdout: journal zero on create = true 2023-11-08T04:52:23.196 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T04:52:23.196 INFO:teuthology.orchestra.run.smithi087.stdout: filestore ondisk finisher threads = 3 2023-11-08T04:52:23.196 INFO:teuthology.orchestra.run.smithi087.stdout: filestore apply finisher threads = 3 2023-11-08T04:52:23.196 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T04:52:23.197 INFO:teuthology.orchestra.run.smithi087.stdout: bdev debug aio = true 2023-11-08T04:52:23.197 INFO:teuthology.orchestra.run.smithi087.stdout: osd debug misdirected ops = true 2023-11-08T04:52:23.197 INFO:teuthology.orchestra.run.smithi087.stdout: bdev async discard = True 2023-11-08T04:52:23.197 INFO:teuthology.orchestra.run.smithi087.stdout: bdev enable discard = True 2023-11-08T04:52:23.197 INFO:teuthology.orchestra.run.smithi087.stdout: bluestore block size = 96636764160 2023-11-08T04:52:23.198 INFO:teuthology.orchestra.run.smithi087.stdout: bluestore fsck on mount = True 2023-11-08T04:52:23.198 INFO:teuthology.orchestra.run.smithi087.stdout: debug bluefs = 1/20 2023-11-08T04:52:23.198 INFO:teuthology.orchestra.run.smithi087.stdout: debug bluestore = 1/20 2023-11-08T04:52:23.198 INFO:teuthology.orchestra.run.smithi087.stdout: debug ms = 1 2023-11-08T04:52:23.198 INFO:teuthology.orchestra.run.smithi087.stdout: debug osd = 20 2023-11-08T04:52:23.199 INFO:teuthology.orchestra.run.smithi087.stdout: debug rocksdb = 4/10 2023-11-08T04:52:23.199 INFO:teuthology.orchestra.run.smithi087.stdout: mon osd backfillfull_ratio = 0.85 2023-11-08T04:52:23.199 INFO:teuthology.orchestra.run.smithi087.stdout: mon osd full ratio = 0.9 2023-11-08T04:52:23.199 INFO:teuthology.orchestra.run.smithi087.stdout: mon osd nearfull ratio = 0.8 2023-11-08T04:52:23.199 INFO:teuthology.orchestra.run.smithi087.stdout: osd failsafe full ratio = 0.95 2023-11-08T04:52:23.199 INFO:teuthology.orchestra.run.smithi087.stdout: osd memory target = 1610612736 2023-11-08T04:52:23.200 INFO:teuthology.orchestra.run.smithi087.stdout: osd objectstore = bluestore 2023-11-08T04:52:23.200 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T04:52:23.200 INFO:teuthology.orchestra.run.smithi087.stdout:[mgr] 2023-11-08T04:52:23.200 INFO:teuthology.orchestra.run.smithi087.stdout: debug ms = 1 2023-11-08T04:52:23.200 INFO:teuthology.orchestra.run.smithi087.stdout: debug mgr = 20 2023-11-08T04:52:23.201 INFO:teuthology.orchestra.run.smithi087.stdout: debug mon = 20 2023-11-08T04:52:23.201 INFO:teuthology.orchestra.run.smithi087.stdout: debug auth = 20 2023-11-08T04:52:23.201 INFO:teuthology.orchestra.run.smithi087.stdout: mon reweight min pgs per osd = 4 2023-11-08T04:52:23.201 INFO:teuthology.orchestra.run.smithi087.stdout: mon reweight min bytes per osd = 10 2023-11-08T04:52:23.201 INFO:teuthology.orchestra.run.smithi087.stdout: mgr/telemetry/nag = false 2023-11-08T04:52:23.201 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T04:52:23.202 INFO:teuthology.orchestra.run.smithi087.stdout:[mon] 2023-11-08T04:52:23.202 INFO:teuthology.orchestra.run.smithi087.stdout: debug ms = 1 2023-11-08T04:52:23.202 INFO:teuthology.orchestra.run.smithi087.stdout: debug mon = 20 2023-11-08T04:52:23.202 INFO:teuthology.orchestra.run.smithi087.stdout: debug paxos = 20 2023-11-08T04:52:23.202 INFO:teuthology.orchestra.run.smithi087.stdout: debug auth = 20 2023-11-08T04:52:23.203 INFO:teuthology.orchestra.run.smithi087.stdout: mon data avail warn = 5 2023-11-08T04:52:23.203 INFO:teuthology.orchestra.run.smithi087.stdout: mon mgr mkfs grace = 240 2023-11-08T04:52:23.203 INFO:teuthology.orchestra.run.smithi087.stdout: mon reweight min pgs per osd = 4 2023-11-08T04:52:23.203 INFO:teuthology.orchestra.run.smithi087.stdout: mon osd reporter subtree level = osd 2023-11-08T04:52:23.203 INFO:teuthology.orchestra.run.smithi087.stdout: mon osd prime pg temp = true 2023-11-08T04:52:23.204 INFO:teuthology.orchestra.run.smithi087.stdout: mon reweight min bytes per osd = 10 2023-11-08T04:52:23.204 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T04:52:23.204 INFO:teuthology.orchestra.run.smithi087.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-11-08T04:52:23.204 INFO:teuthology.orchestra.run.smithi087.stdout: auth mon ticket ttl = 660 # 11m 2023-11-08T04:52:23.204 INFO:teuthology.orchestra.run.smithi087.stdout: auth service ticket ttl = 240 # 4m 2023-11-08T04:52:23.205 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T04:52:23.205 INFO:teuthology.orchestra.run.smithi087.stdout: # don't complain about insecure global_id in the test suite 2023-11-08T04:52:23.205 INFO:teuthology.orchestra.run.smithi087.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-11-08T04:52:23.205 INFO:teuthology.orchestra.run.smithi087.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-11-08T04:52:23.205 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T04:52:23.206 INFO:teuthology.orchestra.run.smithi087.stdout:[client] 2023-11-08T04:52:23.206 INFO:teuthology.orchestra.run.smithi087.stdout: rgw cache enabled = true 2023-11-08T04:52:23.206 INFO:teuthology.orchestra.run.smithi087.stdout: rgw enable ops log = true 2023-11-08T04:52:23.206 INFO:teuthology.orchestra.run.smithi087.stdout: rgw enable usage log = true 2023-11-08T04:52:23.206 INFO:teuthology.orchestra.run.smithi087.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-11-08T04:52:23.207 INFO:teuthology.orchestra.run.smithi087.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-11-08T04:52:23.207 INFO:teuthology.orchestra.run.smithi087.stdout: debug rbd = 20 2023-11-08T04:52:23.207 INFO:teuthology.orchestra.run.smithi087.stdout: debug rbd_mirror = 15 2023-11-08T04:52:23.207 INFO:teuthology.orchestra.run.smithi087.stdout: log to stderr = False 2023-11-08T04:52:23.207 INFO:teuthology.orchestra.run.smithi087.stdout: rbd default features = 125 2023-11-08T04:52:23.207 INFO:teuthology.orchestra.run.smithi087.stdout: rbd mirror image policy type = simple 2023-11-08T04:52:23.208 INFO:teuthology.orchestra.run.smithi087.stdout:[mon.a] 2023-11-08T04:52:23.208 INFO:teuthology.orchestra.run.smithi087.stdout:[client.mirror.0] 2023-11-08T04:52:23.208 INFO:teuthology.orchestra.run.smithi087.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-11-08T04:52:23.208 INFO:teuthology.orchestra.run.smithi087.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-11-08T04:52:23.208 INFO:teuthology.orchestra.run.smithi087.stdout:[client.mirror.1] 2023-11-08T04:52:23.209 INFO:teuthology.orchestra.run.smithi087.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-11-08T04:52:23.209 INFO:teuthology.orchestra.run.smithi087.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-11-08T04:52:23.209 INFO:teuthology.orchestra.run.smithi087.stdout:[client.mirror.2] 2023-11-08T04:52:23.209 INFO:teuthology.orchestra.run.smithi087.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-11-08T04:52:23.209 INFO:teuthology.orchestra.run.smithi087.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-11-08T04:52:23.210 INFO:teuthology.orchestra.run.smithi087.stdout:[client.mirror.3] 2023-11-08T04:52:23.210 INFO:teuthology.orchestra.run.smithi087.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-11-08T04:52:23.210 INFO:teuthology.orchestra.run.smithi087.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-11-08T04:52:23.211 INFO:tasks.ceph:Creating admin key on cluster2.mon.a... 2023-11-08T04:52:23.212 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/cluster2.keyring 2023-11-08T04:52:23.303 INFO:tasks.ceph:Copying monmap to all nodes... 2023-11-08T04:52:23.304 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2023-11-08T04:52:23.304 DEBUG:teuthology.orchestra.run.smithi087:> dd if=/etc/ceph/cluster2.keyring of=/dev/stdout 2023-11-08T04:52:23.321 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2023-11-08T04:52:23.322 DEBUG:teuthology.orchestra.run.smithi087:> dd if=/home/ubuntu/cephtest/cluster2.monmap of=/dev/stdout 2023-11-08T04:52:23.379 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi077.front.sepia.ceph.com 2023-11-08T04:52:23.379 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2023-11-08T04:52:23.380 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd of=/etc/ceph/cluster2.keyring 2023-11-08T04:52:23.380 DEBUG:teuthology.orchestra.run.smithi077:> sudo chmod 0644 /etc/ceph/cluster2.keyring 2023-11-08T04:52:23.421 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2023-11-08T04:52:23.422 DEBUG:teuthology.orchestra.run.smithi077:> dd of=/home/ubuntu/cephtest/cluster2.monmap 2023-11-08T04:52:23.480 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi087.front.sepia.ceph.com 2023-11-08T04:52:23.481 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2023-11-08T04:52:23.481 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd of=/etc/ceph/cluster2.keyring 2023-11-08T04:52:23.481 DEBUG:teuthology.orchestra.run.smithi087:> sudo chmod 0644 /etc/ceph/cluster2.keyring 2023-11-08T04:52:23.517 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2023-11-08T04:52:23.517 DEBUG:teuthology.orchestra.run.smithi087:> dd of=/home/ubuntu/cephtest/cluster2.monmap 2023-11-08T04:52:23.575 INFO:tasks.ceph:Setting up mon nodes... 2023-11-08T04:52:23.575 INFO:tasks.ceph:Setting up mgr nodes... 2023-11-08T04:52:23.576 DEBUG:teuthology.orchestra.run.smithi077:> sudo mkdir -p /var/lib/ceph/mgr/cluster2-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/cluster2-x/keyring 2023-11-08T04:52:23.628 INFO:teuthology.orchestra.run.smithi077.stdout:creating /var/lib/ceph/mgr/cluster2-x/keyring 2023-11-08T04:52:23.631 INFO:tasks.ceph:Setting up mds nodes... 2023-11-08T04:52:23.631 INFO:tasks.ceph_client:Setting up client nodes... 2023-11-08T04:52:23.631 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/cluster2.client.0.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.0.keyring 2023-11-08T04:52:23.671 INFO:teuthology.orchestra.run.smithi077.stdout:creating /etc/ceph/cluster2.client.0.keyring 2023-11-08T04:52:23.683 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror /etc/ceph/cluster2.client.mirror.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.keyring 2023-11-08T04:52:23.723 INFO:teuthology.orchestra.run.smithi087.stdout:creating /etc/ceph/cluster2.client.mirror.keyring 2023-11-08T04:52:23.735 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.0 /etc/ceph/cluster2.client.mirror.0.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.0.keyring 2023-11-08T04:52:23.815 INFO:teuthology.orchestra.run.smithi087.stdout:creating /etc/ceph/cluster2.client.mirror.0.keyring 2023-11-08T04:52:23.825 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.1 /etc/ceph/cluster2.client.mirror.1.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.1.keyring 2023-11-08T04:52:23.903 INFO:teuthology.orchestra.run.smithi087.stdout:creating /etc/ceph/cluster2.client.mirror.1.keyring 2023-11-08T04:52:23.915 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.2 /etc/ceph/cluster2.client.mirror.2.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.2.keyring 2023-11-08T04:52:23.995 INFO:teuthology.orchestra.run.smithi087.stdout:creating /etc/ceph/cluster2.client.mirror.2.keyring 2023-11-08T04:52:24.007 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.3 /etc/ceph/cluster2.client.mirror.3.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.3.keyring 2023-11-08T04:52:24.087 INFO:teuthology.orchestra.run.smithi087.stdout:creating /etc/ceph/cluster2.client.mirror.3.keyring 2023-11-08T04:52:24.098 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.4 /etc/ceph/cluster2.client.mirror.4.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.4.keyring 2023-11-08T04:52:24.177 INFO:teuthology.orchestra.run.smithi087.stdout:creating /etc/ceph/cluster2.client.mirror.4.keyring 2023-11-08T04:52:24.189 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.5 /etc/ceph/cluster2.client.mirror.5.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.5.keyring 2023-11-08T04:52:24.268 INFO:teuthology.orchestra.run.smithi087.stdout:creating /etc/ceph/cluster2.client.mirror.5.keyring 2023-11-08T04:52:24.279 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.6 /etc/ceph/cluster2.client.mirror.6.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.6.keyring 2023-11-08T04:52:24.359 INFO:teuthology.orchestra.run.smithi087.stdout:creating /etc/ceph/cluster2.client.mirror.6.keyring 2023-11-08T04:52:24.370 INFO:tasks.ceph:Running mkfs on osd nodes... 2023-11-08T04:52:24.371 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi077.front.sepia.ceph.com'): {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'}, Remote(name='ubuntu@smithi087.front.sepia.ceph.com'): {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'}} 2023-11-08T04:52:24.371 DEBUG:teuthology.orchestra.run.smithi087:> sudo mkdir -p /var/lib/ceph/osd/cluster2-0 2023-11-08T04:52:24.438 INFO:tasks.ceph:roles_to_devs: {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'} 2023-11-08T04:52:24.438 INFO:tasks.ceph:role: cluster2.osd.0 2023-11-08T04:52:24.438 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi087.front.sepia.ceph.com 2023-11-08T04:52:24.439 DEBUG:teuthology.orchestra.run.smithi087:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-11-08T04:52:24.512 INFO:teuthology.orchestra.run.smithi087.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-11-08T04:52:24.512 INFO:teuthology.orchestra.run.smithi087.stdout: = sectsz=512 attr=2, projid32bit=1 2023-11-08T04:52:24.512 INFO:teuthology.orchestra.run.smithi087.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-11-08T04:52:24.512 INFO:teuthology.orchestra.run.smithi087.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-11-08T04:52:24.512 INFO:teuthology.orchestra.run.smithi087.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-11-08T04:52:24.513 INFO:teuthology.orchestra.run.smithi087.stdout: = sunit=0 swidth=0 blks 2023-11-08T04:52:24.513 INFO:teuthology.orchestra.run.smithi087.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-11-08T04:52:24.513 INFO:teuthology.orchestra.run.smithi087.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-11-08T04:52:24.513 INFO:teuthology.orchestra.run.smithi087.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-11-08T04:52:24.513 INFO:teuthology.orchestra.run.smithi087.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-11-08T04:52:24.952 INFO:teuthology.orchestra.run.smithi087.stdout:Discarding blocks...Done. 2023-11-08T04:52:24.954 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi087.front.sepia.ceph.com -o noatime 2023-11-08T04:52:24.954 DEBUG:teuthology.orchestra.run.smithi087:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/cluster2-0 2023-11-08T04:52:25.121 DEBUG:teuthology.orchestra.run.smithi087:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster2-0 2023-11-08T04:52:25.156 DEBUG:teuthology.orchestra.run.smithi087:> sudo mkdir -p /var/lib/ceph/osd/cluster2-1 2023-11-08T04:52:25.224 INFO:tasks.ceph:roles_to_devs: {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'} 2023-11-08T04:52:25.224 INFO:tasks.ceph:role: cluster2.osd.1 2023-11-08T04:52:25.225 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi087.front.sepia.ceph.com 2023-11-08T04:52:25.225 DEBUG:teuthology.orchestra.run.smithi087:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-11-08T04:52:25.299 INFO:teuthology.orchestra.run.smithi087.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-11-08T04:52:25.300 INFO:teuthology.orchestra.run.smithi087.stdout: = sectsz=512 attr=2, projid32bit=1 2023-11-08T04:52:25.300 INFO:teuthology.orchestra.run.smithi087.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-11-08T04:52:25.300 INFO:teuthology.orchestra.run.smithi087.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-11-08T04:52:25.300 INFO:teuthology.orchestra.run.smithi087.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-11-08T04:52:25.301 INFO:teuthology.orchestra.run.smithi087.stdout: = sunit=0 swidth=0 blks 2023-11-08T04:52:25.301 INFO:teuthology.orchestra.run.smithi087.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-11-08T04:52:25.301 INFO:teuthology.orchestra.run.smithi087.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-11-08T04:52:25.301 INFO:teuthology.orchestra.run.smithi087.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-11-08T04:52:25.301 INFO:teuthology.orchestra.run.smithi087.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-11-08T04:52:25.721 INFO:teuthology.orchestra.run.smithi087.stdout:Discarding blocks...Done. 2023-11-08T04:52:25.723 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi087.front.sepia.ceph.com -o noatime 2023-11-08T04:52:25.723 DEBUG:teuthology.orchestra.run.smithi087:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/cluster2-1 2023-11-08T04:52:25.833 DEBUG:teuthology.orchestra.run.smithi087:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster2-1 2023-11-08T04:52:25.865 DEBUG:teuthology.orchestra.run.smithi087:> sudo mkdir -p /var/lib/ceph/osd/cluster2-2 2023-11-08T04:52:25.931 INFO:tasks.ceph:roles_to_devs: {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'} 2023-11-08T04:52:25.932 INFO:tasks.ceph:role: cluster2.osd.2 2023-11-08T04:52:25.932 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi087.front.sepia.ceph.com 2023-11-08T04:52:25.932 DEBUG:teuthology.orchestra.run.smithi087:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-11-08T04:52:26.006 INFO:teuthology.orchestra.run.smithi087.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-11-08T04:52:26.006 INFO:teuthology.orchestra.run.smithi087.stdout: = sectsz=512 attr=2, projid32bit=1 2023-11-08T04:52:26.007 INFO:teuthology.orchestra.run.smithi087.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-11-08T04:52:26.007 INFO:teuthology.orchestra.run.smithi087.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-11-08T04:52:26.007 INFO:teuthology.orchestra.run.smithi087.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-11-08T04:52:26.007 INFO:teuthology.orchestra.run.smithi087.stdout: = sunit=0 swidth=0 blks 2023-11-08T04:52:26.007 INFO:teuthology.orchestra.run.smithi087.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-11-08T04:52:26.008 INFO:teuthology.orchestra.run.smithi087.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-11-08T04:52:26.008 INFO:teuthology.orchestra.run.smithi087.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-11-08T04:52:26.008 INFO:teuthology.orchestra.run.smithi087.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-11-08T04:52:26.464 INFO:teuthology.orchestra.run.smithi087.stdout:Discarding blocks...Done. 2023-11-08T04:52:26.466 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi087.front.sepia.ceph.com -o noatime 2023-11-08T04:52:26.466 DEBUG:teuthology.orchestra.run.smithi087:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/cluster2-2 2023-11-08T04:52:26.633 DEBUG:teuthology.orchestra.run.smithi087:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster2-2 2023-11-08T04:52:26.666 DEBUG:teuthology.orchestra.run.smithi087:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster2 --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/cluster2.monmap 2023-11-08T04:52:26.765 INFO:teuthology.orchestra.run.smithi087.stderr:2023-11-08T04:52:26.763+0000 7fb92c794380 -1 auth: error reading file: /var/lib/ceph/osd/cluster2-0/keyring: can't open /var/lib/ceph/osd/cluster2-0/keyring: (2) No such file or directory 2023-11-08T04:52:26.766 INFO:teuthology.orchestra.run.smithi087.stderr:2023-11-08T04:52:26.764+0000 7fb92c794380 -1 created new key in keyring /var/lib/ceph/osd/cluster2-0/keyring 2023-11-08T04:52:26.767 INFO:teuthology.orchestra.run.smithi087.stderr:2023-11-08T04:52:26.764+0000 7fb92c794380 -1 bluestore(/var/lib/ceph/osd/cluster2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-0/block: (2) No such file or directory 2023-11-08T04:52:26.767 INFO:teuthology.orchestra.run.smithi087.stderr:2023-11-08T04:52:26.764+0000 7fb92c794380 -1 bluestore(/var/lib/ceph/osd/cluster2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-0/block: (2) No such file or directory 2023-11-08T04:52:26.767 INFO:teuthology.orchestra.run.smithi087.stderr:2023-11-08T04:52:26.764+0000 7fb92c794380 -1 bluestore(/var/lib/ceph/osd/cluster2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-0/block: (2) No such file or directory 2023-11-08T04:52:26.768 INFO:teuthology.orchestra.run.smithi087.stderr:2023-11-08T04:52:26.766+0000 7fb92c794380 -1 bluestore(/var/lib/ceph/osd/cluster2-0) _read_fsid unparsable uuid 2023-11-08T04:52:29.614 DEBUG:teuthology.orchestra.run.smithi087:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster2-0 2023-11-08T04:52:29.646 DEBUG:teuthology.orchestra.run.smithi087:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster2 --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/cluster2.monmap 2023-11-08T04:52:29.734 INFO:teuthology.orchestra.run.smithi087.stderr:2023-11-08T04:52:29.731+0000 7f7895c7a380 -1 auth: error reading file: /var/lib/ceph/osd/cluster2-1/keyring: can't open /var/lib/ceph/osd/cluster2-1/keyring: (2) No such file or directory 2023-11-08T04:52:29.734 INFO:teuthology.orchestra.run.smithi087.stderr:2023-11-08T04:52:29.732+0000 7f7895c7a380 -1 created new key in keyring /var/lib/ceph/osd/cluster2-1/keyring 2023-11-08T04:52:29.734 INFO:teuthology.orchestra.run.smithi087.stderr:2023-11-08T04:52:29.732+0000 7f7895c7a380 -1 bluestore(/var/lib/ceph/osd/cluster2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-1/block: (2) No such file or directory 2023-11-08T04:52:29.735 INFO:teuthology.orchestra.run.smithi087.stderr:2023-11-08T04:52:29.732+0000 7f7895c7a380 -1 bluestore(/var/lib/ceph/osd/cluster2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-1/block: (2) No such file or directory 2023-11-08T04:52:29.735 INFO:teuthology.orchestra.run.smithi087.stderr:2023-11-08T04:52:29.732+0000 7f7895c7a380 -1 bluestore(/var/lib/ceph/osd/cluster2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-1/block: (2) No such file or directory 2023-11-08T04:52:29.735 INFO:teuthology.orchestra.run.smithi087.stderr:2023-11-08T04:52:29.732+0000 7f7895c7a380 -1 bluestore(/var/lib/ceph/osd/cluster2-1) _read_fsid unparsable uuid 2023-11-08T04:52:32.573 DEBUG:teuthology.orchestra.run.smithi087:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster2-1 2023-11-08T04:52:32.604 DEBUG:teuthology.orchestra.run.smithi087:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster2 --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/cluster2.monmap 2023-11-08T04:52:32.691 INFO:teuthology.orchestra.run.smithi087.stderr:2023-11-08T04:52:32.689+0000 7f6956a4f380 -1 auth: error reading file: /var/lib/ceph/osd/cluster2-2/keyring: can't open /var/lib/ceph/osd/cluster2-2/keyring: (2) No such file or directory 2023-11-08T04:52:32.692 INFO:teuthology.orchestra.run.smithi087.stderr:2023-11-08T04:52:32.689+0000 7f6956a4f380 -1 created new key in keyring /var/lib/ceph/osd/cluster2-2/keyring 2023-11-08T04:52:32.692 INFO:teuthology.orchestra.run.smithi087.stderr:2023-11-08T04:52:32.689+0000 7f6956a4f380 -1 bluestore(/var/lib/ceph/osd/cluster2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-2/block: (2) No such file or directory 2023-11-08T04:52:32.692 INFO:teuthology.orchestra.run.smithi087.stderr:2023-11-08T04:52:32.689+0000 7f6956a4f380 -1 bluestore(/var/lib/ceph/osd/cluster2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-2/block: (2) No such file or directory 2023-11-08T04:52:32.692 INFO:teuthology.orchestra.run.smithi087.stderr:2023-11-08T04:52:32.689+0000 7f6956a4f380 -1 bluestore(/var/lib/ceph/osd/cluster2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-2/block: (2) No such file or directory 2023-11-08T04:52:32.693 INFO:teuthology.orchestra.run.smithi087.stderr:2023-11-08T04:52:32.690+0000 7f6956a4f380 -1 bluestore(/var/lib/ceph/osd/cluster2-2) _read_fsid unparsable uuid 2023-11-08T04:52:35.540 DEBUG:teuthology.orchestra.run.smithi087:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster2-2 2023-11-08T04:52:35.572 INFO:tasks.ceph:Reading keys from all nodes... 2023-11-08T04:52:35.573 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2023-11-08T04:52:35.573 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd if=/var/lib/ceph/mgr/cluster2-x/keyring of=/dev/stdout 2023-11-08T04:52:35.606 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2023-11-08T04:52:35.607 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd if=/var/lib/ceph/osd/cluster2-0/keyring of=/dev/stdout 2023-11-08T04:52:35.642 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2023-11-08T04:52:35.643 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd if=/var/lib/ceph/osd/cluster2-1/keyring of=/dev/stdout 2023-11-08T04:52:35.711 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2023-11-08T04:52:35.712 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd if=/var/lib/ceph/osd/cluster2-2/keyring of=/dev/stdout 2023-11-08T04:52:35.780 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2023-11-08T04:52:35.781 DEBUG:teuthology.orchestra.run.smithi077:> dd if=/etc/ceph/cluster2.client.0.keyring of=/dev/stdout 2023-11-08T04:52:35.803 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2023-11-08T04:52:35.803 DEBUG:teuthology.orchestra.run.smithi087:> dd if=/etc/ceph/cluster2.client.mirror.keyring of=/dev/stdout 2023-11-08T04:52:35.839 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2023-11-08T04:52:35.839 DEBUG:teuthology.orchestra.run.smithi087:> dd if=/etc/ceph/cluster2.client.mirror.0.keyring of=/dev/stdout 2023-11-08T04:52:35.897 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2023-11-08T04:52:35.897 DEBUG:teuthology.orchestra.run.smithi087:> dd if=/etc/ceph/cluster2.client.mirror.1.keyring of=/dev/stdout 2023-11-08T04:52:35.954 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2023-11-08T04:52:35.955 DEBUG:teuthology.orchestra.run.smithi087:> dd if=/etc/ceph/cluster2.client.mirror.2.keyring of=/dev/stdout 2023-11-08T04:52:36.012 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2023-11-08T04:52:36.013 DEBUG:teuthology.orchestra.run.smithi087:> dd if=/etc/ceph/cluster2.client.mirror.3.keyring of=/dev/stdout 2023-11-08T04:52:36.070 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2023-11-08T04:52:36.071 DEBUG:teuthology.orchestra.run.smithi087:> dd if=/etc/ceph/cluster2.client.mirror.4.keyring of=/dev/stdout 2023-11-08T04:52:36.128 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2023-11-08T04:52:36.128 DEBUG:teuthology.orchestra.run.smithi087:> dd if=/etc/ceph/cluster2.client.mirror.5.keyring of=/dev/stdout 2023-11-08T04:52:36.185 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2023-11-08T04:52:36.185 DEBUG:teuthology.orchestra.run.smithi087:> dd if=/etc/ceph/cluster2.client.mirror.6.keyring of=/dev/stdout 2023-11-08T04:52:36.242 INFO:tasks.ceph:Adding keys to all mons... 2023-11-08T04:52:36.242 DEBUG:teuthology.orchestra.run.smithi087:> sudo tee -a /etc/ceph/cluster2.keyring 2023-11-08T04:52:36.306 INFO:teuthology.orchestra.run.smithi087.stdout:[mgr.x] 2023-11-08T04:52:36.306 INFO:teuthology.orchestra.run.smithi087.stdout: key = AQAHFEtlJKVpJRAAqEsusEWceCWSu9QFwgeO+g== 2023-11-08T04:52:36.306 INFO:teuthology.orchestra.run.smithi087.stdout:[osd.0] 2023-11-08T04:52:36.307 INFO:teuthology.orchestra.run.smithi087.stdout: key = AQAKFEtllLaULRAAuQrgRIBf+BRWjsAIPs3xOg== 2023-11-08T04:52:36.307 INFO:teuthology.orchestra.run.smithi087.stdout:[osd.1] 2023-11-08T04:52:36.307 INFO:teuthology.orchestra.run.smithi087.stdout: key = AQANFEtlHUWsKxAASkepDUtS/8ImJkcIl4RWTg== 2023-11-08T04:52:36.307 INFO:teuthology.orchestra.run.smithi087.stdout:[osd.2] 2023-11-08T04:52:36.307 INFO:teuthology.orchestra.run.smithi087.stdout: key = AQAQFEtlWi8mKRAA8KQVrvXCLUbyDrMNlsKL/g== 2023-11-08T04:52:36.308 INFO:teuthology.orchestra.run.smithi087.stdout:[client.0] 2023-11-08T04:52:36.308 INFO:teuthology.orchestra.run.smithi087.stdout: key = AQAHFEtlK7j7JxAAI+xVDT2IqM20+n4tPmaHCw== 2023-11-08T04:52:36.308 INFO:teuthology.orchestra.run.smithi087.stdout:[client.mirror] 2023-11-08T04:52:36.308 INFO:teuthology.orchestra.run.smithi087.stdout: key = AQAHFEtlr7MMKxAA/QBaglbUxAJ2D5bFfNKw0Q== 2023-11-08T04:52:36.308 INFO:teuthology.orchestra.run.smithi087.stdout:[client.mirror.0] 2023-11-08T04:52:36.308 INFO:teuthology.orchestra.run.smithi087.stdout: key = AQAHFEtlsHtvMBAA3HFu8xmCjje0HUwOOSdjAQ== 2023-11-08T04:52:36.309 INFO:teuthology.orchestra.run.smithi087.stdout:[client.mirror.1] 2023-11-08T04:52:36.309 INFO:teuthology.orchestra.run.smithi087.stdout: key = AQAHFEtlYtHQNRAAI+2+/LhaIPO8EMIPtXRrMg== 2023-11-08T04:52:36.309 INFO:teuthology.orchestra.run.smithi087.stdout:[client.mirror.2] 2023-11-08T04:52:36.309 INFO:teuthology.orchestra.run.smithi087.stdout: key = AQAHFEtlPMpLOxAAyGR7aMGrc7eMpzAMm08S/Q== 2023-11-08T04:52:36.309 INFO:teuthology.orchestra.run.smithi087.stdout:[client.mirror.3] 2023-11-08T04:52:36.309 INFO:teuthology.orchestra.run.smithi087.stdout: key = AQAIFEtleFYlBRAAaots/gTt4PW72JPov/rsyw== 2023-11-08T04:52:36.310 INFO:teuthology.orchestra.run.smithi087.stdout:[client.mirror.4] 2023-11-08T04:52:36.310 INFO:teuthology.orchestra.run.smithi087.stdout: key = AQAIFEtlgZCFChAA7EC0S453VR7liQ0xyhzyeQ== 2023-11-08T04:52:36.310 INFO:teuthology.orchestra.run.smithi087.stdout:[client.mirror.5] 2023-11-08T04:52:36.310 INFO:teuthology.orchestra.run.smithi087.stdout: key = AQAIFEtlmpHuDxAAIZVbiDehgUEJH19h3A7RHg== 2023-11-08T04:52:36.310 INFO:teuthology.orchestra.run.smithi087.stdout:[client.mirror.6] 2023-11-08T04:52:36.311 INFO:teuthology.orchestra.run.smithi087.stdout: key = AQAIFEtllpJYFRAAmodc+7LT1pD8V7y4/ls/0A== 2023-11-08T04:52:36.312 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-11-08T04:52:36.402 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-08T04:52:36.450 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-08T04:52:36.498 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-08T04:52:36.546 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-11-08T04:52:36.597 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-11-08T04:52:36.648 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-11-08T04:52:36.699 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-11-08T04:52:36.750 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-11-08T04:52:36.801 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-11-08T04:52:36.852 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.4 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-11-08T04:52:36.903 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.5 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-11-08T04:52:36.954 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.6 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-11-08T04:52:37.006 INFO:tasks.ceph:Running mkfs on mon nodes... 2023-11-08T04:52:37.006 DEBUG:teuthology.orchestra.run.smithi087:> sudo mkdir -p /var/lib/ceph/mon/cluster2-a 2023-11-08T04:52:37.033 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster cluster2 --mkfs -i a --monmap /home/ubuntu/cephtest/cluster2.monmap --keyring /etc/ceph/cluster2.keyring 2023-11-08T04:52:37.125 DEBUG:teuthology.orchestra.run.smithi087:> sudo chown -R ceph:ceph /var/lib/ceph/mon/cluster2-a 2023-11-08T04:52:37.152 DEBUG:teuthology.orchestra.run.smithi087:> rm -- /home/ubuntu/cephtest/cluster2.monmap 2023-11-08T04:52:37.211 INFO:tasks.ceph:Starting mon daemons in cluster cluster2... 2023-11-08T04:52:37.211 INFO:tasks.ceph.cluster2.mon.a:Restarting daemon 2023-11-08T04:52:37.211 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster cluster2 -i a 2023-11-08T04:52:37.253 INFO:tasks.ceph.cluster2.mon.a:Started 2023-11-08T04:52:37.253 INFO:tasks.ceph:Starting mgr daemons in cluster cluster2... 2023-11-08T04:52:37.254 INFO:tasks.ceph.cluster2.mgr.x:Restarting daemon 2023-11-08T04:52:37.254 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster cluster2 -i x 2023-11-08T04:52:37.257 INFO:tasks.ceph.cluster2.mgr.x:Started 2023-11-08T04:52:37.257 INFO:tasks.ceph:Setting crush tunables to default 2023-11-08T04:52:37.258 DEBUG:teuthology.orchestra.run.smithi087:> sudo ceph --cluster cluster2 osd crush tunables default 2023-11-08T04:52:37.471 INFO:teuthology.orchestra.run.smithi087.stderr:adjusted tunables profile to default 2023-11-08T04:52:37.482 INFO:tasks.ceph:Starting osd daemons in cluster cluster2... 2023-11-08T04:52:37.482 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2023-11-08T04:52:37.483 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd if=/var/lib/ceph/osd/cluster2-0/fsid of=/dev/stdout 2023-11-08T04:52:37.497 INFO:tasks.ceph.cluster2.mgr.x.smithi077.stderr:2023-11-08T04:52:37.494+0000 7f172f437680 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-11-08T04:52:37.509 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2023-11-08T04:52:37.509 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd if=/var/lib/ceph/osd/cluster2-1/fsid of=/dev/stdout 2023-11-08T04:52:37.558 INFO:tasks.ceph.cluster2.mgr.x.smithi077.stderr:2023-11-08T04:52:37.556+0000 7f172f437680 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-11-08T04:52:37.576 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2023-11-08T04:52:37.577 DEBUG:teuthology.orchestra.run.smithi087:> sudo dd if=/var/lib/ceph/osd/cluster2-2/fsid of=/dev/stdout 2023-11-08T04:52:37.643 DEBUG:teuthology.orchestra.run.smithi087:> sudo ceph --cluster cluster2 osd new 871ecae4-6fa4-41b3-beab-e6d861330b33 0 2023-11-08T04:52:37.825 INFO:tasks.ceph.cluster2.mgr.x.smithi077.stderr:2023-11-08T04:52:37.822+0000 7f172f437680 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-11-08T04:52:37.881 INFO:teuthology.orchestra.run.smithi087.stdout:0 2023-11-08T04:52:37.893 DEBUG:teuthology.orchestra.run.smithi087:> sudo ceph --cluster cluster2 osd new 6146f4ae-0032-4ec1-b1af-f785a8265290 1 2023-11-08T04:52:38.093 INFO:teuthology.orchestra.run.smithi087.stdout:1 2023-11-08T04:52:38.104 DEBUG:teuthology.orchestra.run.smithi087:> sudo ceph --cluster cluster2 osd new f6939e19-a2a3-4f8d-a25b-41f149cf05b1 2 2023-11-08T04:52:38.302 INFO:teuthology.orchestra.run.smithi087.stdout:2 2023-11-08T04:52:38.314 INFO:tasks.ceph.cluster2.osd.0:Restarting daemon 2023-11-08T04:52:38.314 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster2 -i 0 2023-11-08T04:52:38.317 INFO:tasks.ceph.cluster2.osd.0:Started 2023-11-08T04:52:38.317 INFO:tasks.ceph.cluster2.osd.1:Restarting daemon 2023-11-08T04:52:38.317 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster2 -i 1 2023-11-08T04:52:38.319 INFO:tasks.ceph.cluster2.osd.1:Started 2023-11-08T04:52:38.320 INFO:tasks.ceph.cluster2.osd.2:Restarting daemon 2023-11-08T04:52:38.320 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster2 -i 2 2023-11-08T04:52:38.322 INFO:tasks.ceph.cluster2.osd.2:Started 2023-11-08T04:52:38.323 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd dump --format=json 2023-11-08T04:52:38.422 INFO:tasks.ceph.cluster2.mgr.x.smithi077.stderr:2023-11-08T04:52:38.419+0000 7f172f437680 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-11-08T04:52:38.492 INFO:tasks.ceph.cluster2.mgr.x.smithi077.stderr:2023-11-08T04:52:38.490+0000 7f172f437680 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-11-08T04:52:38.516 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T04:52:38.516 INFO:teuthology.orchestra.run.smithi087.stdout:{"epoch":5,"fsid":"ab4cbc06-449d-497d-b513-220d9db2b590","created":"2023-11-08T04:52:37.369406+0000","modified":"2023-11-08T04:52:38.299675+0000","last_up_change":"0.000000","last_in_change":"2023-11-08T04:52:38.299675+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[],"osds":[{"osd":0,"uuid":"871ecae4-6fa4-41b3-beab-e6d861330b33","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6146f4ae-0032-4ec1-b1af-f785a8265290","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f6939e19-a2a3-4f8d-a25b-41f149cf05b1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-08T04:52:38.528 INFO:tasks.ceph.ceph_manager.cluster2:[] 2023-11-08T04:52:38.528 INFO:tasks.ceph:Waiting for OSDs to come up 2023-11-08T04:52:38.675 INFO:tasks.ceph.cluster2.mgr.x.smithi077.stderr:2023-11-08T04:52:38.673+0000 7f172f437680 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-11-08T04:52:38.831 DEBUG:teuthology.orchestra.run.smithi087:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster2 osd dump --format=json 2023-11-08T04:52:38.912 INFO:tasks.ceph.cluster2.osd.0.smithi087.stderr:2023-11-08T04:52:38.910+0000 7f1adce32380 -1 Falling back to public interface 2023-11-08T04:52:38.917 INFO:tasks.ceph.cluster2.osd.1.smithi087.stderr:2023-11-08T04:52:38.915+0000 7fbb91cab380 -1 Falling back to public interface 2023-11-08T04:52:38.922 INFO:tasks.ceph.cluster2.osd.2.smithi087.stderr:2023-11-08T04:52:38.920+0000 7fa25aac6380 -1 Falling back to public interface 2023-11-08T04:52:39.011 INFO:teuthology.misc.health.smithi087.stdout: 2023-11-08T04:52:39.011 INFO:teuthology.misc.health.smithi087.stdout:{"epoch":5,"fsid":"ab4cbc06-449d-497d-b513-220d9db2b590","created":"2023-11-08T04:52:37.369406+0000","modified":"2023-11-08T04:52:38.299675+0000","last_up_change":"0.000000","last_in_change":"2023-11-08T04:52:38.299675+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[],"osds":[{"osd":0,"uuid":"871ecae4-6fa4-41b3-beab-e6d861330b33","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6146f4ae-0032-4ec1-b1af-f785a8265290","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f6939e19-a2a3-4f8d-a25b-41f149cf05b1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-08T04:52:39.021 DEBUG:teuthology.misc:0 of 3 OSDs are up 2023-11-08T04:52:39.084 INFO:tasks.ceph.cluster2.mgr.x.smithi077.stderr:2023-11-08T04:52:39.081+0000 7f172f437680 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-11-08T04:52:39.185 INFO:tasks.ceph.cluster2.mgr.x.smithi077.stderr:2023-11-08T04:52:39.183+0000 7f172f437680 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-11-08T04:52:39.336 INFO:tasks.ceph.cluster2.mgr.x.smithi077.stderr:2023-11-08T04:52:39.334+0000 7f172f437680 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-11-08T04:52:39.409 INFO:tasks.ceph.cluster2.mgr.x.smithi077.stderr:2023-11-08T04:52:39.406+0000 7f172f437680 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-11-08T04:52:39.474 INFO:tasks.ceph.cluster2.mgr.x.smithi077.stderr:2023-11-08T04:52:39.471+0000 7f172f437680 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-11-08T04:52:39.721 INFO:tasks.ceph.cluster2.mgr.x.smithi077.stderr:2023-11-08T04:52:39.718+0000 7f172f437680 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-11-08T04:52:39.899 INFO:tasks.ceph.cluster2.mgr.x.smithi077.stderr:2023-11-08T04:52:39.896+0000 7f172f437680 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-11-08T04:52:40.536 INFO:tasks.ceph.cluster2.mgr.x.smithi077.stderr:2023-11-08T04:52:40.533+0000 7f172f437680 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-11-08T04:52:40.602 INFO:tasks.ceph.cluster2.mgr.x.smithi077.stderr:2023-11-08T04:52:40.599+0000 7f172f437680 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-11-08T04:52:40.674 INFO:tasks.ceph.cluster2.mgr.x.smithi077.stderr:2023-11-08T04:52:40.672+0000 7f172f437680 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-11-08T04:52:40.814 INFO:tasks.ceph.cluster2.mgr.x.smithi077.stderr:2023-11-08T04:52:40.812+0000 7f172f437680 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-11-08T04:52:40.882 INFO:tasks.ceph.cluster2.mgr.x.smithi077.stderr:2023-11-08T04:52:40.880+0000 7f172f437680 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-11-08T04:52:40.993 INFO:tasks.ceph.cluster2.mgr.x.smithi077.stderr:2023-11-08T04:52:40.991+0000 7f172f437680 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-11-08T04:52:41.098 INFO:tasks.ceph.cluster2.mgr.x.smithi077.stderr:2023-11-08T04:52:41.095+0000 7f172f437680 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-11-08T04:52:41.385 INFO:tasks.ceph.cluster2.mgr.x.smithi077.stderr:2023-11-08T04:52:41.382+0000 7f172f437680 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-11-08T04:52:41.452 INFO:tasks.ceph.cluster2.mgr.x.smithi077.stderr:2023-11-08T04:52:41.450+0000 7f172f437680 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-11-08T04:52:42.331 INFO:tasks.ceph.cluster2.osd.0.smithi087.stderr:2023-11-08T04:52:42.329+0000 7f1adce32380 -1 osd.0 0 log_to_monitors {default=true} 2023-11-08T04:52:42.368 INFO:tasks.ceph.cluster2.osd.1.smithi087.stderr:2023-11-08T04:52:42.366+0000 7fbb91cab380 -1 osd.1 0 log_to_monitors {default=true} 2023-11-08T04:52:42.404 INFO:tasks.ceph.cluster2.osd.2.smithi087.stderr:2023-11-08T04:52:42.402+0000 7fa25aac6380 -1 osd.2 0 log_to_monitors {default=true} 2023-11-08T04:52:42.471 INFO:tasks.ceph.cluster2.mgr.x.smithi077.stderr:2023-11-08T04:52:42.469+0000 7f16e4990700 -1 mgr.server handle_report got status from non-daemon mon.a 2023-11-08T04:52:43.529 INFO:tasks.ceph.cluster2.osd.0.smithi087.stderr:2023-11-08T04:52:43.527+0000 7f1ad37d3700 -1 osd.0 0 waiting for initial osdmap 2023-11-08T04:52:43.529 INFO:tasks.ceph.cluster2.osd.1.smithi087.stderr:2023-11-08T04:52:43.527+0000 7fbb8864c700 -1 osd.1 0 waiting for initial osdmap 2023-11-08T04:52:43.530 INFO:tasks.ceph.cluster2.osd.2.smithi087.stderr:2023-11-08T04:52:43.529+0000 7fa251467700 -1 osd.2 0 waiting for initial osdmap 2023-11-08T04:52:43.533 INFO:tasks.ceph.cluster2.osd.0.smithi087.stderr:2023-11-08T04:52:43.531+0000 7f1acff4f700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-11-08T04:52:43.533 INFO:tasks.ceph.cluster2.osd.1.smithi087.stderr:2023-11-08T04:52:43.531+0000 7fbb84dc8700 -1 osd.1 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-11-08T04:52:43.534 INFO:tasks.ceph.cluster2.osd.2.smithi087.stderr:2023-11-08T04:52:43.532+0000 7fa24dbe3700 -1 osd.2 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-11-08T04:52:45.326 DEBUG:teuthology.orchestra.run.smithi087:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster2 osd dump --format=json 2023-11-08T04:52:45.619 INFO:teuthology.misc.health.smithi087.stdout: 2023-11-08T04:52:45.619 INFO:teuthology.misc.health.smithi087.stdout:{"epoch":9,"fsid":"ab4cbc06-449d-497d-b513-220d9db2b590","created":"2023-11-08T04:52:37.369406+0000","modified":"2023-11-08T04:52:45.528796+0000","last_up_change":"2023-11-08T04:52:44.524151+0000","last_in_change":"2023-11-08T04:52:38.299675+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2023-11-08T04:52:44.563937+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"9","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"871ecae4-6fa4-41b3-beab-e6d861330b33","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6800","nonce":116186},{"type":"v1","addr":"172.21.15.87:6801","nonce":116186}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6802","nonce":116186},{"type":"v1","addr":"172.21.15.87:6803","nonce":116186}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6806","nonce":116186},{"type":"v1","addr":"172.21.15.87:6807","nonce":116186}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6804","nonce":116186},{"type":"v1","addr":"172.21.15.87:6805","nonce":116186}]},"public_addr":"172.21.15.87:6801/116186","cluster_addr":"172.21.15.87:6803/116186","heartbeat_back_addr":"172.21.15.87:6807/116186","heartbeat_front_addr":"172.21.15.87:6805/116186","state":["exists","up"]},{"osd":1,"uuid":"6146f4ae-0032-4ec1-b1af-f785a8265290","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6808","nonce":116187},{"type":"v1","addr":"172.21.15.87:6809","nonce":116187}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6810","nonce":116187},{"type":"v1","addr":"172.21.15.87:6811","nonce":116187}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6814","nonce":116187},{"type":"v1","addr":"172.21.15.87:6815","nonce":116187}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6812","nonce":116187},{"type":"v1","addr":"172.21.15.87:6813","nonce":116187}]},"public_addr":"172.21.15.87:6809/116187","cluster_addr":"172.21.15.87:6811/116187","heartbeat_back_addr":"172.21.15.87:6815/116187","heartbeat_front_addr":"172.21.15.87:6813/116187","state":["exists","up"]},{"osd":2,"uuid":"f6939e19-a2a3-4f8d-a25b-41f149cf05b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6816","nonce":116188},{"type":"v1","addr":"172.21.15.87:6817","nonce":116188}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6818","nonce":116188},{"type":"v1","addr":"172.21.15.87:6819","nonce":116188}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6822","nonce":116188},{"type":"v1","addr":"172.21.15.87:6823","nonce":116188}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6820","nonce":116188},{"type":"v1","addr":"172.21.15.87:6821","nonce":116188}]},"public_addr":"172.21.15.87:6817/116188","cluster_addr":"172.21.15.87:6819/116188","heartbeat_back_addr":"172.21.15.87:6823/116188","heartbeat_front_addr":"172.21.15.87:6821/116188","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T04:52:43.283591+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T04:52:43.416477+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T04:52:43.365275+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-08T04:52:45.631 DEBUG:teuthology.misc:3 of 3 OSDs are up 2023-11-08T04:52:45.631 INFO:tasks.ceph:Creating RBD pool 2023-11-08T04:52:45.632 DEBUG:teuthology.orchestra.run.smithi087:> sudo ceph --cluster cluster2 osd pool create rbd 8 2023-11-08T04:52:46.541 INFO:teuthology.orchestra.run.smithi087.stderr:pool 'rbd' created 2023-11-08T04:52:46.555 DEBUG:teuthology.orchestra.run.smithi087:> sudo ceph --cluster cluster2 osd pool application enable rbd rbd --yes-i-really-mean-it 2023-11-08T04:52:47.547 INFO:teuthology.orchestra.run.smithi087.stderr:enabled application 'rbd' on pool 'rbd' 2023-11-08T04:52:47.561 INFO:tasks.ceph:Starting mds daemons in cluster cluster2... 2023-11-08T04:52:47.561 INFO:tasks.ceph:Waiting until cluster2 daemons up and pgs clean... 2023-11-08T04:52:47.562 INFO:tasks.ceph.ceph_manager.cluster2:waiting for mgr available 2023-11-08T04:52:47.562 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 mgr dump --format=json 2023-11-08T04:52:47.563 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2023-11-08T04:52:47.897 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T04:52:47.909 INFO:teuthology.orchestra.run.smithi087.stdout:{"epoch":4,"active_gid":4099,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":119114},{"type":"v1","addr":"172.21.15.77:6827","nonce":119114}]},"active_addr":"172.21.15.77:6827/119114","active_change":"2023-11-08T04:52:41.464541+0000","active_mgr_features":4540138314316775423,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.23.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:8.3.5","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"docker.io/library/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"docker.io/arcts/keepalived","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.3.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.33.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"str","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"str","level":"advanced","flags":0,"default_value":"30","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"str","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"str","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"str","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"nautilus":["balancer","crash","devicehealth","orchestrator_cli","progress","rbd_support","status","volumes"],"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.77:0","nonce":2600512117}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.77:0","nonce":1107525376}]}]} 2023-11-08T04:52:47.913 INFO:tasks.ceph.ceph_manager.cluster2:mgr available! 2023-11-08T04:52:47.913 INFO:tasks.ceph.ceph_manager.cluster2:waiting for all up 2023-11-08T04:52:47.913 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd dump --format=json 2023-11-08T04:52:48.199 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T04:52:48.199 INFO:teuthology.orchestra.run.smithi087.stdout:{"epoch":11,"fsid":"ab4cbc06-449d-497d-b513-220d9db2b590","created":"2023-11-08T04:52:37.369406+0000","modified":"2023-11-08T04:52:47.532694+0000","last_up_change":"2023-11-08T04:52:44.524151+0000","last_in_change":"2023-11-08T04:52:38.299675+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2023-11-08T04:52:44.563937+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2023-11-08T04:52:45.947193+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"871ecae4-6fa4-41b3-beab-e6d861330b33","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6800","nonce":116186},{"type":"v1","addr":"172.21.15.87:6801","nonce":116186}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6802","nonce":116186},{"type":"v1","addr":"172.21.15.87:6803","nonce":116186}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6806","nonce":116186},{"type":"v1","addr":"172.21.15.87:6807","nonce":116186}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6804","nonce":116186},{"type":"v1","addr":"172.21.15.87:6805","nonce":116186}]},"public_addr":"172.21.15.87:6801/116186","cluster_addr":"172.21.15.87:6803/116186","heartbeat_back_addr":"172.21.15.87:6807/116186","heartbeat_front_addr":"172.21.15.87:6805/116186","state":["exists","up"]},{"osd":1,"uuid":"6146f4ae-0032-4ec1-b1af-f785a8265290","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6808","nonce":116187},{"type":"v1","addr":"172.21.15.87:6809","nonce":116187}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6810","nonce":116187},{"type":"v1","addr":"172.21.15.87:6811","nonce":116187}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6814","nonce":116187},{"type":"v1","addr":"172.21.15.87:6815","nonce":116187}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6812","nonce":116187},{"type":"v1","addr":"172.21.15.87:6813","nonce":116187}]},"public_addr":"172.21.15.87:6809/116187","cluster_addr":"172.21.15.87:6811/116187","heartbeat_back_addr":"172.21.15.87:6815/116187","heartbeat_front_addr":"172.21.15.87:6813/116187","state":["exists","up"]},{"osd":2,"uuid":"f6939e19-a2a3-4f8d-a25b-41f149cf05b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6816","nonce":116188},{"type":"v1","addr":"172.21.15.87:6817","nonce":116188}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6818","nonce":116188},{"type":"v1","addr":"172.21.15.87:6819","nonce":116188}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6822","nonce":116188},{"type":"v1","addr":"172.21.15.87:6823","nonce":116188}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6820","nonce":116188},{"type":"v1","addr":"172.21.15.87:6821","nonce":116188}]},"public_addr":"172.21.15.87:6817/116188","cluster_addr":"172.21.15.87:6819/116188","heartbeat_back_addr":"172.21.15.87:6823/116188","heartbeat_front_addr":"172.21.15.87:6821/116188","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T04:52:43.283591+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T04:52:43.416477+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T04:52:43.365275+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-08T04:52:48.212 INFO:tasks.ceph.ceph_manager.cluster2:all up! 2023-11-08T04:52:48.212 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd dump --format=json 2023-11-08T04:52:48.500 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T04:52:48.500 INFO:teuthology.orchestra.run.smithi087.stdout:{"epoch":11,"fsid":"ab4cbc06-449d-497d-b513-220d9db2b590","created":"2023-11-08T04:52:37.369406+0000","modified":"2023-11-08T04:52:47.532694+0000","last_up_change":"2023-11-08T04:52:44.524151+0000","last_in_change":"2023-11-08T04:52:38.299675+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2023-11-08T04:52:44.563937+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2023-11-08T04:52:45.947193+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"871ecae4-6fa4-41b3-beab-e6d861330b33","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6800","nonce":116186},{"type":"v1","addr":"172.21.15.87:6801","nonce":116186}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6802","nonce":116186},{"type":"v1","addr":"172.21.15.87:6803","nonce":116186}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6806","nonce":116186},{"type":"v1","addr":"172.21.15.87:6807","nonce":116186}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6804","nonce":116186},{"type":"v1","addr":"172.21.15.87:6805","nonce":116186}]},"public_addr":"172.21.15.87:6801/116186","cluster_addr":"172.21.15.87:6803/116186","heartbeat_back_addr":"172.21.15.87:6807/116186","heartbeat_front_addr":"172.21.15.87:6805/116186","state":["exists","up"]},{"osd":1,"uuid":"6146f4ae-0032-4ec1-b1af-f785a8265290","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6808","nonce":116187},{"type":"v1","addr":"172.21.15.87:6809","nonce":116187}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6810","nonce":116187},{"type":"v1","addr":"172.21.15.87:6811","nonce":116187}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6814","nonce":116187},{"type":"v1","addr":"172.21.15.87:6815","nonce":116187}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6812","nonce":116187},{"type":"v1","addr":"172.21.15.87:6813","nonce":116187}]},"public_addr":"172.21.15.87:6809/116187","cluster_addr":"172.21.15.87:6811/116187","heartbeat_back_addr":"172.21.15.87:6815/116187","heartbeat_front_addr":"172.21.15.87:6813/116187","state":["exists","up"]},{"osd":2,"uuid":"f6939e19-a2a3-4f8d-a25b-41f149cf05b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6816","nonce":116188},{"type":"v1","addr":"172.21.15.87:6817","nonce":116188}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6818","nonce":116188},{"type":"v1","addr":"172.21.15.87:6819","nonce":116188}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6822","nonce":116188},{"type":"v1","addr":"172.21.15.87:6823","nonce":116188}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6820","nonce":116188},{"type":"v1","addr":"172.21.15.87:6821","nonce":116188}]},"public_addr":"172.21.15.87:6817/116188","cluster_addr":"172.21.15.87:6819/116188","heartbeat_back_addr":"172.21.15.87:6823/116188","heartbeat_front_addr":"172.21.15.87:6821/116188","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T04:52:43.283591+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T04:52:43.416477+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T04:52:43.365275+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-08T04:52:48.513 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 tell osd.0 flush_pg_stats 2023-11-08T04:52:48.660 INFO:teuthology.orchestra.run.smithi087.stdout:34359738372 2023-11-08T04:52:48.660 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 tell osd.1 flush_pg_stats 2023-11-08T04:52:48.807 INFO:teuthology.orchestra.run.smithi087.stdout:34359738371 2023-11-08T04:52:48.808 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 tell osd.2 flush_pg_stats 2023-11-08T04:52:48.955 INFO:teuthology.orchestra.run.smithi087.stdout:34359738371 2023-11-08T04:52:48.955 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.0 2023-11-08T04:52:49.240 INFO:teuthology.orchestra.run.smithi087.stdout:34359738371 2023-11-08T04:52:49.253 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738372 got 34359738371 for osd.0 2023-11-08T04:52:50.254 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.0 2023-11-08T04:52:50.550 INFO:teuthology.orchestra.run.smithi087.stdout:34359738372 2023-11-08T04:52:50.563 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738372 got 34359738372 for osd.0 2023-11-08T04:52:50.564 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.1 2023-11-08T04:52:50.849 INFO:teuthology.orchestra.run.smithi087.stdout:34359738371 2023-11-08T04:52:50.863 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738371 got 34359738371 for osd.1 2023-11-08T04:52:50.863 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.2 2023-11-08T04:52:51.147 INFO:teuthology.orchestra.run.smithi087.stdout:34359738371 2023-11-08T04:52:51.160 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738371 got 34359738371 for osd.2 2023-11-08T04:52:51.161 INFO:tasks.ceph.ceph_manager.cluster2:waiting for clean 2023-11-08T04:52:51.161 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-11-08T04:52:51.444 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T04:52:51.446 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2023-11-08T04:52:51.456 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":12,"stamp":"2023-11-08T04:52:49.469340+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":14,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":891572,"kb_used_data":284,"kb_used_omap":0,"kb_used_meta":891264,"kb_avail":282223948,"statfs":{"total":289910292480,"available":288997322752,"internally_reserved":0,"allocated":290816,"data_stored":77406,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":912654336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-11-08T04:52:47.553490+0000","last_change":"2023-11-08T04:52:47.549055+0000","last_active":"2023-11-08T04:52:47.553490+0000","last_peered":"2023-11-08T04:52:47.553490+0000","last_clean":"2023-11-08T04:52:47.553490+0000","last_became_active":"2023-11-08T04:52:47.548772+0000","last_became_peered":"2023-11-08T04:52:47.548772+0000","last_unstale":"2023-11-08T04:52:47.553490+0000","last_undegraded":"2023-11-08T04:52:47.553490+0000","last_fullsized":"2023-11-08T04:52:47.553490+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-11-08T04:52:47.549550+0000","last_change":"2023-11-08T04:52:47.548585+0000","last_active":"2023-11-08T04:52:47.549550+0000","last_peered":"2023-11-08T04:52:47.549550+0000","last_clean":"2023-11-08T04:52:47.549550+0000","last_became_active":"2023-11-08T04:52:47.548289+0000","last_became_peered":"2023-11-08T04:52:47.548289+0000","last_unstale":"2023-11-08T04:52:47.549550+0000","last_undegraded":"2023-11-08T04:52:47.549550+0000","last_fullsized":"2023-11-08T04:52:47.549550+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-11-08T04:52:47.553473+0000","last_change":"2023-11-08T04:52:47.548928+0000","last_active":"2023-11-08T04:52:47.553473+0000","last_peered":"2023-11-08T04:52:47.553473+0000","last_clean":"2023-11-08T04:52:47.553473+0000","last_became_active":"2023-11-08T04:52:47.548323+0000","last_became_peered":"2023-11-08T04:52:47.548323+0000","last_unstale":"2023-11-08T04:52:47.553473+0000","last_undegraded":"2023-11-08T04:52:47.553473+0000","last_fullsized":"2023-11-08T04:52:47.553473+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-11-08T04:52:47.549399+0000","last_change":"2023-11-08T04:52:47.546022+0000","last_active":"2023-11-08T04:52:47.549399+0000","last_peered":"2023-11-08T04:52:47.549399+0000","last_clean":"2023-11-08T04:52:47.549399+0000","last_became_active":"2023-11-08T04:52:47.545767+0000","last_became_peered":"2023-11-08T04:52:47.545767+0000","last_unstale":"2023-11-08T04:52:47.549399+0000","last_undegraded":"2023-11-08T04:52:47.549399+0000","last_fullsized":"2023-11-08T04:52:47.549399+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":10,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-11-08T04:52:47.543946+0000","last_change":"2023-11-08T04:52:46.545601+0000","last_active":"2023-11-08T04:52:47.543946+0000","last_peered":"2023-11-08T04:52:47.543946+0000","last_clean":"2023-11-08T04:52:47.543946+0000","last_became_active":"2023-11-08T04:52:46.545321+0000","last_became_peered":"2023-11-08T04:52:46.545321+0000","last_unstale":"2023-11-08T04:52:47.543946+0000","last_undegraded":"2023-11-08T04:52:47.543946+0000","last_fullsized":"2023-11-08T04:52:47.543946+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_clean_scrub_stamp":"2023-11-08T04:52:45.528796+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-11-08T04:52:47.549543+0000","last_change":"2023-11-08T04:52:47.548597+0000","last_active":"2023-11-08T04:52:47.549543+0000","last_peered":"2023-11-08T04:52:47.549543+0000","last_clean":"2023-11-08T04:52:47.549543+0000","last_became_active":"2023-11-08T04:52:47.548294+0000","last_became_peered":"2023-11-08T04:52:47.548294+0000","last_unstale":"2023-11-08T04:52:47.549543+0000","last_undegraded":"2023-11-08T04:52:47.549543+0000","last_fullsized":"2023-11-08T04:52:47.549543+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-11-08T04:52:47.549368+0000","last_change":"2023-11-08T04:52:47.546100+0000","last_active":"2023-11-08T04:52:47.549368+0000","last_peered":"2023-11-08T04:52:47.549368+0000","last_clean":"2023-11-08T04:52:47.549368+0000","last_became_active":"2023-11-08T04:52:47.545840+0000","last_became_peered":"2023-11-08T04:52:47.545840+0000","last_unstale":"2023-11-08T04:52:47.549368+0000","last_undegraded":"2023-11-08T04:52:47.549368+0000","last_fullsized":"2023-11-08T04:52:47.549368+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-11-08T04:52:47.546027+0000","last_change":"2023-11-08T04:52:47.546027+0000","last_active":"2023-11-08T04:52:47.546027+0000","last_peered":"2023-11-08T04:52:47.546027+0000","last_clean":"2023-11-08T04:52:47.546027+0000","last_became_active":"2023-11-08T04:52:47.545761+0000","last_became_peered":"2023-11-08T04:52:47.545761+0000","last_unstale":"2023-11-08T04:52:47.546027+0000","last_undegraded":"2023-11-08T04:52:47.546027+0000","last_fullsized":"2023-11-08T04:52:47.546027+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-11-08T04:52:47.549505+0000","last_change":"2023-11-08T04:52:47.548575+0000","last_active":"2023-11-08T04:52:47.549505+0000","last_peered":"2023-11-08T04:52:47.549505+0000","last_clean":"2023-11-08T04:52:47.549505+0000","last_became_active":"2023-11-08T04:52:47.548265+0000","last_became_peered":"2023-11-08T04:52:47.548265+0000","last_unstale":"2023-11-08T04:52:47.549505+0000","last_undegraded":"2023-11-08T04:52:47.549505+0000","last_fullsized":"2023-11-08T04:52:47.549505+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738371,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297188,"kb_used_data":92,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94074652,"statfs":{"total":96636764160,"available":96332443648,"internally_reserved":0,"allocated":94208,"data_stored":25156,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.24099999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.23200000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738372,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297192,"kb_used_data":96,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94074648,"statfs":{"total":96636764160,"available":96332439552,"internally_reserved":0,"allocated":98304,"data_stored":26125,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.218}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.22600000000000001}]}]},{"osd":1,"up_from":8,"seq":34359738371,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297192,"kb_used_data":96,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94074648,"statfs":{"total":96636764160,"available":96332439552,"internally_reserved":0,"allocated":98304,"data_stored":26125,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36499999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32600000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-08T04:52:51.457 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-11-08T04:52:51.740 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T04:52:51.741 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2023-11-08T04:52:51.752 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":13,"stamp":"2023-11-08T04:52:51.469714+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":891596,"kb_used_data":308,"kb_used_omap":0,"kb_used_meta":891264,"kb_avail":282223924,"statfs":{"total":289910292480,"available":288997298176,"internally_reserved":0,"allocated":315392,"data_stored":87993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":912654336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-11-08T04:52:47.553490+0000","last_change":"2023-11-08T04:52:47.549055+0000","last_active":"2023-11-08T04:52:47.553490+0000","last_peered":"2023-11-08T04:52:47.553490+0000","last_clean":"2023-11-08T04:52:47.553490+0000","last_became_active":"2023-11-08T04:52:47.548772+0000","last_became_peered":"2023-11-08T04:52:47.548772+0000","last_unstale":"2023-11-08T04:52:47.553490+0000","last_undegraded":"2023-11-08T04:52:47.553490+0000","last_fullsized":"2023-11-08T04:52:47.553490+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-11-08T04:52:47.549550+0000","last_change":"2023-11-08T04:52:47.548585+0000","last_active":"2023-11-08T04:52:47.549550+0000","last_peered":"2023-11-08T04:52:47.549550+0000","last_clean":"2023-11-08T04:52:47.549550+0000","last_became_active":"2023-11-08T04:52:47.548289+0000","last_became_peered":"2023-11-08T04:52:47.548289+0000","last_unstale":"2023-11-08T04:52:47.549550+0000","last_undegraded":"2023-11-08T04:52:47.549550+0000","last_fullsized":"2023-11-08T04:52:47.549550+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-11-08T04:52:47.553473+0000","last_change":"2023-11-08T04:52:47.548928+0000","last_active":"2023-11-08T04:52:47.553473+0000","last_peered":"2023-11-08T04:52:47.553473+0000","last_clean":"2023-11-08T04:52:47.553473+0000","last_became_active":"2023-11-08T04:52:47.548323+0000","last_became_peered":"2023-11-08T04:52:47.548323+0000","last_unstale":"2023-11-08T04:52:47.553473+0000","last_undegraded":"2023-11-08T04:52:47.553473+0000","last_fullsized":"2023-11-08T04:52:47.553473+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-11-08T04:52:47.549399+0000","last_change":"2023-11-08T04:52:47.546022+0000","last_active":"2023-11-08T04:52:47.549399+0000","last_peered":"2023-11-08T04:52:47.549399+0000","last_clean":"2023-11-08T04:52:47.549399+0000","last_became_active":"2023-11-08T04:52:47.545767+0000","last_became_peered":"2023-11-08T04:52:47.545767+0000","last_unstale":"2023-11-08T04:52:47.549399+0000","last_undegraded":"2023-11-08T04:52:47.549399+0000","last_fullsized":"2023-11-08T04:52:47.549399+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":10,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-11-08T04:52:47.543946+0000","last_change":"2023-11-08T04:52:46.545601+0000","last_active":"2023-11-08T04:52:47.543946+0000","last_peered":"2023-11-08T04:52:47.543946+0000","last_clean":"2023-11-08T04:52:47.543946+0000","last_became_active":"2023-11-08T04:52:46.545321+0000","last_became_peered":"2023-11-08T04:52:46.545321+0000","last_unstale":"2023-11-08T04:52:47.543946+0000","last_undegraded":"2023-11-08T04:52:47.543946+0000","last_fullsized":"2023-11-08T04:52:47.543946+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_clean_scrub_stamp":"2023-11-08T04:52:45.528796+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-11-08T04:52:47.549543+0000","last_change":"2023-11-08T04:52:47.548597+0000","last_active":"2023-11-08T04:52:47.549543+0000","last_peered":"2023-11-08T04:52:47.549543+0000","last_clean":"2023-11-08T04:52:47.549543+0000","last_became_active":"2023-11-08T04:52:47.548294+0000","last_became_peered":"2023-11-08T04:52:47.548294+0000","last_unstale":"2023-11-08T04:52:47.549543+0000","last_undegraded":"2023-11-08T04:52:47.549543+0000","last_fullsized":"2023-11-08T04:52:47.549543+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-11-08T04:52:47.549368+0000","last_change":"2023-11-08T04:52:47.546100+0000","last_active":"2023-11-08T04:52:47.549368+0000","last_peered":"2023-11-08T04:52:47.549368+0000","last_clean":"2023-11-08T04:52:47.549368+0000","last_became_active":"2023-11-08T04:52:47.545840+0000","last_became_peered":"2023-11-08T04:52:47.545840+0000","last_unstale":"2023-11-08T04:52:47.549368+0000","last_undegraded":"2023-11-08T04:52:47.549368+0000","last_fullsized":"2023-11-08T04:52:47.549368+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-11-08T04:52:47.549373+0000","last_change":"2023-11-08T04:52:47.546027+0000","last_active":"2023-11-08T04:52:47.549373+0000","last_peered":"2023-11-08T04:52:47.549373+0000","last_clean":"2023-11-08T04:52:47.549373+0000","last_became_active":"2023-11-08T04:52:47.545761+0000","last_became_peered":"2023-11-08T04:52:47.545761+0000","last_unstale":"2023-11-08T04:52:47.549373+0000","last_undegraded":"2023-11-08T04:52:47.549373+0000","last_fullsized":"2023-11-08T04:52:47.549373+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-11-08T04:52:47.549505+0000","last_change":"2023-11-08T04:52:47.548575+0000","last_active":"2023-11-08T04:52:47.549505+0000","last_peered":"2023-11-08T04:52:47.549505+0000","last_clean":"2023-11-08T04:52:47.549505+0000","last_became_active":"2023-11-08T04:52:47.548265+0000","last_became_peered":"2023-11-08T04:52:47.548265+0000","last_unstale":"2023-11-08T04:52:47.549505+0000","last_undegraded":"2023-11-08T04:52:47.549505+0000","last_fullsized":"2023-11-08T04:52:47.549505+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738372,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297196,"kb_used_data":100,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94074644,"statfs":{"total":96636764160,"available":96332435456,"internally_reserved":0,"allocated":102400,"data_stored":28685,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.24099999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.23200000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738373,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297200,"kb_used_data":104,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94074640,"statfs":{"total":96636764160,"available":96332431360,"internally_reserved":0,"allocated":106496,"data_stored":29654,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27700000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.22600000000000001}]}]},{"osd":1,"up_from":8,"seq":34359738372,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297200,"kb_used_data":104,"kb_used_omap":0,"kb_used_meta":297088,"kb_avail":94074640,"statfs":{"total":96636764160,"available":96332431360,"internally_reserved":0,"allocated":106496,"data_stored":29654,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304218112},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36499999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32600000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-08T04:52:51.753 INFO:tasks.ceph.ceph_manager.cluster2:clean! 2023-11-08T04:52:51.753 INFO:tasks.ceph:Waiting until ceph cluster cluster2 is healthy... 2023-11-08T04:52:51.753 INFO:tasks.ceph.ceph_manager.cluster2:wait_until_healthy 2023-11-08T04:52:51.754 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 health --format=json 2023-11-08T04:52:52.073 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T04:52:52.085 INFO:teuthology.orchestra.run.smithi087.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2023-11-08T04:52:52.086 INFO:tasks.ceph.ceph_manager.cluster2:wait_until_healthy done 2023-11-08T04:52:52.087 INFO:teuthology.run_tasks:Running task exec... 2023-11-08T04:52:52.097 INFO:teuthology.task.exec:Executing custom commands... 2023-11-08T04:52:52.097 INFO:teuthology.task.exec:Running commands on role cluster1.client.mirror.0 host ubuntu@smithi087.front.sepia.ceph.com 2023-11-08T04:52:52.098 DEBUG:teuthology.orchestra.run.smithi087:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror mon '"'"'profile rbd-mirror-peer'"'"' osd '"'"'profile rbd'"'"'' 2023-11-08T04:52:52.458 INFO:teuthology.orchestra.run.smithi087.stderr:updated caps for client.mirror 2023-11-08T04:52:52.470 DEBUG:teuthology.orchestra.run.smithi087:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror.0 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2023-11-08T04:52:52.827 INFO:teuthology.orchestra.run.smithi087.stderr:updated caps for client.mirror.0 2023-11-08T04:52:52.839 DEBUG:teuthology.orchestra.run.smithi087:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror.1 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2023-11-08T04:52:53.197 INFO:teuthology.orchestra.run.smithi087.stderr:updated caps for client.mirror.1 2023-11-08T04:52:53.210 DEBUG:teuthology.orchestra.run.smithi087:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror.2 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2023-11-08T04:52:53.574 INFO:teuthology.orchestra.run.smithi087.stderr:updated caps for client.mirror.2 2023-11-08T04:52:53.585 DEBUG:teuthology.orchestra.run.smithi087:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror.3 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2023-11-08T04:52:53.945 INFO:teuthology.orchestra.run.smithi087.stderr:updated caps for client.mirror.3 2023-11-08T04:52:53.956 DEBUG:teuthology.orchestra.run.smithi087:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror mon '"'"'profile rbd-mirror-peer'"'"' osd '"'"'profile rbd'"'"'' 2023-11-08T04:52:54.315 INFO:teuthology.orchestra.run.smithi087.stderr:updated caps for client.mirror 2023-11-08T04:52:54.326 DEBUG:teuthology.orchestra.run.smithi087:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror.0 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2023-11-08T04:52:54.690 INFO:teuthology.orchestra.run.smithi087.stderr:updated caps for client.mirror.0 2023-11-08T04:52:54.702 DEBUG:teuthology.orchestra.run.smithi087:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror.1 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2023-11-08T04:52:55.063 INFO:teuthology.orchestra.run.smithi087.stderr:updated caps for client.mirror.1 2023-11-08T04:52:55.075 DEBUG:teuthology.orchestra.run.smithi087:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror.2 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2023-11-08T04:52:55.438 INFO:teuthology.orchestra.run.smithi087.stderr:updated caps for client.mirror.2 2023-11-08T04:52:55.449 DEBUG:teuthology.orchestra.run.smithi087:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror.3 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2023-11-08T04:52:55.813 INFO:teuthology.orchestra.run.smithi087.stderr:updated caps for client.mirror.3 2023-11-08T04:52:55.824 INFO:teuthology.run_tasks:Running task rbd-mirror... 2023-11-08T04:52:55.835 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2023-11-08T04:52:55.836 DEBUG:teuthology.orchestra.run.smithi087:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2023-11-08T04:52:55.839 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2023-11-08T04:52:55.839 INFO:teuthology.run_tasks:Running task rbd-mirror... 2023-11-08T04:52:55.850 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Restarting daemon 2023-11-08T04:52:55.850 DEBUG:teuthology.orchestra.run.smithi087:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 2023-11-08T04:52:55.852 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Started 2023-11-08T04:52:55.853 INFO:teuthology.run_tasks:Running task rbd-mirror... 2023-11-08T04:52:55.865 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Restarting daemon 2023-11-08T04:52:55.865 DEBUG:teuthology.orchestra.run.smithi087:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 2023-11-08T04:52:55.868 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Started 2023-11-08T04:52:55.868 INFO:teuthology.run_tasks:Running task rbd-mirror... 2023-11-08T04:52:55.884 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Restarting daemon 2023-11-08T04:52:55.884 DEBUG:teuthology.orchestra.run.smithi087:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.3 2023-11-08T04:52:55.887 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Started 2023-11-08T04:52:55.887 INFO:teuthology.run_tasks:Running task rbd-mirror... 2023-11-08T04:52:55.900 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Restarting daemon 2023-11-08T04:52:55.900 DEBUG:teuthology.orchestra.run.smithi087:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 2023-11-08T04:52:55.902 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Started 2023-11-08T04:52:55.903 INFO:teuthology.run_tasks:Running task rbd-mirror... 2023-11-08T04:52:55.916 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Restarting daemon 2023-11-08T04:52:55.916 DEBUG:teuthology.orchestra.run.smithi087:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 2023-11-08T04:52:55.919 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Started 2023-11-08T04:52:55.920 INFO:teuthology.run_tasks:Running task rbd-mirror... 2023-11-08T04:52:55.933 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Restarting daemon 2023-11-08T04:52:55.934 DEBUG:teuthology.orchestra.run.smithi087:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 2023-11-08T04:52:55.936 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Started 2023-11-08T04:52:55.936 INFO:teuthology.run_tasks:Running task rbd-mirror... 2023-11-08T04:52:55.951 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Restarting daemon 2023-11-08T04:52:55.951 DEBUG:teuthology.orchestra.run.smithi087:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3 2023-11-08T04:52:55.954 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Started 2023-11-08T04:52:55.955 INFO:teuthology.run_tasks:Running task rbd-mirror-thrash... 2023-11-08T04:52:55.968 INFO:tasks.rbd_mirror_thrash:rbd_mirror_thrash using random seed: 1699419175 2023-11-08T04:52:55.969 DEBUG:tasks.rbd_mirror_thrash:Yielding 2023-11-08T04:52:55.969 INFO:teuthology.run_tasks:Running task rbd-mirror-thrash... 2023-11-08T04:52:55.984 INFO:tasks.rbd_mirror_thrash:rbd_mirror_thrash using random seed: 1699419175 2023-11-08T04:52:55.985 DEBUG:tasks.rbd_mirror_thrash:Yielding 2023-11-08T04:52:55.985 INFO:teuthology.run_tasks:Running task workunit... 2023-11-08T04:52:56.004 INFO:tasks.workunit:Pulling workunits from ref cfb6fd52113be6a0af3fe7fb89b2e08474eadc0f 2023-11-08T04:52:56.004 INFO:tasks.workunit:Making a separate scratch dir for every client... 2023-11-08T04:52:56.005 DEBUG:teuthology.orchestra.run.smithi087:> stat -- /home/ubuntu/cephtest/mnt.cluster1.mirror 2023-11-08T04:52:56.005 INFO:tasks.rbd_mirror_thrash:starting thrash for cluster cluster1 2023-11-08T04:52:56.006 INFO:tasks.rbd_mirror_thrash:waiting for 107 secs before thrashing 2023-11-08T04:52:56.006 INFO:tasks.rbd_mirror_thrash:starting thrash for cluster cluster2 2023-11-08T04:52:56.006 INFO:tasks.rbd_mirror_thrash:waiting for 106 secs before thrashing 2023-11-08T04:52:56.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-08T04:52:56.024 INFO:teuthology.orchestra.run.smithi087.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.cluster1.mirror': No such file or directory 2023-11-08T04:52:56.024 DEBUG:teuthology.orchestra.run.smithi087:> mkdir -- /home/ubuntu/cephtest/mnt.cluster1.mirror 2023-11-08T04:52:56.083 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.cluster1.mirror 2023-11-08T04:52:56.083 DEBUG:teuthology.orchestra.run.smithi087:> cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror && mkdir -- client.mirror 2023-11-08T04:52:56.140 INFO:tasks.workunit:timeout=6h 2023-11-08T04:52:56.140 INFO:tasks.workunit:cleanup=True 2023-11-08T04:52:56.141 DEBUG:teuthology.orchestra.run.smithi087:> rm -rf /home/ubuntu/cephtest/clone.cluster1.client.mirror && git clone https://github.com/ceph/ceph-ci.git /home/ubuntu/cephtest/clone.cluster1.client.mirror && cd /home/ubuntu/cephtest/clone.cluster1.client.mirror && git checkout cfb6fd52113be6a0af3fe7fb89b2e08474eadc0f 2023-11-08T04:52:56.198 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:Cloning into '/home/ubuntu/cephtest/clone.cluster1.client.mirror'... 2023-11-08T04:53:42.766 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:Note: switching to 'cfb6fd52113be6a0af3fe7fb89b2e08474eadc0f'. 2023-11-08T04:53:42.766 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: 2023-11-08T04:53:42.767 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2023-11-08T04:53:42.767 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:changes and commit them, and you can discard any commits you make in this 2023-11-08T04:53:42.767 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:state without impacting any branches by switching back to a branch. 2023-11-08T04:53:42.767 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: 2023-11-08T04:53:42.767 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:If you want to create a new branch to retain commits you create, you may 2023-11-08T04:53:42.768 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:do so (now or later) by using -c with the switch command. Example: 2023-11-08T04:53:42.768 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: 2023-11-08T04:53:42.768 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: git switch -c 2023-11-08T04:53:42.768 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: 2023-11-08T04:53:42.769 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:Or undo this operation with: 2023-11-08T04:53:42.769 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: 2023-11-08T04:53:42.769 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: git switch - 2023-11-08T04:53:42.769 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: 2023-11-08T04:53:42.769 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2023-11-08T04:53:42.770 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: 2023-11-08T04:53:42.770 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:HEAD is now at cfb6fd52113 Merge branch 'wip-63385-pacific' of https://github.com/ajarr/ceph into wip-yuri3-testing-2023-11-07-0801-pacific 2023-11-08T04:53:42.771 DEBUG:teuthology.orchestra.run.smithi087:> cd -- /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.cluster1.client.mirror 2023-11-08T04:53:42.837 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2023-11-08T04:53:42.839 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/direct_io' 2023-11-08T04:53:42.839 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2023-11-08T04:53:43.406 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2023-11-08T04:53:43.465 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2023-11-08T04:53:43.503 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/direct_io' 2023-11-08T04:53:43.504 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/fs' 2023-11-08T04:53:43.504 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2023-11-08T04:53:43.541 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/fs' 2023-11-08T04:53:43.544 DEBUG:teuthology.orchestra.run.smithi087:> set -ex 2023-11-08T04:53:43.545 DEBUG:teuthology.orchestra.run.smithi087:> dd if=/home/ubuntu/cephtest/workunits.list.cluster1.client.mirror of=/dev/stdout 2023-11-08T04:53:43.603 INFO:tasks.workunit:Running workunits matching rbd/rbd_mirror_stress.sh on cluster1.client.mirror... 2023-11-08T04:53:43.604 INFO:tasks.workunit:Running workunit rbd/rbd_mirror_stress.sh... 2023-11-08T04:53:43.605 DEBUG:teuthology.orchestra.run.smithi087:workunit test rbd/rbd_mirror_stress.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=cfb6fd52113be6a0af3fe7fb89b2e08474eadc0f TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS='' MIRROR_IMAGE_MODE=snapshot MIRROR_POOL_MODE=image RBD_IMAGE_FEATURES=layering,exclusive-lock RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1 RBD_MIRROR_USE_RBD_MIRROR=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_stress.sh 2023-11-08T04:53:43.664 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ IMAGE_COUNT=50 2023-11-08T04:53:43.664 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ export LOCKDEP=0 2023-11-08T04:53:43.664 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ LOCKDEP=0 2023-11-08T04:53:43.665 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ dirname /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_stress.sh 2023-11-08T04:53:43.666 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ . /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_helpers.sh 2023-11-08T04:53:43.667 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ type xmlstarlet 2023-11-08T04:53:43.667 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ XMLSTARLET=xmlstarlet 2023-11-08T04:53:43.667 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ RBD_MIRROR_INSTANCES=4 2023-11-08T04:53:43.667 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ CLUSTER1=cluster1 2023-11-08T04:53:43.667 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ CLUSTER2=cluster2 2023-11-08T04:53:43.668 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ PEER_CLUSTER_SUFFIX= 2023-11-08T04:53:43.668 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ POOL=mirror 2023-11-08T04:53:43.668 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ PARENT_POOL=mirror_parent 2023-11-08T04:53:43.668 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ NS1=ns1 2023-11-08T04:53:43.668 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ NS2=ns2 2023-11-08T04:53:43.669 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ TEMPDIR= 2023-11-08T04:53:43.669 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ CEPH_ID=mirror 2023-11-08T04:53:43.669 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ RBD_IMAGE_FEATURES=layering,exclusive-lock 2023-11-08T04:53:43.669 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ MIRROR_USER_ID_PREFIX=mirror. 2023-11-08T04:53:43.669 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ MIRROR_POOL_MODE=image 2023-11-08T04:53:43.669 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ MIRROR_IMAGE_MODE=snapshot 2023-11-08T04:53:43.670 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ export 'CEPH_ARGS=--id mirror' 2023-11-08T04:53:43.670 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ CEPH_ARGS='--id mirror' 2023-11-08T04:53:43.670 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ LAST_MIRROR_INSTANCE=3 2023-11-08T04:53:43.670 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ dirname /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_stress.sh 2023-11-08T04:53:43.670 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ readlink -f /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/../../../src 2023-11-08T04:53:43.672 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror/src 2023-11-08T04:53:43.672 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ CEPH_BIN=. 2023-11-08T04:53:43.672 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ CEPH_SRC=. 2023-11-08T04:53:43.673 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' -e CMakeCache.txt ']' 2023-11-08T04:53:43.673 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' 0 -gt 0 ']' 2023-11-08T04:53:43.673 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ setup 2023-11-08T04:53:43.673 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local c 2023-11-08T04:53:43.673 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ trap 'cleanup $?' INT TERM EXIT 2023-11-08T04:53:43.673 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ setup_tempdir 2023-11-08T04:53:43.674 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' -n '' ']' 2023-11-08T04:53:43.674 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ mktemp -d 2023-11-08T04:53:43.674 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ TEMPDIR=/tmp/tmp.Ludk8ikCAB 2023-11-08T04:53:43.674 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' -z 1 ']' 2023-11-08T04:53:43.674 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ setup_pools cluster1 cluster2 2023-11-08T04:53:43.675 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T04:53:43.675 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local remote_cluster=cluster2 2023-11-08T04:53:43.675 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local mon_map_file 2023-11-08T04:53:43.675 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local mon_addr 2023-11-08T04:53:43.675 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local admin_key_file 2023-11-08T04:53:43.676 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local uuid 2023-11-08T04:53:43.676 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ CEPH_ARGS= 2023-11-08T04:53:43.676 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ ceph --cluster cluster1 osd pool create mirror 64 64 2023-11-08T04:53:44.382 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:pool 'mirror' already exists 2023-11-08T04:53:44.391 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ CEPH_ARGS= 2023-11-08T04:53:44.391 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ ceph --cluster cluster1 osd pool create mirror_parent 64 64 2023-11-08T04:53:45.385 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:pool 'mirror_parent' already exists 2023-11-08T04:53:45.394 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ CEPH_ARGS= 2023-11-08T04:53:45.394 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 pool init mirror 2023-11-08T04:53:48.303 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ CEPH_ARGS= 2023-11-08T04:53:48.303 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 pool init mirror_parent 2023-11-08T04:53:51.306 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' -n '' ']' 2023-11-08T04:53:51.306 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ CEPH_ARGS= 2023-11-08T04:53:51.306 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 mirror pool enable --site-name cluster1 mirror image 2023-11-08T04:53:51.352 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 mirror pool enable mirror_parent image 2023-11-08T04:53:51.388 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 namespace create mirror/ns1 2023-11-08T04:53:51.419 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 namespace create mirror/ns2 2023-11-08T04:53:51.448 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 mirror pool enable mirror/ns1 image 2023-11-08T04:53:51.480 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 mirror pool enable mirror/ns2 image 2023-11-08T04:53:51.514 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' -z ']' 2023-11-08T04:53:51.514 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' -z ']' 2023-11-08T04:53:51.514 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ peer_add cluster1 mirror cluster2 2023-11-08T04:53:51.514 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T04:53:51.515 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T04:53:51.516 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T04:53:51.516 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T04:53:51.516 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local client_cluster=cluster2 2023-11-08T04:53:51.516 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T04:53:51.516 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local remote_cluster=cluster2 2023-11-08T04:53:51.516 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local uuid_var_name 2023-11-08T04:53:51.517 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' -n '' ']' 2023-11-08T04:53:51.517 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local error_code 2023-11-08T04:53:51.517 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local peer_uuid 2023-11-08T04:53:51.517 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 1 2 4 8 16 32 2023-11-08T04:53:51.517 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ set +e 2023-11-08T04:53:51.518 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ rbd --cluster cluster1 mirror pool peer add mirror cluster2 2023-11-08T04:53:51.545 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ peer_uuid=ced8946a-4f5f-45fb-85a7-96dfecfa648e 2023-11-08T04:53:51.545 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ error_code=0 2023-11-08T04:53:51.545 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ set -e 2023-11-08T04:53:51.546 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' 0 -eq 17 ']' 2023-11-08T04:53:51.546 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test 0 -eq 0 2023-11-08T04:53:51.546 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' -n '' ']' 2023-11-08T04:53:51.546 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T04:53:51.546 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ peer_add cluster1 mirror_parent cluster2 2023-11-08T04:53:51.547 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T04:53:51.547 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T04:53:51.547 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror_parent 2023-11-08T04:53:51.547 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T04:53:51.547 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local client_cluster=cluster2 2023-11-08T04:53:51.548 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T04:53:51.548 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local remote_cluster=cluster2 2023-11-08T04:53:51.548 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local uuid_var_name 2023-11-08T04:53:51.548 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' -n '' ']' 2023-11-08T04:53:51.548 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local error_code 2023-11-08T04:53:51.549 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local peer_uuid 2023-11-08T04:53:51.549 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 1 2 4 8 16 32 2023-11-08T04:53:51.549 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ set +e 2023-11-08T04:53:51.549 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ rbd --cluster cluster1 mirror pool peer add mirror_parent cluster2 2023-11-08T04:53:51.576 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ peer_uuid=9db76edd-b9f1-4835-b9a2-df269f7d6c96 2023-11-08T04:53:51.576 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ error_code=0 2023-11-08T04:53:51.576 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ set -e 2023-11-08T04:53:51.577 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' 0 -eq 17 ']' 2023-11-08T04:53:51.577 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test 0 -eq 0 2023-11-08T04:53:51.577 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' -n '' ']' 2023-11-08T04:53:51.577 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T04:53:51.577 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ setup_pools cluster2 cluster1 2023-11-08T04:53:51.578 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T04:53:51.578 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local remote_cluster=cluster1 2023-11-08T04:53:51.578 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local mon_map_file 2023-11-08T04:53:51.578 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local mon_addr 2023-11-08T04:53:51.578 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local admin_key_file 2023-11-08T04:53:51.579 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local uuid 2023-11-08T04:53:51.579 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ CEPH_ARGS= 2023-11-08T04:53:51.579 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ ceph --cluster cluster2 osd pool create mirror 64 64 2023-11-08T04:53:52.503 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:pool 'mirror' already exists 2023-11-08T04:53:52.512 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ CEPH_ARGS= 2023-11-08T04:53:52.512 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ ceph --cluster cluster2 osd pool create mirror_parent 64 64 2023-11-08T04:53:53.528 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:pool 'mirror_parent' already exists 2023-11-08T04:53:53.538 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ CEPH_ARGS= 2023-11-08T04:53:53.538 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 pool init mirror 2023-11-08T04:53:55.983 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi087.stderr:2023-11-08T04:53:55.981+0000 7f11e23146c0 -1 rbd::mirror::PoolReplayer: 0x55d5fc1e8000 init: failed to initialize remote pool poller: (2) No such file or directory 2023-11-08T04:53:55.986 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi087.stderr:2023-11-08T04:53:55.985+0000 7fa17c5ef6c0 -1 rbd::mirror::PoolReplayer: 0x55f33da06000 init: failed to initialize remote pool poller: (2) No such file or directory 2023-11-08T04:53:56.022 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi087.stderr:2023-11-08T04:53:56.020+0000 7f11e23146c0 -1 rbd::mirror::PoolReplayer: 0x55d5fd8e2c00 init: failed to initialize remote pool poller: (2) No such file or directory 2023-11-08T04:53:56.023 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi087.stderr:2023-11-08T04:53:56.021+0000 7f7470c8d6c0 -1 rbd::mirror::PoolReplayer: 0x55c27b2c6000 init: failed to initialize remote pool poller: (2) No such file or directory 2023-11-08T04:53:56.033 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi087.stderr:2023-11-08T04:53:56.031+0000 7f89df1fc6c0 -1 rbd::mirror::PoolReplayer: 0x55c78f86a000 init: failed to initialize remote pool poller: (2) No such file or directory 2023-11-08T04:53:56.067 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi087.stderr:2023-11-08T04:53:56.066+0000 7f7470c8d6c0 -1 rbd::mirror::PoolReplayer: 0x55c27c9bcc00 init: failed to initialize remote pool poller: (2) No such file or directory 2023-11-08T04:53:56.228 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi087.stderr:2023-11-08T04:53:56.227+0000 7fa17c5ef6c0 -1 rbd::mirror::PoolReplayer: 0x55f33f100c00 init: failed to initialize remote pool poller: (2) No such file or directory 2023-11-08T04:53:56.273 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi087.stderr:2023-11-08T04:53:56.271+0000 7f89df1fc6c0 -1 rbd::mirror::PoolReplayer: 0x55c790f64c00 init: failed to initialize remote pool poller: (2) No such file or directory 2023-11-08T04:53:56.416 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ CEPH_ARGS= 2023-11-08T04:53:56.417 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 pool init mirror_parent 2023-11-08T04:53:59.430 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' -n '' ']' 2023-11-08T04:53:59.430 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ CEPH_ARGS= 2023-11-08T04:53:59.430 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 mirror pool enable --site-name cluster2 mirror image 2023-11-08T04:53:59.477 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 mirror pool enable mirror_parent image 2023-11-08T04:53:59.511 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 namespace create mirror/ns1 2023-11-08T04:53:59.542 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 namespace create mirror/ns2 2023-11-08T04:53:59.572 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 mirror pool enable mirror/ns1 image 2023-11-08T04:53:59.605 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 mirror pool enable mirror/ns2 image 2023-11-08T04:53:59.637 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' -z ']' 2023-11-08T04:53:59.637 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' -z ']' 2023-11-08T04:53:59.638 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ peer_add cluster2 mirror cluster1 2023-11-08T04:53:59.638 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T04:53:59.638 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T04:53:59.638 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T04:53:59.639 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T04:53:59.639 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local client_cluster=cluster1 2023-11-08T04:53:59.639 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T04:53:59.639 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local remote_cluster=cluster1 2023-11-08T04:53:59.639 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local uuid_var_name 2023-11-08T04:53:59.639 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' -n '' ']' 2023-11-08T04:53:59.640 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local error_code 2023-11-08T04:53:59.640 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local peer_uuid 2023-11-08T04:53:59.640 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 1 2 4 8 16 32 2023-11-08T04:53:59.640 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ set +e 2023-11-08T04:53:59.640 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ rbd --cluster cluster2 mirror pool peer add mirror cluster1 2023-11-08T04:53:59.668 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ peer_uuid=238fee2f-658f-47b2-b895-7fcda39fcdeb 2023-11-08T04:53:59.668 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ error_code=0 2023-11-08T04:53:59.668 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ set -e 2023-11-08T04:53:59.668 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' 0 -eq 17 ']' 2023-11-08T04:53:59.669 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test 0 -eq 0 2023-11-08T04:53:59.669 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' -n '' ']' 2023-11-08T04:53:59.669 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T04:53:59.669 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ peer_add cluster2 mirror_parent cluster1 2023-11-08T04:53:59.669 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T04:53:59.669 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T04:53:59.670 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror_parent 2023-11-08T04:53:59.670 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T04:53:59.670 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local client_cluster=cluster1 2023-11-08T04:53:59.670 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T04:53:59.670 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local remote_cluster=cluster1 2023-11-08T04:53:59.671 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local uuid_var_name 2023-11-08T04:53:59.671 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' -n '' ']' 2023-11-08T04:53:59.671 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local error_code 2023-11-08T04:53:59.671 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local peer_uuid 2023-11-08T04:53:59.671 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 1 2 4 8 16 32 2023-11-08T04:53:59.672 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ set +e 2023-11-08T04:53:59.672 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ rbd --cluster cluster2 mirror pool peer add mirror_parent cluster1 2023-11-08T04:53:59.698 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ peer_uuid=aacae26e-b1ab-4692-a5f3-96d468b14914 2023-11-08T04:53:59.698 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ error_code=0 2023-11-08T04:53:59.698 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ set -e 2023-11-08T04:53:59.699 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' 0 -eq 17 ']' 2023-11-08T04:53:59.699 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test 0 -eq 0 2023-11-08T04:53:59.699 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' -n '' ']' 2023-11-08T04:53:59.699 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T04:53:59.699 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' -n '' ']' 2023-11-08T04:53:59.700 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ start_mirrors cluster1 2023-11-08T04:53:59.700 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T04:53:59.700 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ seq 0 3 2023-11-08T04:53:59.716 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T04:53:59.716 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ start_mirror cluster1:0 2023-11-08T04:53:59.717 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1:0 2023-11-08T04:53:59.717 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local instance 2023-11-08T04:53:59.717 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-11-08T04:53:59.717 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local val=cluster1:0 2023-11-08T04:53:59.717 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster_var_name=cluster 2023-11-08T04:53:59.717 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local instance_var_name=instance 2023-11-08T04:53:59.718 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ cluster=cluster1 2023-11-08T04:53:59.718 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ instance=0 2023-11-08T04:53:59.718 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' 0 = cluster1:0 ']' 2023-11-08T04:53:59.718 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ eval cluster=cluster1 2023-11-08T04:53:59.718 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cluster=cluster1 2023-11-08T04:53:59.719 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ eval instance=0 2023-11-08T04:53:59.719 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ instance=0 2023-11-08T04:53:59.719 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test -n 1 2023-11-08T04:53:59.719 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 2023-11-08T04:53:59.719 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T04:53:59.720 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ start_mirror cluster1:1 2023-11-08T04:53:59.720 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1:1 2023-11-08T04:53:59.720 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local instance 2023-11-08T04:53:59.720 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-11-08T04:53:59.720 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local val=cluster1:1 2023-11-08T04:53:59.721 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster_var_name=cluster 2023-11-08T04:53:59.721 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local instance_var_name=instance 2023-11-08T04:53:59.721 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ cluster=cluster1 2023-11-08T04:53:59.721 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ instance=1 2023-11-08T04:53:59.721 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' 1 = cluster1:1 ']' 2023-11-08T04:53:59.722 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ eval cluster=cluster1 2023-11-08T04:53:59.722 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cluster=cluster1 2023-11-08T04:53:59.722 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ eval instance=1 2023-11-08T04:53:59.722 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ instance=1 2023-11-08T04:53:59.722 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test -n 1 2023-11-08T04:53:59.723 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 2023-11-08T04:53:59.723 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T04:53:59.723 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ start_mirror cluster1:2 2023-11-08T04:53:59.723 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1:2 2023-11-08T04:53:59.723 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local instance 2023-11-08T04:53:59.723 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ set_cluster_instance cluster1:2 cluster instance 2023-11-08T04:53:59.724 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local val=cluster1:2 2023-11-08T04:53:59.724 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster_var_name=cluster 2023-11-08T04:53:59.724 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local instance_var_name=instance 2023-11-08T04:53:59.724 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ cluster=cluster1 2023-11-08T04:53:59.724 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ instance=2 2023-11-08T04:53:59.725 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' 2 = cluster1:2 ']' 2023-11-08T04:53:59.725 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ eval cluster=cluster1 2023-11-08T04:53:59.725 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cluster=cluster1 2023-11-08T04:53:59.725 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ eval instance=2 2023-11-08T04:53:59.725 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ instance=2 2023-11-08T04:53:59.725 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test -n 1 2023-11-08T04:53:59.726 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 2023-11-08T04:53:59.726 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T04:53:59.726 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ start_mirror cluster1:3 2023-11-08T04:53:59.726 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1:3 2023-11-08T04:53:59.726 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local instance 2023-11-08T04:53:59.727 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ set_cluster_instance cluster1:3 cluster instance 2023-11-08T04:53:59.727 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local val=cluster1:3 2023-11-08T04:53:59.727 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster_var_name=cluster 2023-11-08T04:53:59.727 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local instance_var_name=instance 2023-11-08T04:53:59.727 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ cluster=cluster1 2023-11-08T04:53:59.727 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ instance=3 2023-11-08T04:53:59.728 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' 3 = cluster1:3 ']' 2023-11-08T04:53:59.728 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ eval cluster=cluster1 2023-11-08T04:53:59.728 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cluster=cluster1 2023-11-08T04:53:59.728 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ eval instance=3 2023-11-08T04:53:59.728 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ instance=3 2023-11-08T04:53:59.728 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test -n 1 2023-11-08T04:53:59.729 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 2023-11-08T04:53:59.729 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ start_mirrors cluster2 2023-11-08T04:53:59.729 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T04:53:59.729 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ seq 0 3 2023-11-08T04:53:59.729 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T04:53:59.729 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ start_mirror cluster2:0 2023-11-08T04:53:59.730 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2:0 2023-11-08T04:53:59.730 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local instance 2023-11-08T04:53:59.730 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ set_cluster_instance cluster2:0 cluster instance 2023-11-08T04:53:59.730 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local val=cluster2:0 2023-11-08T04:53:59.730 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster_var_name=cluster 2023-11-08T04:53:59.731 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local instance_var_name=instance 2023-11-08T04:53:59.731 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ cluster=cluster2 2023-11-08T04:53:59.731 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ instance=0 2023-11-08T04:53:59.731 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' 0 = cluster2:0 ']' 2023-11-08T04:53:59.731 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ eval cluster=cluster2 2023-11-08T04:53:59.731 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cluster=cluster2 2023-11-08T04:53:59.732 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ eval instance=0 2023-11-08T04:53:59.732 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ instance=0 2023-11-08T04:53:59.732 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test -n 1 2023-11-08T04:53:59.732 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 2023-11-08T04:53:59.732 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T04:53:59.733 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ start_mirror cluster2:1 2023-11-08T04:53:59.733 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2:1 2023-11-08T04:53:59.733 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local instance 2023-11-08T04:53:59.733 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ set_cluster_instance cluster2:1 cluster instance 2023-11-08T04:53:59.733 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local val=cluster2:1 2023-11-08T04:53:59.734 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster_var_name=cluster 2023-11-08T04:53:59.734 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local instance_var_name=instance 2023-11-08T04:53:59.734 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ cluster=cluster2 2023-11-08T04:53:59.734 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ instance=1 2023-11-08T04:53:59.734 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' 1 = cluster2:1 ']' 2023-11-08T04:53:59.735 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ eval cluster=cluster2 2023-11-08T04:53:59.735 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cluster=cluster2 2023-11-08T04:53:59.735 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ eval instance=1 2023-11-08T04:53:59.735 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ instance=1 2023-11-08T04:53:59.735 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test -n 1 2023-11-08T04:53:59.735 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 2023-11-08T04:53:59.736 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T04:53:59.736 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ start_mirror cluster2:2 2023-11-08T04:53:59.736 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2:2 2023-11-08T04:53:59.736 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local instance 2023-11-08T04:53:59.736 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ set_cluster_instance cluster2:2 cluster instance 2023-11-08T04:53:59.737 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local val=cluster2:2 2023-11-08T04:53:59.737 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster_var_name=cluster 2023-11-08T04:53:59.737 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local instance_var_name=instance 2023-11-08T04:53:59.737 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ cluster=cluster2 2023-11-08T04:53:59.737 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ instance=2 2023-11-08T04:53:59.737 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' 2 = cluster2:2 ']' 2023-11-08T04:53:59.738 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ eval cluster=cluster2 2023-11-08T04:53:59.738 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cluster=cluster2 2023-11-08T04:53:59.738 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ eval instance=2 2023-11-08T04:53:59.738 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ instance=2 2023-11-08T04:53:59.738 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test -n 1 2023-11-08T04:53:59.739 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 2023-11-08T04:53:59.739 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T04:53:59.739 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ start_mirror cluster2:3 2023-11-08T04:53:59.739 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2:3 2023-11-08T04:53:59.739 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local instance 2023-11-08T04:53:59.740 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ set_cluster_instance cluster2:3 cluster instance 2023-11-08T04:53:59.740 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local val=cluster2:3 2023-11-08T04:53:59.740 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster_var_name=cluster 2023-11-08T04:53:59.740 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local instance_var_name=instance 2023-11-08T04:53:59.740 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ cluster=cluster2 2023-11-08T04:53:59.740 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ instance=3 2023-11-08T04:53:59.741 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' 3 = cluster2:3 ']' 2023-11-08T04:53:59.741 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ eval cluster=cluster2 2023-11-08T04:53:59.741 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cluster=cluster2 2023-11-08T04:53:59.741 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ eval instance=3 2023-11-08T04:53:59.741 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ instance=3 2023-11-08T04:53:59.742 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test -n 1 2023-11-08T04:53:59.742 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 2023-11-08T04:53:59.742 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ testlog 'TEST: add image and test replay after client crashes' 2023-11-08T04:53:59.742 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ tee -a /tmp/tmp.Ludk8ikCAB/rbd-mirror.test.log 2023-11-08T04:53:59.742 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ date '+%F %T' 2023-11-08T04:53:59.743 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ echo 2023-11-08 04:53:59 TEST: add image and test replay after client crashes 2023-11-08T04:53:59.743 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:2023-11-08 04:53:59 TEST: add image and test replay after client crashes 2023-11-08T04:53:59.743 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image=test 2023-11-08T04:53:59.743 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ create_image_and_enable_mirror cluster2 mirror test snapshot 512M 2023-11-08T04:53:59.743 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T04:53:59.743 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T04:53:59.744 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T04:53:59.744 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T04:53:59.744 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T04:53:59.744 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T04:53:59.744 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local mode=snapshot 2023-11-08T04:53:59.745 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' -n snapshot ']' 2023-11-08T04:53:59.745 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T04:53:59.745 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ create_image cluster2 mirror test 512M 2023-11-08T04:53:59.745 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T04:53:59.745 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T04:53:59.746 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T04:53:59.746 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T04:53:59.746 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T04:53:59.746 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T04:53:59.746 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local size=128 2023-11-08T04:53:59.747 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' -n 512M ']' 2023-11-08T04:53:59.747 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ size=512M 2023-11-08T04:53:59.747 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T04:53:59.747 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock mirror/test 2023-11-08T04:53:59.748 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' image = image ']' 2023-11-08T04:53:59.748 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ enable_mirror cluster2 mirror test snapshot 2023-11-08T04:53:59.748 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T04:53:59.749 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T04:53:59.749 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T04:53:59.749 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local mode=snapshot 2023-11-08T04:53:59.749 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/test snapshot 2023-11-08T04:54:00.500 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:Mirroring enabled 2023-11-08T04:54:00.505 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster=cluster2 info mirror/test 2023-11-08T04:54:00.537 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:rbd image 'test': 2023-11-08T04:54:00.538 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: size 512 MiB in 128 objects 2023-11-08T04:54:00.538 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: order 22 (4 MiB objects) 2023-11-08T04:54:00.538 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: snapshot_count: 1 2023-11-08T04:54:00.538 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: id: 10a1a6009ec2 2023-11-08T04:54:00.539 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: block_name_prefix: rbd_data.10a1a6009ec2 2023-11-08T04:54:00.539 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: format: 2 2023-11-08T04:54:00.539 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: features: layering, exclusive-lock 2023-11-08T04:54:00.539 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: op_features: 2023-11-08T04:54:00.539 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: flags: 2023-11-08T04:54:00.540 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: create_timestamp: Wed Nov 8 04:53:59 2023 2023-11-08T04:54:00.540 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: access_timestamp: Wed Nov 8 04:53:59 2023 2023-11-08T04:54:00.540 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: modify_timestamp: Wed Nov 8 04:53:59 2023 2023-11-08T04:54:00.540 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring state: enabled 2023-11-08T04:54:00.540 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring mode: snapshot 2023-11-08T04:54:00.541 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring global id: 88ced7f9-2739-4a27-ad7a-30edec05edb1 2023-11-08T04:54:00.541 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring primary: true 2023-11-08T04:54:00.542 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_image_replay_started cluster1 mirror test 2023-11-08T04:54:00.542 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T04:54:00.542 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T04:54:00.543 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T04:54:00.543 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2023-11-08T04:54:00.543 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T04:54:00.543 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T04:54:00.543 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T04:54:00.544 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local state=started 2023-11-08T04:54:00.544 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local s 2023-11-08T04:54:00.544 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-11-08T04:54:00.544 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 1 2023-11-08T04:54:01.544 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test_image_replay_state cluster1 mirror test started 2023-11-08T04:54:01.544 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T04:54:01.545 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T04:54:01.545 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T04:54:01.545 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local test_state=started 2023-11-08T04:54:01.546 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local status_result 2023-11-08T04:54:01.546 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local current_state=stopped 2023-11-08T04:54:01.546 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2023-11-08T04:54:01.547 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_instance=cluster1 2023-11-08T04:54:01.547 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T04:54:01.548 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ grep -i state 2023-11-08T04:54:01.548 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1 2023-11-08T04:54:01.548 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance=cluster1 2023-11-08T04:54:01.549 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local loop_instance 2023-11-08T04:54:01.549 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-11-08T04:54:01.550 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sleep 0 2023-11-08T04:54:01.550 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-11-08T04:54:01.550 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ seq 0 3 2023-11-08T04:54:01.556 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T04:54:01.556 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2023-11-08T04:54:01.557 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1:0 2023-11-08T04:54:01.557 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T04:54:01.558 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance 2023-11-08T04:54:01.558 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-11-08T04:54:01.558 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local val=cluster1:0 2023-11-08T04:54:01.559 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_var_name=cluster 2023-11-08T04:54:01.559 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance_var_name=instance 2023-11-08T04:54:01.560 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cluster=cluster1 2023-11-08T04:54:01.560 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ instance=0 2023-11-08T04:54:01.560 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' 0 = cluster1:0 ']' 2023-11-08T04:54:01.561 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval cluster=cluster1 2023-11-08T04:54:01.561 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T04:54:01.561 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval instance=0 2023-11-08T04:54:01.562 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=0 2023-11-08T04:54:01.562 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-11-08T04:54:01.563 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local local_cluster=cluster1:0 2023-11-08T04:54:01.563 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster=cluster1 2023-11-08T04:54:01.563 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance 2023-11-08T04:54:01.564 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-11-08T04:54:01.564 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local val=cluster1:0 2023-11-08T04:54:01.565 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster_var_name=local_cluster 2023-11-08T04:54:01.565 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance_var_name=instance 2023-11-08T04:54:01.565 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T04:54:01.566 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=0 2023-11-08T04:54:01.566 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ '[' 0 = cluster1:0 ']' 2023-11-08T04:54:01.566 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval local_cluster=cluster1 2023-11-08T04:54:01.567 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ local_cluster=cluster1 2023-11-08T04:54:01.567 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval instance=0 2023-11-08T04:54:01.567 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ instance=0 2023-11-08T04:54:01.568 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-11-08T04:54:01.569 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-08T04:54:01.570 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-08T04:54:01.570 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-08T04:54:01.570 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2023-11-08T04:54:01.655 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-11-08T04:54:01.655 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd mirror status : get status for rbd mirror 2023-11-08T04:54:01.656 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:admin_socket: invalid command 2023-11-08T04:54:01.658 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T04:54:01.658 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2023-11-08T04:54:01.658 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1:1 2023-11-08T04:54:01.659 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T04:54:01.659 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance 2023-11-08T04:54:01.659 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-11-08T04:54:01.659 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local val=cluster1:1 2023-11-08T04:54:01.659 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_var_name=cluster 2023-11-08T04:54:01.660 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance_var_name=instance 2023-11-08T04:54:01.660 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cluster=cluster1 2023-11-08T04:54:01.660 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ instance=1 2023-11-08T04:54:01.660 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' 1 = cluster1:1 ']' 2023-11-08T04:54:01.660 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval cluster=cluster1 2023-11-08T04:54:01.660 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T04:54:01.661 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval instance=1 2023-11-08T04:54:01.661 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=1 2023-11-08T04:54:01.661 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-11-08T04:54:01.661 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local local_cluster=cluster1:1 2023-11-08T04:54:01.661 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster=cluster1 2023-11-08T04:54:01.662 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance 2023-11-08T04:54:01.662 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-11-08T04:54:01.662 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local val=cluster1:1 2023-11-08T04:54:01.662 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster_var_name=local_cluster 2023-11-08T04:54:01.662 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance_var_name=instance 2023-11-08T04:54:01.663 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T04:54:01.663 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=1 2023-11-08T04:54:01.663 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ '[' 1 = cluster1:1 ']' 2023-11-08T04:54:01.663 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval local_cluster=cluster1 2023-11-08T04:54:01.663 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ local_cluster=cluster1 2023-11-08T04:54:01.664 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval instance=1 2023-11-08T04:54:01.664 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ instance=1 2023-11-08T04:54:01.664 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-11-08T04:54:01.672 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-08T04:54:01.672 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-08T04:54:01.673 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-08T04:54:01.673 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2023-11-08T04:54:01.758 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-11-08T04:54:01.758 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd mirror status : get status for rbd mirror 2023-11-08T04:54:01.759 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:admin_socket: invalid command 2023-11-08T04:54:01.761 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T04:54:01.762 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2023-11-08T04:54:01.762 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1:2 2023-11-08T04:54:01.762 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T04:54:01.762 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance 2023-11-08T04:54:01.762 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-11-08T04:54:01.763 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local val=cluster1:2 2023-11-08T04:54:01.763 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_var_name=cluster 2023-11-08T04:54:01.763 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance_var_name=instance 2023-11-08T04:54:01.763 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cluster=cluster1 2023-11-08T04:54:01.763 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ instance=2 2023-11-08T04:54:01.764 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' 2 = cluster1:2 ']' 2023-11-08T04:54:01.764 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval cluster=cluster1 2023-11-08T04:54:01.764 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T04:54:01.764 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval instance=2 2023-11-08T04:54:01.764 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=2 2023-11-08T04:54:01.765 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-11-08T04:54:01.765 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local local_cluster=cluster1:2 2023-11-08T04:54:01.765 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster=cluster1 2023-11-08T04:54:01.765 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance 2023-11-08T04:54:01.765 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-11-08T04:54:01.765 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local val=cluster1:2 2023-11-08T04:54:01.766 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster_var_name=local_cluster 2023-11-08T04:54:01.766 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance_var_name=instance 2023-11-08T04:54:01.766 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T04:54:01.766 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=2 2023-11-08T04:54:01.766 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ '[' 2 = cluster1:2 ']' 2023-11-08T04:54:01.767 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval local_cluster=cluster1 2023-11-08T04:54:01.767 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ local_cluster=cluster1 2023-11-08T04:54:01.767 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval instance=2 2023-11-08T04:54:01.767 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ instance=2 2023-11-08T04:54:01.767 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-11-08T04:54:01.775 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-08T04:54:01.775 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-08T04:54:01.775 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-08T04:54:01.775 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2023-11-08T04:54:01.860 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-11-08T04:54:01.861 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd mirror status : get status for rbd mirror 2023-11-08T04:54:01.861 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:admin_socket: invalid command 2023-11-08T04:54:01.864 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T04:54:01.864 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2023-11-08T04:54:01.864 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1:3 2023-11-08T04:54:01.864 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T04:54:01.864 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance 2023-11-08T04:54:01.865 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ set_cluster_instance cluster1:3 cluster instance 2023-11-08T04:54:01.865 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local val=cluster1:3 2023-11-08T04:54:01.865 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_var_name=cluster 2023-11-08T04:54:01.865 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance_var_name=instance 2023-11-08T04:54:01.865 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cluster=cluster1 2023-11-08T04:54:01.866 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ instance=3 2023-11-08T04:54:01.866 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' 3 = cluster1:3 ']' 2023-11-08T04:54:01.866 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval cluster=cluster1 2023-11-08T04:54:01.866 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T04:54:01.866 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval instance=3 2023-11-08T04:54:01.867 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=3 2023-11-08T04:54:01.867 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ daemon_asok_file cluster1:3 cluster1 2023-11-08T04:54:01.867 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local local_cluster=cluster1:3 2023-11-08T04:54:01.867 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster=cluster1 2023-11-08T04:54:01.867 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance 2023-11-08T04:54:01.868 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2023-11-08T04:54:01.868 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local val=cluster1:3 2023-11-08T04:54:01.868 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster_var_name=local_cluster 2023-11-08T04:54:01.868 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance_var_name=instance 2023-11-08T04:54:01.868 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T04:54:01.868 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=3 2023-11-08T04:54:01.869 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ '[' 3 = cluster1:3 ']' 2023-11-08T04:54:01.869 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval local_cluster=cluster1 2023-11-08T04:54:01.869 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ local_cluster=cluster1 2023-11-08T04:54:01.869 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval instance=3 2023-11-08T04:54:01.869 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ instance=3 2023-11-08T04:54:01.870 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2023-11-08T04:54:01.877 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-11-08T04:54:01.878 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-11-08T04:54:01.878 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-11-08T04:54:01.878 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2023-11-08T04:54:01.962 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-11-08T04:54:01.963 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd mirror status : get status for rbd mirror 2023-11-08T04:54:01.963 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:admin_socket: invalid command 2023-11-08T04:54:01.963 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-11-08T04:54:01.964 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sleep 1 2023-11-08T04:54:02.964 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-11-08T04:54:02.965 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ seq 0 3 2023-11-08T04:54:02.968 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T04:54:02.968 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2023-11-08T04:54:02.968 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1:0 2023-11-08T04:54:02.969 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T04:54:02.969 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance 2023-11-08T04:54:02.969 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-11-08T04:54:02.969 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local val=cluster1:0 2023-11-08T04:54:02.969 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_var_name=cluster 2023-11-08T04:54:02.969 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance_var_name=instance 2023-11-08T04:54:02.970 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cluster=cluster1 2023-11-08T04:54:02.970 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ instance=0 2023-11-08T04:54:02.970 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' 0 = cluster1:0 ']' 2023-11-08T04:54:02.970 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval cluster=cluster1 2023-11-08T04:54:02.970 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T04:54:02.971 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval instance=0 2023-11-08T04:54:02.971 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=0 2023-11-08T04:54:02.971 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-11-08T04:54:02.971 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local local_cluster=cluster1:0 2023-11-08T04:54:02.971 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster=cluster1 2023-11-08T04:54:02.972 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance 2023-11-08T04:54:02.972 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-11-08T04:54:02.972 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local val=cluster1:0 2023-11-08T04:54:02.972 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster_var_name=local_cluster 2023-11-08T04:54:02.972 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance_var_name=instance 2023-11-08T04:54:02.973 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T04:54:02.973 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=0 2023-11-08T04:54:02.973 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ '[' 0 = cluster1:0 ']' 2023-11-08T04:54:02.974 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval local_cluster=cluster1 2023-11-08T04:54:02.974 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ local_cluster=cluster1 2023-11-08T04:54:02.974 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval instance=0 2023-11-08T04:54:02.974 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ instance=0 2023-11-08T04:54:02.975 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-11-08T04:54:02.984 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-08T04:54:02.985 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-08T04:54:02.985 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-08T04:54:02.985 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2023-11-08T04:54:03.077 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-11-08T04:54:03.078 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd mirror status : get status for rbd mirror 2023-11-08T04:54:03.078 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:admin_socket: invalid command 2023-11-08T04:54:03.081 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T04:54:03.081 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2023-11-08T04:54:03.081 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1:1 2023-11-08T04:54:03.081 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T04:54:03.081 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance 2023-11-08T04:54:03.082 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-11-08T04:54:03.082 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local val=cluster1:1 2023-11-08T04:54:03.082 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_var_name=cluster 2023-11-08T04:54:03.082 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance_var_name=instance 2023-11-08T04:54:03.082 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cluster=cluster1 2023-11-08T04:54:03.083 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ instance=1 2023-11-08T04:54:03.083 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' 1 = cluster1:1 ']' 2023-11-08T04:54:03.083 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval cluster=cluster1 2023-11-08T04:54:03.083 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T04:54:03.083 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval instance=1 2023-11-08T04:54:03.084 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=1 2023-11-08T04:54:03.084 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-11-08T04:54:03.084 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local local_cluster=cluster1:1 2023-11-08T04:54:03.084 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster=cluster1 2023-11-08T04:54:03.084 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance 2023-11-08T04:54:03.084 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-11-08T04:54:03.085 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local val=cluster1:1 2023-11-08T04:54:03.085 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster_var_name=local_cluster 2023-11-08T04:54:03.085 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance_var_name=instance 2023-11-08T04:54:03.085 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T04:54:03.085 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=1 2023-11-08T04:54:03.086 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ '[' 1 = cluster1:1 ']' 2023-11-08T04:54:03.086 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval local_cluster=cluster1 2023-11-08T04:54:03.086 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ local_cluster=cluster1 2023-11-08T04:54:03.086 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval instance=1 2023-11-08T04:54:03.086 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ instance=1 2023-11-08T04:54:03.087 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-11-08T04:54:03.095 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-08T04:54:03.095 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-08T04:54:03.095 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-08T04:54:03.095 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2023-11-08T04:54:03.180 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-11-08T04:54:03.180 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd mirror status : get status for rbd mirror 2023-11-08T04:54:03.181 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:admin_socket: invalid command 2023-11-08T04:54:03.183 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T04:54:03.183 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2023-11-08T04:54:03.183 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1:2 2023-11-08T04:54:03.184 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T04:54:03.184 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance 2023-11-08T04:54:03.184 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-11-08T04:54:03.184 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local val=cluster1:2 2023-11-08T04:54:03.184 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_var_name=cluster 2023-11-08T04:54:03.185 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance_var_name=instance 2023-11-08T04:54:03.185 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cluster=cluster1 2023-11-08T04:54:03.185 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ instance=2 2023-11-08T04:54:03.185 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' 2 = cluster1:2 ']' 2023-11-08T04:54:03.185 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval cluster=cluster1 2023-11-08T04:54:03.186 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T04:54:03.186 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval instance=2 2023-11-08T04:54:03.186 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=2 2023-11-08T04:54:03.187 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-11-08T04:54:03.187 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local local_cluster=cluster1:2 2023-11-08T04:54:03.187 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster=cluster1 2023-11-08T04:54:03.187 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance 2023-11-08T04:54:03.187 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-11-08T04:54:03.188 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local val=cluster1:2 2023-11-08T04:54:03.188 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster_var_name=local_cluster 2023-11-08T04:54:03.188 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance_var_name=instance 2023-11-08T04:54:03.188 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T04:54:03.188 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=2 2023-11-08T04:54:03.189 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ '[' 2 = cluster1:2 ']' 2023-11-08T04:54:03.189 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval local_cluster=cluster1 2023-11-08T04:54:03.189 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ local_cluster=cluster1 2023-11-08T04:54:03.189 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval instance=2 2023-11-08T04:54:03.189 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ instance=2 2023-11-08T04:54:03.190 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-11-08T04:54:03.197 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-08T04:54:03.198 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-08T04:54:03.198 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-08T04:54:03.198 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2023-11-08T04:54:03.282 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-11-08T04:54:03.282 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd mirror status : get status for rbd mirror 2023-11-08T04:54:03.282 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:admin_socket: invalid command 2023-11-08T04:54:03.285 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T04:54:03.285 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2023-11-08T04:54:03.285 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1:3 2023-11-08T04:54:03.285 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T04:54:03.286 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance 2023-11-08T04:54:03.286 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ set_cluster_instance cluster1:3 cluster instance 2023-11-08T04:54:03.286 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local val=cluster1:3 2023-11-08T04:54:03.286 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_var_name=cluster 2023-11-08T04:54:03.286 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance_var_name=instance 2023-11-08T04:54:03.287 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cluster=cluster1 2023-11-08T04:54:03.287 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ instance=3 2023-11-08T04:54:03.287 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' 3 = cluster1:3 ']' 2023-11-08T04:54:03.287 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval cluster=cluster1 2023-11-08T04:54:03.287 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T04:54:03.288 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval instance=3 2023-11-08T04:54:03.288 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=3 2023-11-08T04:54:03.288 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ daemon_asok_file cluster1:3 cluster1 2023-11-08T04:54:03.288 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local local_cluster=cluster1:3 2023-11-08T04:54:03.288 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster=cluster1 2023-11-08T04:54:03.289 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance 2023-11-08T04:54:03.289 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2023-11-08T04:54:03.289 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local val=cluster1:3 2023-11-08T04:54:03.289 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster_var_name=local_cluster 2023-11-08T04:54:03.289 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance_var_name=instance 2023-11-08T04:54:03.290 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T04:54:03.290 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=3 2023-11-08T04:54:03.290 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ '[' 3 = cluster1:3 ']' 2023-11-08T04:54:03.290 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval local_cluster=cluster1 2023-11-08T04:54:03.290 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ local_cluster=cluster1 2023-11-08T04:54:03.291 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval instance=3 2023-11-08T04:54:03.291 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ instance=3 2023-11-08T04:54:03.291 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2023-11-08T04:54:03.299 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-11-08T04:54:03.299 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-11-08T04:54:03.299 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-11-08T04:54:03.300 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2023-11-08T04:54:03.384 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-11-08T04:54:03.384 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd mirror status : get status for rbd mirror 2023-11-08T04:54:03.385 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:admin_socket: invalid command 2023-11-08T04:54:03.385 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-11-08T04:54:03.385 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sleep 2 2023-11-08T04:54:05.386 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-11-08T04:54:05.387 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ seq 0 3 2023-11-08T04:54:05.391 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T04:54:05.392 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2023-11-08T04:54:05.392 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1:0 2023-11-08T04:54:05.392 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T04:54:05.393 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance 2023-11-08T04:54:05.393 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-11-08T04:54:05.393 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local val=cluster1:0 2023-11-08T04:54:05.394 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_var_name=cluster 2023-11-08T04:54:05.394 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance_var_name=instance 2023-11-08T04:54:05.394 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cluster=cluster1 2023-11-08T04:54:05.395 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ instance=0 2023-11-08T04:54:05.395 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' 0 = cluster1:0 ']' 2023-11-08T04:54:05.395 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval cluster=cluster1 2023-11-08T04:54:05.396 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T04:54:05.396 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval instance=0 2023-11-08T04:54:05.396 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=0 2023-11-08T04:54:05.397 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-11-08T04:54:05.397 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local local_cluster=cluster1:0 2023-11-08T04:54:05.397 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster=cluster1 2023-11-08T04:54:05.397 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance 2023-11-08T04:54:05.398 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-11-08T04:54:05.398 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local val=cluster1:0 2023-11-08T04:54:05.398 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster_var_name=local_cluster 2023-11-08T04:54:05.398 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance_var_name=instance 2023-11-08T04:54:05.398 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T04:54:05.399 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=0 2023-11-08T04:54:05.399 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ '[' 0 = cluster1:0 ']' 2023-11-08T04:54:05.399 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval local_cluster=cluster1 2023-11-08T04:54:05.399 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ local_cluster=cluster1 2023-11-08T04:54:05.399 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval instance=0 2023-11-08T04:54:05.399 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ instance=0 2023-11-08T04:54:05.400 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-11-08T04:54:05.406 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-08T04:54:05.406 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-08T04:54:05.407 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-08T04:54:05.407 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2023-11-08T04:54:05.499 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-11-08T04:54:05.499 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd mirror status : get status for rbd mirror 2023-11-08T04:54:05.500 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:admin_socket: invalid command 2023-11-08T04:54:05.503 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T04:54:05.503 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2023-11-08T04:54:05.503 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1:1 2023-11-08T04:54:05.503 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T04:54:05.504 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance 2023-11-08T04:54:05.504 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-11-08T04:54:05.504 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local val=cluster1:1 2023-11-08T04:54:05.504 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_var_name=cluster 2023-11-08T04:54:05.504 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance_var_name=instance 2023-11-08T04:54:05.505 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cluster=cluster1 2023-11-08T04:54:05.505 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ instance=1 2023-11-08T04:54:05.505 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' 1 = cluster1:1 ']' 2023-11-08T04:54:05.505 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval cluster=cluster1 2023-11-08T04:54:05.506 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T04:54:05.506 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval instance=1 2023-11-08T04:54:05.506 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=1 2023-11-08T04:54:05.506 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-11-08T04:54:05.506 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local local_cluster=cluster1:1 2023-11-08T04:54:05.507 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster=cluster1 2023-11-08T04:54:05.507 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance 2023-11-08T04:54:05.507 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-11-08T04:54:05.507 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local val=cluster1:1 2023-11-08T04:54:05.507 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster_var_name=local_cluster 2023-11-08T04:54:05.508 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance_var_name=instance 2023-11-08T04:54:05.508 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T04:54:05.508 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=1 2023-11-08T04:54:05.509 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ '[' 1 = cluster1:1 ']' 2023-11-08T04:54:05.509 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval local_cluster=cluster1 2023-11-08T04:54:05.509 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ local_cluster=cluster1 2023-11-08T04:54:05.509 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval instance=1 2023-11-08T04:54:05.510 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ instance=1 2023-11-08T04:54:05.510 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-11-08T04:54:05.516 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-08T04:54:05.516 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-08T04:54:05.516 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-08T04:54:05.517 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2023-11-08T04:54:05.601 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-11-08T04:54:05.602 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd mirror status : get status for rbd mirror 2023-11-08T04:54:05.602 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:admin_socket: invalid command 2023-11-08T04:54:05.605 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T04:54:05.605 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2023-11-08T04:54:05.605 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1:2 2023-11-08T04:54:05.606 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T04:54:05.606 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance 2023-11-08T04:54:05.606 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-11-08T04:54:05.606 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local val=cluster1:2 2023-11-08T04:54:05.606 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_var_name=cluster 2023-11-08T04:54:05.607 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance_var_name=instance 2023-11-08T04:54:05.607 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cluster=cluster1 2023-11-08T04:54:05.607 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ instance=2 2023-11-08T04:54:05.607 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' 2 = cluster1:2 ']' 2023-11-08T04:54:05.607 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval cluster=cluster1 2023-11-08T04:54:05.608 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T04:54:05.608 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval instance=2 2023-11-08T04:54:05.608 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=2 2023-11-08T04:54:05.608 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-11-08T04:54:05.609 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local local_cluster=cluster1:2 2023-11-08T04:54:05.609 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster=cluster1 2023-11-08T04:54:05.609 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance 2023-11-08T04:54:05.609 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-11-08T04:54:05.609 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local val=cluster1:2 2023-11-08T04:54:05.609 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster_var_name=local_cluster 2023-11-08T04:54:05.610 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance_var_name=instance 2023-11-08T04:54:05.610 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T04:54:05.610 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=2 2023-11-08T04:54:05.610 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ '[' 2 = cluster1:2 ']' 2023-11-08T04:54:05.611 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval local_cluster=cluster1 2023-11-08T04:54:05.611 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ local_cluster=cluster1 2023-11-08T04:54:05.611 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval instance=2 2023-11-08T04:54:05.611 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ instance=2 2023-11-08T04:54:05.611 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-11-08T04:54:05.618 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-08T04:54:05.618 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-08T04:54:05.619 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-08T04:54:05.619 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2023-11-08T04:54:05.703 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-11-08T04:54:05.703 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd mirror status : get status for rbd mirror 2023-11-08T04:54:05.703 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:admin_socket: invalid command 2023-11-08T04:54:05.706 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T04:54:05.707 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2023-11-08T04:54:05.707 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1:3 2023-11-08T04:54:05.707 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T04:54:05.707 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance 2023-11-08T04:54:05.707 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ set_cluster_instance cluster1:3 cluster instance 2023-11-08T04:54:05.708 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local val=cluster1:3 2023-11-08T04:54:05.708 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_var_name=cluster 2023-11-08T04:54:05.708 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance_var_name=instance 2023-11-08T04:54:05.708 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cluster=cluster1 2023-11-08T04:54:05.708 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ instance=3 2023-11-08T04:54:05.709 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' 3 = cluster1:3 ']' 2023-11-08T04:54:05.709 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval cluster=cluster1 2023-11-08T04:54:05.709 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T04:54:05.709 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval instance=3 2023-11-08T04:54:05.709 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=3 2023-11-08T04:54:05.710 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ daemon_asok_file cluster1:3 cluster1 2023-11-08T04:54:05.710 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local local_cluster=cluster1:3 2023-11-08T04:54:05.710 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster=cluster1 2023-11-08T04:54:05.710 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance 2023-11-08T04:54:05.711 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2023-11-08T04:54:05.711 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local val=cluster1:3 2023-11-08T04:54:05.711 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster_var_name=local_cluster 2023-11-08T04:54:05.711 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance_var_name=instance 2023-11-08T04:54:05.711 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T04:54:05.711 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=3 2023-11-08T04:54:05.712 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ '[' 3 = cluster1:3 ']' 2023-11-08T04:54:05.712 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval local_cluster=cluster1 2023-11-08T04:54:05.712 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ local_cluster=cluster1 2023-11-08T04:54:05.712 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval instance=3 2023-11-08T04:54:05.712 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ instance=3 2023-11-08T04:54:05.713 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2023-11-08T04:54:05.720 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-11-08T04:54:05.720 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-11-08T04:54:05.720 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-11-08T04:54:05.720 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2023-11-08T04:54:05.805 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-11-08T04:54:05.805 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd mirror status : get status for rbd mirror 2023-11-08T04:54:05.805 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:admin_socket: invalid command 2023-11-08T04:54:05.806 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-11-08T04:54:05.806 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sleep 4 2023-11-08T04:54:09.807 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-11-08T04:54:09.807 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ seq 0 3 2023-11-08T04:54:09.810 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T04:54:09.810 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2023-11-08T04:54:09.811 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1:0 2023-11-08T04:54:09.811 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T04:54:09.811 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance 2023-11-08T04:54:09.811 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-11-08T04:54:09.811 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local val=cluster1:0 2023-11-08T04:54:09.812 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_var_name=cluster 2023-11-08T04:54:09.812 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance_var_name=instance 2023-11-08T04:54:09.812 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cluster=cluster1 2023-11-08T04:54:09.812 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ instance=0 2023-11-08T04:54:09.812 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' 0 = cluster1:0 ']' 2023-11-08T04:54:09.813 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval cluster=cluster1 2023-11-08T04:54:09.813 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T04:54:09.813 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval instance=0 2023-11-08T04:54:09.813 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=0 2023-11-08T04:54:09.813 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-11-08T04:54:09.814 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local local_cluster=cluster1:0 2023-11-08T04:54:09.814 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster=cluster1 2023-11-08T04:54:09.814 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance 2023-11-08T04:54:09.814 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-11-08T04:54:09.814 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local val=cluster1:0 2023-11-08T04:54:09.815 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster_var_name=local_cluster 2023-11-08T04:54:09.815 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance_var_name=instance 2023-11-08T04:54:09.815 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T04:54:09.815 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=0 2023-11-08T04:54:09.815 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ '[' 0 = cluster1:0 ']' 2023-11-08T04:54:09.816 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval local_cluster=cluster1 2023-11-08T04:54:09.816 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ local_cluster=cluster1 2023-11-08T04:54:09.816 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval instance=0 2023-11-08T04:54:09.816 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ instance=0 2023-11-08T04:54:09.816 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-11-08T04:54:09.827 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-08T04:54:09.827 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-08T04:54:09.827 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-08T04:54:09.827 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2023-11-08T04:54:09.920 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-11-08T04:54:09.920 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd mirror status : get status for rbd mirror 2023-11-08T04:54:09.920 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:admin_socket: invalid command 2023-11-08T04:54:09.923 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T04:54:09.923 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2023-11-08T04:54:09.923 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1:1 2023-11-08T04:54:09.923 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T04:54:09.923 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance 2023-11-08T04:54:09.924 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-11-08T04:54:09.924 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local val=cluster1:1 2023-11-08T04:54:09.924 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_var_name=cluster 2023-11-08T04:54:09.924 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance_var_name=instance 2023-11-08T04:54:09.924 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cluster=cluster1 2023-11-08T04:54:09.925 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ instance=1 2023-11-08T04:54:09.925 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' 1 = cluster1:1 ']' 2023-11-08T04:54:09.925 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval cluster=cluster1 2023-11-08T04:54:09.925 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T04:54:09.925 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval instance=1 2023-11-08T04:54:09.926 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=1 2023-11-08T04:54:09.926 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-11-08T04:54:09.926 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local local_cluster=cluster1:1 2023-11-08T04:54:09.926 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster=cluster1 2023-11-08T04:54:09.926 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance 2023-11-08T04:54:09.927 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-11-08T04:54:09.927 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local val=cluster1:1 2023-11-08T04:54:09.927 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster_var_name=local_cluster 2023-11-08T04:54:09.927 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance_var_name=instance 2023-11-08T04:54:09.927 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T04:54:09.927 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=1 2023-11-08T04:54:09.928 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ '[' 1 = cluster1:1 ']' 2023-11-08T04:54:09.928 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval local_cluster=cluster1 2023-11-08T04:54:09.928 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ local_cluster=cluster1 2023-11-08T04:54:09.928 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval instance=1 2023-11-08T04:54:09.928 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ instance=1 2023-11-08T04:54:09.929 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-11-08T04:54:09.937 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-08T04:54:09.937 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-08T04:54:09.937 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-08T04:54:09.937 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2023-11-08T04:54:10.023 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-11-08T04:54:10.023 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd mirror status : get status for rbd mirror 2023-11-08T04:54:10.023 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:admin_socket: invalid command 2023-11-08T04:54:10.026 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T04:54:10.027 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2023-11-08T04:54:10.027 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1:2 2023-11-08T04:54:10.027 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T04:54:10.028 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance 2023-11-08T04:54:10.028 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-11-08T04:54:10.028 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local val=cluster1:2 2023-11-08T04:54:10.028 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_var_name=cluster 2023-11-08T04:54:10.028 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance_var_name=instance 2023-11-08T04:54:10.029 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cluster=cluster1 2023-11-08T04:54:10.029 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ instance=2 2023-11-08T04:54:10.029 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' 2 = cluster1:2 ']' 2023-11-08T04:54:10.029 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval cluster=cluster1 2023-11-08T04:54:10.030 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T04:54:10.030 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval instance=2 2023-11-08T04:54:10.030 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=2 2023-11-08T04:54:10.030 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-11-08T04:54:10.030 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local local_cluster=cluster1:2 2023-11-08T04:54:10.031 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster=cluster1 2023-11-08T04:54:10.031 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance 2023-11-08T04:54:10.031 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-11-08T04:54:10.031 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local val=cluster1:2 2023-11-08T04:54:10.031 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster_var_name=local_cluster 2023-11-08T04:54:10.031 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance_var_name=instance 2023-11-08T04:54:10.032 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T04:54:10.032 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=2 2023-11-08T04:54:10.032 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ '[' 2 = cluster1:2 ']' 2023-11-08T04:54:10.032 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval local_cluster=cluster1 2023-11-08T04:54:10.032 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ local_cluster=cluster1 2023-11-08T04:54:10.033 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval instance=2 2023-11-08T04:54:10.033 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ instance=2 2023-11-08T04:54:10.033 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-11-08T04:54:10.039 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-08T04:54:10.039 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-08T04:54:10.039 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-08T04:54:10.040 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2023-11-08T04:54:10.124 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-11-08T04:54:10.125 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd mirror status : get status for rbd mirror 2023-11-08T04:54:10.125 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:admin_socket: invalid command 2023-11-08T04:54:10.128 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T04:54:10.128 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2023-11-08T04:54:10.129 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1:3 2023-11-08T04:54:10.129 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T04:54:10.129 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance 2023-11-08T04:54:10.129 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ set_cluster_instance cluster1:3 cluster instance 2023-11-08T04:54:10.129 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local val=cluster1:3 2023-11-08T04:54:10.130 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_var_name=cluster 2023-11-08T04:54:10.130 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance_var_name=instance 2023-11-08T04:54:10.130 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cluster=cluster1 2023-11-08T04:54:10.130 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ instance=3 2023-11-08T04:54:10.131 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' 3 = cluster1:3 ']' 2023-11-08T04:54:10.131 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval cluster=cluster1 2023-11-08T04:54:10.131 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T04:54:10.131 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval instance=3 2023-11-08T04:54:10.131 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=3 2023-11-08T04:54:10.132 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ daemon_asok_file cluster1:3 cluster1 2023-11-08T04:54:10.132 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local local_cluster=cluster1:3 2023-11-08T04:54:10.132 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster=cluster1 2023-11-08T04:54:10.132 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance 2023-11-08T04:54:10.132 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2023-11-08T04:54:10.133 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local val=cluster1:3 2023-11-08T04:54:10.133 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster_var_name=local_cluster 2023-11-08T04:54:10.133 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance_var_name=instance 2023-11-08T04:54:10.134 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T04:54:10.134 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=3 2023-11-08T04:54:10.134 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ '[' 3 = cluster1:3 ']' 2023-11-08T04:54:10.135 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval local_cluster=cluster1 2023-11-08T04:54:10.135 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ local_cluster=cluster1 2023-11-08T04:54:10.135 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval instance=3 2023-11-08T04:54:10.136 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ instance=3 2023-11-08T04:54:10.136 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2023-11-08T04:54:10.141 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-11-08T04:54:10.141 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-11-08T04:54:10.141 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-11-08T04:54:10.141 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2023-11-08T04:54:10.226 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-11-08T04:54:10.226 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd mirror status : get status for rbd mirror 2023-11-08T04:54:10.226 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:admin_socket: invalid command 2023-11-08T04:54:10.227 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-11-08T04:54:10.227 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sleep 8 2023-11-08T04:54:18.228 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-11-08T04:54:18.228 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ seq 0 3 2023-11-08T04:54:18.232 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T04:54:18.232 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2023-11-08T04:54:18.232 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1:0 2023-11-08T04:54:18.233 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T04:54:18.233 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance 2023-11-08T04:54:18.233 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-11-08T04:54:18.233 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local val=cluster1:0 2023-11-08T04:54:18.233 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_var_name=cluster 2023-11-08T04:54:18.234 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance_var_name=instance 2023-11-08T04:54:18.234 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cluster=cluster1 2023-11-08T04:54:18.234 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ instance=0 2023-11-08T04:54:18.234 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' 0 = cluster1:0 ']' 2023-11-08T04:54:18.234 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval cluster=cluster1 2023-11-08T04:54:18.235 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T04:54:18.235 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval instance=0 2023-11-08T04:54:18.235 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=0 2023-11-08T04:54:18.235 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-11-08T04:54:18.236 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local local_cluster=cluster1:0 2023-11-08T04:54:18.236 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster=cluster1 2023-11-08T04:54:18.236 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance 2023-11-08T04:54:18.236 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-11-08T04:54:18.236 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local val=cluster1:0 2023-11-08T04:54:18.237 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster_var_name=local_cluster 2023-11-08T04:54:18.237 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance_var_name=instance 2023-11-08T04:54:18.237 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T04:54:18.237 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=0 2023-11-08T04:54:18.237 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ '[' 0 = cluster1:0 ']' 2023-11-08T04:54:18.238 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval local_cluster=cluster1 2023-11-08T04:54:18.238 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ local_cluster=cluster1 2023-11-08T04:54:18.238 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval instance=0 2023-11-08T04:54:18.238 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ instance=0 2023-11-08T04:54:18.238 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-11-08T04:54:18.248 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-08T04:54:18.248 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-08T04:54:18.249 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-08T04:54:18.249 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2023-11-08T04:54:18.341 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-11-08T04:54:18.341 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd mirror status : get status for rbd mirror 2023-11-08T04:54:18.342 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:admin_socket: invalid command 2023-11-08T04:54:18.347 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T04:54:18.347 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2023-11-08T04:54:18.347 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1:1 2023-11-08T04:54:18.347 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T04:54:18.348 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance 2023-11-08T04:54:18.348 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-11-08T04:54:18.348 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local val=cluster1:1 2023-11-08T04:54:18.349 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_var_name=cluster 2023-11-08T04:54:18.349 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance_var_name=instance 2023-11-08T04:54:18.349 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cluster=cluster1 2023-11-08T04:54:18.350 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ instance=1 2023-11-08T04:54:18.350 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' 1 = cluster1:1 ']' 2023-11-08T04:54:18.350 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval cluster=cluster1 2023-11-08T04:54:18.351 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T04:54:18.351 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval instance=1 2023-11-08T04:54:18.351 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=1 2023-11-08T04:54:18.351 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-11-08T04:54:18.351 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local local_cluster=cluster1:1 2023-11-08T04:54:18.352 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster=cluster1 2023-11-08T04:54:18.352 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance 2023-11-08T04:54:18.352 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-11-08T04:54:18.352 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local val=cluster1:1 2023-11-08T04:54:18.352 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster_var_name=local_cluster 2023-11-08T04:54:18.353 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance_var_name=instance 2023-11-08T04:54:18.353 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T04:54:18.353 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=1 2023-11-08T04:54:18.353 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ '[' 1 = cluster1:1 ']' 2023-11-08T04:54:18.353 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval local_cluster=cluster1 2023-11-08T04:54:18.354 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ local_cluster=cluster1 2023-11-08T04:54:18.354 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval instance=1 2023-11-08T04:54:18.354 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ instance=1 2023-11-08T04:54:18.354 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-11-08T04:54:18.359 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-08T04:54:18.359 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-08T04:54:18.360 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-08T04:54:18.360 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2023-11-08T04:54:18.447 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-11-08T04:54:18.447 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd mirror status : get status for rbd mirror 2023-11-08T04:54:18.448 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:admin_socket: invalid command 2023-11-08T04:54:18.452 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T04:54:18.452 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2023-11-08T04:54:18.452 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1:2 2023-11-08T04:54:18.452 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T04:54:18.452 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance 2023-11-08T04:54:18.453 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-11-08T04:54:18.453 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local val=cluster1:2 2023-11-08T04:54:18.453 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_var_name=cluster 2023-11-08T04:54:18.453 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance_var_name=instance 2023-11-08T04:54:18.454 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cluster=cluster1 2023-11-08T04:54:18.454 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ instance=2 2023-11-08T04:54:18.454 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' 2 = cluster1:2 ']' 2023-11-08T04:54:18.454 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval cluster=cluster1 2023-11-08T04:54:18.455 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T04:54:18.455 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval instance=2 2023-11-08T04:54:18.455 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=2 2023-11-08T04:54:18.455 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-11-08T04:54:18.455 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local local_cluster=cluster1:2 2023-11-08T04:54:18.455 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster=cluster1 2023-11-08T04:54:18.456 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance 2023-11-08T04:54:18.456 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-11-08T04:54:18.456 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local val=cluster1:2 2023-11-08T04:54:18.456 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster_var_name=local_cluster 2023-11-08T04:54:18.456 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance_var_name=instance 2023-11-08T04:54:18.457 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T04:54:18.457 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=2 2023-11-08T04:54:18.457 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ '[' 2 = cluster1:2 ']' 2023-11-08T04:54:18.457 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval local_cluster=cluster1 2023-11-08T04:54:18.457 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ local_cluster=cluster1 2023-11-08T04:54:18.458 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval instance=2 2023-11-08T04:54:18.458 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ instance=2 2023-11-08T04:54:18.458 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-11-08T04:54:18.464 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-08T04:54:18.464 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-08T04:54:18.465 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-08T04:54:18.465 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2023-11-08T04:54:18.548 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-11-08T04:54:18.549 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd mirror status : get status for rbd mirror 2023-11-08T04:54:18.549 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:admin_socket: invalid command 2023-11-08T04:54:18.552 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T04:54:18.552 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2023-11-08T04:54:18.552 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1:3 2023-11-08T04:54:18.552 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T04:54:18.553 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance 2023-11-08T04:54:18.553 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ set_cluster_instance cluster1:3 cluster instance 2023-11-08T04:54:18.553 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local val=cluster1:3 2023-11-08T04:54:18.554 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_var_name=cluster 2023-11-08T04:54:18.554 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance_var_name=instance 2023-11-08T04:54:18.554 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cluster=cluster1 2023-11-08T04:54:18.554 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ instance=3 2023-11-08T04:54:18.555 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' 3 = cluster1:3 ']' 2023-11-08T04:54:18.555 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval cluster=cluster1 2023-11-08T04:54:18.555 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T04:54:18.555 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval instance=3 2023-11-08T04:54:18.555 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=3 2023-11-08T04:54:18.556 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ daemon_asok_file cluster1:3 cluster1 2023-11-08T04:54:18.556 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local local_cluster=cluster1:3 2023-11-08T04:54:18.556 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster=cluster1 2023-11-08T04:54:18.556 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance 2023-11-08T04:54:18.556 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2023-11-08T04:54:18.557 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local val=cluster1:3 2023-11-08T04:54:18.557 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster_var_name=local_cluster 2023-11-08T04:54:18.557 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance_var_name=instance 2023-11-08T04:54:18.557 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T04:54:18.557 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=3 2023-11-08T04:54:18.558 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ '[' 3 = cluster1:3 ']' 2023-11-08T04:54:18.558 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval local_cluster=cluster1 2023-11-08T04:54:18.558 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ local_cluster=cluster1 2023-11-08T04:54:18.558 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval instance=3 2023-11-08T04:54:18.558 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ instance=3 2023-11-08T04:54:18.559 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2023-11-08T04:54:18.565 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-11-08T04:54:18.566 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-11-08T04:54:18.566 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-11-08T04:54:18.566 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2023-11-08T04:54:18.650 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-11-08T04:54:18.651 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd mirror status : get status for rbd mirror 2023-11-08T04:54:18.651 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:admin_socket: invalid command 2023-11-08T04:54:18.651 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-11-08T04:54:18.652 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sleep 8 2023-11-08T04:54:26.030 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi087.stderr:2023-11-08T04:54:26.027+0000 7f11e23146c0 -1 rbd::mirror::Mirror: 0x55d5fb48f9e0 update_pool_replayers: restarting failed pool replayer for uuid: ced8946a-4f5f-45fb-85a7-96dfecfa648e cluster: cluster2 client: client.mirror 2023-11-08T04:54:26.080 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi087.stderr:2023-11-08T04:54:26.077+0000 7f7470c8d6c0 -1 rbd::mirror::Mirror: 0x55c27a55bb00 update_pool_replayers: restarting failed pool replayer for uuid: ced8946a-4f5f-45fb-85a7-96dfecfa648e cluster: cluster2 client: client.mirror 2023-11-08T04:54:26.112 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi087.stderr:2023-11-08T04:54:26.109+0000 7f11e23146c0 -1 rbd::mirror::Mirror: 0x55d5fb48f9e0 update_pool_replayers: restarting failed pool replayer for uuid: 9db76edd-b9f1-4835-b9a2-df269f7d6c96 cluster: cluster2 client: client.mirror 2023-11-08T04:54:26.165 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi087.stderr:2023-11-08T04:54:26.162+0000 7f7470c8d6c0 -1 rbd::mirror::Mirror: 0x55c27a55bb00 update_pool_replayers: restarting failed pool replayer for uuid: 9db76edd-b9f1-4835-b9a2-df269f7d6c96 cluster: cluster2 client: client.mirror 2023-11-08T04:54:26.235 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi087.stderr:2023-11-08T04:54:26.232+0000 7fa17c5ef6c0 -1 rbd::mirror::Mirror: 0x55f33cca39e0 update_pool_replayers: restarting failed pool replayer for uuid: ced8946a-4f5f-45fb-85a7-96dfecfa648e cluster: cluster2 client: client.mirror 2023-11-08T04:54:26.277 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi087.stderr:2023-11-08T04:54:26.274+0000 7fa17c5ef6c0 -1 rbd::mirror::Mirror: 0x55f33cca39e0 update_pool_replayers: restarting failed pool replayer for uuid: 9db76edd-b9f1-4835-b9a2-df269f7d6c96 cluster: cluster2 client: client.mirror 2023-11-08T04:54:26.282 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi087.stderr:2023-11-08T04:54:26.278+0000 7f89df1fc6c0 -1 rbd::mirror::Mirror: 0x55c78eaf99e0 update_pool_replayers: restarting failed pool replayer for uuid: ced8946a-4f5f-45fb-85a7-96dfecfa648e cluster: cluster2 client: client.mirror 2023-11-08T04:54:26.342 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi087.stderr:2023-11-08T04:54:26.339+0000 7f89df1fc6c0 -1 rbd::mirror::Mirror: 0x55c78eaf99e0 update_pool_replayers: restarting failed pool replayer for uuid: 9db76edd-b9f1-4835-b9a2-df269f7d6c96 cluster: cluster2 client: client.mirror 2023-11-08T04:54:26.652 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-11-08T04:54:26.653 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ seq 0 3 2023-11-08T04:54:26.656 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T04:54:26.656 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2023-11-08T04:54:26.656 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1:0 2023-11-08T04:54:26.657 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T04:54:26.657 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance 2023-11-08T04:54:26.657 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-11-08T04:54:26.657 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local val=cluster1:0 2023-11-08T04:54:26.657 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_var_name=cluster 2023-11-08T04:54:26.658 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance_var_name=instance 2023-11-08T04:54:26.658 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cluster=cluster1 2023-11-08T04:54:26.658 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ instance=0 2023-11-08T04:54:26.658 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' 0 = cluster1:0 ']' 2023-11-08T04:54:26.658 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval cluster=cluster1 2023-11-08T04:54:26.659 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T04:54:26.659 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval instance=0 2023-11-08T04:54:26.659 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=0 2023-11-08T04:54:26.659 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-11-08T04:54:26.659 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local local_cluster=cluster1:0 2023-11-08T04:54:26.659 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster=cluster1 2023-11-08T04:54:26.660 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance 2023-11-08T04:54:26.660 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-11-08T04:54:26.660 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local val=cluster1:0 2023-11-08T04:54:26.660 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster_var_name=local_cluster 2023-11-08T04:54:26.660 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance_var_name=instance 2023-11-08T04:54:26.661 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T04:54:26.661 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=0 2023-11-08T04:54:26.661 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ '[' 0 = cluster1:0 ']' 2023-11-08T04:54:26.661 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval local_cluster=cluster1 2023-11-08T04:54:26.661 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ local_cluster=cluster1 2023-11-08T04:54:26.662 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval instance=0 2023-11-08T04:54:26.662 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ instance=0 2023-11-08T04:54:26.662 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-11-08T04:54:26.670 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-08T04:54:26.670 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-08T04:54:26.670 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-08T04:54:26.670 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2023-11-08T04:54:26.757 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-11-08T04:54:26.758 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd mirror status : get status for rbd mirror 2023-11-08T04:54:26.758 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:admin_socket: invalid command 2023-11-08T04:54:26.761 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T04:54:26.761 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2023-11-08T04:54:26.761 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1:1 2023-11-08T04:54:26.761 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T04:54:26.761 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance 2023-11-08T04:54:26.762 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-11-08T04:54:26.762 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local val=cluster1:1 2023-11-08T04:54:26.762 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_var_name=cluster 2023-11-08T04:54:26.762 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance_var_name=instance 2023-11-08T04:54:26.762 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cluster=cluster1 2023-11-08T04:54:26.763 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ instance=1 2023-11-08T04:54:26.763 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' 1 = cluster1:1 ']' 2023-11-08T04:54:26.763 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval cluster=cluster1 2023-11-08T04:54:26.763 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T04:54:26.763 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval instance=1 2023-11-08T04:54:26.764 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=1 2023-11-08T04:54:26.764 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-11-08T04:54:26.764 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local local_cluster=cluster1:1 2023-11-08T04:54:26.764 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster=cluster1 2023-11-08T04:54:26.764 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance 2023-11-08T04:54:26.765 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-11-08T04:54:26.765 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local val=cluster1:1 2023-11-08T04:54:26.765 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster_var_name=local_cluster 2023-11-08T04:54:26.765 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance_var_name=instance 2023-11-08T04:54:26.765 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T04:54:26.765 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=1 2023-11-08T04:54:26.766 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ '[' 1 = cluster1:1 ']' 2023-11-08T04:54:26.766 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval local_cluster=cluster1 2023-11-08T04:54:26.766 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ local_cluster=cluster1 2023-11-08T04:54:26.766 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval instance=1 2023-11-08T04:54:26.766 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ instance=1 2023-11-08T04:54:26.767 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-11-08T04:54:26.774 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-08T04:54:26.775 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-08T04:54:26.775 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-08T04:54:26.775 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2023-11-08T04:54:26.862 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-11-08T04:54:26.863 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd mirror status : get status for rbd mirror 2023-11-08T04:54:26.863 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:admin_socket: invalid command 2023-11-08T04:54:26.868 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T04:54:26.869 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2023-11-08T04:54:26.869 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1:2 2023-11-08T04:54:26.869 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T04:54:26.870 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance 2023-11-08T04:54:26.870 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-11-08T04:54:26.870 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local val=cluster1:2 2023-11-08T04:54:26.871 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_var_name=cluster 2023-11-08T04:54:26.871 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance_var_name=instance 2023-11-08T04:54:26.871 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cluster=cluster1 2023-11-08T04:54:26.872 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ instance=2 2023-11-08T04:54:26.872 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' 2 = cluster1:2 ']' 2023-11-08T04:54:26.872 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval cluster=cluster1 2023-11-08T04:54:26.873 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T04:54:26.873 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval instance=2 2023-11-08T04:54:26.874 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=2 2023-11-08T04:54:26.874 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-11-08T04:54:26.874 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local local_cluster=cluster1:2 2023-11-08T04:54:26.875 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster=cluster1 2023-11-08T04:54:26.875 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance 2023-11-08T04:54:26.875 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-11-08T04:54:26.876 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local val=cluster1:2 2023-11-08T04:54:26.876 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster_var_name=local_cluster 2023-11-08T04:54:26.877 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance_var_name=instance 2023-11-08T04:54:26.877 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T04:54:26.877 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=2 2023-11-08T04:54:26.878 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ '[' 2 = cluster1:2 ']' 2023-11-08T04:54:26.878 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval local_cluster=cluster1 2023-11-08T04:54:26.878 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ local_cluster=cluster1 2023-11-08T04:54:26.879 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval instance=2 2023-11-08T04:54:26.879 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ instance=2 2023-11-08T04:54:26.879 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-11-08T04:54:26.881 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-08T04:54:26.881 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-08T04:54:26.882 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-08T04:54:26.882 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2023-11-08T04:54:26.967 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-11-08T04:54:26.967 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd mirror status : get status for rbd mirror 2023-11-08T04:54:26.968 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:admin_socket: invalid command 2023-11-08T04:54:26.970 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T04:54:26.971 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2023-11-08T04:54:26.971 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1:3 2023-11-08T04:54:26.971 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T04:54:26.971 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance 2023-11-08T04:54:26.972 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ set_cluster_instance cluster1:3 cluster instance 2023-11-08T04:54:26.972 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local val=cluster1:3 2023-11-08T04:54:26.972 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_var_name=cluster 2023-11-08T04:54:26.972 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance_var_name=instance 2023-11-08T04:54:26.972 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cluster=cluster1 2023-11-08T04:54:26.973 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ instance=3 2023-11-08T04:54:26.973 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' 3 = cluster1:3 ']' 2023-11-08T04:54:26.973 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval cluster=cluster1 2023-11-08T04:54:26.973 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T04:54:26.973 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval instance=3 2023-11-08T04:54:26.973 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=3 2023-11-08T04:54:26.974 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ daemon_asok_file cluster1:3 cluster1 2023-11-08T04:54:26.974 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local local_cluster=cluster1:3 2023-11-08T04:54:26.974 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster=cluster1 2023-11-08T04:54:26.974 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance 2023-11-08T04:54:26.975 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2023-11-08T04:54:26.975 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local val=cluster1:3 2023-11-08T04:54:26.975 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster_var_name=local_cluster 2023-11-08T04:54:26.975 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance_var_name=instance 2023-11-08T04:54:26.975 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T04:54:26.975 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=3 2023-11-08T04:54:26.976 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ '[' 3 = cluster1:3 ']' 2023-11-08T04:54:26.976 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval local_cluster=cluster1 2023-11-08T04:54:26.976 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ local_cluster=cluster1 2023-11-08T04:54:26.976 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval instance=3 2023-11-08T04:54:26.977 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ instance=3 2023-11-08T04:54:26.977 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2023-11-08T04:54:26.984 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-11-08T04:54:26.984 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-11-08T04:54:26.985 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-11-08T04:54:26.985 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2023-11-08T04:54:27.071 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-11-08T04:54:27.071 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd mirror status : get status for rbd mirror 2023-11-08T04:54:27.071 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:admin_socket: invalid command 2023-11-08T04:54:27.072 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-11-08T04:54:27.072 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sleep 8 2023-11-08T04:54:35.073 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-11-08T04:54:35.073 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ seq 0 3 2023-11-08T04:54:35.077 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T04:54:35.077 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2023-11-08T04:54:35.077 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1:0 2023-11-08T04:54:35.077 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T04:54:35.078 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance 2023-11-08T04:54:35.078 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-11-08T04:54:35.078 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local val=cluster1:0 2023-11-08T04:54:35.078 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_var_name=cluster 2023-11-08T04:54:35.079 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance_var_name=instance 2023-11-08T04:54:35.079 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cluster=cluster1 2023-11-08T04:54:35.079 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ instance=0 2023-11-08T04:54:35.079 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' 0 = cluster1:0 ']' 2023-11-08T04:54:35.080 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval cluster=cluster1 2023-11-08T04:54:35.080 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T04:54:35.080 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval instance=0 2023-11-08T04:54:35.080 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=0 2023-11-08T04:54:35.080 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-11-08T04:54:35.081 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local local_cluster=cluster1:0 2023-11-08T04:54:35.081 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster=cluster1 2023-11-08T04:54:35.081 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance 2023-11-08T04:54:35.082 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-11-08T04:54:35.082 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local val=cluster1:0 2023-11-08T04:54:35.082 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster_var_name=local_cluster 2023-11-08T04:54:35.082 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance_var_name=instance 2023-11-08T04:54:35.082 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T04:54:35.083 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=0 2023-11-08T04:54:35.083 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ '[' 0 = cluster1:0 ']' 2023-11-08T04:54:35.083 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval local_cluster=cluster1 2023-11-08T04:54:35.083 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ local_cluster=cluster1 2023-11-08T04:54:35.083 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval instance=0 2023-11-08T04:54:35.084 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ instance=0 2023-11-08T04:54:35.084 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-11-08T04:54:35.093 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-08T04:54:35.093 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-08T04:54:35.093 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-08T04:54:35.093 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2023-11-08T04:54:35.199 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ return 0 2023-11-08T04:54:35.200 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ status_result=' "state": "Replaying"' 2023-11-08T04:54:35.200 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ echo ' "state": "Replaying"' 2023-11-08T04:54:35.200 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ grep -i Replaying 2023-11-08T04:54:35.201 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: "state": "Replaying" 2023-11-08T04:54:35.202 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ current_state=started 2023-11-08T04:54:35.202 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test started = started 2023-11-08T04:54:35.202 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T04:54:35.203 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ clean_snap_name= 2023-11-08T04:54:35.203 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ seq 1 10 2023-11-08T04:54:35.203 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for i in `seq 1 10` 2023-11-08T04:54:35.203 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ stress_write_image cluster2 mirror test 2023-11-08T04:54:35.203 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T04:54:35.204 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T04:54:35.204 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T04:54:35.204 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-11-08T04:54:35.205 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local duration=5 2023-11-08T04:54:35.205 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ set +e 2023-11-08T04:54:35.205 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2023-11-08T04:54:35.355 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:100 IOs, 25676800 bytes 2023-11-08T04:54:35.557 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:200 IOs, 99514368 bytes 2023-11-08T04:54:35.687 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:300 IOs, 130502656 bytes 2023-11-08T04:54:35.797 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:400 IOs, 160233472 bytes 2023-11-08T04:54:35.972 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:500 IOs, 217288704 bytes 2023-11-08T04:54:36.093 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:600 IOs, 242060288 bytes 2023-11-08T04:54:36.161 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:700 IOs, 246968320 bytes 2023-11-08T04:54:36.303 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:800 IOs, 281319424 bytes 2023-11-08T04:54:36.454 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:900 IOs, 330042368 bytes 2023-11-08T04:54:36.656 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1000 IOs, 401017856 bytes 2023-11-08T04:54:36.716 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1100 IOs, 401635328 bytes 2023-11-08T04:54:36.810 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1200 IOs, 414030848 bytes 2023-11-08T04:54:36.947 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1300 IOs, 469069824 bytes 2023-11-08T04:54:37.061 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1400 IOs, 483287040 bytes 2023-11-08T04:54:37.256 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1500 IOs, 543348736 bytes 2023-11-08T04:54:37.328 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1600 IOs, 543951872 bytes 2023-11-08T04:54:37.491 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1700 IOs, 599325696 bytes 2023-11-08T04:54:37.619 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1800 IOs, 630411264 bytes 2023-11-08T04:54:37.712 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1900 IOs, 658421760 bytes 2023-11-08T04:54:37.877 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2000 IOs, 710859776 bytes 2023-11-08T04:54:38.030 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2100 IOs, 746906624 bytes 2023-11-08T04:54:38.087 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2200 IOs, 762452992 bytes 2023-11-08T04:54:38.329 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2300 IOs, 833664000 bytes 2023-11-08T04:54:38.485 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2400 IOs, 882741248 bytes 2023-11-08T04:54:38.605 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2500 IOs, 910291968 bytes 2023-11-08T04:54:38.800 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2600 IOs, 966196224 bytes 2023-11-08T04:54:38.929 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2700 IOs, 1003941888 bytes 2023-11-08T04:54:39.053 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2800 IOs, 1053402112 bytes 2023-11-08T04:54:39.304 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2900 IOs, 1094440960 bytes 2023-11-08T04:54:39.401 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3000 IOs, 1107868672 bytes 2023-11-08T04:54:39.560 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3100 IOs, 1152523264 bytes 2023-11-08T04:54:39.679 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3200 IOs, 1176458240 bytes 2023-11-08T04:54:39.772 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3300 IOs, 1189234688 bytes 2023-11-08T04:54:39.898 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3400 IOs, 1220976640 bytes 2023-11-08T04:54:39.967 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3500 IOs, 1221578752 bytes 2023-11-08T04:54:40.112 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3600 IOs, 1261283328 bytes 2023-11-08T04:54:40.207 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ error_code=124 2023-11-08T04:54:40.207 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ set -e 2023-11-08T04:54:40.208 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' 124 -eq 124 ']' 2023-11-08T04:54:40.208 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T04:54:40.208 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2023-11-08T04:54:40.208 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T04:54:40.208 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T04:54:40.209 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T04:54:40.209 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local state_pattern=up+replaying 2023-11-08T04:54:40.209 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local description_pattern= 2023-11-08T04:54:40.209 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local service_pattern= 2023-11-08T04:54:40.209 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-11-08T04:54:40.210 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 1 2023-11-08T04:54:41.208 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2023-11-08T04:54:41.209 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T04:54:41.209 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T04:54:41.209 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T04:54:41.209 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local state_pattern=up+replaying 2023-11-08T04:54:41.210 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local description_pattern= 2023-11-08T04:54:41.210 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local service_pattern= 2023-11-08T04:54:41.210 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ mkfname cluster1-mirror-test.mirror_status 2023-11-08T04:54:41.210 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ echo cluster1-mirror-test.mirror_status 2023-11-08T04:54:41.210 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -e 's|[/ ]|_|g' 2023-11-08T04:54:41.211 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local status_log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.mirror_status 2023-11-08T04:54:41.211 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ CEPH_ARGS= 2023-11-08T04:54:41.211 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 mirror image status mirror/test 2023-11-08T04:54:41.212 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ tee /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.mirror_status 2023-11-08T04:54:41.254 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:test: 2023-11-08T04:54:41.255 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: global_id: 88ced7f9-2739-4a27-ad7a-30edec05edb1 2023-11-08T04:54:41.255 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: state: down+unknown 2023-11-08T04:54:41.255 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: description: status not found 2023-11-08T04:54:41.257 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: last_update: 2023-11-08T04:54:41.257 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: peer_sites: 2023-11-08T04:54:41.258 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: name: cluster2 2023-11-08T04:54:41.258 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: state: down+unknown 2023-11-08T04:54:41.258 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: description: status not found 2023-11-08T04:54:41.258 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: last_update: 2023-11-08T04:54:41.259 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ grep '^ state: .*up+replaying' /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.mirror_status 2023-11-08T04:54:41.261 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 1 2023-11-08T04:54:41.261 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-11-08T04:54:41.261 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 2 2023-11-08T04:54:42.007 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.6612008287152717) > weight (0.25) 2023-11-08T04:54:42.007 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.8151172028648757) > weight (0.25) 2023-11-08T04:54:42.007 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.5294435776147467) > weight (0.25) 2023-11-08T04:54:42.008 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.3 with skip (0.6945717446781622) > weight (0.25) 2023-11-08T04:54:42.008 INFO:tasks.rbd_mirror_thrash:waiting for 93 secs before thrashing 2023-11-08T04:54:43.007 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.726093990190493) > weight (0.25) 2023-11-08T04:54:43.007 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.7576486346557091) > weight (0.25) 2023-11-08T04:54:43.007 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.2 2023-11-08T04:54:43.008 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Sent signal 15 2023-11-08T04:54:43.008 INFO:tasks.rbd_mirror_thrash:waiting for 7 secs before reviving daemons 2023-11-08T04:54:43.009 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi087.stderr:2023-11-08T04:54:43.007+0000 7f745fa2c700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 (PID: 120056) UID: 1000 2023-11-08T04:54:43.209 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi087.stderr:daemon-helper: command failed with exit status 1 2023-11-08T04:54:43.262 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2023-11-08T04:54:43.263 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T04:54:43.263 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T04:54:43.263 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T04:54:43.264 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local state_pattern=up+replaying 2023-11-08T04:54:43.264 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local description_pattern= 2023-11-08T04:54:43.264 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local service_pattern= 2023-11-08T04:54:43.264 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ mkfname cluster1-mirror-test.mirror_status 2023-11-08T04:54:43.264 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ echo cluster1-mirror-test.mirror_status 2023-11-08T04:54:43.265 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -e 's|[/ ]|_|g' 2023-11-08T04:54:43.265 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local status_log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.mirror_status 2023-11-08T04:54:43.266 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ CEPH_ARGS= 2023-11-08T04:54:43.266 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 mirror image status mirror/test 2023-11-08T04:54:43.266 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ tee /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.mirror_status 2023-11-08T04:54:43.306 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:test: 2023-11-08T04:54:43.306 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: global_id: 88ced7f9-2739-4a27-ad7a-30edec05edb1 2023-11-08T04:54:43.306 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: state: down+unknown 2023-11-08T04:54:43.306 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: description: status not found 2023-11-08T04:54:43.307 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: last_update: 2023-11-08T04:54:43.307 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: peer_sites: 2023-11-08T04:54:43.307 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: name: cluster2 2023-11-08T04:54:43.307 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: state: down+unknown 2023-11-08T04:54:43.307 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: description: status not found 2023-11-08T04:54:43.308 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: last_update: 2023-11-08T04:54:43.310 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ grep '^ state: .*up+replaying' /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.mirror_status 2023-11-08T04:54:43.311 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 1 2023-11-08T04:54:43.311 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-11-08T04:54:43.311 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 4 2023-11-08T04:54:47.313 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2023-11-08T04:54:47.313 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T04:54:47.313 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T04:54:47.314 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T04:54:47.314 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local state_pattern=up+replaying 2023-11-08T04:54:47.314 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local description_pattern= 2023-11-08T04:54:47.314 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local service_pattern= 2023-11-08T04:54:47.315 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ mkfname cluster1-mirror-test.mirror_status 2023-11-08T04:54:47.315 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ echo cluster1-mirror-test.mirror_status 2023-11-08T04:54:47.315 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -e 's|[/ ]|_|g' 2023-11-08T04:54:47.316 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local status_log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.mirror_status 2023-11-08T04:54:47.316 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ CEPH_ARGS= 2023-11-08T04:54:47.316 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 mirror image status mirror/test 2023-11-08T04:54:47.316 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ tee /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.mirror_status 2023-11-08T04:54:47.354 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:test: 2023-11-08T04:54:47.355 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: global_id: 88ced7f9-2739-4a27-ad7a-30edec05edb1 2023-11-08T04:54:47.355 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: state: down+unknown 2023-11-08T04:54:47.355 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: description: status not found 2023-11-08T04:54:47.355 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: last_update: 2023-11-08T04:54:47.355 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: peer_sites: 2023-11-08T04:54:47.356 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: name: cluster2 2023-11-08T04:54:47.356 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: state: down+unknown 2023-11-08T04:54:47.356 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: description: status not found 2023-11-08T04:54:47.356 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: last_update: 2023-11-08T04:54:47.358 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ grep '^ state: .*up+replaying' /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.mirror_status 2023-11-08T04:54:47.360 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 1 2023-11-08T04:54:47.360 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-11-08T04:54:47.360 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 8 2023-11-08T04:54:50.009 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.2 2023-11-08T04:54:50.010 INFO:teuthology.orchestra.run:waiting for 600 2023-11-08T04:54:50.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-08T04:54:50.011 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.2 2023-11-08T04:54:50.011 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Restarting daemon 2023-11-08T04:54:50.011 DEBUG:teuthology.orchestra.run.smithi087:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 2023-11-08T04:54:50.014 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Started 2023-11-08T04:54:50.015 INFO:tasks.rbd_mirror_thrash:waiting for 64 secs before thrashing 2023-11-08T04:54:55.361 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2023-11-08T04:54:55.362 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T04:54:55.362 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T04:54:55.362 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T04:54:55.362 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local state_pattern=up+replaying 2023-11-08T04:54:55.363 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local description_pattern= 2023-11-08T04:54:55.363 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local service_pattern= 2023-11-08T04:54:55.363 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ mkfname cluster1-mirror-test.mirror_status 2023-11-08T04:54:55.363 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ echo cluster1-mirror-test.mirror_status 2023-11-08T04:54:55.364 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -e 's|[/ ]|_|g' 2023-11-08T04:54:55.364 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local status_log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.mirror_status 2023-11-08T04:54:55.364 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ CEPH_ARGS= 2023-11-08T04:54:55.365 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 mirror image status mirror/test 2023-11-08T04:54:55.365 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ tee /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.mirror_status 2023-11-08T04:54:55.405 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:test: 2023-11-08T04:54:55.406 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: global_id: 88ced7f9-2739-4a27-ad7a-30edec05edb1 2023-11-08T04:54:55.406 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: state: down+unknown 2023-11-08T04:54:55.406 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: description: status not found 2023-11-08T04:54:55.406 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: last_update: 2023-11-08T04:54:55.407 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: peer_sites: 2023-11-08T04:54:55.407 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: name: cluster2 2023-11-08T04:54:55.407 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: state: down+unknown 2023-11-08T04:54:55.407 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: description: status not found 2023-11-08T04:54:55.407 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: last_update: 2023-11-08T04:54:55.410 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ grep '^ state: .*up+replaying' /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.mirror_status 2023-11-08T04:54:55.414 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 1 2023-11-08T04:54:55.414 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-11-08T04:54:55.414 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 8 2023-11-08T04:55:03.413 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2023-11-08T04:55:03.413 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T04:55:03.413 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T04:55:03.414 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T04:55:03.414 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local state_pattern=up+replaying 2023-11-08T04:55:03.414 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local description_pattern= 2023-11-08T04:55:03.414 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local service_pattern= 2023-11-08T04:55:03.414 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ mkfname cluster1-mirror-test.mirror_status 2023-11-08T04:55:03.415 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ echo cluster1-mirror-test.mirror_status 2023-11-08T04:55:03.416 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -e 's|[/ ]|_|g' 2023-11-08T04:55:03.416 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local status_log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.mirror_status 2023-11-08T04:55:03.416 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ CEPH_ARGS= 2023-11-08T04:55:03.416 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 mirror image status mirror/test 2023-11-08T04:55:03.416 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ tee /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.mirror_status 2023-11-08T04:55:03.456 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:test: 2023-11-08T04:55:03.457 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: global_id: 88ced7f9-2739-4a27-ad7a-30edec05edb1 2023-11-08T04:55:03.457 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: state: up+replaying 2023-11-08T04:55:03.457 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1699419240,"remote_snapshot_timestamp":1699419240,"replay_state":"idle"} 2023-11-08T04:55:03.457 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: service: mirror.0 on smithi087 2023-11-08T04:55:03.458 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: last_update: 2023-11-08 04:54:56 2023-11-08T04:55:03.458 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: peer_sites: 2023-11-08T04:55:03.458 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: name: cluster2 2023-11-08T04:55:03.458 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: state: up+stopped 2023-11-08T04:55:03.459 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: description: local image is primary 2023-11-08T04:55:03.459 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: last_update: 2023-11-08 04:54:56 2023-11-08T04:55:03.460 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ grep '^ state: .*up+replaying' /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.mirror_status 2023-11-08T04:55:03.461 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: state: up+replaying 2023-11-08T04:55:03.462 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ grep '^ description: .*' /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.mirror_status 2023-11-08T04:55:03.462 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1699419240,"remote_snapshot_timestamp":1699419240,"replay_state":"idle"} 2023-11-08T04:55:03.463 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' -n '' ']' 2023-11-08T04:55:03.463 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ echo up+replaying 2023-11-08T04:55:03.464 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ grep '^up+' 2023-11-08T04:55:03.464 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:up+replaying 2023-11-08T04:55:03.465 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ grep 'service: *mirror..* on ' /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.mirror_status 2023-11-08T04:55:03.465 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: service: mirror.0 on smithi087 2023-11-08T04:55:03.466 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.mirror_status 2023-11-08T04:55:03.467 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local 'last_update=2023-11-08 04:54:56' 2023-11-08T04:55:03.467 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2023-11-08 04:54:56' 2023-11-08T04:55:03.467 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T04:55:03.468 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T04:55:03.468 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T04:55:03.468 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local state_pattern=up+replaying 2023-11-08T04:55:03.468 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local 'prev_last_update=2023-11-08 04:54:56' 2023-11-08T04:55:03.469 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ mkfname cluster1-mirror.mirror_status 2023-11-08T04:55:03.469 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ echo cluster1-mirror.mirror_status 2023-11-08T04:55:03.469 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -e 's|[/ ]|_|g' 2023-11-08T04:55:03.470 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local status_log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror.mirror_status 2023-11-08T04:55:03.470 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2023-11-08T04:55:03.499 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd: failed to query services: (13) Permission denied 2023-11-08T04:55:03.518 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local last_update state 2023-11-08T04:55:03.518 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2023-11-08T04:55:03.535 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ last_update='2023-11-08 04:54:56' 2023-11-08T04:55:03.535 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2023-11-08T04:55:03.537 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ state=up+replaying 2023-11-08T04:55:03.537 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ echo up+replaying 2023-11-08T04:55:03.537 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ grep up+replaying 2023-11-08T04:55:03.538 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:up+replaying 2023-11-08T04:55:03.539 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T04:55:03.539 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T04:55:03.539 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ snap_name=snap1 2023-11-08T04:55:03.540 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ create_snap cluster2 mirror test snap1 2023-11-08T04:55:03.540 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T04:55:03.540 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T04:55:03.540 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T04:55:03.541 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap_name=snap1 2023-11-08T04:55:03.541 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap1 --debug-rbd=20 --debug-journaler=20 2023-11-08T04:55:11.413 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_image_replay_started cluster1 mirror test 2023-11-08T04:55:11.414 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T04:55:11.414 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T04:55:11.414 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T04:55:11.415 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2023-11-08T04:55:11.415 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T04:55:11.415 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T04:55:11.415 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T04:55:11.415 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local state=started 2023-11-08T04:55:11.416 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local s 2023-11-08T04:55:11.416 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-11-08T04:55:11.416 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 1 2023-11-08T04:55:12.415 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test_image_replay_state cluster1 mirror test started 2023-11-08T04:55:12.415 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T04:55:12.415 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T04:55:12.415 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T04:55:12.416 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local test_state=started 2023-11-08T04:55:12.416 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local status_result 2023-11-08T04:55:12.416 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local current_state=stopped 2023-11-08T04:55:12.420 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2023-11-08T04:55:12.420 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_instance=cluster1 2023-11-08T04:55:12.420 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T04:55:12.420 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ grep -i state 2023-11-08T04:55:12.420 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1 2023-11-08T04:55:12.421 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance=cluster1 2023-11-08T04:55:12.421 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local loop_instance 2023-11-08T04:55:12.421 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-11-08T04:55:12.421 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sleep 0 2023-11-08T04:55:12.421 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-11-08T04:55:12.422 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ seq 0 3 2023-11-08T04:55:12.422 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T04:55:12.422 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2023-11-08T04:55:12.422 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1:0 2023-11-08T04:55:12.422 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T04:55:12.423 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance 2023-11-08T04:55:12.423 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-11-08T04:55:12.423 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local val=cluster1:0 2023-11-08T04:55:12.423 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_var_name=cluster 2023-11-08T04:55:12.423 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance_var_name=instance 2023-11-08T04:55:12.423 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cluster=cluster1 2023-11-08T04:55:12.424 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ instance=0 2023-11-08T04:55:12.424 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' 0 = cluster1:0 ']' 2023-11-08T04:55:12.424 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval cluster=cluster1 2023-11-08T04:55:12.424 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T04:55:12.424 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval instance=0 2023-11-08T04:55:12.425 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=0 2023-11-08T04:55:12.425 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-11-08T04:55:12.425 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local local_cluster=cluster1:0 2023-11-08T04:55:12.425 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster=cluster1 2023-11-08T04:55:12.425 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance 2023-11-08T04:55:12.426 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-11-08T04:55:12.426 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local val=cluster1:0 2023-11-08T04:55:12.426 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster_var_name=local_cluster 2023-11-08T04:55:12.426 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance_var_name=instance 2023-11-08T04:55:12.426 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T04:55:12.427 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=0 2023-11-08T04:55:12.427 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ '[' 0 = cluster1:0 ']' 2023-11-08T04:55:12.427 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval local_cluster=cluster1 2023-11-08T04:55:12.427 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ local_cluster=cluster1 2023-11-08T04:55:12.427 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval instance=0 2023-11-08T04:55:12.427 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ instance=0 2023-11-08T04:55:12.428 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-11-08T04:55:12.436 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-08T04:55:12.436 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-08T04:55:12.436 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-08T04:55:12.437 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2023-11-08T04:55:12.542 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ return 0 2023-11-08T04:55:12.543 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ status_result=' "state": "Replaying"' 2023-11-08T04:55:12.543 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ echo ' "state": "Replaying"' 2023-11-08T04:55:12.543 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ grep -i Replaying 2023-11-08T04:55:12.543 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: "state": "Replaying" 2023-11-08T04:55:12.545 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ current_state=started 2023-11-08T04:55:12.545 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test started = started 2023-11-08T04:55:12.546 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T04:55:12.546 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2023-11-08T04:55:12.546 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local local_cluster=cluster1 2023-11-08T04:55:12.546 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T04:55:12.546 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T04:55:12.547 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T04:55:12.547 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' snapshot = journal ']' 2023-11-08T04:55:12.547 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' snapshot = snapshot ']' 2023-11-08T04:55:12.547 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2023-11-08T04:55:12.547 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local local_cluster=cluster1 2023-11-08T04:55:12.548 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T04:55:12.548 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T04:55:12.548 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T04:55:12.548 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ mkfname cluster2-mirror-test.status 2023-11-08T04:55:12.548 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ echo cluster2-mirror-test.status 2023-11-08T04:55:12.548 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -e 's|[/ ]|_|g' 2023-11-08T04:55:12.549 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local status_log=/tmp/tmp.Ludk8ikCAB/cluster2-mirror-test.status 2023-11-08T04:55:12.550 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ mkfname cluster1-mirror-test.status 2023-11-08T04:55:12.550 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ echo cluster1-mirror-test.status 2023-11-08T04:55:12.550 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -e 's|[/ ]|_|g' 2023-11-08T04:55:12.550 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local local_status_log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.status 2023-11-08T04:55:12.551 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ mirror_image_snapshot cluster2 mirror test 2023-11-08T04:55:12.551 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T04:55:12.551 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T04:55:12.551 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T04:55:12.551 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2023-11-08T04:55:13.434 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:Snapshot ID: 6 2023-11-08T04:55:13.441 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.Ludk8ikCAB/cluster2-mirror-test.status 2023-11-08T04:55:13.441 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T04:55:13.442 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T04:55:13.442 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T04:55:13.442 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster2-mirror-test.status 2023-11-08T04:55:13.442 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2023-11-08T04:55:13.442 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T04:55:13.479 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-11-08T04:55:13.480 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snapshot_id=6 2023-11-08T04:55:13.481 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ true 2023-11-08T04:55:13.481 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-11-08T04:55:13.481 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0.2 2023-11-08T04:55:13.682 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.status 2023-11-08T04:55:13.682 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T04:55:13.683 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T04:55:13.683 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T04:55:13.683 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.status 2023-11-08T04:55:13.683 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-11-08T04:55:13.683 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T04:55:13.723 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-08T04:55:13.724 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local primary_snapshot_id=4 2023-11-08T04:55:13.725 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test 6 = 4 2023-11-08T04:55:13.725 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-11-08T04:55:13.725 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0.4 2023-11-08T04:55:14.126 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.status 2023-11-08T04:55:14.126 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T04:55:14.126 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T04:55:14.127 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T04:55:14.127 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.status 2023-11-08T04:55:14.127 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-11-08T04:55:14.127 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T04:55:14.169 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-08T04:55:14.171 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local primary_snapshot_id=4 2023-11-08T04:55:14.171 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test 6 = 4 2023-11-08T04:55:14.171 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-11-08T04:55:14.172 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0.8 2023-11-08T04:55:14.973 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.status 2023-11-08T04:55:14.973 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T04:55:14.973 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T04:55:14.973 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T04:55:14.974 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.status 2023-11-08T04:55:14.974 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-11-08T04:55:14.974 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T04:55:15.016 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ true 2023-11-08T04:55:15.016 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-08T04:55:15.017 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:-:1.1: Document is empty 2023-11-08T04:55:15.017 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: 2023-11-08T04:55:15.018 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:^ 2023-11-08T04:55:15.018 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local primary_snapshot_id= 2023-11-08T04:55:15.018 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test 6 = '' 2023-11-08T04:55:15.019 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-11-08T04:55:15.019 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 1.6 2023-11-08T04:55:16.619 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.status 2023-11-08T04:55:16.620 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T04:55:16.620 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T04:55:16.620 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T04:55:16.620 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.status 2023-11-08T04:55:16.620 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-11-08T04:55:16.621 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T04:55:16.657 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-08T04:55:16.659 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local primary_snapshot_id=6 2023-11-08T04:55:16.659 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test 6 = 6 2023-11-08T04:55:16.659 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T04:55:16.660 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_snap_present cluster1 mirror test snap1 2023-11-08T04:55:16.660 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T04:55:16.660 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T04:55:16.660 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T04:55:16.660 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap_name=snap1 2023-11-08T04:55:16.661 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local s 2023-11-08T04:55:16.661 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-11-08T04:55:16.661 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 1 2023-11-08T04:55:17.660 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 info mirror/test@snap1 2023-11-08T04:55:17.699 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:rbd image 'test': 2023-11-08T04:55:17.699 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: size 512 MiB in 128 objects 2023-11-08T04:55:17.699 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: order 22 (4 MiB objects) 2023-11-08T04:55:17.699 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: snapshot_count: 2 2023-11-08T04:55:17.700 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: id: 10afad8fe72b 2023-11-08T04:55:17.700 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: block_name_prefix: rbd_data.10afad8fe72b 2023-11-08T04:55:17.700 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: format: 2 2023-11-08T04:55:17.700 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: features: layering, exclusive-lock, non-primary 2023-11-08T04:55:17.700 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: op_features: 2023-11-08T04:55:17.701 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: flags: 2023-11-08T04:55:17.701 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: create_timestamp: Wed Nov 8 04:54:28 2023 2023-11-08T04:55:17.701 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: access_timestamp: Wed Nov 8 04:54:28 2023 2023-11-08T04:55:17.701 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: modify_timestamp: Wed Nov 8 04:54:28 2023 2023-11-08T04:55:17.701 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: protected: False 2023-11-08T04:55:17.702 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring state: enabled 2023-11-08T04:55:17.702 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring mode: snapshot 2023-11-08T04:55:17.702 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring global id: 88ced7f9-2739-4a27-ad7a-30edec05edb1 2023-11-08T04:55:17.702 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring primary: false 2023-11-08T04:55:17.704 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T04:55:17.704 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' -n '' ']' 2023-11-08T04:55:17.704 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ compare_image_snaps mirror test snap1 2023-11-08T04:55:17.704 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T04:55:17.704 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T04:55:17.705 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap_name=snap1 2023-11-08T04:55:17.705 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local ret=0 2023-11-08T04:55:17.705 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local rmt_export=/tmp/tmp.Ludk8ikCAB/cluster2-mirror-test.export 2023-11-08T04:55:17.705 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local loc_export=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.export 2023-11-08T04:55:17.705 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rm -f /tmp/tmp.Ludk8ikCAB/cluster2-mirror-test.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.export 2023-11-08T04:55:17.706 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 -p mirror export test@snap1 /tmp/tmp.Ludk8ikCAB/cluster2-mirror-test.export 2023-11-08T04:55:18.266 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-11-08T04:55:18.270 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 -p mirror export test@snap1 /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.export 2023-11-08T04:55:18.989 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-11-08T04:55:18.993 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ cmp /tmp/tmp.Ludk8ikCAB/cluster2-mirror-test.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.export 2023-11-08T04:55:19.203 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rm -f /tmp/tmp.Ludk8ikCAB/cluster2-mirror-test.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.export 2023-11-08T04:55:19.343 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T04:55:19.343 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ clean_snap_name=snap1-clean 2023-11-08T04:55:19.344 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ create_snap cluster2 mirror test snap1-clean 2023-11-08T04:55:19.344 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T04:55:19.344 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T04:55:19.344 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T04:55:19.344 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap_name=snap1-clean 2023-11-08T04:55:19.345 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap1-clean --debug-rbd=20 --debug-journaler=20 2023-11-08T04:55:19.433 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for i in `seq 1 10` 2023-11-08T04:55:19.434 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ stress_write_image cluster2 mirror test 2023-11-08T04:55:19.434 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T04:55:19.434 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T04:55:19.434 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T04:55:19.435 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-11-08T04:55:19.435 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local duration=8 2023-11-08T04:55:19.436 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ set +e 2023-11-08T04:55:19.436 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2023-11-08T04:55:19.634 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:100 IOs, 27944960 bytes 2023-11-08T04:55:19.762 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:200 IOs, 57960448 bytes 2023-11-08T04:55:19.933 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:300 IOs, 103434240 bytes 2023-11-08T04:55:20.021 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:400 IOs, 113136640 bytes 2023-11-08T04:55:20.145 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:500 IOs, 141099008 bytes 2023-11-08T04:55:20.318 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:600 IOs, 192202752 bytes 2023-11-08T04:55:20.469 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:700 IOs, 229082112 bytes 2023-11-08T04:55:20.615 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:800 IOs, 280560640 bytes 2023-11-08T04:55:20.790 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:900 IOs, 321540096 bytes 2023-11-08T04:55:20.930 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1000 IOs, 355700736 bytes 2023-11-08T04:55:21.060 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1100 IOs, 382950400 bytes 2023-11-08T04:55:21.242 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1200 IOs, 435573760 bytes 2023-11-08T04:55:21.478 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1300 IOs, 516978688 bytes 2023-11-08T04:55:21.583 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1400 IOs, 541792256 bytes 2023-11-08T04:55:21.764 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1500 IOs, 621537280 bytes 2023-11-08T04:55:21.946 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1600 IOs, 650424320 bytes 2023-11-08T04:55:22.054 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1700 IOs, 670710784 bytes 2023-11-08T04:55:22.121 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1800 IOs, 680617984 bytes 2023-11-08T04:55:22.226 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1900 IOs, 711495680 bytes 2023-11-08T04:55:22.417 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2000 IOs, 744268800 bytes 2023-11-08T04:55:22.587 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2100 IOs, 789600256 bytes 2023-11-08T04:55:22.828 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2200 IOs, 866411520 bytes 2023-11-08T04:55:22.989 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2300 IOs, 910884864 bytes 2023-11-08T04:55:23.183 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2400 IOs, 982429696 bytes 2023-11-08T04:55:23.337 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2500 IOs, 1022825472 bytes 2023-11-08T04:55:23.503 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2600 IOs, 1068308480 bytes 2023-11-08T04:55:23.703 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2700 IOs, 1133464576 bytes 2023-11-08T04:55:23.853 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2800 IOs, 1178891264 bytes 2023-11-08T04:55:24.104 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2900 IOs, 1263872000 bytes 2023-11-08T04:55:24.286 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3000 IOs, 1316762624 bytes 2023-11-08T04:55:24.350 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3100 IOs, 1317368832 bytes 2023-11-08T04:55:24.546 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3200 IOs, 1385053184 bytes 2023-11-08T04:55:24.964 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3300 IOs, 1483901952 bytes 2023-11-08T04:55:25.180 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3400 IOs, 1555077120 bytes 2023-11-08T04:55:25.344 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3500 IOs, 1599629312 bytes 2023-11-08T04:55:25.519 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3600 IOs, 1654414336 bytes 2023-11-08T04:55:25.650 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3700 IOs, 1674301440 bytes 2023-11-08T04:55:25.770 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3800 IOs, 1701526528 bytes 2023-11-08T04:55:25.925 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3900 IOs, 1743569920 bytes 2023-11-08T04:55:26.084 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4000 IOs, 1779034112 bytes 2023-11-08T04:55:26.252 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4100 IOs, 1824424960 bytes 2023-11-08T04:55:26.313 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4200 IOs, 1825028096 bytes 2023-11-08T04:55:26.430 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4300 IOs, 1840385024 bytes 2023-11-08T04:55:26.547 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4400 IOs, 1865694208 bytes 2023-11-08T04:55:26.637 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4500 IOs, 1902982144 bytes 2023-11-08T04:55:26.830 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4600 IOs, 1942119424 bytes 2023-11-08T04:55:27.033 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4700 IOs, 2007788544 bytes 2023-11-08T04:55:27.189 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4800 IOs, 2056333312 bytes 2023-11-08T04:55:27.340 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4900 IOs, 2091912192 bytes 2023-11-08T04:55:27.440 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ error_code=124 2023-11-08T04:55:27.440 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ set -e 2023-11-08T04:55:27.440 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' 124 -eq 124 ']' 2023-11-08T04:55:27.441 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T04:55:27.441 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2023-11-08T04:55:27.441 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T04:55:27.441 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T04:55:27.441 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T04:55:27.442 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local state_pattern=up+replaying 2023-11-08T04:55:27.442 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local description_pattern= 2023-11-08T04:55:27.442 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local service_pattern= 2023-11-08T04:55:27.442 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-11-08T04:55:27.443 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 1 2023-11-08T04:55:28.441 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2023-11-08T04:55:28.441 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T04:55:28.442 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T04:55:28.442 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T04:55:28.442 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local state_pattern=up+replaying 2023-11-08T04:55:28.442 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local description_pattern= 2023-11-08T04:55:28.443 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local service_pattern= 2023-11-08T04:55:28.443 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ mkfname cluster1-mirror-test.mirror_status 2023-11-08T04:55:28.443 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ echo cluster1-mirror-test.mirror_status 2023-11-08T04:55:28.443 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -e 's|[/ ]|_|g' 2023-11-08T04:55:28.444 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local status_log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.mirror_status 2023-11-08T04:55:28.444 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ CEPH_ARGS= 2023-11-08T04:55:28.445 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 mirror image status mirror/test 2023-11-08T04:55:28.445 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ tee /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.mirror_status 2023-11-08T04:55:28.487 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:test: 2023-11-08T04:55:28.487 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: global_id: 88ced7f9-2739-4a27-ad7a-30edec05edb1 2023-11-08T04:55:28.487 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: state: up+replaying 2023-11-08T04:55:28.488 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: description: replaying, {"bytes_per_second":16241971.2,"bytes_per_snapshot":243629568.0,"last_snapshot_bytes":487259136,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1699419313,"remote_snapshot_timestamp":1699419313,"replay_state":"idle"} 2023-11-08T04:55:28.488 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: service: mirror.0 on smithi087 2023-11-08T04:55:28.488 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: last_update: 2023-11-08 04:55:26 2023-11-08T04:55:28.488 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: peer_sites: 2023-11-08T04:55:28.488 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: name: cluster2 2023-11-08T04:55:28.489 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: state: up+stopped 2023-11-08T04:55:28.489 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: description: local image is primary 2023-11-08T04:55:28.489 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: last_update: 2023-11-08 04:55:26 2023-11-08T04:55:28.491 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ grep '^ state: .*up+replaying' /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.mirror_status 2023-11-08T04:55:28.493 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: state: up+replaying 2023-11-08T04:55:28.493 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ grep '^ description: .*' /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.mirror_status 2023-11-08T04:55:28.494 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: description: replaying, {"bytes_per_second":16241971.2,"bytes_per_snapshot":243629568.0,"last_snapshot_bytes":487259136,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1699419313,"remote_snapshot_timestamp":1699419313,"replay_state":"idle"} 2023-11-08T04:55:28.495 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' -n '' ']' 2023-11-08T04:55:28.495 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ echo up+replaying 2023-11-08T04:55:28.495 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ grep '^up+' 2023-11-08T04:55:28.496 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:up+replaying 2023-11-08T04:55:28.496 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ grep 'service: *mirror..* on ' /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.mirror_status 2023-11-08T04:55:28.496 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: service: mirror.0 on smithi087 2023-11-08T04:55:28.497 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.mirror_status 2023-11-08T04:55:28.498 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local 'last_update=2023-11-08 04:55:26' 2023-11-08T04:55:28.498 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2023-11-08 04:55:26' 2023-11-08T04:55:28.499 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T04:55:28.499 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T04:55:28.499 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T04:55:28.499 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local state_pattern=up+replaying 2023-11-08T04:55:28.500 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local 'prev_last_update=2023-11-08 04:55:26' 2023-11-08T04:55:28.500 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ mkfname cluster1-mirror.mirror_status 2023-11-08T04:55:28.500 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ echo cluster1-mirror.mirror_status 2023-11-08T04:55:28.501 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -e 's|[/ ]|_|g' 2023-11-08T04:55:28.502 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local status_log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror.mirror_status 2023-11-08T04:55:28.502 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2023-11-08T04:55:28.531 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd: failed to query services: (13) Permission denied 2023-11-08T04:55:28.549 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local last_update state 2023-11-08T04:55:28.550 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2023-11-08T04:55:28.551 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ last_update='2023-11-08 04:55:26' 2023-11-08T04:55:28.552 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2023-11-08T04:55:28.553 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ state=up+replaying 2023-11-08T04:55:28.554 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ echo up+replaying 2023-11-08T04:55:28.554 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ grep up+replaying 2023-11-08T04:55:28.554 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:up+replaying 2023-11-08T04:55:28.556 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T04:55:28.556 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T04:55:28.556 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ snap_name=snap2 2023-11-08T04:55:28.556 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ create_snap cluster2 mirror test snap2 2023-11-08T04:55:28.556 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T04:55:28.557 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T04:55:28.557 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T04:55:28.557 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap_name=snap2 2023-11-08T04:55:28.557 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap2 --debug-rbd=20 --debug-journaler=20 2023-11-08T04:55:54.016 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.497466688820851) > weight (0.25) 2023-11-08T04:55:54.016 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.1 2023-11-08T04:55:54.017 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Sent signal 15 2023-11-08T04:55:54.017 INFO:tasks.rbd_mirror_thrash:waiting for 6 secs before reviving daemons 2023-11-08T04:55:54.018 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi087.stderr:2023-11-08T04:55:54.016+0000 7fa16b38e700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 (PID: 120032) UID: 1000 2023-11-08T04:55:54.219 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi087.stderr:daemon-helper: command failed with exit status 1 2023-11-08T04:55:57.416 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_image_replay_started cluster1 mirror test 2023-11-08T04:55:57.416 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T04:55:57.416 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T04:55:57.417 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T04:55:57.417 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2023-11-08T04:55:57.417 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T04:55:57.417 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T04:55:57.417 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T04:55:57.418 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local state=started 2023-11-08T04:55:57.418 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local s 2023-11-08T04:55:57.418 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-11-08T04:55:57.418 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 1 2023-11-08T04:55:58.417 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test_image_replay_state cluster1 mirror test started 2023-11-08T04:55:58.417 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T04:55:58.417 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T04:55:58.418 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T04:55:58.418 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local test_state=started 2023-11-08T04:55:58.418 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local status_result 2023-11-08T04:55:58.418 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local current_state=stopped 2023-11-08T04:55:58.423 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2023-11-08T04:55:58.423 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_instance=cluster1 2023-11-08T04:55:58.424 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T04:55:58.424 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ grep -i state 2023-11-08T04:55:58.424 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1 2023-11-08T04:55:58.424 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance=cluster1 2023-11-08T04:55:58.424 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local loop_instance 2023-11-08T04:55:58.425 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-11-08T04:55:58.425 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sleep 0 2023-11-08T04:55:58.425 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-11-08T04:55:58.425 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ seq 0 3 2023-11-08T04:55:58.425 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T04:55:58.426 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2023-11-08T04:55:58.426 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1:0 2023-11-08T04:55:58.426 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T04:55:58.426 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance 2023-11-08T04:55:58.427 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-11-08T04:55:58.427 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local val=cluster1:0 2023-11-08T04:55:58.427 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_var_name=cluster 2023-11-08T04:55:58.427 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance_var_name=instance 2023-11-08T04:55:58.427 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cluster=cluster1 2023-11-08T04:55:58.428 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ instance=0 2023-11-08T04:55:58.428 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' 0 = cluster1:0 ']' 2023-11-08T04:55:58.428 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval cluster=cluster1 2023-11-08T04:55:58.428 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T04:55:58.428 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval instance=0 2023-11-08T04:55:58.429 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=0 2023-11-08T04:55:58.429 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-11-08T04:55:58.429 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local local_cluster=cluster1:0 2023-11-08T04:55:58.429 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster=cluster1 2023-11-08T04:55:58.429 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance 2023-11-08T04:55:58.429 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-11-08T04:55:58.430 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local val=cluster1:0 2023-11-08T04:55:58.430 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster_var_name=local_cluster 2023-11-08T04:55:58.430 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance_var_name=instance 2023-11-08T04:55:58.430 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T04:55:58.430 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=0 2023-11-08T04:55:58.431 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ '[' 0 = cluster1:0 ']' 2023-11-08T04:55:58.431 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval local_cluster=cluster1 2023-11-08T04:55:58.431 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ local_cluster=cluster1 2023-11-08T04:55:58.431 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval instance=0 2023-11-08T04:55:58.431 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ instance=0 2023-11-08T04:55:58.432 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-11-08T04:55:58.440 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-08T04:55:58.441 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-08T04:55:58.441 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-08T04:55:58.442 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2023-11-08T04:55:58.546 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ return 0 2023-11-08T04:55:58.547 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ status_result=' "state": "Replaying"' 2023-11-08T04:55:58.547 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ echo ' "state": "Replaying"' 2023-11-08T04:55:58.547 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ grep -i Replaying 2023-11-08T04:55:58.548 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: "state": "Replaying" 2023-11-08T04:55:58.550 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ current_state=started 2023-11-08T04:55:58.551 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test started = started 2023-11-08T04:55:58.551 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T04:55:58.551 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2023-11-08T04:55:58.551 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local local_cluster=cluster1 2023-11-08T04:55:58.551 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T04:55:58.552 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T04:55:58.552 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T04:55:58.552 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' snapshot = journal ']' 2023-11-08T04:55:58.552 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' snapshot = snapshot ']' 2023-11-08T04:55:58.552 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2023-11-08T04:55:58.553 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local local_cluster=cluster1 2023-11-08T04:55:58.553 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T04:55:58.553 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T04:55:58.553 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T04:55:58.554 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ mkfname cluster2-mirror-test.status 2023-11-08T04:55:58.554 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ echo cluster2-mirror-test.status 2023-11-08T04:55:58.554 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -e 's|[/ ]|_|g' 2023-11-08T04:55:58.554 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local status_log=/tmp/tmp.Ludk8ikCAB/cluster2-mirror-test.status 2023-11-08T04:55:58.555 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ mkfname cluster1-mirror-test.status 2023-11-08T04:55:58.555 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ echo cluster1-mirror-test.status 2023-11-08T04:55:58.555 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -e 's|[/ ]|_|g' 2023-11-08T04:55:58.556 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local local_status_log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.status 2023-11-08T04:55:58.556 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ mirror_image_snapshot cluster2 mirror test 2023-11-08T04:55:58.556 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T04:55:58.557 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T04:55:58.557 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T04:55:58.557 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2023-11-08T04:56:00.018 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.1 2023-11-08T04:56:00.019 INFO:teuthology.orchestra.run:waiting for 600 2023-11-08T04:56:00.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-08T04:56:00.020 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.1 2023-11-08T04:56:00.020 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Restarting daemon 2023-11-08T04:56:00.020 DEBUG:teuthology.orchestra.run.smithi087:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 2023-11-08T04:56:00.023 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Started 2023-11-08T04:56:00.024 INFO:tasks.rbd_mirror_thrash:waiting for 90 secs before thrashing 2023-11-08T04:56:00.428 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:Snapshot ID: 9 2023-11-08T04:56:00.441 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.Ludk8ikCAB/cluster2-mirror-test.status 2023-11-08T04:56:00.441 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T04:56:00.441 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T04:56:00.442 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T04:56:00.442 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster2-mirror-test.status 2023-11-08T04:56:00.442 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2023-11-08T04:56:00.442 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T04:56:00.476 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-11-08T04:56:00.478 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snapshot_id=9 2023-11-08T04:56:00.479 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ true 2023-11-08T04:56:00.479 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-11-08T04:56:00.479 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0.2 2023-11-08T04:56:00.680 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.status 2023-11-08T04:56:00.680 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T04:56:00.680 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T04:56:00.680 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T04:56:00.681 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.status 2023-11-08T04:56:00.681 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-11-08T04:56:00.681 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T04:56:00.717 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ true 2023-11-08T04:56:00.718 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-08T04:56:00.719 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:-:1.1: Document is empty 2023-11-08T04:56:00.719 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: 2023-11-08T04:56:00.719 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:^ 2023-11-08T04:56:00.720 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local primary_snapshot_id= 2023-11-08T04:56:00.720 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test 9 = '' 2023-11-08T04:56:00.720 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-11-08T04:56:00.721 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0.4 2023-11-08T04:56:01.121 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.status 2023-11-08T04:56:01.121 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T04:56:01.121 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T04:56:01.122 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T04:56:01.122 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.status 2023-11-08T04:56:01.122 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-11-08T04:56:01.122 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T04:56:01.159 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ true 2023-11-08T04:56:01.159 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-08T04:56:01.160 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:-:1.1: Document is empty 2023-11-08T04:56:01.160 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: 2023-11-08T04:56:01.161 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:^ 2023-11-08T04:56:01.161 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local primary_snapshot_id= 2023-11-08T04:56:01.162 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test 9 = '' 2023-11-08T04:56:01.162 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-11-08T04:56:01.162 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0.8 2023-11-08T04:56:01.962 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.status 2023-11-08T04:56:01.963 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T04:56:01.963 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T04:56:01.963 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T04:56:01.963 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.status 2023-11-08T04:56:01.963 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-11-08T04:56:01.964 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T04:56:02.003 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ true 2023-11-08T04:56:02.003 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-08T04:56:02.004 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:-:1.1: Document is empty 2023-11-08T04:56:02.004 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: 2023-11-08T04:56:02.004 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:^ 2023-11-08T04:56:02.005 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local primary_snapshot_id= 2023-11-08T04:56:02.005 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test 9 = '' 2023-11-08T04:56:02.006 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-11-08T04:56:02.006 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 1.6 2023-11-08T04:56:03.606 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.status 2023-11-08T04:56:03.606 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T04:56:03.607 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T04:56:03.607 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T04:56:03.607 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.status 2023-11-08T04:56:03.607 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-11-08T04:56:03.607 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T04:56:03.647 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-08T04:56:03.649 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local primary_snapshot_id=9 2023-11-08T04:56:03.649 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test 9 = 9 2023-11-08T04:56:03.649 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T04:56:03.650 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_snap_present cluster1 mirror test snap2 2023-11-08T04:56:03.650 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T04:56:03.650 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T04:56:03.650 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T04:56:03.651 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap_name=snap2 2023-11-08T04:56:03.651 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local s 2023-11-08T04:56:03.651 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-11-08T04:56:03.651 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 1 2023-11-08T04:56:04.649 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 info mirror/test@snap2 2023-11-08T04:56:04.686 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:rbd image 'test': 2023-11-08T04:56:04.687 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: size 512 MiB in 128 objects 2023-11-08T04:56:04.687 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: order 22 (4 MiB objects) 2023-11-08T04:56:04.687 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: snapshot_count: 4 2023-11-08T04:56:04.687 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: id: 10afad8fe72b 2023-11-08T04:56:04.688 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: block_name_prefix: rbd_data.10afad8fe72b 2023-11-08T04:56:04.688 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: format: 2 2023-11-08T04:56:04.688 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: features: layering, exclusive-lock, non-primary 2023-11-08T04:56:04.688 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: op_features: 2023-11-08T04:56:04.688 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: flags: 2023-11-08T04:56:04.689 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: create_timestamp: Wed Nov 8 04:54:28 2023 2023-11-08T04:56:04.689 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: access_timestamp: Wed Nov 8 04:54:28 2023 2023-11-08T04:56:04.689 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: modify_timestamp: Wed Nov 8 04:54:28 2023 2023-11-08T04:56:04.689 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: protected: False 2023-11-08T04:56:04.689 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring state: enabled 2023-11-08T04:56:04.690 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring mode: snapshot 2023-11-08T04:56:04.690 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring global id: 88ced7f9-2739-4a27-ad7a-30edec05edb1 2023-11-08T04:56:04.690 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring primary: false 2023-11-08T04:56:04.691 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T04:56:04.691 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' -n snap1-clean ']' 2023-11-08T04:56:04.692 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ compare_image_snaps mirror test snap1-clean 2023-11-08T04:56:04.692 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T04:56:04.692 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T04:56:04.692 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap_name=snap1-clean 2023-11-08T04:56:04.692 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local ret=0 2023-11-08T04:56:04.693 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local rmt_export=/tmp/tmp.Ludk8ikCAB/cluster2-mirror-test.export 2023-11-08T04:56:04.693 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local loc_export=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.export 2023-11-08T04:56:04.693 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rm -f /tmp/tmp.Ludk8ikCAB/cluster2-mirror-test.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.export 2023-11-08T04:56:04.693 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 -p mirror export test@snap1-clean /tmp/tmp.Ludk8ikCAB/cluster2-mirror-test.export 2023-11-08T04:56:05.238 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-11-08T04:56:05.242 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 -p mirror export test@snap1-clean /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.export 2023-11-08T04:56:05.974 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-11-08T04:56:05.977 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ cmp /tmp/tmp.Ludk8ikCAB/cluster2-mirror-test.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.export 2023-11-08T04:56:06.190 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rm -f /tmp/tmp.Ludk8ikCAB/cluster2-mirror-test.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.export 2023-11-08T04:56:06.332 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T04:56:06.332 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ compare_image_snaps mirror test snap2 2023-11-08T04:56:06.332 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T04:56:06.333 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T04:56:06.333 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap_name=snap2 2023-11-08T04:56:06.333 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local ret=0 2023-11-08T04:56:06.333 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local rmt_export=/tmp/tmp.Ludk8ikCAB/cluster2-mirror-test.export 2023-11-08T04:56:06.333 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local loc_export=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.export 2023-11-08T04:56:06.334 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rm -f /tmp/tmp.Ludk8ikCAB/cluster2-mirror-test.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.export 2023-11-08T04:56:06.334 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 -p mirror export test@snap2 /tmp/tmp.Ludk8ikCAB/cluster2-mirror-test.export 2023-11-08T04:56:06.931 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-11-08T04:56:06.934 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 -p mirror export test@snap2 /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.export 2023-11-08T04:56:07.694 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-11-08T04:56:07.697 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ cmp /tmp/tmp.Ludk8ikCAB/cluster2-mirror-test.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.export 2023-11-08T04:56:07.908 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rm -f /tmp/tmp.Ludk8ikCAB/cluster2-mirror-test.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.export 2023-11-08T04:56:08.049 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T04:56:08.049 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ clean_snap_name=snap2-clean 2023-11-08T04:56:08.049 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ create_snap cluster2 mirror test snap2-clean 2023-11-08T04:56:08.050 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T04:56:08.050 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T04:56:08.050 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T04:56:08.050 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap_name=snap2-clean 2023-11-08T04:56:08.050 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap2-clean --debug-rbd=20 --debug-journaler=20 2023-11-08T04:56:08.404 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for i in `seq 1 10` 2023-11-08T04:56:08.404 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ stress_write_image cluster2 mirror test 2023-11-08T04:56:08.404 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T04:56:08.405 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T04:56:08.405 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T04:56:08.405 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-11-08T04:56:08.406 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local duration=8 2023-11-08T04:56:08.406 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ set +e 2023-11-08T04:56:08.406 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2023-11-08T04:56:08.634 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:100 IOs, 37436416 bytes 2023-11-08T04:56:08.929 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:200 IOs, 139275264 bytes 2023-11-08T04:56:09.100 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:300 IOs, 172914688 bytes 2023-11-08T04:56:09.195 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:400 IOs, 182424576 bytes 2023-11-08T04:56:09.403 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:500 IOs, 253956096 bytes 2023-11-08T04:56:09.510 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:600 IOs, 277040128 bytes 2023-11-08T04:56:09.727 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:700 IOs, 351475712 bytes 2023-11-08T04:56:09.834 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:800 IOs, 352069632 bytes 2023-11-08T04:56:10.011 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:900 IOs, 411458560 bytes 2023-11-08T04:56:10.135 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1000 IOs, 435336192 bytes 2023-11-08T04:56:10.440 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1100 IOs, 572575744 bytes 2023-11-08T04:56:10.688 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1200 IOs, 646072320 bytes 2023-11-08T04:56:10.851 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1300 IOs, 692249600 bytes 2023-11-08T04:56:10.990 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1400 IOs, 740048896 bytes 2023-11-08T04:56:11.176 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1500 IOs, 778425344 bytes 2023-11-08T04:56:11.302 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1600 IOs, 795337728 bytes 2023-11-08T04:56:11.441 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1700 IOs, 826739712 bytes 2023-11-08T04:56:11.547 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1800 IOs, 846633984 bytes 2023-11-08T04:56:11.746 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1900 IOs, 938634240 bytes 2023-11-08T04:56:11.898 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2000 IOs, 944006144 bytes 2023-11-08T04:56:12.024 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2100 IOs, 969573376 bytes 2023-11-08T04:56:12.121 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2200 IOs, 978678784 bytes 2023-11-08T04:56:12.267 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2300 IOs, 1003177984 bytes 2023-11-08T04:56:12.419 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2400 IOs, 1043302400 bytes 2023-11-08T04:56:12.632 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2500 IOs, 1097334784 bytes 2023-11-08T04:56:12.790 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2600 IOs, 1141834752 bytes 2023-11-08T04:56:12.997 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2700 IOs, 1215099904 bytes 2023-11-08T04:56:13.170 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2800 IOs, 1242054656 bytes 2023-11-08T04:56:13.309 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2900 IOs, 1269842944 bytes 2023-11-08T04:56:13.485 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3000 IOs, 1322344448 bytes 2023-11-08T04:56:13.565 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3100 IOs, 1327391744 bytes 2023-11-08T04:56:13.671 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3200 IOs, 1342361600 bytes 2023-11-08T04:56:13.742 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3300 IOs, 1349686272 bytes 2023-11-08T04:56:13.937 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3400 IOs, 1409586176 bytes 2023-11-08T04:56:14.156 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3500 IOs, 1481006080 bytes 2023-11-08T04:56:14.349 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3600 IOs, 1525014528 bytes 2023-11-08T04:56:14.517 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3700 IOs, 1559285760 bytes 2023-11-08T04:56:14.610 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3800 IOs, 1563245568 bytes 2023-11-08T04:56:14.757 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3900 IOs, 1608859648 bytes 2023-11-08T04:56:14.855 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4000 IOs, 1624758272 bytes 2023-11-08T04:56:15.009 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.4396986462400788) > weight (0.25) 2023-11-08T04:56:15.010 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.535225179563372) > weight (0.25) 2023-11-08T04:56:15.010 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.4217999675908999) > weight (0.25) 2023-11-08T04:56:15.010 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.3 2023-11-08T04:56:15.011 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Sent signal 15 2023-11-08T04:56:15.011 INFO:tasks.rbd_mirror_thrash:waiting for 3 secs before reviving daemons 2023-11-08T04:56:15.011 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi087.stderr:2023-11-08T04:56:15.010+0000 7fb737005700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3 (PID: 120241) UID: 1000 2023-11-08T04:56:15.032 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4100 IOs, 1668897792 bytes 2023-11-08T04:56:15.212 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi087.stderr:daemon-helper: command failed with exit status 1 2023-11-08T04:56:15.230 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4200 IOs, 1730196480 bytes 2023-11-08T04:56:15.424 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4300 IOs, 1811077120 bytes 2023-11-08T04:56:15.669 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4400 IOs, 1896588288 bytes 2023-11-08T04:56:15.910 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4500 IOs, 1971356672 bytes 2023-11-08T04:56:16.137 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4600 IOs, 2048506880 bytes 2023-11-08T04:56:16.245 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4700 IOs, 2060065792 bytes 2023-11-08T04:56:16.410 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ error_code=124 2023-11-08T04:56:16.411 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ set -e 2023-11-08T04:56:16.411 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' 124 -eq 124 ']' 2023-11-08T04:56:16.411 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T04:56:16.412 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2023-11-08T04:56:16.412 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T04:56:16.412 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T04:56:16.412 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T04:56:16.413 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local state_pattern=up+replaying 2023-11-08T04:56:16.413 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local description_pattern= 2023-11-08T04:56:16.413 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local service_pattern= 2023-11-08T04:56:16.413 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-11-08T04:56:16.414 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 1 2023-11-08T04:56:17.412 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2023-11-08T04:56:17.412 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T04:56:17.413 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T04:56:17.413 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T04:56:17.413 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local state_pattern=up+replaying 2023-11-08T04:56:17.413 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local description_pattern= 2023-11-08T04:56:17.414 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local service_pattern= 2023-11-08T04:56:17.414 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ mkfname cluster1-mirror-test.mirror_status 2023-11-08T04:56:17.414 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ echo cluster1-mirror-test.mirror_status 2023-11-08T04:56:17.414 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -e 's|[/ ]|_|g' 2023-11-08T04:56:17.415 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local status_log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.mirror_status 2023-11-08T04:56:17.416 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ CEPH_ARGS= 2023-11-08T04:56:17.416 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 mirror image status mirror/test 2023-11-08T04:56:17.416 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ tee /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.mirror_status 2023-11-08T04:56:17.460 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:test: 2023-11-08T04:56:17.460 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: global_id: 88ced7f9-2739-4a27-ad7a-30edec05edb1 2023-11-08T04:56:17.461 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: state: up+replaying 2023-11-08T04:56:17.461 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":243629568.0,"last_snapshot_bytes":487259136,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1699419313,"remote_snapshot_timestamp":1699419313,"replay_state":"idle"} 2023-11-08T04:56:17.461 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: service: mirror.0 on smithi087 2023-11-08T04:56:17.461 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: last_update: 2023-11-08 04:55:56 2023-11-08T04:56:17.462 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: peer_sites: 2023-11-08T04:56:17.462 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: name: cluster2 2023-11-08T04:56:17.462 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: state: up+stopped 2023-11-08T04:56:17.462 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: description: local image is primary 2023-11-08T04:56:17.462 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: last_update: 2023-11-08 04:55:56 2023-11-08T04:56:17.465 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ grep '^ state: .*up+replaying' /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.mirror_status 2023-11-08T04:56:17.466 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: state: up+replaying 2023-11-08T04:56:17.466 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ grep '^ description: .*' /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.mirror_status 2023-11-08T04:56:17.467 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":243629568.0,"last_snapshot_bytes":487259136,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1699419313,"remote_snapshot_timestamp":1699419313,"replay_state":"idle"} 2023-11-08T04:56:17.467 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' -n '' ']' 2023-11-08T04:56:17.467 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ echo up+replaying 2023-11-08T04:56:17.468 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ grep '^up+' 2023-11-08T04:56:17.468 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:up+replaying 2023-11-08T04:56:17.469 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ grep 'service: *mirror..* on ' /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.mirror_status 2023-11-08T04:56:17.469 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: service: mirror.0 on smithi087 2023-11-08T04:56:17.469 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.mirror_status 2023-11-08T04:56:17.471 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local 'last_update=2023-11-08 04:55:56' 2023-11-08T04:56:17.471 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2023-11-08 04:55:56' 2023-11-08T04:56:17.472 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T04:56:17.472 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T04:56:17.472 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T04:56:17.472 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local state_pattern=up+replaying 2023-11-08T04:56:17.472 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local 'prev_last_update=2023-11-08 04:55:56' 2023-11-08T04:56:17.473 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ mkfname cluster1-mirror.mirror_status 2023-11-08T04:56:17.473 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ echo cluster1-mirror.mirror_status 2023-11-08T04:56:17.473 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -e 's|[/ ]|_|g' 2023-11-08T04:56:17.473 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local status_log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror.mirror_status 2023-11-08T04:56:17.474 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2023-11-08T04:56:17.504 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd: failed to query services: (13) Permission denied 2023-11-08T04:56:17.522 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local last_update state 2023-11-08T04:56:17.523 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2023-11-08T04:56:17.524 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ last_update='2023-11-08 04:55:56' 2023-11-08T04:56:17.525 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2023-11-08T04:56:17.526 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ state=up+replaying 2023-11-08T04:56:17.527 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ echo up+replaying 2023-11-08T04:56:17.527 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ grep up+replaying 2023-11-08T04:56:17.527 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:up+replaying 2023-11-08T04:56:17.528 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T04:56:17.529 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T04:56:17.529 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ snap_name=snap3 2023-11-08T04:56:17.529 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ create_snap cluster2 mirror test snap3 2023-11-08T04:56:17.529 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T04:56:17.530 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T04:56:17.530 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T04:56:17.530 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap_name=snap3 2023-11-08T04:56:17.530 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap3 --debug-rbd=20 --debug-journaler=20 2023-11-08T04:56:18.012 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.3 2023-11-08T04:56:18.012 INFO:teuthology.orchestra.run:waiting for 600 2023-11-08T04:56:18.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-08T04:56:18.013 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.3 2023-11-08T04:56:18.014 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Restarting daemon 2023-11-08T04:56:18.014 DEBUG:teuthology.orchestra.run.smithi087:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3 2023-11-08T04:56:18.018 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Started 2023-11-08T04:56:18.018 INFO:tasks.rbd_mirror_thrash:waiting for 85 secs before thrashing 2023-11-08T04:56:45.421 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_image_replay_started cluster1 mirror test 2023-11-08T04:56:45.421 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T04:56:45.422 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T04:56:45.422 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T04:56:45.422 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2023-11-08T04:56:45.422 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T04:56:45.422 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T04:56:45.423 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T04:56:45.423 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local state=started 2023-11-08T04:56:45.423 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local s 2023-11-08T04:56:45.423 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-11-08T04:56:45.423 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 1 2023-11-08T04:56:46.422 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test_image_replay_state cluster1 mirror test started 2023-11-08T04:56:46.423 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T04:56:46.423 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T04:56:46.423 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T04:56:46.423 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local test_state=started 2023-11-08T04:56:46.424 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local status_result 2023-11-08T04:56:46.424 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local current_state=stopped 2023-11-08T04:56:46.424 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2023-11-08T04:56:46.424 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_instance=cluster1 2023-11-08T04:56:46.424 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T04:56:46.425 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ grep -i state 2023-11-08T04:56:46.425 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1 2023-11-08T04:56:46.425 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance=cluster1 2023-11-08T04:56:46.425 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local loop_instance 2023-11-08T04:56:46.425 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-11-08T04:56:46.426 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sleep 0 2023-11-08T04:56:46.428 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-11-08T04:56:46.428 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ seq 0 3 2023-11-08T04:56:46.429 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T04:56:46.429 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2023-11-08T04:56:46.429 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1:0 2023-11-08T04:56:46.429 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T04:56:46.429 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance 2023-11-08T04:56:46.429 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-11-08T04:56:46.430 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local val=cluster1:0 2023-11-08T04:56:46.430 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_var_name=cluster 2023-11-08T04:56:46.430 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance_var_name=instance 2023-11-08T04:56:46.430 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cluster=cluster1 2023-11-08T04:56:46.430 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ instance=0 2023-11-08T04:56:46.431 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' 0 = cluster1:0 ']' 2023-11-08T04:56:46.431 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval cluster=cluster1 2023-11-08T04:56:46.431 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T04:56:46.431 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval instance=0 2023-11-08T04:56:46.431 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=0 2023-11-08T04:56:46.431 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-11-08T04:56:46.432 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local local_cluster=cluster1:0 2023-11-08T04:56:46.432 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster=cluster1 2023-11-08T04:56:46.432 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance 2023-11-08T04:56:46.432 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-11-08T04:56:46.432 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local val=cluster1:0 2023-11-08T04:56:46.433 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster_var_name=local_cluster 2023-11-08T04:56:46.433 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance_var_name=instance 2023-11-08T04:56:46.433 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T04:56:46.433 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=0 2023-11-08T04:56:46.433 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ '[' 0 = cluster1:0 ']' 2023-11-08T04:56:46.433 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval local_cluster=cluster1 2023-11-08T04:56:46.434 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ local_cluster=cluster1 2023-11-08T04:56:46.434 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval instance=0 2023-11-08T04:56:46.434 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ instance=0 2023-11-08T04:56:46.434 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-11-08T04:56:46.443 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-08T04:56:46.443 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-08T04:56:46.444 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-08T04:56:46.444 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2023-11-08T04:56:46.549 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ return 0 2023-11-08T04:56:46.550 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ status_result=' "state": "Replaying"' 2023-11-08T04:56:46.551 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ echo ' "state": "Replaying"' 2023-11-08T04:56:46.551 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ grep -i Replaying 2023-11-08T04:56:46.551 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: "state": "Replaying" 2023-11-08T04:56:46.553 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ current_state=started 2023-11-08T04:56:46.553 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test started = started 2023-11-08T04:56:46.553 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T04:56:46.554 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2023-11-08T04:56:46.554 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local local_cluster=cluster1 2023-11-08T04:56:46.554 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T04:56:46.554 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T04:56:46.554 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T04:56:46.555 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' snapshot = journal ']' 2023-11-08T04:56:46.555 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' snapshot = snapshot ']' 2023-11-08T04:56:46.555 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2023-11-08T04:56:46.555 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local local_cluster=cluster1 2023-11-08T04:56:46.555 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T04:56:46.556 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T04:56:46.556 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T04:56:46.556 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ mkfname cluster2-mirror-test.status 2023-11-08T04:56:46.556 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ echo cluster2-mirror-test.status 2023-11-08T04:56:46.556 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -e 's|[/ ]|_|g' 2023-11-08T04:56:46.557 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local status_log=/tmp/tmp.Ludk8ikCAB/cluster2-mirror-test.status 2023-11-08T04:56:46.558 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ mkfname cluster1-mirror-test.status 2023-11-08T04:56:46.559 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ echo cluster1-mirror-test.status 2023-11-08T04:56:46.559 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -e 's|[/ ]|_|g' 2023-11-08T04:56:46.560 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local local_status_log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.status 2023-11-08T04:56:46.560 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ mirror_image_snapshot cluster2 mirror test 2023-11-08T04:56:46.560 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T04:56:46.560 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T04:56:46.560 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T04:56:46.561 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2023-11-08T04:56:48.421 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:Snapshot ID: 13 2023-11-08T04:56:48.439 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.Ludk8ikCAB/cluster2-mirror-test.status 2023-11-08T04:56:48.439 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T04:56:48.440 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T04:56:48.440 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T04:56:48.440 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster2-mirror-test.status 2023-11-08T04:56:48.441 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2023-11-08T04:56:48.441 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T04:56:48.495 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-11-08T04:56:48.499 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snapshot_id=13 2023-11-08T04:56:48.499 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ true 2023-11-08T04:56:48.499 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-11-08T04:56:48.499 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0.2 2023-11-08T04:56:48.700 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.status 2023-11-08T04:56:48.701 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T04:56:48.701 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T04:56:48.701 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T04:56:48.701 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.status 2023-11-08T04:56:48.701 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-11-08T04:56:48.702 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T04:56:48.742 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ true 2023-11-08T04:56:48.743 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-08T04:56:48.744 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:-:1.1: Document is empty 2023-11-08T04:56:48.744 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: 2023-11-08T04:56:48.745 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:^ 2023-11-08T04:56:48.745 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local primary_snapshot_id= 2023-11-08T04:56:48.745 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test 13 = '' 2023-11-08T04:56:48.745 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-11-08T04:56:48.745 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0.4 2023-11-08T04:56:49.146 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.status 2023-11-08T04:56:49.146 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T04:56:49.147 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T04:56:49.147 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T04:56:49.147 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.status 2023-11-08T04:56:49.147 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-11-08T04:56:49.147 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T04:56:49.187 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ true 2023-11-08T04:56:49.188 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-08T04:56:49.189 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:-:1.1: Document is empty 2023-11-08T04:56:49.189 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: 2023-11-08T04:56:49.189 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:^ 2023-11-08T04:56:49.190 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local primary_snapshot_id= 2023-11-08T04:56:49.190 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test 13 = '' 2023-11-08T04:56:49.191 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-11-08T04:56:49.191 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0.8 2023-11-08T04:56:49.991 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.status 2023-11-08T04:56:49.991 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T04:56:49.992 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T04:56:49.992 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T04:56:49.992 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.status 2023-11-08T04:56:49.992 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-11-08T04:56:49.993 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T04:56:50.038 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ true 2023-11-08T04:56:50.038 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-08T04:56:50.039 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:-:1.1: Document is empty 2023-11-08T04:56:50.039 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: 2023-11-08T04:56:50.040 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:^ 2023-11-08T04:56:50.040 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local primary_snapshot_id= 2023-11-08T04:56:50.040 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test 13 = '' 2023-11-08T04:56:50.041 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-11-08T04:56:50.041 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 1.6 2023-11-08T04:56:51.641 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.status 2023-11-08T04:56:51.642 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T04:56:51.642 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T04:56:51.642 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T04:56:51.642 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.status 2023-11-08T04:56:51.643 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-11-08T04:56:51.643 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T04:56:51.681 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-08T04:56:51.683 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local primary_snapshot_id=13 2023-11-08T04:56:51.683 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test 13 = 13 2023-11-08T04:56:51.683 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T04:56:51.683 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_snap_present cluster1 mirror test snap3 2023-11-08T04:56:51.684 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T04:56:51.684 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T04:56:51.684 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T04:56:51.684 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap_name=snap3 2023-11-08T04:56:51.684 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local s 2023-11-08T04:56:51.685 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-11-08T04:56:51.685 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 1 2023-11-08T04:56:52.683 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 info mirror/test@snap3 2023-11-08T04:56:52.724 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:rbd image 'test': 2023-11-08T04:56:52.724 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: size 512 MiB in 128 objects 2023-11-08T04:56:52.724 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: order 22 (4 MiB objects) 2023-11-08T04:56:52.724 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: snapshot_count: 6 2023-11-08T04:56:52.725 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: id: 10afad8fe72b 2023-11-08T04:56:52.725 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: block_name_prefix: rbd_data.10afad8fe72b 2023-11-08T04:56:52.725 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: format: 2 2023-11-08T04:56:52.725 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: features: layering, exclusive-lock, non-primary 2023-11-08T04:56:52.726 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: op_features: 2023-11-08T04:56:52.726 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: flags: 2023-11-08T04:56:52.726 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: create_timestamp: Wed Nov 8 04:54:28 2023 2023-11-08T04:56:52.727 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: access_timestamp: Wed Nov 8 04:56:05 2023 2023-11-08T04:56:52.727 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: modify_timestamp: Wed Nov 8 04:54:28 2023 2023-11-08T04:56:52.727 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: protected: False 2023-11-08T04:56:52.728 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring state: enabled 2023-11-08T04:56:52.728 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring mode: snapshot 2023-11-08T04:56:52.728 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring global id: 88ced7f9-2739-4a27-ad7a-30edec05edb1 2023-11-08T04:56:52.729 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring primary: false 2023-11-08T04:56:52.731 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T04:56:52.731 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' -n snap2-clean ']' 2023-11-08T04:56:52.731 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ compare_image_snaps mirror test snap2-clean 2023-11-08T04:56:52.732 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T04:56:52.732 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T04:56:52.732 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap_name=snap2-clean 2023-11-08T04:56:52.733 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local ret=0 2023-11-08T04:56:52.733 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local rmt_export=/tmp/tmp.Ludk8ikCAB/cluster2-mirror-test.export 2023-11-08T04:56:52.733 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local loc_export=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.export 2023-11-08T04:56:52.734 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rm -f /tmp/tmp.Ludk8ikCAB/cluster2-mirror-test.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.export 2023-11-08T04:56:52.734 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 -p mirror export test@snap2-clean /tmp/tmp.Ludk8ikCAB/cluster2-mirror-test.export 2023-11-08T04:56:53.348 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-11-08T04:56:53.351 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 -p mirror export test@snap2-clean /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.export 2023-11-08T04:56:54.138 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-11-08T04:56:54.141 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ cmp /tmp/tmp.Ludk8ikCAB/cluster2-mirror-test.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.export 2023-11-08T04:56:54.353 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rm -f /tmp/tmp.Ludk8ikCAB/cluster2-mirror-test.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.export 2023-11-08T04:56:54.495 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T04:56:54.496 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ compare_image_snaps mirror test snap3 2023-11-08T04:56:54.496 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T04:56:54.496 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T04:56:54.496 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap_name=snap3 2023-11-08T04:56:54.497 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local ret=0 2023-11-08T04:56:54.497 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local rmt_export=/tmp/tmp.Ludk8ikCAB/cluster2-mirror-test.export 2023-11-08T04:56:54.497 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local loc_export=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.export 2023-11-08T04:56:54.497 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rm -f /tmp/tmp.Ludk8ikCAB/cluster2-mirror-test.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.export 2023-11-08T04:56:54.497 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 -p mirror export test@snap3 /tmp/tmp.Ludk8ikCAB/cluster2-mirror-test.export 2023-11-08T04:56:55.097 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-11-08T04:56:55.101 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 -p mirror export test@snap3 /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.export 2023-11-08T04:56:55.872 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-11-08T04:56:55.875 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ cmp /tmp/tmp.Ludk8ikCAB/cluster2-mirror-test.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.export 2023-11-08T04:56:56.086 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rm -f /tmp/tmp.Ludk8ikCAB/cluster2-mirror-test.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.export 2023-11-08T04:56:56.232 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T04:56:56.232 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ clean_snap_name=snap3-clean 2023-11-08T04:56:56.232 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ create_snap cluster2 mirror test snap3-clean 2023-11-08T04:56:56.232 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T04:56:56.232 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T04:56:56.233 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T04:56:56.233 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap_name=snap3-clean 2023-11-08T04:56:56.233 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap3-clean --debug-rbd=20 --debug-journaler=20 2023-11-08T04:56:56.477 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for i in `seq 1 10` 2023-11-08T04:56:56.477 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ stress_write_image cluster2 mirror test 2023-11-08T04:56:56.478 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T04:56:56.478 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T04:56:56.478 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T04:56:56.478 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-11-08T04:56:56.479 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local duration=9 2023-11-08T04:56:56.479 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ set +e 2023-11-08T04:56:56.479 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2023-11-08T04:56:56.886 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:100 IOs, 99308544 bytes 2023-11-08T04:56:57.074 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:200 IOs, 153315328 bytes 2023-11-08T04:56:57.313 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:300 IOs, 232223744 bytes 2023-11-08T04:56:57.416 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:400 IOs, 243237888 bytes 2023-11-08T04:56:57.578 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:500 IOs, 278281216 bytes 2023-11-08T04:56:57.708 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:600 IOs, 301563904 bytes 2023-11-08T04:56:57.868 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:700 IOs, 347574272 bytes 2023-11-08T04:56:58.012 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:800 IOs, 386183168 bytes 2023-11-08T04:56:58.186 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:900 IOs, 446383104 bytes 2023-11-08T04:56:58.311 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1000 IOs, 477688832 bytes 2023-11-08T04:56:58.459 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1100 IOs, 495873024 bytes 2023-11-08T04:56:58.590 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1200 IOs, 525639680 bytes 2023-11-08T04:56:58.728 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1300 IOs, 561580032 bytes 2023-11-08T04:56:58.842 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1400 IOs, 586433536 bytes 2023-11-08T04:56:58.986 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1500 IOs, 617562112 bytes 2023-11-08T04:56:59.170 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1600 IOs, 677569536 bytes 2023-11-08T04:56:59.367 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1700 IOs, 725588992 bytes 2023-11-08T04:56:59.515 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1800 IOs, 765830144 bytes 2023-11-08T04:56:59.616 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1900 IOs, 789403648 bytes 2023-11-08T04:56:59.726 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2000 IOs, 825965568 bytes 2023-11-08T04:56:59.897 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2100 IOs, 864626688 bytes 2023-11-08T04:57:00.065 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2200 IOs, 904535040 bytes 2023-11-08T04:57:00.160 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2300 IOs, 917554176 bytes 2023-11-08T04:57:00.290 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2400 IOs, 945912832 bytes 2023-11-08T04:57:00.483 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2500 IOs, 1027808256 bytes 2023-11-08T04:57:00.724 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2600 IOs, 1088306176 bytes 2023-11-08T04:57:00.915 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2700 IOs, 1145673728 bytes 2023-11-08T04:57:01.149 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2800 IOs, 1219540992 bytes 2023-11-08T04:57:01.220 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2900 IOs, 1224125440 bytes 2023-11-08T04:57:01.386 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3000 IOs, 1269673984 bytes 2023-11-08T04:57:01.546 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3100 IOs, 1333807104 bytes 2023-11-08T04:57:01.878 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3200 IOs, 1447933952 bytes 2023-11-08T04:57:01.975 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3300 IOs, 1448543232 bytes 2023-11-08T04:57:02.229 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3400 IOs, 1529016320 bytes 2023-11-08T04:57:02.325 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3500 IOs, 1541659648 bytes 2023-11-08T04:57:02.540 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3600 IOs, 1611596800 bytes 2023-11-08T04:57:02.698 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3700 IOs, 1650806784 bytes 2023-11-08T04:57:02.844 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3800 IOs, 1677537280 bytes 2023-11-08T04:57:03.000 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3900 IOs, 1724150784 bytes 2023-11-08T04:57:03.214 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4000 IOs, 1793925120 bytes 2023-11-08T04:57:03.408 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4100 IOs, 1858333696 bytes 2023-11-08T04:57:03.494 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4200 IOs, 1873993728 bytes 2023-11-08T04:57:03.615 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4300 IOs, 1893123072 bytes 2023-11-08T04:57:03.738 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4400 IOs, 1917845504 bytes 2023-11-08T04:57:03.831 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4500 IOs, 1934103552 bytes 2023-11-08T04:57:04.200 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4600 IOs, 2078249984 bytes 2023-11-08T04:57:04.315 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4700 IOs, 2102964224 bytes 2023-11-08T04:57:04.559 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4800 IOs, 2180783104 bytes 2023-11-08T04:57:04.673 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4900 IOs, 2201494528 bytes 2023-11-08T04:57:04.942 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5000 IOs, 2297805824 bytes 2023-11-08T04:57:05.074 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5100 IOs, 2325017600 bytes 2023-11-08T04:57:05.198 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5200 IOs, 2352850944 bytes 2023-11-08T04:57:05.341 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5300 IOs, 2378790912 bytes 2023-11-08T04:57:05.445 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5400 IOs, 2381852672 bytes 2023-11-08T04:57:05.484 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ error_code=124 2023-11-08T04:57:05.484 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ set -e 2023-11-08T04:57:05.484 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' 124 -eq 124 ']' 2023-11-08T04:57:05.484 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T04:57:05.485 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2023-11-08T04:57:05.485 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T04:57:05.485 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T04:57:05.485 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T04:57:05.485 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local state_pattern=up+replaying 2023-11-08T04:57:05.486 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local description_pattern= 2023-11-08T04:57:05.486 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local service_pattern= 2023-11-08T04:57:05.486 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-11-08T04:57:05.486 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 1 2023-11-08T04:57:06.485 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2023-11-08T04:57:06.485 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T04:57:06.486 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T04:57:06.486 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T04:57:06.486 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local state_pattern=up+replaying 2023-11-08T04:57:06.486 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local description_pattern= 2023-11-08T04:57:06.487 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local service_pattern= 2023-11-08T04:57:06.487 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ mkfname cluster1-mirror-test.mirror_status 2023-11-08T04:57:06.487 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ echo cluster1-mirror-test.mirror_status 2023-11-08T04:57:06.487 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -e 's|[/ ]|_|g' 2023-11-08T04:57:06.488 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local status_log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.mirror_status 2023-11-08T04:57:06.488 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ CEPH_ARGS= 2023-11-08T04:57:06.488 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 mirror image status mirror/test 2023-11-08T04:57:06.488 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ tee /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.mirror_status 2023-11-08T04:57:06.530 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:test: 2023-11-08T04:57:06.531 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: global_id: 88ced7f9-2739-4a27-ad7a-30edec05edb1 2023-11-08T04:57:06.531 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: state: up+replaying 2023-11-08T04:57:06.531 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: description: replaying, {"bytes_per_second":17398408.53,"bytes_per_snapshot":518865920.0,"last_snapshot_bytes":521952256,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1699419407,"remote_snapshot_timestamp":1699419407,"replay_state":"idle"} 2023-11-08T04:57:06.532 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: service: mirror.0 on smithi087 2023-11-08T04:57:06.532 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: last_update: 2023-11-08 04:56:56 2023-11-08T04:57:06.532 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: peer_sites: 2023-11-08T04:57:06.532 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: name: cluster2 2023-11-08T04:57:06.532 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: state: up+stopped 2023-11-08T04:57:06.533 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: description: local image is primary 2023-11-08T04:57:06.533 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: last_update: 2023-11-08 04:56:56 2023-11-08T04:57:06.535 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ grep '^ state: .*up+replaying' /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.mirror_status 2023-11-08T04:57:06.536 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: state: up+replaying 2023-11-08T04:57:06.537 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ grep '^ description: .*' /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.mirror_status 2023-11-08T04:57:06.537 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: description: replaying, {"bytes_per_second":17398408.53,"bytes_per_snapshot":518865920.0,"last_snapshot_bytes":521952256,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1699419407,"remote_snapshot_timestamp":1699419407,"replay_state":"idle"} 2023-11-08T04:57:06.538 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' -n '' ']' 2023-11-08T04:57:06.538 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ echo up+replaying 2023-11-08T04:57:06.538 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ grep '^up+' 2023-11-08T04:57:06.539 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:up+replaying 2023-11-08T04:57:06.539 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ grep 'service: *mirror..* on ' /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.mirror_status 2023-11-08T04:57:06.540 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: service: mirror.0 on smithi087 2023-11-08T04:57:06.540 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.mirror_status 2023-11-08T04:57:06.542 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local 'last_update=2023-11-08 04:56:56' 2023-11-08T04:57:06.542 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2023-11-08 04:56:56' 2023-11-08T04:57:06.542 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T04:57:06.542 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T04:57:06.543 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T04:57:06.543 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local state_pattern=up+replaying 2023-11-08T04:57:06.543 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local 'prev_last_update=2023-11-08 04:56:56' 2023-11-08T04:57:06.543 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ mkfname cluster1-mirror.mirror_status 2023-11-08T04:57:06.543 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ echo cluster1-mirror.mirror_status 2023-11-08T04:57:06.544 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -e 's|[/ ]|_|g' 2023-11-08T04:57:06.544 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local status_log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror.mirror_status 2023-11-08T04:57:06.544 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2023-11-08T04:57:06.574 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd: failed to query services: (13) Permission denied 2023-11-08T04:57:06.595 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local last_update state 2023-11-08T04:57:06.595 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2023-11-08T04:57:06.597 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ last_update='2023-11-08 04:56:56' 2023-11-08T04:57:06.597 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2023-11-08T04:57:06.599 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ state=up+replaying 2023-11-08T04:57:06.599 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ echo up+replaying 2023-11-08T04:57:06.599 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ grep up+replaying 2023-11-08T04:57:06.600 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:up+replaying 2023-11-08T04:57:06.601 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T04:57:06.601 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T04:57:06.602 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ snap_name=snap4 2023-11-08T04:57:06.602 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ create_snap cluster2 mirror test snap4 2023-11-08T04:57:06.602 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T04:57:06.602 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T04:57:06.602 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T04:57:06.603 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap_name=snap4 2023-11-08T04:57:06.603 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap4 --debug-rbd=20 --debug-journaler=20 2023-11-08T04:57:30.025 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.0 2023-11-08T04:57:30.027 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Sent signal 15 2023-11-08T04:57:30.027 INFO:tasks.rbd_mirror_thrash:waiting for 7 secs before reviving daemons 2023-11-08T04:57:30.028 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi087.stderr:2023-11-08T04:57:30.026+0000 7f11d10b3700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 120010) UID: 1000 2023-11-08T04:57:30.228 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi087.stderr:daemon-helper: command failed with exit status 1 2023-11-08T04:57:35.473 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_image_replay_started cluster1 mirror test 2023-11-08T04:57:35.474 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T04:57:35.474 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T04:57:35.474 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T04:57:35.475 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2023-11-08T04:57:35.475 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T04:57:35.475 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T04:57:35.475 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T04:57:35.475 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local state=started 2023-11-08T04:57:35.475 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local s 2023-11-08T04:57:35.476 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-11-08T04:57:35.476 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 1 2023-11-08T04:57:36.476 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test_image_replay_state cluster1 mirror test started 2023-11-08T04:57:36.476 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T04:57:36.476 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T04:57:36.476 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T04:57:36.477 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local test_state=started 2023-11-08T04:57:36.477 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local status_result 2023-11-08T04:57:36.477 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local current_state=stopped 2023-11-08T04:57:36.477 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2023-11-08T04:57:36.477 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_instance=cluster1 2023-11-08T04:57:36.478 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T04:57:36.478 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ grep -i state 2023-11-08T04:57:36.478 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1 2023-11-08T04:57:36.478 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance=cluster1 2023-11-08T04:57:36.478 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local loop_instance 2023-11-08T04:57:36.479 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-11-08T04:57:36.479 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sleep 0 2023-11-08T04:57:36.482 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-11-08T04:57:36.482 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ seq 0 3 2023-11-08T04:57:36.482 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T04:57:36.482 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2023-11-08T04:57:36.483 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1:0 2023-11-08T04:57:36.483 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T04:57:36.483 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance 2023-11-08T04:57:36.483 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-11-08T04:57:36.483 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local val=cluster1:0 2023-11-08T04:57:36.483 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_var_name=cluster 2023-11-08T04:57:36.484 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance_var_name=instance 2023-11-08T04:57:36.484 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cluster=cluster1 2023-11-08T04:57:36.484 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ instance=0 2023-11-08T04:57:36.484 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' 0 = cluster1:0 ']' 2023-11-08T04:57:36.484 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval cluster=cluster1 2023-11-08T04:57:36.485 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T04:57:36.485 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval instance=0 2023-11-08T04:57:36.485 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=0 2023-11-08T04:57:36.485 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-11-08T04:57:36.485 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local local_cluster=cluster1:0 2023-11-08T04:57:36.486 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster=cluster1 2023-11-08T04:57:36.486 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance 2023-11-08T04:57:36.486 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-11-08T04:57:36.486 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local val=cluster1:0 2023-11-08T04:57:36.486 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster_var_name=local_cluster 2023-11-08T04:57:36.487 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance_var_name=instance 2023-11-08T04:57:36.487 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T04:57:36.487 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=0 2023-11-08T04:57:36.487 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ '[' 0 = cluster1:0 ']' 2023-11-08T04:57:36.487 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval local_cluster=cluster1 2023-11-08T04:57:36.487 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ local_cluster=cluster1 2023-11-08T04:57:36.488 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval instance=0 2023-11-08T04:57:36.488 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ instance=0 2023-11-08T04:57:36.488 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-11-08T04:57:36.496 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-08T04:57:36.496 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-08T04:57:36.497 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-08T04:57:36.497 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2023-11-08T04:57:36.582 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:admin_socket: exception getting command descriptions: [Errno 2] No such file or directory 2023-11-08T04:57:36.586 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T04:57:36.586 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2023-11-08T04:57:36.586 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1:1 2023-11-08T04:57:36.587 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T04:57:36.587 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance 2023-11-08T04:57:36.587 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-11-08T04:57:36.587 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local val=cluster1:1 2023-11-08T04:57:36.587 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_var_name=cluster 2023-11-08T04:57:36.588 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance_var_name=instance 2023-11-08T04:57:36.588 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cluster=cluster1 2023-11-08T04:57:36.588 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ instance=1 2023-11-08T04:57:36.588 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' 1 = cluster1:1 ']' 2023-11-08T04:57:36.588 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval cluster=cluster1 2023-11-08T04:57:36.590 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T04:57:36.590 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval instance=1 2023-11-08T04:57:36.590 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=1 2023-11-08T04:57:36.591 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-11-08T04:57:36.591 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local local_cluster=cluster1:1 2023-11-08T04:57:36.591 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster=cluster1 2023-11-08T04:57:36.591 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance 2023-11-08T04:57:36.591 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-11-08T04:57:36.592 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local val=cluster1:1 2023-11-08T04:57:36.592 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster_var_name=local_cluster 2023-11-08T04:57:36.593 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance_var_name=instance 2023-11-08T04:57:36.593 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T04:57:36.593 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=1 2023-11-08T04:57:36.593 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ '[' 1 = cluster1:1 ']' 2023-11-08T04:57:36.593 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval local_cluster=cluster1 2023-11-08T04:57:36.594 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ local_cluster=cluster1 2023-11-08T04:57:36.594 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval instance=1 2023-11-08T04:57:36.594 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ instance=1 2023-11-08T04:57:36.594 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-11-08T04:57:36.600 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-08T04:57:36.600 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-08T04:57:36.601 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-08T04:57:36.601 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2023-11-08T04:57:36.689 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-11-08T04:57:36.721 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd mirror status : get status for rbd mirror 2023-11-08T04:57:36.722 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:admin_socket: invalid command 2023-11-08T04:57:36.725 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T04:57:36.725 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2023-11-08T04:57:36.726 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1:2 2023-11-08T04:57:36.726 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T04:57:36.726 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance 2023-11-08T04:57:36.726 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-11-08T04:57:36.726 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local val=cluster1:2 2023-11-08T04:57:36.727 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_var_name=cluster 2023-11-08T04:57:36.727 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance_var_name=instance 2023-11-08T04:57:36.727 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cluster=cluster1 2023-11-08T04:57:36.727 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ instance=2 2023-11-08T04:57:36.727 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' 2 = cluster1:2 ']' 2023-11-08T04:57:36.728 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval cluster=cluster1 2023-11-08T04:57:36.728 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T04:57:36.728 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval instance=2 2023-11-08T04:57:36.728 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=2 2023-11-08T04:57:36.728 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-11-08T04:57:36.728 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local local_cluster=cluster1:2 2023-11-08T04:57:36.729 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster=cluster1 2023-11-08T04:57:36.729 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance 2023-11-08T04:57:36.729 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-11-08T04:57:36.729 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local val=cluster1:2 2023-11-08T04:57:36.729 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster_var_name=local_cluster 2023-11-08T04:57:36.730 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance_var_name=instance 2023-11-08T04:57:36.730 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T04:57:36.730 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=2 2023-11-08T04:57:36.730 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ '[' 2 = cluster1:2 ']' 2023-11-08T04:57:36.731 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval local_cluster=cluster1 2023-11-08T04:57:36.731 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ local_cluster=cluster1 2023-11-08T04:57:36.731 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval instance=2 2023-11-08T04:57:36.731 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ instance=2 2023-11-08T04:57:36.731 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-11-08T04:57:36.732 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-08T04:57:36.732 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-08T04:57:36.732 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-08T04:57:36.732 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2023-11-08T04:57:36.793 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-11-08T04:57:36.793 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd mirror status : get status for rbd mirror 2023-11-08T04:57:36.793 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:admin_socket: invalid command 2023-11-08T04:57:36.797 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T04:57:36.797 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2023-11-08T04:57:36.797 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1:3 2023-11-08T04:57:36.797 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T04:57:36.797 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance 2023-11-08T04:57:36.798 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ set_cluster_instance cluster1:3 cluster instance 2023-11-08T04:57:36.798 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local val=cluster1:3 2023-11-08T04:57:36.798 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_var_name=cluster 2023-11-08T04:57:36.798 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance_var_name=instance 2023-11-08T04:57:36.799 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cluster=cluster1 2023-11-08T04:57:36.799 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ instance=3 2023-11-08T04:57:36.799 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' 3 = cluster1:3 ']' 2023-11-08T04:57:36.799 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval cluster=cluster1 2023-11-08T04:57:36.800 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T04:57:36.800 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval instance=3 2023-11-08T04:57:36.800 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=3 2023-11-08T04:57:36.800 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ daemon_asok_file cluster1:3 cluster1 2023-11-08T04:57:36.800 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local local_cluster=cluster1:3 2023-11-08T04:57:36.801 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster=cluster1 2023-11-08T04:57:36.801 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance 2023-11-08T04:57:36.801 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2023-11-08T04:57:36.801 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local val=cluster1:3 2023-11-08T04:57:36.801 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster_var_name=local_cluster 2023-11-08T04:57:36.802 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance_var_name=instance 2023-11-08T04:57:36.802 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T04:57:36.802 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=3 2023-11-08T04:57:36.802 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ '[' 3 = cluster1:3 ']' 2023-11-08T04:57:36.803 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval local_cluster=cluster1 2023-11-08T04:57:36.803 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ local_cluster=cluster1 2023-11-08T04:57:36.803 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval instance=3 2023-11-08T04:57:36.803 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ instance=3 2023-11-08T04:57:36.803 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2023-11-08T04:57:36.811 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-11-08T04:57:36.811 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-11-08T04:57:36.812 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-11-08T04:57:36.812 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2023-11-08T04:57:36.899 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-11-08T04:57:36.899 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd mirror status : get status for rbd mirror 2023-11-08T04:57:36.899 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:admin_socket: invalid command 2023-11-08T04:57:36.900 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-11-08T04:57:36.900 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sleep 1 2023-11-08T04:57:37.028 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.0 2023-11-08T04:57:37.029 INFO:teuthology.orchestra.run:waiting for 600 2023-11-08T04:57:37.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-08T04:57:37.029 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.0 2023-11-08T04:57:37.030 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2023-11-08T04:57:37.030 DEBUG:teuthology.orchestra.run.smithi087:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2023-11-08T04:57:37.033 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2023-11-08T04:57:37.033 INFO:tasks.rbd_mirror_thrash:waiting for 84 secs before thrashing 2023-11-08T04:57:37.901 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-11-08T04:57:37.901 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ seq 0 3 2023-11-08T04:57:37.904 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T04:57:37.905 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2023-11-08T04:57:37.905 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1:0 2023-11-08T04:57:37.905 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T04:57:37.905 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance 2023-11-08T04:57:37.906 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-11-08T04:57:37.906 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local val=cluster1:0 2023-11-08T04:57:37.906 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_var_name=cluster 2023-11-08T04:57:37.906 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance_var_name=instance 2023-11-08T04:57:37.906 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cluster=cluster1 2023-11-08T04:57:37.907 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ instance=0 2023-11-08T04:57:37.907 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' 0 = cluster1:0 ']' 2023-11-08T04:57:37.907 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval cluster=cluster1 2023-11-08T04:57:37.907 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T04:57:37.907 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval instance=0 2023-11-08T04:57:37.908 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=0 2023-11-08T04:57:37.908 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-11-08T04:57:37.908 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local local_cluster=cluster1:0 2023-11-08T04:57:37.908 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster=cluster1 2023-11-08T04:57:37.908 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance 2023-11-08T04:57:37.909 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-11-08T04:57:37.909 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local val=cluster1:0 2023-11-08T04:57:37.909 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster_var_name=local_cluster 2023-11-08T04:57:37.909 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance_var_name=instance 2023-11-08T04:57:37.909 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T04:57:37.910 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=0 2023-11-08T04:57:37.910 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ '[' 0 = cluster1:0 ']' 2023-11-08T04:57:37.910 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval local_cluster=cluster1 2023-11-08T04:57:37.910 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ local_cluster=cluster1 2023-11-08T04:57:37.910 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval instance=0 2023-11-08T04:57:37.911 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ instance=0 2023-11-08T04:57:37.911 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-11-08T04:57:37.921 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-08T04:57:37.921 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-08T04:57:37.922 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-08T04:57:37.922 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2023-11-08T04:57:38.015 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-11-08T04:57:38.016 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd mirror status : get status for rbd mirror 2023-11-08T04:57:38.016 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:admin_socket: invalid command 2023-11-08T04:57:38.019 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T04:57:38.019 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2023-11-08T04:57:38.019 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1:1 2023-11-08T04:57:38.020 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T04:57:38.020 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance 2023-11-08T04:57:38.020 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-11-08T04:57:38.020 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local val=cluster1:1 2023-11-08T04:57:38.020 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_var_name=cluster 2023-11-08T04:57:38.021 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance_var_name=instance 2023-11-08T04:57:38.021 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cluster=cluster1 2023-11-08T04:57:38.021 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ instance=1 2023-11-08T04:57:38.021 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' 1 = cluster1:1 ']' 2023-11-08T04:57:38.021 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval cluster=cluster1 2023-11-08T04:57:38.022 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T04:57:38.022 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval instance=1 2023-11-08T04:57:38.022 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=1 2023-11-08T04:57:38.022 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-11-08T04:57:38.022 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local local_cluster=cluster1:1 2023-11-08T04:57:38.023 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster=cluster1 2023-11-08T04:57:38.023 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance 2023-11-08T04:57:38.023 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-11-08T04:57:38.023 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local val=cluster1:1 2023-11-08T04:57:38.023 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster_var_name=local_cluster 2023-11-08T04:57:38.024 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance_var_name=instance 2023-11-08T04:57:38.024 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T04:57:38.024 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=1 2023-11-08T04:57:38.024 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ '[' 1 = cluster1:1 ']' 2023-11-08T04:57:38.024 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval local_cluster=cluster1 2023-11-08T04:57:38.025 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ local_cluster=cluster1 2023-11-08T04:57:38.025 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval instance=1 2023-11-08T04:57:38.025 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ instance=1 2023-11-08T04:57:38.025 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-11-08T04:57:38.032 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-08T04:57:38.033 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-08T04:57:38.033 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-08T04:57:38.033 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2023-11-08T04:57:38.121 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-11-08T04:57:38.121 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd mirror status : get status for rbd mirror 2023-11-08T04:57:38.121 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:admin_socket: invalid command 2023-11-08T04:57:38.124 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T04:57:38.125 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2023-11-08T04:57:38.125 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1:2 2023-11-08T04:57:38.125 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T04:57:38.125 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance 2023-11-08T04:57:38.125 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-11-08T04:57:38.126 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local val=cluster1:2 2023-11-08T04:57:38.126 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_var_name=cluster 2023-11-08T04:57:38.126 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance_var_name=instance 2023-11-08T04:57:38.126 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cluster=cluster1 2023-11-08T04:57:38.126 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ instance=2 2023-11-08T04:57:38.127 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' 2 = cluster1:2 ']' 2023-11-08T04:57:38.127 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval cluster=cluster1 2023-11-08T04:57:38.127 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T04:57:38.127 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval instance=2 2023-11-08T04:57:38.127 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=2 2023-11-08T04:57:38.128 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-11-08T04:57:38.128 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local local_cluster=cluster1:2 2023-11-08T04:57:38.128 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster=cluster1 2023-11-08T04:57:38.128 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance 2023-11-08T04:57:38.128 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-11-08T04:57:38.128 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local val=cluster1:2 2023-11-08T04:57:38.129 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster_var_name=local_cluster 2023-11-08T04:57:38.129 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance_var_name=instance 2023-11-08T04:57:38.129 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T04:57:38.129 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=2 2023-11-08T04:57:38.129 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ '[' 2 = cluster1:2 ']' 2023-11-08T04:57:38.130 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval local_cluster=cluster1 2023-11-08T04:57:38.130 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ local_cluster=cluster1 2023-11-08T04:57:38.130 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval instance=2 2023-11-08T04:57:38.130 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ instance=2 2023-11-08T04:57:38.130 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-11-08T04:57:38.139 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-08T04:57:38.139 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-08T04:57:38.139 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-08T04:57:38.139 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2023-11-08T04:57:38.227 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-11-08T04:57:38.228 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd mirror status : get status for rbd mirror 2023-11-08T04:57:38.228 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:admin_socket: invalid command 2023-11-08T04:57:38.231 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T04:57:38.231 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2023-11-08T04:57:38.231 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1:3 2023-11-08T04:57:38.232 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T04:57:38.232 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance 2023-11-08T04:57:38.232 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ set_cluster_instance cluster1:3 cluster instance 2023-11-08T04:57:38.232 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local val=cluster1:3 2023-11-08T04:57:38.232 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_var_name=cluster 2023-11-08T04:57:38.233 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance_var_name=instance 2023-11-08T04:57:38.233 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cluster=cluster1 2023-11-08T04:57:38.233 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ instance=3 2023-11-08T04:57:38.233 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' 3 = cluster1:3 ']' 2023-11-08T04:57:38.233 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval cluster=cluster1 2023-11-08T04:57:38.234 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T04:57:38.234 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval instance=3 2023-11-08T04:57:38.234 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=3 2023-11-08T04:57:38.234 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ daemon_asok_file cluster1:3 cluster1 2023-11-08T04:57:38.235 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local local_cluster=cluster1:3 2023-11-08T04:57:38.235 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster=cluster1 2023-11-08T04:57:38.235 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance 2023-11-08T04:57:38.235 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2023-11-08T04:57:38.235 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local val=cluster1:3 2023-11-08T04:57:38.236 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster_var_name=local_cluster 2023-11-08T04:57:38.236 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance_var_name=instance 2023-11-08T04:57:38.236 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T04:57:38.236 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=3 2023-11-08T04:57:38.236 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ '[' 3 = cluster1:3 ']' 2023-11-08T04:57:38.237 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval local_cluster=cluster1 2023-11-08T04:57:38.237 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ local_cluster=cluster1 2023-11-08T04:57:38.237 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval instance=3 2023-11-08T04:57:38.237 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ instance=3 2023-11-08T04:57:38.237 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2023-11-08T04:57:38.244 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-11-08T04:57:38.245 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-11-08T04:57:38.245 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-11-08T04:57:38.245 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2023-11-08T04:57:38.332 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-11-08T04:57:38.333 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd mirror status : get status for rbd mirror 2023-11-08T04:57:38.333 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:admin_socket: invalid command 2023-11-08T04:57:38.333 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-11-08T04:57:38.334 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sleep 2 2023-11-08T04:57:40.335 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-11-08T04:57:40.335 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ seq 0 3 2023-11-08T04:57:40.339 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T04:57:40.339 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2023-11-08T04:57:40.339 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1:0 2023-11-08T04:57:40.339 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T04:57:40.340 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance 2023-11-08T04:57:40.340 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-11-08T04:57:40.340 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local val=cluster1:0 2023-11-08T04:57:40.340 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_var_name=cluster 2023-11-08T04:57:40.340 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance_var_name=instance 2023-11-08T04:57:40.341 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cluster=cluster1 2023-11-08T04:57:40.341 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ instance=0 2023-11-08T04:57:40.341 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' 0 = cluster1:0 ']' 2023-11-08T04:57:40.341 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval cluster=cluster1 2023-11-08T04:57:40.341 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T04:57:40.342 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval instance=0 2023-11-08T04:57:40.342 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=0 2023-11-08T04:57:40.342 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-11-08T04:57:40.342 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local local_cluster=cluster1:0 2023-11-08T04:57:40.343 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster=cluster1 2023-11-08T04:57:40.343 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance 2023-11-08T04:57:40.343 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-11-08T04:57:40.343 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local val=cluster1:0 2023-11-08T04:57:40.343 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster_var_name=local_cluster 2023-11-08T04:57:40.344 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance_var_name=instance 2023-11-08T04:57:40.344 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T04:57:40.344 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=0 2023-11-08T04:57:40.344 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ '[' 0 = cluster1:0 ']' 2023-11-08T04:57:40.344 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval local_cluster=cluster1 2023-11-08T04:57:40.345 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ local_cluster=cluster1 2023-11-08T04:57:40.345 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval instance=0 2023-11-08T04:57:40.345 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ instance=0 2023-11-08T04:57:40.345 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-11-08T04:57:40.355 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-08T04:57:40.356 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-08T04:57:40.356 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-08T04:57:40.356 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2023-11-08T04:57:40.451 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-11-08T04:57:40.451 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd mirror status : get status for rbd mirror 2023-11-08T04:57:40.451 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:admin_socket: invalid command 2023-11-08T04:57:40.454 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T04:57:40.454 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2023-11-08T04:57:40.455 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1:1 2023-11-08T04:57:40.455 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T04:57:40.455 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance 2023-11-08T04:57:40.455 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-11-08T04:57:40.455 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local val=cluster1:1 2023-11-08T04:57:40.456 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_var_name=cluster 2023-11-08T04:57:40.456 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance_var_name=instance 2023-11-08T04:57:40.456 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cluster=cluster1 2023-11-08T04:57:40.456 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ instance=1 2023-11-08T04:57:40.457 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' 1 = cluster1:1 ']' 2023-11-08T04:57:40.457 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval cluster=cluster1 2023-11-08T04:57:40.457 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T04:57:40.457 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval instance=1 2023-11-08T04:57:40.457 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=1 2023-11-08T04:57:40.457 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-11-08T04:57:40.458 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local local_cluster=cluster1:1 2023-11-08T04:57:40.458 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster=cluster1 2023-11-08T04:57:40.458 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance 2023-11-08T04:57:40.458 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-11-08T04:57:40.458 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local val=cluster1:1 2023-11-08T04:57:40.459 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster_var_name=local_cluster 2023-11-08T04:57:40.459 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance_var_name=instance 2023-11-08T04:57:40.459 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T04:57:40.459 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=1 2023-11-08T04:57:40.459 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ '[' 1 = cluster1:1 ']' 2023-11-08T04:57:40.460 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval local_cluster=cluster1 2023-11-08T04:57:40.460 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ local_cluster=cluster1 2023-11-08T04:57:40.460 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval instance=1 2023-11-08T04:57:40.460 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ instance=1 2023-11-08T04:57:40.460 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-11-08T04:57:40.469 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-08T04:57:40.469 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-08T04:57:40.470 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-08T04:57:40.470 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2023-11-08T04:57:40.558 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-11-08T04:57:40.558 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd mirror status : get status for rbd mirror 2023-11-08T04:57:40.558 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:admin_socket: invalid command 2023-11-08T04:57:40.561 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T04:57:40.561 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2023-11-08T04:57:40.562 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1:2 2023-11-08T04:57:40.562 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T04:57:40.562 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance 2023-11-08T04:57:40.562 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-11-08T04:57:40.562 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local val=cluster1:2 2023-11-08T04:57:40.563 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_var_name=cluster 2023-11-08T04:57:40.563 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance_var_name=instance 2023-11-08T04:57:40.563 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cluster=cluster1 2023-11-08T04:57:40.563 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ instance=2 2023-11-08T04:57:40.563 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' 2 = cluster1:2 ']' 2023-11-08T04:57:40.564 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval cluster=cluster1 2023-11-08T04:57:40.564 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T04:57:40.564 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval instance=2 2023-11-08T04:57:40.564 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=2 2023-11-08T04:57:40.564 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-11-08T04:57:40.565 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local local_cluster=cluster1:2 2023-11-08T04:57:40.565 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster=cluster1 2023-11-08T04:57:40.565 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance 2023-11-08T04:57:40.565 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-11-08T04:57:40.565 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local val=cluster1:2 2023-11-08T04:57:40.566 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster_var_name=local_cluster 2023-11-08T04:57:40.566 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance_var_name=instance 2023-11-08T04:57:40.566 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T04:57:40.566 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=2 2023-11-08T04:57:40.566 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ '[' 2 = cluster1:2 ']' 2023-11-08T04:57:40.567 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval local_cluster=cluster1 2023-11-08T04:57:40.567 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ local_cluster=cluster1 2023-11-08T04:57:40.567 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval instance=2 2023-11-08T04:57:40.567 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ instance=2 2023-11-08T04:57:40.567 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-11-08T04:57:40.575 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-08T04:57:40.575 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-08T04:57:40.575 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-08T04:57:40.575 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2023-11-08T04:57:40.662 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-11-08T04:57:40.662 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd mirror status : get status for rbd mirror 2023-11-08T04:57:40.662 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:admin_socket: invalid command 2023-11-08T04:57:40.665 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T04:57:40.665 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2023-11-08T04:57:40.665 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1:3 2023-11-08T04:57:40.666 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T04:57:40.666 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance 2023-11-08T04:57:40.666 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ set_cluster_instance cluster1:3 cluster instance 2023-11-08T04:57:40.666 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local val=cluster1:3 2023-11-08T04:57:40.666 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_var_name=cluster 2023-11-08T04:57:40.667 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance_var_name=instance 2023-11-08T04:57:40.667 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cluster=cluster1 2023-11-08T04:57:40.667 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ instance=3 2023-11-08T04:57:40.667 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' 3 = cluster1:3 ']' 2023-11-08T04:57:40.667 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval cluster=cluster1 2023-11-08T04:57:40.667 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T04:57:40.668 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval instance=3 2023-11-08T04:57:40.668 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=3 2023-11-08T04:57:40.668 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ daemon_asok_file cluster1:3 cluster1 2023-11-08T04:57:40.668 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local local_cluster=cluster1:3 2023-11-08T04:57:40.669 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster=cluster1 2023-11-08T04:57:40.669 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance 2023-11-08T04:57:40.669 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2023-11-08T04:57:40.669 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local val=cluster1:3 2023-11-08T04:57:40.669 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster_var_name=local_cluster 2023-11-08T04:57:40.669 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance_var_name=instance 2023-11-08T04:57:40.670 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T04:57:40.670 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=3 2023-11-08T04:57:40.670 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ '[' 3 = cluster1:3 ']' 2023-11-08T04:57:40.670 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval local_cluster=cluster1 2023-11-08T04:57:40.670 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ local_cluster=cluster1 2023-11-08T04:57:40.671 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval instance=3 2023-11-08T04:57:40.671 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ instance=3 2023-11-08T04:57:40.671 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2023-11-08T04:57:40.679 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-11-08T04:57:40.679 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-11-08T04:57:40.679 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-11-08T04:57:40.680 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2023-11-08T04:57:40.766 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-11-08T04:57:40.767 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd mirror status : get status for rbd mirror 2023-11-08T04:57:40.767 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:admin_socket: invalid command 2023-11-08T04:57:40.767 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-11-08T04:57:40.768 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sleep 4 2023-11-08T04:57:43.019 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.25882099459763686) > weight (0.25) 2023-11-08T04:57:43.019 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.43545678705816904) > weight (0.25) 2023-11-08T04:57:43.019 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.2 2023-11-08T04:57:43.020 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Sent signal 15 2023-11-08T04:57:43.020 INFO:tasks.rbd_mirror_thrash:waiting for 9 secs before reviving daemons 2023-11-08T04:57:43.021 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi087.stderr:2023-11-08T04:57:43.019+0000 7efcfb120700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 (PID: 120193) UID: 1000 2023-11-08T04:57:43.221 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi087.stderr:daemon-helper: command failed with exit status 1 2023-11-08T04:57:44.768 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-11-08T04:57:44.769 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ seq 0 3 2023-11-08T04:57:44.774 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T04:57:44.774 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2023-11-08T04:57:44.775 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1:0 2023-11-08T04:57:44.775 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T04:57:44.775 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance 2023-11-08T04:57:44.776 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-11-08T04:57:44.776 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local val=cluster1:0 2023-11-08T04:57:44.776 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_var_name=cluster 2023-11-08T04:57:44.777 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance_var_name=instance 2023-11-08T04:57:44.777 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cluster=cluster1 2023-11-08T04:57:44.777 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ instance=0 2023-11-08T04:57:44.778 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' 0 = cluster1:0 ']' 2023-11-08T04:57:44.778 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval cluster=cluster1 2023-11-08T04:57:44.779 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T04:57:44.779 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval instance=0 2023-11-08T04:57:44.779 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=0 2023-11-08T04:57:44.780 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-11-08T04:57:44.780 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local local_cluster=cluster1:0 2023-11-08T04:57:44.780 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster=cluster1 2023-11-08T04:57:44.781 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance 2023-11-08T04:57:44.781 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-11-08T04:57:44.781 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local val=cluster1:0 2023-11-08T04:57:44.781 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster_var_name=local_cluster 2023-11-08T04:57:44.782 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance_var_name=instance 2023-11-08T04:57:44.782 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T04:57:44.782 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=0 2023-11-08T04:57:44.782 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ '[' 0 = cluster1:0 ']' 2023-11-08T04:57:44.783 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval local_cluster=cluster1 2023-11-08T04:57:44.783 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ local_cluster=cluster1 2023-11-08T04:57:44.783 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval instance=0 2023-11-08T04:57:44.783 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ instance=0 2023-11-08T04:57:44.784 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-11-08T04:57:44.788 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-08T04:57:44.789 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-08T04:57:44.789 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-08T04:57:44.789 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2023-11-08T04:57:44.885 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-11-08T04:57:44.885 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd mirror status : get status for rbd mirror 2023-11-08T04:57:44.885 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:admin_socket: invalid command 2023-11-08T04:57:44.888 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T04:57:44.888 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2023-11-08T04:57:44.888 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1:1 2023-11-08T04:57:44.889 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T04:57:44.889 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance 2023-11-08T04:57:44.889 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-11-08T04:57:44.889 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local val=cluster1:1 2023-11-08T04:57:44.889 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_var_name=cluster 2023-11-08T04:57:44.890 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance_var_name=instance 2023-11-08T04:57:44.890 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cluster=cluster1 2023-11-08T04:57:44.890 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ instance=1 2023-11-08T04:57:44.890 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' 1 = cluster1:1 ']' 2023-11-08T04:57:44.890 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval cluster=cluster1 2023-11-08T04:57:44.891 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T04:57:44.891 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval instance=1 2023-11-08T04:57:44.891 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=1 2023-11-08T04:57:44.891 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-11-08T04:57:44.891 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local local_cluster=cluster1:1 2023-11-08T04:57:44.892 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster=cluster1 2023-11-08T04:57:44.892 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance 2023-11-08T04:57:44.892 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-11-08T04:57:44.892 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local val=cluster1:1 2023-11-08T04:57:44.892 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster_var_name=local_cluster 2023-11-08T04:57:44.893 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance_var_name=instance 2023-11-08T04:57:44.893 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T04:57:44.893 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=1 2023-11-08T04:57:44.893 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ '[' 1 = cluster1:1 ']' 2023-11-08T04:57:44.893 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval local_cluster=cluster1 2023-11-08T04:57:44.894 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ local_cluster=cluster1 2023-11-08T04:57:44.894 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval instance=1 2023-11-08T04:57:44.894 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ instance=1 2023-11-08T04:57:44.894 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-11-08T04:57:44.902 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-08T04:57:44.902 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-08T04:57:44.903 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-08T04:57:44.903 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2023-11-08T04:57:44.991 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-11-08T04:57:44.991 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd mirror status : get status for rbd mirror 2023-11-08T04:57:44.991 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:admin_socket: invalid command 2023-11-08T04:57:44.994 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T04:57:44.994 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2023-11-08T04:57:44.994 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1:2 2023-11-08T04:57:44.995 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T04:57:44.995 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance 2023-11-08T04:57:44.995 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-11-08T04:57:44.995 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local val=cluster1:2 2023-11-08T04:57:44.995 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_var_name=cluster 2023-11-08T04:57:44.996 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance_var_name=instance 2023-11-08T04:57:44.996 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cluster=cluster1 2023-11-08T04:57:44.996 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ instance=2 2023-11-08T04:57:44.996 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' 2 = cluster1:2 ']' 2023-11-08T04:57:44.997 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval cluster=cluster1 2023-11-08T04:57:44.997 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T04:57:44.997 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval instance=2 2023-11-08T04:57:44.997 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=2 2023-11-08T04:57:44.997 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-11-08T04:57:44.998 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local local_cluster=cluster1:2 2023-11-08T04:57:44.998 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster=cluster1 2023-11-08T04:57:44.998 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance 2023-11-08T04:57:44.998 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-11-08T04:57:44.998 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local val=cluster1:2 2023-11-08T04:57:44.999 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster_var_name=local_cluster 2023-11-08T04:57:44.999 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance_var_name=instance 2023-11-08T04:57:44.999 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T04:57:44.999 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=2 2023-11-08T04:57:44.999 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ '[' 2 = cluster1:2 ']' 2023-11-08T04:57:44.999 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval local_cluster=cluster1 2023-11-08T04:57:45.000 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ local_cluster=cluster1 2023-11-08T04:57:45.000 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval instance=2 2023-11-08T04:57:45.000 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ instance=2 2023-11-08T04:57:45.000 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-11-08T04:57:45.007 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-08T04:57:45.008 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-08T04:57:45.008 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-08T04:57:45.008 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2023-11-08T04:57:45.097 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-11-08T04:57:45.098 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd mirror status : get status for rbd mirror 2023-11-08T04:57:45.098 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:admin_socket: invalid command 2023-11-08T04:57:45.100 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T04:57:45.101 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2023-11-08T04:57:45.101 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1:3 2023-11-08T04:57:45.101 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T04:57:45.102 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance 2023-11-08T04:57:45.102 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ set_cluster_instance cluster1:3 cluster instance 2023-11-08T04:57:45.102 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local val=cluster1:3 2023-11-08T04:57:45.102 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_var_name=cluster 2023-11-08T04:57:45.102 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance_var_name=instance 2023-11-08T04:57:45.103 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cluster=cluster1 2023-11-08T04:57:45.103 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ instance=3 2023-11-08T04:57:45.103 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' 3 = cluster1:3 ']' 2023-11-08T04:57:45.103 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval cluster=cluster1 2023-11-08T04:57:45.104 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T04:57:45.104 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval instance=3 2023-11-08T04:57:45.104 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=3 2023-11-08T04:57:45.104 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ daemon_asok_file cluster1:3 cluster1 2023-11-08T04:57:45.104 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local local_cluster=cluster1:3 2023-11-08T04:57:45.104 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster=cluster1 2023-11-08T04:57:45.105 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance 2023-11-08T04:57:45.105 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2023-11-08T04:57:45.105 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local val=cluster1:3 2023-11-08T04:57:45.105 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster_var_name=local_cluster 2023-11-08T04:57:45.105 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance_var_name=instance 2023-11-08T04:57:45.106 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T04:57:45.106 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=3 2023-11-08T04:57:45.106 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ '[' 3 = cluster1:3 ']' 2023-11-08T04:57:45.106 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval local_cluster=cluster1 2023-11-08T04:57:45.106 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ local_cluster=cluster1 2023-11-08T04:57:45.107 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval instance=3 2023-11-08T04:57:45.107 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ instance=3 2023-11-08T04:57:45.107 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2023-11-08T04:57:45.114 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-11-08T04:57:45.115 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-11-08T04:57:45.115 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-11-08T04:57:45.115 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2023-11-08T04:57:45.204 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-11-08T04:57:45.204 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd mirror status : get status for rbd mirror 2023-11-08T04:57:45.204 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:admin_socket: invalid command 2023-11-08T04:57:45.205 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-11-08T04:57:45.205 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sleep 8 2023-11-08T04:57:52.021 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.2 2023-11-08T04:57:52.021 INFO:teuthology.orchestra.run:waiting for 600 2023-11-08T04:57:52.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-08T04:57:52.022 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.2 2023-11-08T04:57:52.022 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Restarting daemon 2023-11-08T04:57:52.022 DEBUG:teuthology.orchestra.run.smithi087:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 2023-11-08T04:57:52.025 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Started 2023-11-08T04:57:52.026 INFO:tasks.rbd_mirror_thrash:waiting for 98 secs before thrashing 2023-11-08T04:57:53.206 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-11-08T04:57:53.206 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ seq 0 3 2023-11-08T04:57:53.209 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T04:57:53.209 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2023-11-08T04:57:53.210 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1:0 2023-11-08T04:57:53.210 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T04:57:53.210 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance 2023-11-08T04:57:53.210 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-11-08T04:57:53.210 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local val=cluster1:0 2023-11-08T04:57:53.211 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_var_name=cluster 2023-11-08T04:57:53.211 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance_var_name=instance 2023-11-08T04:57:53.211 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cluster=cluster1 2023-11-08T04:57:53.211 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ instance=0 2023-11-08T04:57:53.211 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' 0 = cluster1:0 ']' 2023-11-08T04:57:53.212 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval cluster=cluster1 2023-11-08T04:57:53.212 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T04:57:53.212 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval instance=0 2023-11-08T04:57:53.212 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=0 2023-11-08T04:57:53.213 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-11-08T04:57:53.213 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local local_cluster=cluster1:0 2023-11-08T04:57:53.213 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster=cluster1 2023-11-08T04:57:53.213 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance 2023-11-08T04:57:53.213 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-11-08T04:57:53.214 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local val=cluster1:0 2023-11-08T04:57:53.214 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster_var_name=local_cluster 2023-11-08T04:57:53.214 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance_var_name=instance 2023-11-08T04:57:53.214 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T04:57:53.214 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=0 2023-11-08T04:57:53.215 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ '[' 0 = cluster1:0 ']' 2023-11-08T04:57:53.215 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval local_cluster=cluster1 2023-11-08T04:57:53.215 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ local_cluster=cluster1 2023-11-08T04:57:53.215 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval instance=0 2023-11-08T04:57:53.215 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ instance=0 2023-11-08T04:57:53.216 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-11-08T04:57:53.227 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-08T04:57:53.227 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-08T04:57:53.227 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-08T04:57:53.227 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2023-11-08T04:57:53.323 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-11-08T04:57:53.323 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd mirror status : get status for rbd mirror 2023-11-08T04:57:53.323 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:admin_socket: invalid command 2023-11-08T04:57:53.326 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T04:57:53.326 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2023-11-08T04:57:53.326 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1:1 2023-11-08T04:57:53.327 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T04:57:53.327 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance 2023-11-08T04:57:53.327 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-11-08T04:57:53.327 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local val=cluster1:1 2023-11-08T04:57:53.327 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_var_name=cluster 2023-11-08T04:57:53.328 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance_var_name=instance 2023-11-08T04:57:53.328 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cluster=cluster1 2023-11-08T04:57:53.328 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ instance=1 2023-11-08T04:57:53.328 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' 1 = cluster1:1 ']' 2023-11-08T04:57:53.328 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval cluster=cluster1 2023-11-08T04:57:53.329 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T04:57:53.329 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval instance=1 2023-11-08T04:57:53.329 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=1 2023-11-08T04:57:53.329 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-11-08T04:57:53.329 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local local_cluster=cluster1:1 2023-11-08T04:57:53.330 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster=cluster1 2023-11-08T04:57:53.330 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance 2023-11-08T04:57:53.330 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-11-08T04:57:53.330 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local val=cluster1:1 2023-11-08T04:57:53.330 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster_var_name=local_cluster 2023-11-08T04:57:53.331 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance_var_name=instance 2023-11-08T04:57:53.331 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T04:57:53.331 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=1 2023-11-08T04:57:53.331 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ '[' 1 = cluster1:1 ']' 2023-11-08T04:57:53.331 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval local_cluster=cluster1 2023-11-08T04:57:53.331 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ local_cluster=cluster1 2023-11-08T04:57:53.332 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval instance=1 2023-11-08T04:57:53.332 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ instance=1 2023-11-08T04:57:53.332 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-11-08T04:57:53.340 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-08T04:57:53.340 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-08T04:57:53.340 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-08T04:57:53.341 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2023-11-08T04:57:53.431 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-11-08T04:57:53.431 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd mirror status : get status for rbd mirror 2023-11-08T04:57:53.431 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:admin_socket: invalid command 2023-11-08T04:57:53.434 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T04:57:53.434 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2023-11-08T04:57:53.434 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1:2 2023-11-08T04:57:53.435 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T04:57:53.435 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance 2023-11-08T04:57:53.435 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-11-08T04:57:53.435 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local val=cluster1:2 2023-11-08T04:57:53.436 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_var_name=cluster 2023-11-08T04:57:53.436 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance_var_name=instance 2023-11-08T04:57:53.436 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cluster=cluster1 2023-11-08T04:57:53.436 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ instance=2 2023-11-08T04:57:53.436 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' 2 = cluster1:2 ']' 2023-11-08T04:57:53.437 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval cluster=cluster1 2023-11-08T04:57:53.437 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T04:57:53.437 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval instance=2 2023-11-08T04:57:53.437 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=2 2023-11-08T04:57:53.437 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-11-08T04:57:53.438 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local local_cluster=cluster1:2 2023-11-08T04:57:53.438 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster=cluster1 2023-11-08T04:57:53.438 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance 2023-11-08T04:57:53.438 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-11-08T04:57:53.438 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local val=cluster1:2 2023-11-08T04:57:53.439 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster_var_name=local_cluster 2023-11-08T04:57:53.439 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance_var_name=instance 2023-11-08T04:57:53.439 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T04:57:53.439 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=2 2023-11-08T04:57:53.439 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ '[' 2 = cluster1:2 ']' 2023-11-08T04:57:53.440 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval local_cluster=cluster1 2023-11-08T04:57:53.440 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ local_cluster=cluster1 2023-11-08T04:57:53.440 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval instance=2 2023-11-08T04:57:53.440 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ instance=2 2023-11-08T04:57:53.440 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-11-08T04:57:53.448 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-08T04:57:53.449 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-08T04:57:53.449 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-08T04:57:53.449 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2023-11-08T04:57:53.537 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-11-08T04:57:53.537 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd mirror status : get status for rbd mirror 2023-11-08T04:57:53.537 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:admin_socket: invalid command 2023-11-08T04:57:53.540 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T04:57:53.540 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2023-11-08T04:57:53.541 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1:3 2023-11-08T04:57:53.541 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T04:57:53.541 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance 2023-11-08T04:57:53.541 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ set_cluster_instance cluster1:3 cluster instance 2023-11-08T04:57:53.541 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local val=cluster1:3 2023-11-08T04:57:53.542 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_var_name=cluster 2023-11-08T04:57:53.542 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance_var_name=instance 2023-11-08T04:57:53.542 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cluster=cluster1 2023-11-08T04:57:53.542 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ instance=3 2023-11-08T04:57:53.542 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' 3 = cluster1:3 ']' 2023-11-08T04:57:53.543 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval cluster=cluster1 2023-11-08T04:57:53.543 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T04:57:53.543 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval instance=3 2023-11-08T04:57:53.543 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=3 2023-11-08T04:57:53.543 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ daemon_asok_file cluster1:3 cluster1 2023-11-08T04:57:53.544 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local local_cluster=cluster1:3 2023-11-08T04:57:53.544 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster=cluster1 2023-11-08T04:57:53.544 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance 2023-11-08T04:57:53.544 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2023-11-08T04:57:53.544 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local val=cluster1:3 2023-11-08T04:57:53.545 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster_var_name=local_cluster 2023-11-08T04:57:53.545 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance_var_name=instance 2023-11-08T04:57:53.545 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T04:57:53.545 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=3 2023-11-08T04:57:53.546 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ '[' 3 = cluster1:3 ']' 2023-11-08T04:57:53.546 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval local_cluster=cluster1 2023-11-08T04:57:53.546 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ local_cluster=cluster1 2023-11-08T04:57:53.546 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval instance=3 2023-11-08T04:57:53.547 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ instance=3 2023-11-08T04:57:53.547 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2023-11-08T04:57:53.554 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-11-08T04:57:53.554 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-11-08T04:57:53.554 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-11-08T04:57:53.555 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2023-11-08T04:57:53.641 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-11-08T04:57:53.642 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd mirror status : get status for rbd mirror 2023-11-08T04:57:53.642 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:admin_socket: invalid command 2023-11-08T04:57:53.643 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-11-08T04:57:53.643 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sleep 8 2023-11-08T04:58:01.643 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-11-08T04:58:01.644 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ seq 0 3 2023-11-08T04:58:01.647 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T04:58:01.647 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2023-11-08T04:58:01.648 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1:0 2023-11-08T04:58:01.648 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T04:58:01.648 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance 2023-11-08T04:58:01.648 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-11-08T04:58:01.648 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local val=cluster1:0 2023-11-08T04:58:01.649 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_var_name=cluster 2023-11-08T04:58:01.649 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance_var_name=instance 2023-11-08T04:58:01.649 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cluster=cluster1 2023-11-08T04:58:01.649 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ instance=0 2023-11-08T04:58:01.649 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' 0 = cluster1:0 ']' 2023-11-08T04:58:01.650 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval cluster=cluster1 2023-11-08T04:58:01.650 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T04:58:01.650 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval instance=0 2023-11-08T04:58:01.650 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=0 2023-11-08T04:58:01.650 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-11-08T04:58:01.651 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local local_cluster=cluster1:0 2023-11-08T04:58:01.651 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster=cluster1 2023-11-08T04:58:01.651 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance 2023-11-08T04:58:01.651 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-11-08T04:58:01.651 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local val=cluster1:0 2023-11-08T04:58:01.652 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster_var_name=local_cluster 2023-11-08T04:58:01.652 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance_var_name=instance 2023-11-08T04:58:01.652 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T04:58:01.652 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=0 2023-11-08T04:58:01.652 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ '[' 0 = cluster1:0 ']' 2023-11-08T04:58:01.652 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval local_cluster=cluster1 2023-11-08T04:58:01.653 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ local_cluster=cluster1 2023-11-08T04:58:01.653 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval instance=0 2023-11-08T04:58:01.653 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ instance=0 2023-11-08T04:58:01.653 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-11-08T04:58:01.665 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-08T04:58:01.666 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-08T04:58:01.666 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-08T04:58:01.666 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2023-11-08T04:58:01.762 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-11-08T04:58:01.762 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd mirror status : get status for rbd mirror 2023-11-08T04:58:01.762 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:admin_socket: invalid command 2023-11-08T04:58:01.765 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T04:58:01.765 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2023-11-08T04:58:01.765 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1:1 2023-11-08T04:58:01.766 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T04:58:01.766 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance 2023-11-08T04:58:01.766 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-11-08T04:58:01.766 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local val=cluster1:1 2023-11-08T04:58:01.767 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_var_name=cluster 2023-11-08T04:58:01.767 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance_var_name=instance 2023-11-08T04:58:01.767 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cluster=cluster1 2023-11-08T04:58:01.767 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ instance=1 2023-11-08T04:58:01.767 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' 1 = cluster1:1 ']' 2023-11-08T04:58:01.768 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval cluster=cluster1 2023-11-08T04:58:01.768 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T04:58:01.768 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval instance=1 2023-11-08T04:58:01.768 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=1 2023-11-08T04:58:01.768 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-11-08T04:58:01.769 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local local_cluster=cluster1:1 2023-11-08T04:58:01.769 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster=cluster1 2023-11-08T04:58:01.769 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance 2023-11-08T04:58:01.769 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-11-08T04:58:01.769 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local val=cluster1:1 2023-11-08T04:58:01.770 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster_var_name=local_cluster 2023-11-08T04:58:01.770 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance_var_name=instance 2023-11-08T04:58:01.770 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T04:58:01.770 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=1 2023-11-08T04:58:01.770 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ '[' 1 = cluster1:1 ']' 2023-11-08T04:58:01.770 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval local_cluster=cluster1 2023-11-08T04:58:01.771 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ local_cluster=cluster1 2023-11-08T04:58:01.771 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval instance=1 2023-11-08T04:58:01.771 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ instance=1 2023-11-08T04:58:01.771 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-11-08T04:58:01.779 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-08T04:58:01.779 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-08T04:58:01.780 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-08T04:58:01.780 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2023-11-08T04:58:01.868 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-11-08T04:58:01.869 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd mirror status : get status for rbd mirror 2023-11-08T04:58:01.869 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:admin_socket: invalid command 2023-11-08T04:58:01.872 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T04:58:01.873 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2023-11-08T04:58:01.873 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1:2 2023-11-08T04:58:01.873 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T04:58:01.873 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance 2023-11-08T04:58:01.874 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-11-08T04:58:01.874 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local val=cluster1:2 2023-11-08T04:58:01.874 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_var_name=cluster 2023-11-08T04:58:01.874 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance_var_name=instance 2023-11-08T04:58:01.875 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cluster=cluster1 2023-11-08T04:58:01.875 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ instance=2 2023-11-08T04:58:01.875 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' 2 = cluster1:2 ']' 2023-11-08T04:58:01.876 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval cluster=cluster1 2023-11-08T04:58:01.876 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T04:58:01.876 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval instance=2 2023-11-08T04:58:01.877 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=2 2023-11-08T04:58:01.877 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-11-08T04:58:01.878 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local local_cluster=cluster1:2 2023-11-08T04:58:01.878 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster=cluster1 2023-11-08T04:58:01.878 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance 2023-11-08T04:58:01.879 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-11-08T04:58:01.879 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local val=cluster1:2 2023-11-08T04:58:01.879 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster_var_name=local_cluster 2023-11-08T04:58:01.880 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance_var_name=instance 2023-11-08T04:58:01.880 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T04:58:01.880 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=2 2023-11-08T04:58:01.881 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ '[' 2 = cluster1:2 ']' 2023-11-08T04:58:01.881 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval local_cluster=cluster1 2023-11-08T04:58:01.881 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ local_cluster=cluster1 2023-11-08T04:58:01.882 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval instance=2 2023-11-08T04:58:01.882 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ instance=2 2023-11-08T04:58:01.882 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-11-08T04:58:01.885 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-08T04:58:01.885 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-08T04:58:01.886 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-08T04:58:01.886 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2023-11-08T04:58:01.973 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-11-08T04:58:01.973 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd mirror status : get status for rbd mirror 2023-11-08T04:58:01.974 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:admin_socket: invalid command 2023-11-08T04:58:01.977 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T04:58:01.977 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2023-11-08T04:58:01.977 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1:3 2023-11-08T04:58:01.977 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T04:58:01.978 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance 2023-11-08T04:58:01.978 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ set_cluster_instance cluster1:3 cluster instance 2023-11-08T04:58:01.978 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local val=cluster1:3 2023-11-08T04:58:01.978 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_var_name=cluster 2023-11-08T04:58:01.979 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance_var_name=instance 2023-11-08T04:58:01.979 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cluster=cluster1 2023-11-08T04:58:01.979 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ instance=3 2023-11-08T04:58:01.979 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' 3 = cluster1:3 ']' 2023-11-08T04:58:01.980 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval cluster=cluster1 2023-11-08T04:58:01.980 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T04:58:01.980 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval instance=3 2023-11-08T04:58:01.980 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=3 2023-11-08T04:58:01.980 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ daemon_asok_file cluster1:3 cluster1 2023-11-08T04:58:01.981 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local local_cluster=cluster1:3 2023-11-08T04:58:01.981 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster=cluster1 2023-11-08T04:58:01.981 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance 2023-11-08T04:58:01.981 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2023-11-08T04:58:01.981 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local val=cluster1:3 2023-11-08T04:58:01.981 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster_var_name=local_cluster 2023-11-08T04:58:01.982 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance_var_name=instance 2023-11-08T04:58:01.985 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T04:58:01.985 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=3 2023-11-08T04:58:01.985 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ '[' 3 = cluster1:3 ']' 2023-11-08T04:58:01.986 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval local_cluster=cluster1 2023-11-08T04:58:01.986 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ local_cluster=cluster1 2023-11-08T04:58:01.986 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval instance=3 2023-11-08T04:58:01.986 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ instance=3 2023-11-08T04:58:01.986 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2023-11-08T04:58:01.990 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-11-08T04:58:01.991 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-11-08T04:58:01.991 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-11-08T04:58:01.991 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2023-11-08T04:58:02.079 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-11-08T04:58:02.079 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd mirror status : get status for rbd mirror 2023-11-08T04:58:02.079 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:admin_socket: invalid command 2023-11-08T04:58:02.080 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-11-08T04:58:02.080 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sleep 8 2023-11-08T04:58:10.081 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-11-08T04:58:10.082 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ seq 0 3 2023-11-08T04:58:10.085 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T04:58:10.085 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2023-11-08T04:58:10.086 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1:0 2023-11-08T04:58:10.086 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T04:58:10.086 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance 2023-11-08T04:58:10.086 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-11-08T04:58:10.087 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local val=cluster1:0 2023-11-08T04:58:10.087 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_var_name=cluster 2023-11-08T04:58:10.087 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance_var_name=instance 2023-11-08T04:58:10.088 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cluster=cluster1 2023-11-08T04:58:10.088 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ instance=0 2023-11-08T04:58:10.088 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' 0 = cluster1:0 ']' 2023-11-08T04:58:10.088 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval cluster=cluster1 2023-11-08T04:58:10.089 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T04:58:10.089 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval instance=0 2023-11-08T04:58:10.089 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=0 2023-11-08T04:58:10.089 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-11-08T04:58:10.089 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local local_cluster=cluster1:0 2023-11-08T04:58:10.090 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster=cluster1 2023-11-08T04:58:10.090 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance 2023-11-08T04:58:10.090 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-11-08T04:58:10.090 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local val=cluster1:0 2023-11-08T04:58:10.090 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster_var_name=local_cluster 2023-11-08T04:58:10.091 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance_var_name=instance 2023-11-08T04:58:10.091 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T04:58:10.091 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=0 2023-11-08T04:58:10.091 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ '[' 0 = cluster1:0 ']' 2023-11-08T04:58:10.091 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval local_cluster=cluster1 2023-11-08T04:58:10.092 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ local_cluster=cluster1 2023-11-08T04:58:10.092 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval instance=0 2023-11-08T04:58:10.092 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ instance=0 2023-11-08T04:58:10.092 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-11-08T04:58:10.102 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-08T04:58:10.102 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-08T04:58:10.102 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-08T04:58:10.102 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2023-11-08T04:58:10.199 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-11-08T04:58:10.199 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd mirror status : get status for rbd mirror 2023-11-08T04:58:10.199 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:admin_socket: invalid command 2023-11-08T04:58:10.202 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T04:58:10.202 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2023-11-08T04:58:10.203 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1:1 2023-11-08T04:58:10.203 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T04:58:10.203 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance 2023-11-08T04:58:10.203 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-11-08T04:58:10.203 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local val=cluster1:1 2023-11-08T04:58:10.204 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_var_name=cluster 2023-11-08T04:58:10.204 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance_var_name=instance 2023-11-08T04:58:10.204 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cluster=cluster1 2023-11-08T04:58:10.204 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ instance=1 2023-11-08T04:58:10.204 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' 1 = cluster1:1 ']' 2023-11-08T04:58:10.205 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval cluster=cluster1 2023-11-08T04:58:10.205 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T04:58:10.205 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval instance=1 2023-11-08T04:58:10.205 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=1 2023-11-08T04:58:10.205 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-11-08T04:58:10.206 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local local_cluster=cluster1:1 2023-11-08T04:58:10.206 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster=cluster1 2023-11-08T04:58:10.206 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance 2023-11-08T04:58:10.206 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-11-08T04:58:10.206 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local val=cluster1:1 2023-11-08T04:58:10.207 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster_var_name=local_cluster 2023-11-08T04:58:10.207 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance_var_name=instance 2023-11-08T04:58:10.207 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T04:58:10.207 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=1 2023-11-08T04:58:10.207 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ '[' 1 = cluster1:1 ']' 2023-11-08T04:58:10.208 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval local_cluster=cluster1 2023-11-08T04:58:10.208 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ local_cluster=cluster1 2023-11-08T04:58:10.208 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval instance=1 2023-11-08T04:58:10.208 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ instance=1 2023-11-08T04:58:10.208 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-11-08T04:58:10.216 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-08T04:58:10.216 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-08T04:58:10.216 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-08T04:58:10.216 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2023-11-08T04:58:10.305 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-11-08T04:58:10.305 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd mirror status : get status for rbd mirror 2023-11-08T04:58:10.305 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:admin_socket: invalid command 2023-11-08T04:58:10.308 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T04:58:10.308 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2023-11-08T04:58:10.308 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1:2 2023-11-08T04:58:10.308 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T04:58:10.308 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance 2023-11-08T04:58:10.309 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-11-08T04:58:10.309 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local val=cluster1:2 2023-11-08T04:58:10.309 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_var_name=cluster 2023-11-08T04:58:10.309 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance_var_name=instance 2023-11-08T04:58:10.309 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cluster=cluster1 2023-11-08T04:58:10.310 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ instance=2 2023-11-08T04:58:10.310 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' 2 = cluster1:2 ']' 2023-11-08T04:58:10.310 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval cluster=cluster1 2023-11-08T04:58:10.310 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T04:58:10.310 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval instance=2 2023-11-08T04:58:10.311 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=2 2023-11-08T04:58:10.311 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-11-08T04:58:10.311 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local local_cluster=cluster1:2 2023-11-08T04:58:10.311 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster=cluster1 2023-11-08T04:58:10.312 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance 2023-11-08T04:58:10.312 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-11-08T04:58:10.312 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local val=cluster1:2 2023-11-08T04:58:10.312 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster_var_name=local_cluster 2023-11-08T04:58:10.313 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance_var_name=instance 2023-11-08T04:58:10.313 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T04:58:10.313 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=2 2023-11-08T04:58:10.313 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ '[' 2 = cluster1:2 ']' 2023-11-08T04:58:10.313 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval local_cluster=cluster1 2023-11-08T04:58:10.314 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ local_cluster=cluster1 2023-11-08T04:58:10.314 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval instance=2 2023-11-08T04:58:10.314 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ instance=2 2023-11-08T04:58:10.314 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-11-08T04:58:10.321 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-08T04:58:10.322 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-08T04:58:10.322 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-08T04:58:10.322 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2023-11-08T04:58:10.411 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-11-08T04:58:10.411 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd mirror status : get status for rbd mirror 2023-11-08T04:58:10.411 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:admin_socket: invalid command 2023-11-08T04:58:10.414 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T04:58:10.414 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2023-11-08T04:58:10.414 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1:3 2023-11-08T04:58:10.415 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T04:58:10.415 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance 2023-11-08T04:58:10.415 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ set_cluster_instance cluster1:3 cluster instance 2023-11-08T04:58:10.415 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local val=cluster1:3 2023-11-08T04:58:10.415 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_var_name=cluster 2023-11-08T04:58:10.416 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance_var_name=instance 2023-11-08T04:58:10.416 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cluster=cluster1 2023-11-08T04:58:10.416 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ instance=3 2023-11-08T04:58:10.417 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' 3 = cluster1:3 ']' 2023-11-08T04:58:10.417 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval cluster=cluster1 2023-11-08T04:58:10.417 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T04:58:10.417 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval instance=3 2023-11-08T04:58:10.417 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=3 2023-11-08T04:58:10.418 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ daemon_asok_file cluster1:3 cluster1 2023-11-08T04:58:10.418 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local local_cluster=cluster1:3 2023-11-08T04:58:10.418 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster=cluster1 2023-11-08T04:58:10.418 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance 2023-11-08T04:58:10.418 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2023-11-08T04:58:10.419 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local val=cluster1:3 2023-11-08T04:58:10.419 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster_var_name=local_cluster 2023-11-08T04:58:10.419 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance_var_name=instance 2023-11-08T04:58:10.419 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T04:58:10.419 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=3 2023-11-08T04:58:10.420 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ '[' 3 = cluster1:3 ']' 2023-11-08T04:58:10.420 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval local_cluster=cluster1 2023-11-08T04:58:10.420 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ local_cluster=cluster1 2023-11-08T04:58:10.420 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval instance=3 2023-11-08T04:58:10.420 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ instance=3 2023-11-08T04:58:10.420 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2023-11-08T04:58:10.429 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-11-08T04:58:10.429 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-11-08T04:58:10.429 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-11-08T04:58:10.429 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2023-11-08T04:58:10.527 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ return 0 2023-11-08T04:58:10.528 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ status_result=' "state": "Replaying"' 2023-11-08T04:58:10.528 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ echo ' "state": "Replaying"' 2023-11-08T04:58:10.529 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ grep -i Replaying 2023-11-08T04:58:10.529 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: "state": "Replaying" 2023-11-08T04:58:10.531 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ current_state=started 2023-11-08T04:58:10.531 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test started = started 2023-11-08T04:58:10.531 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T04:58:10.532 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2023-11-08T04:58:10.532 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local local_cluster=cluster1 2023-11-08T04:58:10.532 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T04:58:10.532 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T04:58:10.532 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T04:58:10.533 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' snapshot = journal ']' 2023-11-08T04:58:10.533 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' snapshot = snapshot ']' 2023-11-08T04:58:10.533 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2023-11-08T04:58:10.533 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local local_cluster=cluster1 2023-11-08T04:58:10.533 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T04:58:10.534 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T04:58:10.534 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T04:58:10.534 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ mkfname cluster2-mirror-test.status 2023-11-08T04:58:10.534 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ echo cluster2-mirror-test.status 2023-11-08T04:58:10.534 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -e 's|[/ ]|_|g' 2023-11-08T04:58:10.535 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local status_log=/tmp/tmp.Ludk8ikCAB/cluster2-mirror-test.status 2023-11-08T04:58:10.535 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ mkfname cluster1-mirror-test.status 2023-11-08T04:58:10.535 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ echo cluster1-mirror-test.status 2023-11-08T04:58:10.535 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -e 's|[/ ]|_|g' 2023-11-08T04:58:10.536 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local local_status_log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.status 2023-11-08T04:58:10.536 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ mirror_image_snapshot cluster2 mirror test 2023-11-08T04:58:10.536 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T04:58:10.537 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T04:58:10.537 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T04:58:10.537 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2023-11-08T04:58:12.456 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:Snapshot ID: 17 2023-11-08T04:58:12.469 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.Ludk8ikCAB/cluster2-mirror-test.status 2023-11-08T04:58:12.469 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T04:58:12.469 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T04:58:12.469 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T04:58:12.469 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster2-mirror-test.status 2023-11-08T04:58:12.470 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2023-11-08T04:58:12.470 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T04:58:12.518 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-11-08T04:58:12.520 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snapshot_id=17 2023-11-08T04:58:12.520 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ true 2023-11-08T04:58:12.521 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-11-08T04:58:12.521 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0.2 2023-11-08T04:58:12.721 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.status 2023-11-08T04:58:12.722 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T04:58:12.722 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T04:58:12.722 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T04:58:12.722 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.status 2023-11-08T04:58:12.723 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-11-08T04:58:12.723 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T04:58:12.762 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ true 2023-11-08T04:58:12.763 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-08T04:58:12.764 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:-:1.1: Document is empty 2023-11-08T04:58:12.764 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: 2023-11-08T04:58:12.764 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:^ 2023-11-08T04:58:12.765 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local primary_snapshot_id= 2023-11-08T04:58:12.765 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test 17 = '' 2023-11-08T04:58:12.765 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-11-08T04:58:12.766 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0.4 2023-11-08T04:58:13.166 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.status 2023-11-08T04:58:13.166 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T04:58:13.166 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T04:58:13.167 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T04:58:13.167 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.status 2023-11-08T04:58:13.167 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-11-08T04:58:13.167 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T04:58:13.209 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ true 2023-11-08T04:58:13.209 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-08T04:58:13.210 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:-:1.1: Document is empty 2023-11-08T04:58:13.210 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: 2023-11-08T04:58:13.210 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:^ 2023-11-08T04:58:13.211 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local primary_snapshot_id= 2023-11-08T04:58:13.211 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test 17 = '' 2023-11-08T04:58:13.211 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-11-08T04:58:13.211 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0.8 2023-11-08T04:58:14.012 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.status 2023-11-08T04:58:14.012 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T04:58:14.012 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T04:58:14.013 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T04:58:14.013 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.status 2023-11-08T04:58:14.013 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-11-08T04:58:14.013 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T04:58:14.057 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ true 2023-11-08T04:58:14.057 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-08T04:58:14.058 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:-:1.1: Document is empty 2023-11-08T04:58:14.059 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: 2023-11-08T04:58:14.059 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:^ 2023-11-08T04:58:14.060 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local primary_snapshot_id= 2023-11-08T04:58:14.060 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test 17 = '' 2023-11-08T04:58:14.060 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-11-08T04:58:14.060 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 1.6 2023-11-08T04:58:15.660 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.status 2023-11-08T04:58:15.661 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T04:58:15.661 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T04:58:15.661 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T04:58:15.661 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.status 2023-11-08T04:58:15.662 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-11-08T04:58:15.662 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T04:58:15.703 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-08T04:58:15.705 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local primary_snapshot_id=17 2023-11-08T04:58:15.705 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test 17 = 17 2023-11-08T04:58:15.706 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T04:58:15.706 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_snap_present cluster1 mirror test snap4 2023-11-08T04:58:15.706 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T04:58:15.706 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T04:58:15.707 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T04:58:15.707 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap_name=snap4 2023-11-08T04:58:15.707 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local s 2023-11-08T04:58:15.707 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-11-08T04:58:15.707 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 1 2023-11-08T04:58:16.706 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 info mirror/test@snap4 2023-11-08T04:58:16.746 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:rbd image 'test': 2023-11-08T04:58:16.747 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: size 512 MiB in 128 objects 2023-11-08T04:58:16.747 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: order 22 (4 MiB objects) 2023-11-08T04:58:16.747 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: snapshot_count: 8 2023-11-08T04:58:16.747 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: id: 10afad8fe72b 2023-11-08T04:58:16.748 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: block_name_prefix: rbd_data.10afad8fe72b 2023-11-08T04:58:16.748 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: format: 2 2023-11-08T04:58:16.748 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: features: layering, exclusive-lock, non-primary 2023-11-08T04:58:16.748 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: op_features: 2023-11-08T04:58:16.748 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: flags: 2023-11-08T04:58:16.749 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: create_timestamp: Wed Nov 8 04:54:28 2023 2023-11-08T04:58:16.749 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: access_timestamp: Wed Nov 8 04:56:05 2023 2023-11-08T04:58:16.749 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: modify_timestamp: Wed Nov 8 04:54:28 2023 2023-11-08T04:58:16.749 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: protected: False 2023-11-08T04:58:16.749 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring state: enabled 2023-11-08T04:58:16.750 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring mode: snapshot 2023-11-08T04:58:16.750 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring global id: 88ced7f9-2739-4a27-ad7a-30edec05edb1 2023-11-08T04:58:16.750 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring primary: false 2023-11-08T04:58:16.751 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T04:58:16.752 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' -n snap3-clean ']' 2023-11-08T04:58:16.752 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ compare_image_snaps mirror test snap3-clean 2023-11-08T04:58:16.752 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T04:58:16.752 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T04:58:16.752 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap_name=snap3-clean 2023-11-08T04:58:16.753 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local ret=0 2023-11-08T04:58:16.753 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local rmt_export=/tmp/tmp.Ludk8ikCAB/cluster2-mirror-test.export 2023-11-08T04:58:16.753 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local loc_export=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.export 2023-11-08T04:58:16.753 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rm -f /tmp/tmp.Ludk8ikCAB/cluster2-mirror-test.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.export 2023-11-08T04:58:16.753 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 -p mirror export test@snap3-clean /tmp/tmp.Ludk8ikCAB/cluster2-mirror-test.export 2023-11-08T04:58:17.375 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-11-08T04:58:17.378 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 -p mirror export test@snap3-clean /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.export 2023-11-08T04:58:18.087 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-11-08T04:58:18.090 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ cmp /tmp/tmp.Ludk8ikCAB/cluster2-mirror-test.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.export 2023-11-08T04:58:18.304 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rm -f /tmp/tmp.Ludk8ikCAB/cluster2-mirror-test.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.export 2023-11-08T04:58:18.447 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T04:58:18.448 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ compare_image_snaps mirror test snap4 2023-11-08T04:58:18.448 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T04:58:18.448 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T04:58:18.448 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap_name=snap4 2023-11-08T04:58:18.449 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local ret=0 2023-11-08T04:58:18.449 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local rmt_export=/tmp/tmp.Ludk8ikCAB/cluster2-mirror-test.export 2023-11-08T04:58:18.449 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local loc_export=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.export 2023-11-08T04:58:18.449 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rm -f /tmp/tmp.Ludk8ikCAB/cluster2-mirror-test.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.export 2023-11-08T04:58:18.450 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 -p mirror export test@snap4 /tmp/tmp.Ludk8ikCAB/cluster2-mirror-test.export 2023-11-08T04:58:19.042 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-11-08T04:58:19.045 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 -p mirror export test@snap4 /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.export 2023-11-08T04:58:19.853 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-11-08T04:58:19.856 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ cmp /tmp/tmp.Ludk8ikCAB/cluster2-mirror-test.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.export 2023-11-08T04:58:20.069 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rm -f /tmp/tmp.Ludk8ikCAB/cluster2-mirror-test.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.export 2023-11-08T04:58:20.213 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T04:58:20.214 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ clean_snap_name=snap4-clean 2023-11-08T04:58:20.214 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ create_snap cluster2 mirror test snap4-clean 2023-11-08T04:58:20.214 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T04:58:20.214 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T04:58:20.215 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T04:58:20.215 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap_name=snap4-clean 2023-11-08T04:58:20.215 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap4-clean --debug-rbd=20 --debug-journaler=20 2023-11-08T04:58:21.469 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for i in `seq 1 10` 2023-11-08T04:58:21.470 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ stress_write_image cluster2 mirror test 2023-11-08T04:58:21.470 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T04:58:21.470 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T04:58:21.470 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T04:58:21.470 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-11-08T04:58:21.471 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local duration=8 2023-11-08T04:58:21.471 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ set +e 2023-11-08T04:58:21.472 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2023-11-08T04:58:21.662 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:100 IOs, 12878848 bytes 2023-11-08T04:58:21.833 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:200 IOs, 50663424 bytes 2023-11-08T04:58:22.202 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:300 IOs, 97589248 bytes 2023-11-08T04:58:22.306 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:400 IOs, 110548992 bytes 2023-11-08T04:58:22.409 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:500 IOs, 129092608 bytes 2023-11-08T04:58:22.487 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:600 IOs, 130757632 bytes 2023-11-08T04:58:22.705 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:700 IOs, 203412480 bytes 2023-11-08T04:58:22.815 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:800 IOs, 227691520 bytes 2023-11-08T04:58:22.942 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:900 IOs, 257725440 bytes 2023-11-08T04:58:23.052 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1000 IOs, 266345472 bytes 2023-11-08T04:58:23.293 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1100 IOs, 364437504 bytes 2023-11-08T04:58:23.491 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1200 IOs, 418388992 bytes 2023-11-08T04:58:23.592 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1300 IOs, 427319296 bytes 2023-11-08T04:58:23.702 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1400 IOs, 447669248 bytes 2023-11-08T04:58:23.795 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1500 IOs, 485733376 bytes 2023-11-08T04:58:23.970 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1600 IOs, 525761536 bytes 2023-11-08T04:58:24.154 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1700 IOs, 568329216 bytes 2023-11-08T04:58:24.339 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1800 IOs, 610328576 bytes 2023-11-08T04:58:24.486 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1900 IOs, 672010240 bytes 2023-11-08T04:58:24.637 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2000 IOs, 689481728 bytes 2023-11-08T04:58:24.748 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2100 IOs, 711809024 bytes 2023-11-08T04:58:24.976 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2200 IOs, 792905728 bytes 2023-11-08T04:58:25.054 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2300 IOs, 793541632 bytes 2023-11-08T04:58:25.180 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2400 IOs, 818479104 bytes 2023-11-08T04:58:25.330 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2500 IOs, 853305344 bytes 2023-11-08T04:58:25.517 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2600 IOs, 909052928 bytes 2023-11-08T04:58:25.729 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2700 IOs, 962915328 bytes 2023-11-08T04:58:25.799 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2800 IOs, 963570688 bytes 2023-11-08T04:58:26.016 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2900 IOs, 1028886528 bytes 2023-11-08T04:58:26.136 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3000 IOs, 1051112448 bytes 2023-11-08T04:58:26.383 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3100 IOs, 1139893248 bytes 2023-11-08T04:58:26.503 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3200 IOs, 1154156544 bytes 2023-11-08T04:58:26.679 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3300 IOs, 1202393088 bytes 2023-11-08T04:58:26.925 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3400 IOs, 1297089536 bytes 2023-11-08T04:58:27.201 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3500 IOs, 1391632384 bytes 2023-11-08T04:58:27.389 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3600 IOs, 1449889792 bytes 2023-11-08T04:58:27.574 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3700 IOs, 1504884736 bytes 2023-11-08T04:58:27.673 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3800 IOs, 1515590656 bytes 2023-11-08T04:58:27.858 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3900 IOs, 1566885888 bytes 2023-11-08T04:58:28.055 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4000 IOs, 1626881024 bytes 2023-11-08T04:58:28.251 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4100 IOs, 1656176640 bytes 2023-11-08T04:58:28.470 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4200 IOs, 1719884800 bytes 2023-11-08T04:58:28.633 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4300 IOs, 1767682048 bytes 2023-11-08T04:58:28.869 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4400 IOs, 1842577408 bytes 2023-11-08T04:58:29.029 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4500 IOs, 1879569408 bytes 2023-11-08T04:58:29.118 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4600 IOs, 1882349568 bytes 2023-11-08T04:58:29.294 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4700 IOs, 1930395648 bytes 2023-11-08T04:58:29.378 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4800 IOs, 1938608128 bytes 2023-11-08T04:58:29.462 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4900 IOs, 1939200000 bytes 2023-11-08T04:58:29.477 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ error_code=124 2023-11-08T04:58:29.477 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ set -e 2023-11-08T04:58:29.478 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' 124 -eq 124 ']' 2023-11-08T04:58:29.478 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T04:58:29.478 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2023-11-08T04:58:29.479 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T04:58:29.479 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T04:58:29.479 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T04:58:29.480 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local state_pattern=up+replaying 2023-11-08T04:58:29.480 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local description_pattern= 2023-11-08T04:58:29.480 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local service_pattern= 2023-11-08T04:58:29.481 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-11-08T04:58:29.481 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 1 2023-11-08T04:58:30.478 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2023-11-08T04:58:30.478 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T04:58:30.478 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T04:58:30.478 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T04:58:30.479 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local state_pattern=up+replaying 2023-11-08T04:58:30.479 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local description_pattern= 2023-11-08T04:58:30.479 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local service_pattern= 2023-11-08T04:58:30.479 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ mkfname cluster1-mirror-test.mirror_status 2023-11-08T04:58:30.479 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ echo cluster1-mirror-test.mirror_status 2023-11-08T04:58:30.480 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -e 's|[/ ]|_|g' 2023-11-08T04:58:30.480 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local status_log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.mirror_status 2023-11-08T04:58:30.481 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ CEPH_ARGS= 2023-11-08T04:58:30.481 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 mirror image status mirror/test 2023-11-08T04:58:30.481 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ tee /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.mirror_status 2023-11-08T04:58:30.524 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:test: 2023-11-08T04:58:30.524 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: global_id: 88ced7f9-2739-4a27-ad7a-30edec05edb1 2023-11-08T04:58:30.525 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: state: up+replaying 2023-11-08T04:58:30.525 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: description: replaying, {"bytes_per_second":26590259.2,"bytes_per_snapshot":531805184.0,"last_snapshot_bytes":531805184,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1699419491,"remote_snapshot_timestamp":1699419491,"replay_state":"idle"} 2023-11-08T04:58:30.526 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: service: mirror.3 on smithi087 2023-11-08T04:58:30.526 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: last_update: 2023-11-08 04:58:26 2023-11-08T04:58:30.526 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: peer_sites: 2023-11-08T04:58:30.527 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: name: cluster2 2023-11-08T04:58:30.527 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: state: up+stopped 2023-11-08T04:58:30.527 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: description: local image is primary 2023-11-08T04:58:30.528 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: last_update: 2023-11-08 04:58:26 2023-11-08T04:58:30.528 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ grep '^ state: .*up+replaying' /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.mirror_status 2023-11-08T04:58:30.529 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: state: up+replaying 2023-11-08T04:58:30.530 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ grep '^ description: .*' /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.mirror_status 2023-11-08T04:58:30.530 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: description: replaying, {"bytes_per_second":26590259.2,"bytes_per_snapshot":531805184.0,"last_snapshot_bytes":531805184,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1699419491,"remote_snapshot_timestamp":1699419491,"replay_state":"idle"} 2023-11-08T04:58:30.531 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' -n '' ']' 2023-11-08T04:58:30.532 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ echo up+replaying 2023-11-08T04:58:30.532 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ grep '^up+' 2023-11-08T04:58:30.532 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:up+replaying 2023-11-08T04:58:30.533 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ grep 'service: *mirror..* on ' /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.mirror_status 2023-11-08T04:58:30.533 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: service: mirror.3 on smithi087 2023-11-08T04:58:30.534 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.mirror_status 2023-11-08T04:58:30.535 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local 'last_update=2023-11-08 04:58:26' 2023-11-08T04:58:30.535 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2023-11-08 04:58:26' 2023-11-08T04:58:30.535 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T04:58:30.535 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T04:58:30.536 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T04:58:30.536 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local state_pattern=up+replaying 2023-11-08T04:58:30.536 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local 'prev_last_update=2023-11-08 04:58:26' 2023-11-08T04:58:30.537 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ mkfname cluster1-mirror.mirror_status 2023-11-08T04:58:30.537 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ echo cluster1-mirror.mirror_status 2023-11-08T04:58:30.538 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -e 's|[/ ]|_|g' 2023-11-08T04:58:30.540 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local status_log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror.mirror_status 2023-11-08T04:58:30.540 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2023-11-08T04:58:30.567 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd: failed to query services: (13) Permission denied 2023-11-08T04:58:30.588 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local last_update state 2023-11-08T04:58:30.588 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2023-11-08T04:58:30.590 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ last_update='2023-11-08 04:58:26' 2023-11-08T04:58:30.590 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2023-11-08T04:58:30.592 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ state=up+replaying 2023-11-08T04:58:30.592 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ echo up+replaying 2023-11-08T04:58:30.592 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ grep up+replaying 2023-11-08T04:58:30.593 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:up+replaying 2023-11-08T04:58:30.594 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T04:58:30.594 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T04:58:30.594 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ snap_name=snap5 2023-11-08T04:58:30.595 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ create_snap cluster2 mirror test snap5 2023-11-08T04:58:30.595 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T04:58:30.595 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T04:58:30.595 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T04:58:30.595 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap_name=snap5 2023-11-08T04:58:30.596 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap5 --debug-rbd=20 --debug-journaler=20 2023-11-08T04:58:58.469 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_image_replay_started cluster1 mirror test 2023-11-08T04:58:58.469 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T04:58:58.470 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T04:58:58.470 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T04:58:58.470 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2023-11-08T04:58:58.470 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T04:58:58.471 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T04:58:58.471 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T04:58:58.471 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local state=started 2023-11-08T04:58:58.471 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local s 2023-11-08T04:58:58.471 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-11-08T04:58:58.471 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 1 2023-11-08T04:58:59.471 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test_image_replay_state cluster1 mirror test started 2023-11-08T04:58:59.471 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T04:58:59.471 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T04:58:59.471 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T04:58:59.472 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local test_state=started 2023-11-08T04:58:59.472 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local status_result 2023-11-08T04:58:59.472 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local current_state=stopped 2023-11-08T04:58:59.472 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2023-11-08T04:58:59.472 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_instance=cluster1 2023-11-08T04:58:59.473 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T04:58:59.473 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ grep -i state 2023-11-08T04:58:59.473 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1 2023-11-08T04:58:59.473 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance=cluster1 2023-11-08T04:58:59.473 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local loop_instance 2023-11-08T04:58:59.474 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-11-08T04:58:59.474 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sleep 0 2023-11-08T04:58:59.477 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-11-08T04:58:59.477 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ seq 0 3 2023-11-08T04:58:59.477 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T04:58:59.478 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2023-11-08T04:58:59.478 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1:0 2023-11-08T04:58:59.478 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T04:58:59.478 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance 2023-11-08T04:58:59.478 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-11-08T04:58:59.479 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local val=cluster1:0 2023-11-08T04:58:59.479 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_var_name=cluster 2023-11-08T04:58:59.479 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance_var_name=instance 2023-11-08T04:58:59.479 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cluster=cluster1 2023-11-08T04:58:59.479 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ instance=0 2023-11-08T04:58:59.480 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' 0 = cluster1:0 ']' 2023-11-08T04:58:59.480 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval cluster=cluster1 2023-11-08T04:58:59.480 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T04:58:59.480 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval instance=0 2023-11-08T04:58:59.480 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=0 2023-11-08T04:58:59.481 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-11-08T04:58:59.481 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local local_cluster=cluster1:0 2023-11-08T04:58:59.481 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster=cluster1 2023-11-08T04:58:59.481 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance 2023-11-08T04:58:59.481 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-11-08T04:58:59.482 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local val=cluster1:0 2023-11-08T04:58:59.482 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster_var_name=local_cluster 2023-11-08T04:58:59.482 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance_var_name=instance 2023-11-08T04:58:59.482 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T04:58:59.482 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=0 2023-11-08T04:58:59.483 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ '[' 0 = cluster1:0 ']' 2023-11-08T04:58:59.483 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval local_cluster=cluster1 2023-11-08T04:58:59.483 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ local_cluster=cluster1 2023-11-08T04:58:59.483 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval instance=0 2023-11-08T04:58:59.483 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ instance=0 2023-11-08T04:58:59.483 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-11-08T04:58:59.492 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-08T04:58:59.492 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-08T04:58:59.492 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-08T04:58:59.492 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2023-11-08T04:58:59.587 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-11-08T04:58:59.588 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd mirror status : get status for rbd mirror 2023-11-08T04:58:59.588 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:admin_socket: invalid command 2023-11-08T04:58:59.591 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T04:58:59.591 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2023-11-08T04:58:59.591 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1:1 2023-11-08T04:58:59.591 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T04:58:59.592 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance 2023-11-08T04:58:59.592 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-11-08T04:58:59.592 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local val=cluster1:1 2023-11-08T04:58:59.592 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_var_name=cluster 2023-11-08T04:58:59.592 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance_var_name=instance 2023-11-08T04:58:59.592 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cluster=cluster1 2023-11-08T04:58:59.593 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ instance=1 2023-11-08T04:58:59.593 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' 1 = cluster1:1 ']' 2023-11-08T04:58:59.593 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval cluster=cluster1 2023-11-08T04:58:59.593 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T04:58:59.593 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval instance=1 2023-11-08T04:58:59.594 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=1 2023-11-08T04:58:59.594 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-11-08T04:58:59.594 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local local_cluster=cluster1:1 2023-11-08T04:58:59.594 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster=cluster1 2023-11-08T04:58:59.594 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance 2023-11-08T04:58:59.594 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-11-08T04:58:59.595 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local val=cluster1:1 2023-11-08T04:58:59.595 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster_var_name=local_cluster 2023-11-08T04:58:59.595 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance_var_name=instance 2023-11-08T04:58:59.595 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T04:58:59.595 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=1 2023-11-08T04:58:59.596 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ '[' 1 = cluster1:1 ']' 2023-11-08T04:58:59.596 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval local_cluster=cluster1 2023-11-08T04:58:59.596 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ local_cluster=cluster1 2023-11-08T04:58:59.596 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval instance=1 2023-11-08T04:58:59.596 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ instance=1 2023-11-08T04:58:59.596 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-11-08T04:58:59.605 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-08T04:58:59.605 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-08T04:58:59.605 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-08T04:58:59.605 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2023-11-08T04:58:59.693 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-11-08T04:58:59.694 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd mirror status : get status for rbd mirror 2023-11-08T04:58:59.694 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:admin_socket: invalid command 2023-11-08T04:58:59.697 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T04:58:59.697 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2023-11-08T04:58:59.697 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1:2 2023-11-08T04:58:59.697 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T04:58:59.698 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance 2023-11-08T04:58:59.698 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-11-08T04:58:59.698 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local val=cluster1:2 2023-11-08T04:58:59.698 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_var_name=cluster 2023-11-08T04:58:59.698 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance_var_name=instance 2023-11-08T04:58:59.699 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cluster=cluster1 2023-11-08T04:58:59.699 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ instance=2 2023-11-08T04:58:59.699 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' 2 = cluster1:2 ']' 2023-11-08T04:58:59.699 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval cluster=cluster1 2023-11-08T04:58:59.699 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T04:58:59.700 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval instance=2 2023-11-08T04:58:59.700 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=2 2023-11-08T04:58:59.700 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-11-08T04:58:59.700 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local local_cluster=cluster1:2 2023-11-08T04:58:59.700 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster=cluster1 2023-11-08T04:58:59.701 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance 2023-11-08T04:58:59.701 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-11-08T04:58:59.701 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local val=cluster1:2 2023-11-08T04:58:59.701 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster_var_name=local_cluster 2023-11-08T04:58:59.701 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance_var_name=instance 2023-11-08T04:58:59.702 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T04:58:59.702 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=2 2023-11-08T04:58:59.702 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ '[' 2 = cluster1:2 ']' 2023-11-08T04:58:59.702 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval local_cluster=cluster1 2023-11-08T04:58:59.702 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ local_cluster=cluster1 2023-11-08T04:58:59.702 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval instance=2 2023-11-08T04:58:59.703 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ instance=2 2023-11-08T04:58:59.703 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-11-08T04:58:59.710 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-08T04:58:59.711 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-08T04:58:59.711 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-08T04:58:59.711 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2023-11-08T04:58:59.798 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-11-08T04:58:59.799 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd mirror status : get status for rbd mirror 2023-11-08T04:58:59.799 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:admin_socket: invalid command 2023-11-08T04:58:59.802 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T04:58:59.802 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2023-11-08T04:58:59.802 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1:3 2023-11-08T04:58:59.802 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T04:58:59.803 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance 2023-11-08T04:58:59.803 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ set_cluster_instance cluster1:3 cluster instance 2023-11-08T04:58:59.803 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local val=cluster1:3 2023-11-08T04:58:59.803 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_var_name=cluster 2023-11-08T04:58:59.803 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance_var_name=instance 2023-11-08T04:58:59.804 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cluster=cluster1 2023-11-08T04:58:59.804 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ instance=3 2023-11-08T04:58:59.804 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' 3 = cluster1:3 ']' 2023-11-08T04:58:59.804 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval cluster=cluster1 2023-11-08T04:58:59.804 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T04:58:59.805 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval instance=3 2023-11-08T04:58:59.805 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=3 2023-11-08T04:58:59.805 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ daemon_asok_file cluster1:3 cluster1 2023-11-08T04:58:59.805 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local local_cluster=cluster1:3 2023-11-08T04:58:59.805 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster=cluster1 2023-11-08T04:58:59.805 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance 2023-11-08T04:58:59.806 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2023-11-08T04:58:59.806 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local val=cluster1:3 2023-11-08T04:58:59.806 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster_var_name=local_cluster 2023-11-08T04:58:59.806 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance_var_name=instance 2023-11-08T04:58:59.806 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T04:58:59.807 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=3 2023-11-08T04:58:59.807 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ '[' 3 = cluster1:3 ']' 2023-11-08T04:58:59.807 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval local_cluster=cluster1 2023-11-08T04:58:59.807 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ local_cluster=cluster1 2023-11-08T04:58:59.807 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval instance=3 2023-11-08T04:58:59.808 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ instance=3 2023-11-08T04:58:59.808 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2023-11-08T04:58:59.816 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-11-08T04:58:59.816 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-11-08T04:58:59.816 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-11-08T04:58:59.816 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2023-11-08T04:58:59.913 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ return 0 2023-11-08T04:58:59.914 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ status_result=' "state": "Replaying"' 2023-11-08T04:58:59.914 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ echo ' "state": "Replaying"' 2023-11-08T04:58:59.914 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ grep -i Replaying 2023-11-08T04:58:59.915 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: "state": "Replaying" 2023-11-08T04:58:59.917 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ current_state=started 2023-11-08T04:58:59.917 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test started = started 2023-11-08T04:58:59.918 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T04:58:59.918 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2023-11-08T04:58:59.918 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local local_cluster=cluster1 2023-11-08T04:58:59.918 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T04:58:59.918 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T04:58:59.919 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T04:58:59.919 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' snapshot = journal ']' 2023-11-08T04:58:59.919 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' snapshot = snapshot ']' 2023-11-08T04:58:59.919 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2023-11-08T04:58:59.919 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local local_cluster=cluster1 2023-11-08T04:58:59.920 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T04:58:59.920 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T04:58:59.920 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T04:58:59.920 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ mkfname cluster2-mirror-test.status 2023-11-08T04:58:59.920 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ echo cluster2-mirror-test.status 2023-11-08T04:58:59.921 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -e 's|[/ ]|_|g' 2023-11-08T04:58:59.921 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local status_log=/tmp/tmp.Ludk8ikCAB/cluster2-mirror-test.status 2023-11-08T04:58:59.921 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ mkfname cluster1-mirror-test.status 2023-11-08T04:58:59.921 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ echo cluster1-mirror-test.status 2023-11-08T04:58:59.921 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -e 's|[/ ]|_|g' 2023-11-08T04:58:59.922 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local local_status_log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.status 2023-11-08T04:58:59.923 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ mirror_image_snapshot cluster2 mirror test 2023-11-08T04:58:59.923 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T04:58:59.923 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T04:58:59.923 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T04:58:59.924 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2023-11-08T04:59:01.034 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.8526133669419269) > weight (0.25) 2023-11-08T04:59:01.035 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.3226249393924049) > weight (0.25) 2023-11-08T04:59:01.035 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.4935773686131475) > weight (0.25) 2023-11-08T04:59:01.035 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.3 with skip (0.30777506428883983) > weight (0.25) 2023-11-08T04:59:01.035 INFO:tasks.rbd_mirror_thrash:waiting for 101 secs before thrashing 2023-11-08T04:59:01.473 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:Snapshot ID: 21 2023-11-08T04:59:01.494 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.Ludk8ikCAB/cluster2-mirror-test.status 2023-11-08T04:59:01.494 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T04:59:01.494 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T04:59:01.494 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T04:59:01.495 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster2-mirror-test.status 2023-11-08T04:59:01.495 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T04:59:01.495 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2023-11-08T04:59:01.553 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-11-08T04:59:01.554 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snapshot_id=21 2023-11-08T04:59:01.555 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ true 2023-11-08T04:59:01.555 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-11-08T04:59:01.555 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0.2 2023-11-08T04:59:01.756 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.status 2023-11-08T04:59:01.757 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T04:59:01.757 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T04:59:01.757 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T04:59:01.757 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.status 2023-11-08T04:59:01.757 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-11-08T04:59:01.758 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T04:59:01.797 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ true 2023-11-08T04:59:01.798 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-08T04:59:01.799 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:-:1.1: Document is empty 2023-11-08T04:59:01.799 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: 2023-11-08T04:59:01.799 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:^ 2023-11-08T04:59:01.800 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local primary_snapshot_id= 2023-11-08T04:59:01.800 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test 21 = '' 2023-11-08T04:59:01.800 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-11-08T04:59:01.800 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0.4 2023-11-08T04:59:02.201 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.status 2023-11-08T04:59:02.201 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T04:59:02.201 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T04:59:02.201 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T04:59:02.202 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.status 2023-11-08T04:59:02.202 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-11-08T04:59:02.202 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T04:59:02.240 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ true 2023-11-08T04:59:02.241 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-08T04:59:02.242 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:-:1.1: Document is empty 2023-11-08T04:59:02.242 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: 2023-11-08T04:59:02.243 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:^ 2023-11-08T04:59:02.243 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local primary_snapshot_id= 2023-11-08T04:59:02.243 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test 21 = '' 2023-11-08T04:59:02.243 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-11-08T04:59:02.243 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0.8 2023-11-08T04:59:03.044 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.status 2023-11-08T04:59:03.044 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T04:59:03.044 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T04:59:03.045 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T04:59:03.045 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.status 2023-11-08T04:59:03.045 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-11-08T04:59:03.045 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T04:59:03.090 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ true 2023-11-08T04:59:03.090 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-08T04:59:03.091 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:-:1.1: Document is empty 2023-11-08T04:59:03.091 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: 2023-11-08T04:59:03.092 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:^ 2023-11-08T04:59:03.092 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local primary_snapshot_id= 2023-11-08T04:59:03.092 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test 21 = '' 2023-11-08T04:59:03.093 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-11-08T04:59:03.093 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 1.6 2023-11-08T04:59:04.693 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.status 2023-11-08T04:59:04.694 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T04:59:04.694 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T04:59:04.694 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T04:59:04.695 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.status 2023-11-08T04:59:04.695 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-11-08T04:59:04.695 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T04:59:04.734 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-08T04:59:04.736 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local primary_snapshot_id=21 2023-11-08T04:59:04.736 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test 21 = 21 2023-11-08T04:59:04.737 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T04:59:04.737 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_snap_present cluster1 mirror test snap5 2023-11-08T04:59:04.737 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T04:59:04.737 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T04:59:04.737 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T04:59:04.738 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap_name=snap5 2023-11-08T04:59:04.738 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local s 2023-11-08T04:59:04.738 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-11-08T04:59:04.738 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 1 2023-11-08T04:59:05.737 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 info mirror/test@snap5 2023-11-08T04:59:05.777 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:rbd image 'test': 2023-11-08T04:59:05.778 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: size 512 MiB in 128 objects 2023-11-08T04:59:05.778 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: order 22 (4 MiB objects) 2023-11-08T04:59:05.778 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: snapshot_count: 10 2023-11-08T04:59:05.778 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: id: 10afad8fe72b 2023-11-08T04:59:05.778 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: block_name_prefix: rbd_data.10afad8fe72b 2023-11-08T04:59:05.779 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: format: 2 2023-11-08T04:59:05.779 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: features: layering, exclusive-lock, non-primary 2023-11-08T04:59:05.779 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: op_features: 2023-11-08T04:59:05.779 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: flags: 2023-11-08T04:59:05.779 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: create_timestamp: Wed Nov 8 04:54:28 2023 2023-11-08T04:59:05.780 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: access_timestamp: Wed Nov 8 04:58:17 2023 2023-11-08T04:59:05.780 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: modify_timestamp: Wed Nov 8 04:54:28 2023 2023-11-08T04:59:05.780 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: protected: False 2023-11-08T04:59:05.780 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring state: enabled 2023-11-08T04:59:05.780 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring mode: snapshot 2023-11-08T04:59:05.781 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring global id: 88ced7f9-2739-4a27-ad7a-30edec05edb1 2023-11-08T04:59:05.781 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring primary: false 2023-11-08T04:59:05.782 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T04:59:05.782 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' -n snap4-clean ']' 2023-11-08T04:59:05.783 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ compare_image_snaps mirror test snap4-clean 2023-11-08T04:59:05.783 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T04:59:05.783 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T04:59:05.783 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap_name=snap4-clean 2023-11-08T04:59:05.783 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local ret=0 2023-11-08T04:59:05.784 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local rmt_export=/tmp/tmp.Ludk8ikCAB/cluster2-mirror-test.export 2023-11-08T04:59:05.784 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local loc_export=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.export 2023-11-08T04:59:05.784 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rm -f /tmp/tmp.Ludk8ikCAB/cluster2-mirror-test.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.export 2023-11-08T04:59:05.784 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 -p mirror export test@snap4-clean /tmp/tmp.Ludk8ikCAB/cluster2-mirror-test.export 2023-11-08T04:59:06.409 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-11-08T04:59:06.412 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 -p mirror export test@snap4-clean /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.export 2023-11-08T04:59:07.256 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-11-08T04:59:07.259 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ cmp /tmp/tmp.Ludk8ikCAB/cluster2-mirror-test.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.export 2023-11-08T04:59:07.473 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rm -f /tmp/tmp.Ludk8ikCAB/cluster2-mirror-test.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.export 2023-11-08T04:59:07.614 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T04:59:07.615 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ compare_image_snaps mirror test snap5 2023-11-08T04:59:07.615 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T04:59:07.615 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T04:59:07.615 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap_name=snap5 2023-11-08T04:59:07.615 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local ret=0 2023-11-08T04:59:07.616 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local rmt_export=/tmp/tmp.Ludk8ikCAB/cluster2-mirror-test.export 2023-11-08T04:59:07.616 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local loc_export=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.export 2023-11-08T04:59:07.617 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rm -f /tmp/tmp.Ludk8ikCAB/cluster2-mirror-test.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.export 2023-11-08T04:59:07.617 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 -p mirror export test@snap5 /tmp/tmp.Ludk8ikCAB/cluster2-mirror-test.export 2023-11-08T04:59:08.204 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-11-08T04:59:08.207 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 -p mirror export test@snap5 /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.export 2023-11-08T04:59:08.965 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-11-08T04:59:08.968 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ cmp /tmp/tmp.Ludk8ikCAB/cluster2-mirror-test.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.export 2023-11-08T04:59:09.179 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rm -f /tmp/tmp.Ludk8ikCAB/cluster2-mirror-test.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.export 2023-11-08T04:59:09.320 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T04:59:09.321 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ clean_snap_name=snap5-clean 2023-11-08T04:59:09.321 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ create_snap cluster2 mirror test snap5-clean 2023-11-08T04:59:09.321 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T04:59:09.321 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T04:59:09.321 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T04:59:09.321 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap_name=snap5-clean 2023-11-08T04:59:09.322 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap5-clean --debug-rbd=20 --debug-journaler=20 2023-11-08T04:59:09.483 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for i in `seq 1 10` 2023-11-08T04:59:09.483 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ stress_write_image cluster2 mirror test 2023-11-08T04:59:09.484 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T04:59:09.484 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T04:59:09.484 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T04:59:09.484 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-11-08T04:59:09.485 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local duration=5 2023-11-08T04:59:09.485 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ set +e 2023-11-08T04:59:09.485 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2023-11-08T04:59:09.734 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:100 IOs, 58466304 bytes 2023-11-08T04:59:09.947 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:200 IOs, 110870528 bytes 2023-11-08T04:59:10.093 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:300 IOs, 155379712 bytes 2023-11-08T04:59:10.243 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:400 IOs, 175078400 bytes 2023-11-08T04:59:10.398 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:500 IOs, 224843776 bytes 2023-11-08T04:59:10.630 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:600 IOs, 290486272 bytes 2023-11-08T04:59:10.802 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:700 IOs, 339234816 bytes 2023-11-08T04:59:10.910 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:800 IOs, 353478656 bytes 2023-11-08T04:59:11.022 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:900 IOs, 377158656 bytes 2023-11-08T04:59:11.176 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1000 IOs, 409945088 bytes 2023-11-08T04:59:11.254 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1100 IOs, 419368960 bytes 2023-11-08T04:59:11.493 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1200 IOs, 486864896 bytes 2023-11-08T04:59:11.705 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1300 IOs, 552358912 bytes 2023-11-08T04:59:11.803 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1400 IOs, 560109568 bytes 2023-11-08T04:59:12.044 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1500 IOs, 652005376 bytes 2023-11-08T04:59:12.219 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1600 IOs, 690952192 bytes 2023-11-08T04:59:12.342 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1700 IOs, 714002432 bytes 2023-11-08T04:59:12.454 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1800 IOs, 727079936 bytes 2023-11-08T04:59:12.604 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1900 IOs, 754074624 bytes 2023-11-08T04:59:12.869 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2000 IOs, 844651520 bytes 2023-11-08T04:59:13.008 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2100 IOs, 880556032 bytes 2023-11-08T04:59:13.098 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2200 IOs, 889186304 bytes 2023-11-08T04:59:13.244 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2300 IOs, 920730624 bytes 2023-11-08T04:59:13.372 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2400 IOs, 959371264 bytes 2023-11-08T04:59:13.487 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2500 IOs, 975282176 bytes 2023-11-08T04:59:13.662 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2600 IOs, 1025422336 bytes 2023-11-08T04:59:13.864 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2700 IOs, 1095517184 bytes 2023-11-08T04:59:13.946 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2800 IOs, 1101778944 bytes 2023-11-08T04:59:14.015 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2900 IOs, 1104612352 bytes 2023-11-08T04:59:14.135 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3000 IOs, 1128398848 bytes 2023-11-08T04:59:14.216 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3100 IOs, 1138541568 bytes 2023-11-08T04:59:14.337 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3200 IOs, 1147170816 bytes 2023-11-08T04:59:14.489 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ error_code=124 2023-11-08T04:59:14.490 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ set -e 2023-11-08T04:59:14.490 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' 124 -eq 124 ']' 2023-11-08T04:59:14.490 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T04:59:14.490 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2023-11-08T04:59:14.491 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T04:59:14.491 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T04:59:14.491 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T04:59:14.491 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local state_pattern=up+replaying 2023-11-08T04:59:14.491 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local description_pattern= 2023-11-08T04:59:14.491 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local service_pattern= 2023-11-08T04:59:14.492 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-11-08T04:59:14.492 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 1 2023-11-08T04:59:15.491 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2023-11-08T04:59:15.491 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T04:59:15.491 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T04:59:15.492 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T04:59:15.492 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local state_pattern=up+replaying 2023-11-08T04:59:15.492 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local description_pattern= 2023-11-08T04:59:15.492 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local service_pattern= 2023-11-08T04:59:15.492 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ mkfname cluster1-mirror-test.mirror_status 2023-11-08T04:59:15.493 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ echo cluster1-mirror-test.mirror_status 2023-11-08T04:59:15.493 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -e 's|[/ ]|_|g' 2023-11-08T04:59:15.493 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local status_log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.mirror_status 2023-11-08T04:59:15.494 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ CEPH_ARGS= 2023-11-08T04:59:15.494 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 mirror image status mirror/test 2023-11-08T04:59:15.494 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ tee /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.mirror_status 2023-11-08T04:59:15.538 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:test: 2023-11-08T04:59:15.539 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: global_id: 88ced7f9-2739-4a27-ad7a-30edec05edb1 2023-11-08T04:59:15.539 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: state: up+replaying 2023-11-08T04:59:15.539 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":531805184.0,"last_snapshot_bytes":531805184,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1699419491,"remote_snapshot_timestamp":1699419491,"replay_state":"idle"} 2023-11-08T04:59:15.539 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: service: mirror.3 on smithi087 2023-11-08T04:59:15.539 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: last_update: 2023-11-08 04:58:56 2023-11-08T04:59:15.540 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: peer_sites: 2023-11-08T04:59:15.540 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: name: cluster2 2023-11-08T04:59:15.540 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: state: up+stopped 2023-11-08T04:59:15.540 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: description: local image is primary 2023-11-08T04:59:15.540 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: last_update: 2023-11-08 04:58:56 2023-11-08T04:59:15.542 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ grep '^ state: .*up+replaying' /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.mirror_status 2023-11-08T04:59:15.544 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: state: up+replaying 2023-11-08T04:59:15.544 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ grep '^ description: .*' /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.mirror_status 2023-11-08T04:59:15.544 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":531805184.0,"last_snapshot_bytes":531805184,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1699419491,"remote_snapshot_timestamp":1699419491,"replay_state":"idle"} 2023-11-08T04:59:15.545 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' -n '' ']' 2023-11-08T04:59:15.545 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ echo up+replaying 2023-11-08T04:59:15.546 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ grep '^up+' 2023-11-08T04:59:15.546 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:up+replaying 2023-11-08T04:59:15.547 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ grep 'service: *mirror..* on ' /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.mirror_status 2023-11-08T04:59:15.547 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: service: mirror.3 on smithi087 2023-11-08T04:59:15.547 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.mirror_status 2023-11-08T04:59:15.549 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local 'last_update=2023-11-08 04:58:56' 2023-11-08T04:59:15.549 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2023-11-08 04:58:56' 2023-11-08T04:59:15.549 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T04:59:15.550 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T04:59:15.550 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T04:59:15.550 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local state_pattern=up+replaying 2023-11-08T04:59:15.550 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local 'prev_last_update=2023-11-08 04:58:56' 2023-11-08T04:59:15.550 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ mkfname cluster1-mirror.mirror_status 2023-11-08T04:59:15.551 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ echo cluster1-mirror.mirror_status 2023-11-08T04:59:15.551 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -e 's|[/ ]|_|g' 2023-11-08T04:59:15.551 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local status_log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror.mirror_status 2023-11-08T04:59:15.552 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2023-11-08T04:59:15.582 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd: failed to query services: (13) Permission denied 2023-11-08T04:59:15.606 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local last_update state 2023-11-08T04:59:15.607 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2023-11-08T04:59:15.608 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ last_update='2023-11-08 04:58:56' 2023-11-08T04:59:15.609 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2023-11-08T04:59:15.610 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ state=up+replaying 2023-11-08T04:59:15.611 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ echo up+replaying 2023-11-08T04:59:15.611 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ grep up+replaying 2023-11-08T04:59:15.611 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:up+replaying 2023-11-08T04:59:15.612 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T04:59:15.613 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T04:59:15.613 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ snap_name=snap6 2023-11-08T04:59:15.613 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ create_snap cluster2 mirror test snap6 2023-11-08T04:59:15.613 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T04:59:15.613 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T04:59:15.614 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T04:59:15.614 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap_name=snap6 2023-11-08T04:59:15.614 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap6 --debug-rbd=20 --debug-journaler=20 2023-11-08T04:59:30.027 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.0 2023-11-08T04:59:30.027 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Sent signal 15 2023-11-08T04:59:30.028 INFO:tasks.rbd_mirror_thrash:waiting for 3 secs before reviving daemons 2023-11-08T04:59:30.029 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi087.stderr:2023-11-08T04:59:30.027+0000 7fa888bb8700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 (PID: 120103) UID: 1000 2023-11-08T04:59:30.229 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi087.stderr:daemon-helper: command failed with exit status 1 2023-11-08T04:59:33.028 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.0 2023-11-08T04:59:33.029 INFO:teuthology.orchestra.run:waiting for 600 2023-11-08T04:59:33.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-08T04:59:33.030 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.0 2023-11-08T04:59:33.030 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Restarting daemon 2023-11-08T04:59:33.030 DEBUG:teuthology.orchestra.run.smithi087:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 2023-11-08T04:59:33.033 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Started 2023-11-08T04:59:33.033 INFO:tasks.rbd_mirror_thrash:waiting for 68 secs before thrashing 2023-11-08T04:59:44.483 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_image_replay_started cluster1 mirror test 2023-11-08T04:59:44.484 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T04:59:44.484 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T04:59:44.484 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T04:59:44.485 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2023-11-08T04:59:44.485 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T04:59:44.485 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T04:59:44.485 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T04:59:44.485 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local state=started 2023-11-08T04:59:44.486 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local s 2023-11-08T04:59:44.486 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-11-08T04:59:44.486 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 1 2023-11-08T04:59:45.485 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test_image_replay_state cluster1 mirror test started 2023-11-08T04:59:45.486 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T04:59:45.486 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T04:59:45.486 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T04:59:45.486 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local test_state=started 2023-11-08T04:59:45.486 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local status_result 2023-11-08T04:59:45.487 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local current_state=stopped 2023-11-08T04:59:45.487 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2023-11-08T04:59:45.487 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_instance=cluster1 2023-11-08T04:59:45.487 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T04:59:45.487 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ grep -i state 2023-11-08T04:59:45.488 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1 2023-11-08T04:59:45.488 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance=cluster1 2023-11-08T04:59:45.488 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local loop_instance 2023-11-08T04:59:45.488 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-11-08T04:59:45.488 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sleep 0 2023-11-08T04:59:45.491 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-11-08T04:59:45.491 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ seq 0 3 2023-11-08T04:59:45.491 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T04:59:45.491 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2023-11-08T04:59:45.492 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1:0 2023-11-08T04:59:45.492 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T04:59:45.492 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance 2023-11-08T04:59:45.492 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-11-08T04:59:45.492 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local val=cluster1:0 2023-11-08T04:59:45.493 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_var_name=cluster 2023-11-08T04:59:45.493 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance_var_name=instance 2023-11-08T04:59:45.493 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cluster=cluster1 2023-11-08T04:59:45.493 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ instance=0 2023-11-08T04:59:45.493 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' 0 = cluster1:0 ']' 2023-11-08T04:59:45.494 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval cluster=cluster1 2023-11-08T04:59:45.494 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T04:59:45.494 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval instance=0 2023-11-08T04:59:45.494 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=0 2023-11-08T04:59:45.494 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-11-08T04:59:45.495 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local local_cluster=cluster1:0 2023-11-08T04:59:45.496 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster=cluster1 2023-11-08T04:59:45.496 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance 2023-11-08T04:59:45.497 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-11-08T04:59:45.497 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local val=cluster1:0 2023-11-08T04:59:45.497 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster_var_name=local_cluster 2023-11-08T04:59:45.497 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance_var_name=instance 2023-11-08T04:59:45.497 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T04:59:45.498 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=0 2023-11-08T04:59:45.498 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ '[' 0 = cluster1:0 ']' 2023-11-08T04:59:45.498 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval local_cluster=cluster1 2023-11-08T04:59:45.498 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ local_cluster=cluster1 2023-11-08T04:59:45.498 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval instance=0 2023-11-08T04:59:45.499 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ instance=0 2023-11-08T04:59:45.499 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-11-08T04:59:45.505 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-08T04:59:45.506 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-08T04:59:45.506 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-08T04:59:45.506 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2023-11-08T04:59:45.600 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-11-08T04:59:45.600 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd mirror status : get status for rbd mirror 2023-11-08T04:59:45.601 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:admin_socket: invalid command 2023-11-08T04:59:45.603 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T04:59:45.603 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2023-11-08T04:59:45.604 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1:1 2023-11-08T04:59:45.604 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T04:59:45.604 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance 2023-11-08T04:59:45.604 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-11-08T04:59:45.605 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local val=cluster1:1 2023-11-08T04:59:45.605 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_var_name=cluster 2023-11-08T04:59:45.605 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance_var_name=instance 2023-11-08T04:59:45.605 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cluster=cluster1 2023-11-08T04:59:45.605 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ instance=1 2023-11-08T04:59:45.605 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' 1 = cluster1:1 ']' 2023-11-08T04:59:45.606 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval cluster=cluster1 2023-11-08T04:59:45.606 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T04:59:45.606 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval instance=1 2023-11-08T04:59:45.606 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=1 2023-11-08T04:59:45.606 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-11-08T04:59:45.607 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local local_cluster=cluster1:1 2023-11-08T04:59:45.607 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster=cluster1 2023-11-08T04:59:45.607 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance 2023-11-08T04:59:45.607 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-11-08T04:59:45.607 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local val=cluster1:1 2023-11-08T04:59:45.608 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster_var_name=local_cluster 2023-11-08T04:59:45.608 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance_var_name=instance 2023-11-08T04:59:45.608 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T04:59:45.608 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=1 2023-11-08T04:59:45.608 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ '[' 1 = cluster1:1 ']' 2023-11-08T04:59:45.609 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval local_cluster=cluster1 2023-11-08T04:59:45.609 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ local_cluster=cluster1 2023-11-08T04:59:45.609 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval instance=1 2023-11-08T04:59:45.609 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ instance=1 2023-11-08T04:59:45.609 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-11-08T04:59:45.617 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-08T04:59:45.617 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-08T04:59:45.617 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-08T04:59:45.618 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2023-11-08T04:59:45.706 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-11-08T04:59:45.706 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd mirror status : get status for rbd mirror 2023-11-08T04:59:45.706 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:admin_socket: invalid command 2023-11-08T04:59:45.709 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T04:59:45.709 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2023-11-08T04:59:45.710 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1:2 2023-11-08T04:59:45.710 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T04:59:45.710 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance 2023-11-08T04:59:45.710 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-11-08T04:59:45.711 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local val=cluster1:2 2023-11-08T04:59:45.711 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_var_name=cluster 2023-11-08T04:59:45.711 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance_var_name=instance 2023-11-08T04:59:45.711 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cluster=cluster1 2023-11-08T04:59:45.712 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ instance=2 2023-11-08T04:59:45.712 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' 2 = cluster1:2 ']' 2023-11-08T04:59:45.712 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval cluster=cluster1 2023-11-08T04:59:45.712 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T04:59:45.712 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval instance=2 2023-11-08T04:59:45.713 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=2 2023-11-08T04:59:45.713 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-11-08T04:59:45.713 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local local_cluster=cluster1:2 2023-11-08T04:59:45.713 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster=cluster1 2023-11-08T04:59:45.714 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance 2023-11-08T04:59:45.714 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-11-08T04:59:45.714 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local val=cluster1:2 2023-11-08T04:59:45.714 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster_var_name=local_cluster 2023-11-08T04:59:45.715 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance_var_name=instance 2023-11-08T04:59:45.715 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T04:59:45.715 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=2 2023-11-08T04:59:45.715 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ '[' 2 = cluster1:2 ']' 2023-11-08T04:59:45.715 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval local_cluster=cluster1 2023-11-08T04:59:45.716 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ local_cluster=cluster1 2023-11-08T04:59:45.716 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval instance=2 2023-11-08T04:59:45.716 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ instance=2 2023-11-08T04:59:45.716 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-11-08T04:59:45.723 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-08T04:59:45.724 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-08T04:59:45.724 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-08T04:59:45.724 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2023-11-08T04:59:45.812 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-11-08T04:59:45.812 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd mirror status : get status for rbd mirror 2023-11-08T04:59:45.812 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:admin_socket: invalid command 2023-11-08T04:59:45.815 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T04:59:45.815 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2023-11-08T04:59:45.816 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1:3 2023-11-08T04:59:45.816 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T04:59:45.816 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance 2023-11-08T04:59:45.816 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ set_cluster_instance cluster1:3 cluster instance 2023-11-08T04:59:45.816 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local val=cluster1:3 2023-11-08T04:59:45.817 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_var_name=cluster 2023-11-08T04:59:45.817 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance_var_name=instance 2023-11-08T04:59:45.817 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cluster=cluster1 2023-11-08T04:59:45.817 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ instance=3 2023-11-08T04:59:45.817 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' 3 = cluster1:3 ']' 2023-11-08T04:59:45.818 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval cluster=cluster1 2023-11-08T04:59:45.818 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T04:59:45.818 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval instance=3 2023-11-08T04:59:45.818 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=3 2023-11-08T04:59:45.818 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ daemon_asok_file cluster1:3 cluster1 2023-11-08T04:59:45.819 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local local_cluster=cluster1:3 2023-11-08T04:59:45.819 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster=cluster1 2023-11-08T04:59:45.819 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance 2023-11-08T04:59:45.819 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2023-11-08T04:59:45.819 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local val=cluster1:3 2023-11-08T04:59:45.820 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster_var_name=local_cluster 2023-11-08T04:59:45.820 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance_var_name=instance 2023-11-08T04:59:45.820 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T04:59:45.820 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=3 2023-11-08T04:59:45.821 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ '[' 3 = cluster1:3 ']' 2023-11-08T04:59:45.821 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval local_cluster=cluster1 2023-11-08T04:59:45.821 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ local_cluster=cluster1 2023-11-08T04:59:45.821 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval instance=3 2023-11-08T04:59:45.821 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ instance=3 2023-11-08T04:59:45.822 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2023-11-08T04:59:45.829 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-11-08T04:59:45.829 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-11-08T04:59:45.829 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-11-08T04:59:45.830 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2023-11-08T04:59:45.925 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ return 0 2023-11-08T04:59:45.926 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ status_result=' "state": "Replaying"' 2023-11-08T04:59:45.927 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ echo ' "state": "Replaying"' 2023-11-08T04:59:45.927 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ grep -i Replaying 2023-11-08T04:59:45.927 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: "state": "Replaying" 2023-11-08T04:59:45.929 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ current_state=started 2023-11-08T04:59:45.929 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test started = started 2023-11-08T04:59:45.930 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T04:59:45.930 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2023-11-08T04:59:45.930 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local local_cluster=cluster1 2023-11-08T04:59:45.930 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T04:59:45.931 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T04:59:45.931 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T04:59:45.931 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' snapshot = journal ']' 2023-11-08T04:59:45.932 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' snapshot = snapshot ']' 2023-11-08T04:59:45.932 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2023-11-08T04:59:45.932 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local local_cluster=cluster1 2023-11-08T04:59:45.932 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T04:59:45.932 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T04:59:45.933 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T04:59:45.933 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ mkfname cluster2-mirror-test.status 2023-11-08T04:59:45.933 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ echo cluster2-mirror-test.status 2023-11-08T04:59:45.933 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -e 's|[/ ]|_|g' 2023-11-08T04:59:45.933 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local status_log=/tmp/tmp.Ludk8ikCAB/cluster2-mirror-test.status 2023-11-08T04:59:45.934 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ mkfname cluster1-mirror-test.status 2023-11-08T04:59:45.934 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ echo cluster1-mirror-test.status 2023-11-08T04:59:45.934 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -e 's|[/ ]|_|g' 2023-11-08T04:59:45.935 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local local_status_log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.status 2023-11-08T04:59:45.935 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ mirror_image_snapshot cluster2 mirror test 2023-11-08T04:59:45.936 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T04:59:45.936 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T04:59:45.936 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T04:59:45.936 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2023-11-08T04:59:47.489 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:Snapshot ID: 25 2023-11-08T04:59:47.503 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.Ludk8ikCAB/cluster2-mirror-test.status 2023-11-08T04:59:47.503 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T04:59:47.504 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T04:59:47.504 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T04:59:47.504 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster2-mirror-test.status 2023-11-08T04:59:47.505 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T04:59:47.505 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2023-11-08T04:59:47.565 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-11-08T04:59:47.567 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snapshot_id=25 2023-11-08T04:59:47.567 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ true 2023-11-08T04:59:47.567 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-11-08T04:59:47.568 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0.2 2023-11-08T04:59:47.768 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.status 2023-11-08T04:59:47.769 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T04:59:47.769 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T04:59:47.769 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T04:59:47.769 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.status 2023-11-08T04:59:47.769 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-11-08T04:59:47.770 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T04:59:47.808 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ true 2023-11-08T04:59:47.808 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-08T04:59:47.809 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:-:1.1: Document is empty 2023-11-08T04:59:47.809 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: 2023-11-08T04:59:47.809 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:^ 2023-11-08T04:59:47.810 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local primary_snapshot_id= 2023-11-08T04:59:47.810 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test 25 = '' 2023-11-08T04:59:47.811 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-11-08T04:59:47.811 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0.4 2023-11-08T04:59:48.211 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.status 2023-11-08T04:59:48.211 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T04:59:48.212 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T04:59:48.212 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T04:59:48.212 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.status 2023-11-08T04:59:48.212 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-11-08T04:59:48.212 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T04:59:48.252 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ true 2023-11-08T04:59:48.252 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-08T04:59:48.254 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:-:1.1: Document is empty 2023-11-08T04:59:48.254 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: 2023-11-08T04:59:48.254 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:^ 2023-11-08T04:59:48.255 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local primary_snapshot_id= 2023-11-08T04:59:48.255 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test 25 = '' 2023-11-08T04:59:48.255 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-11-08T04:59:48.256 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0.8 2023-11-08T04:59:49.056 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.status 2023-11-08T04:59:49.056 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T04:59:49.056 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T04:59:49.056 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T04:59:49.057 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.status 2023-11-08T04:59:49.057 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-11-08T04:59:49.057 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T04:59:49.103 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ true 2023-11-08T04:59:49.103 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-08T04:59:49.104 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:-:1.1: Document is empty 2023-11-08T04:59:49.104 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: 2023-11-08T04:59:49.104 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:^ 2023-11-08T04:59:49.105 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local primary_snapshot_id= 2023-11-08T04:59:49.105 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test 25 = '' 2023-11-08T04:59:49.106 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-11-08T04:59:49.106 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 1.6 2023-11-08T04:59:50.707 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.status 2023-11-08T04:59:50.707 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T04:59:50.707 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T04:59:50.707 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T04:59:50.708 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.status 2023-11-08T04:59:50.708 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-11-08T04:59:50.708 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T04:59:50.748 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-08T04:59:50.748 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local primary_snapshot_id=25 2023-11-08T04:59:50.748 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test 25 = 25 2023-11-08T04:59:50.749 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T04:59:50.749 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_snap_present cluster1 mirror test snap6 2023-11-08T04:59:50.749 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T04:59:50.749 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T04:59:50.749 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T04:59:50.750 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap_name=snap6 2023-11-08T04:59:50.750 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local s 2023-11-08T04:59:50.750 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-11-08T04:59:50.750 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 1 2023-11-08T04:59:51.748 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 info mirror/test@snap6 2023-11-08T04:59:51.791 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:rbd image 'test': 2023-11-08T04:59:51.791 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: size 512 MiB in 128 objects 2023-11-08T04:59:51.792 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: order 22 (4 MiB objects) 2023-11-08T04:59:51.792 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: snapshot_count: 12 2023-11-08T04:59:51.792 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: id: 10afad8fe72b 2023-11-08T04:59:51.792 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: block_name_prefix: rbd_data.10afad8fe72b 2023-11-08T04:59:51.792 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: format: 2 2023-11-08T04:59:51.793 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: features: layering, exclusive-lock, non-primary 2023-11-08T04:59:51.793 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: op_features: 2023-11-08T04:59:51.793 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: flags: 2023-11-08T04:59:51.793 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: create_timestamp: Wed Nov 8 04:54:28 2023 2023-11-08T04:59:51.793 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: access_timestamp: Wed Nov 8 04:58:17 2023 2023-11-08T04:59:51.793 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: modify_timestamp: Wed Nov 8 04:54:28 2023 2023-11-08T04:59:51.794 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: protected: False 2023-11-08T04:59:51.794 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring state: enabled 2023-11-08T04:59:51.794 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring mode: snapshot 2023-11-08T04:59:51.794 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring global id: 88ced7f9-2739-4a27-ad7a-30edec05edb1 2023-11-08T04:59:51.795 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring primary: false 2023-11-08T04:59:51.796 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T04:59:51.796 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' -n snap5-clean ']' 2023-11-08T04:59:51.796 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ compare_image_snaps mirror test snap5-clean 2023-11-08T04:59:51.797 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T04:59:51.797 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T04:59:51.797 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap_name=snap5-clean 2023-11-08T04:59:51.797 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local ret=0 2023-11-08T04:59:51.797 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local rmt_export=/tmp/tmp.Ludk8ikCAB/cluster2-mirror-test.export 2023-11-08T04:59:51.797 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local loc_export=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.export 2023-11-08T04:59:51.798 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rm -f /tmp/tmp.Ludk8ikCAB/cluster2-mirror-test.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.export 2023-11-08T04:59:51.798 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 -p mirror export test@snap5-clean /tmp/tmp.Ludk8ikCAB/cluster2-mirror-test.export 2023-11-08T04:59:52.400 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-11-08T04:59:52.403 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 -p mirror export test@snap5-clean /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.export 2023-11-08T04:59:53.186 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-11-08T04:59:53.189 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ cmp /tmp/tmp.Ludk8ikCAB/cluster2-mirror-test.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.export 2023-11-08T04:59:53.401 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rm -f /tmp/tmp.Ludk8ikCAB/cluster2-mirror-test.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.export 2023-11-08T04:59:53.544 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T04:59:53.545 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ compare_image_snaps mirror test snap6 2023-11-08T04:59:53.545 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T04:59:53.545 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T04:59:53.545 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap_name=snap6 2023-11-08T04:59:53.546 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local ret=0 2023-11-08T04:59:53.546 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local rmt_export=/tmp/tmp.Ludk8ikCAB/cluster2-mirror-test.export 2023-11-08T04:59:53.546 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local loc_export=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.export 2023-11-08T04:59:53.546 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rm -f /tmp/tmp.Ludk8ikCAB/cluster2-mirror-test.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.export 2023-11-08T04:59:53.547 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 -p mirror export test@snap6 /tmp/tmp.Ludk8ikCAB/cluster2-mirror-test.export 2023-11-08T04:59:54.148 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-11-08T04:59:54.151 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 -p mirror export test@snap6 /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.export 2023-11-08T04:59:54.888 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-11-08T04:59:54.892 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ cmp /tmp/tmp.Ludk8ikCAB/cluster2-mirror-test.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.export 2023-11-08T04:59:55.100 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rm -f /tmp/tmp.Ludk8ikCAB/cluster2-mirror-test.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.export 2023-11-08T04:59:55.244 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T04:59:55.244 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ clean_snap_name=snap6-clean 2023-11-08T04:59:55.244 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ create_snap cluster2 mirror test snap6-clean 2023-11-08T04:59:55.245 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T04:59:55.245 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T04:59:55.245 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T04:59:55.245 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap_name=snap6-clean 2023-11-08T04:59:55.245 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap6-clean --debug-rbd=20 --debug-journaler=20 2023-11-08T04:59:55.448 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for i in `seq 1 10` 2023-11-08T04:59:55.448 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ stress_write_image cluster2 mirror test 2023-11-08T04:59:55.448 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T04:59:55.448 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T04:59:55.449 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T04:59:55.449 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-11-08T04:59:55.449 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local duration=8 2023-11-08T04:59:55.450 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ set +e 2023-11-08T04:59:55.450 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2023-11-08T04:59:55.682 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:100 IOs, 25625600 bytes 2023-11-08T04:59:55.828 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:200 IOs, 43838464 bytes 2023-11-08T04:59:56.060 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:300 IOs, 116994048 bytes 2023-11-08T04:59:56.178 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:400 IOs, 129613824 bytes 2023-11-08T04:59:56.333 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:500 IOs, 164549632 bytes 2023-11-08T04:59:56.518 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:600 IOs, 212399104 bytes 2023-11-08T04:59:56.663 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:700 IOs, 242976768 bytes 2023-11-08T04:59:56.830 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:800 IOs, 289041408 bytes 2023-11-08T04:59:56.997 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:900 IOs, 334860288 bytes 2023-11-08T04:59:57.169 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1000 IOs, 384252928 bytes 2023-11-08T04:59:57.273 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1100 IOs, 398330880 bytes 2023-11-08T04:59:57.352 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1200 IOs, 405860352 bytes 2023-11-08T04:59:57.462 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1300 IOs, 427520000 bytes 2023-11-08T04:59:57.541 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1400 IOs, 448737280 bytes 2023-11-08T04:59:57.689 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1500 IOs, 463983616 bytes 2023-11-08T04:59:57.859 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1600 IOs, 501343232 bytes 2023-11-08T04:59:58.062 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1700 IOs, 563302400 bytes 2023-11-08T04:59:58.229 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1800 IOs, 601149440 bytes 2023-11-08T04:59:58.537 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1900 IOs, 716596224 bytes 2023-11-08T04:59:58.691 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2000 IOs, 754455552 bytes 2023-11-08T04:59:58.964 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2100 IOs, 846954496 bytes 2023-11-08T04:59:59.043 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2200 IOs, 850913280 bytes 2023-11-08T04:59:59.184 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2300 IOs, 885475328 bytes 2023-11-08T04:59:59.362 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2400 IOs, 926315520 bytes 2023-11-08T04:59:59.480 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2500 IOs, 946854912 bytes 2023-11-08T04:59:59.608 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2600 IOs, 967695360 bytes 2023-11-08T04:59:59.822 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2700 IOs, 1037150208 bytes 2023-11-08T04:59:59.905 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2800 IOs, 1044637696 bytes 2023-11-08T05:00:00.029 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2900 IOs, 1067653120 bytes 2023-11-08T05:00:00.155 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3000 IOs, 1111324672 bytes 2023-11-08T05:00:00.341 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3100 IOs, 1168727040 bytes 2023-11-08T05:00:00.488 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3200 IOs, 1181719552 bytes 2023-11-08T05:00:00.585 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3300 IOs, 1191328768 bytes 2023-11-08T05:00:00.767 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3400 IOs, 1234666496 bytes 2023-11-08T05:00:01.082 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3500 IOs, 1349440512 bytes 2023-11-08T05:00:01.187 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3600 IOs, 1352009728 bytes 2023-11-08T05:00:01.430 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3700 IOs, 1438329856 bytes 2023-11-08T05:00:01.593 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3800 IOs, 1482031104 bytes 2023-11-08T05:00:01.742 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3900 IOs, 1532368896 bytes 2023-11-08T05:00:01.915 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4000 IOs, 1555168256 bytes 2023-11-08T05:00:02.102 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4100 IOs, 1600317440 bytes 2023-11-08T05:00:02.271 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4200 IOs, 1651555328 bytes 2023-11-08T05:00:02.373 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4300 IOs, 1660152832 bytes 2023-11-08T05:00:02.497 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4400 IOs, 1693215744 bytes 2023-11-08T05:00:02.711 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4500 IOs, 1754568704 bytes 2023-11-08T05:00:02.815 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4600 IOs, 1763020800 bytes 2023-11-08T05:00:03.023 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4700 IOs, 1822793728 bytes 2023-11-08T05:00:03.103 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4800 IOs, 1830387712 bytes 2023-11-08T05:00:03.225 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4900 IOs, 1845745664 bytes 2023-11-08T05:00:03.350 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5000 IOs, 1869778944 bytes 2023-11-08T05:00:03.454 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ error_code=124 2023-11-08T05:00:03.454 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ set -e 2023-11-08T05:00:03.454 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' 124 -eq 124 ']' 2023-11-08T05:00:03.454 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:00:03.455 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2023-11-08T05:00:03.455 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:00:03.455 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:00:03.455 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T05:00:03.455 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local state_pattern=up+replaying 2023-11-08T05:00:03.456 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local description_pattern= 2023-11-08T05:00:03.456 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local service_pattern= 2023-11-08T05:00:03.456 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-11-08T05:00:03.456 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 1 2023-11-08T05:00:04.455 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2023-11-08T05:00:04.456 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:00:04.456 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:00:04.456 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T05:00:04.456 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local state_pattern=up+replaying 2023-11-08T05:00:04.456 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local description_pattern= 2023-11-08T05:00:04.457 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local service_pattern= 2023-11-08T05:00:04.457 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ mkfname cluster1-mirror-test.mirror_status 2023-11-08T05:00:04.457 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ echo cluster1-mirror-test.mirror_status 2023-11-08T05:00:04.457 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -e 's|[/ ]|_|g' 2023-11-08T05:00:04.458 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local status_log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.mirror_status 2023-11-08T05:00:04.458 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ CEPH_ARGS= 2023-11-08T05:00:04.458 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 mirror image status mirror/test 2023-11-08T05:00:04.459 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ tee /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.mirror_status 2023-11-08T05:00:04.500 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:test: 2023-11-08T05:00:04.500 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: global_id: 88ced7f9-2739-4a27-ad7a-30edec05edb1 2023-11-08T05:00:04.501 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: state: up+replaying 2023-11-08T05:00:04.502 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: description: replaying, {"bytes_per_second":15767756.8,"bytes_per_snapshot":500993536.0,"last_snapshot_bytes":473032704,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1699419586,"remote_snapshot_timestamp":1699419586,"replay_state":"idle"} 2023-11-08T05:00:04.502 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: service: mirror.3 on smithi087 2023-11-08T05:00:04.502 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: last_update: 2023-11-08 04:59:56 2023-11-08T05:00:04.502 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: peer_sites: 2023-11-08T05:00:04.503 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: name: cluster2 2023-11-08T05:00:04.503 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: state: up+stopped 2023-11-08T05:00:04.503 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: description: local image is primary 2023-11-08T05:00:04.503 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: last_update: 2023-11-08 04:59:59 2023-11-08T05:00:04.504 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ grep '^ state: .*up+replaying' /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.mirror_status 2023-11-08T05:00:04.505 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: state: up+replaying 2023-11-08T05:00:04.506 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ grep '^ description: .*' /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.mirror_status 2023-11-08T05:00:04.506 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: description: replaying, {"bytes_per_second":15767756.8,"bytes_per_snapshot":500993536.0,"last_snapshot_bytes":473032704,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1699419586,"remote_snapshot_timestamp":1699419586,"replay_state":"idle"} 2023-11-08T05:00:04.507 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' -n '' ']' 2023-11-08T05:00:04.507 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ echo up+replaying 2023-11-08T05:00:04.507 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ grep '^up+' 2023-11-08T05:00:04.508 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:up+replaying 2023-11-08T05:00:04.508 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ grep 'service: *mirror..* on ' /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.mirror_status 2023-11-08T05:00:04.509 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: service: mirror.3 on smithi087 2023-11-08T05:00:04.509 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.mirror_status 2023-11-08T05:00:04.511 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local 'last_update=2023-11-08 04:59:56' 2023-11-08T05:00:04.511 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2023-11-08 04:59:56' 2023-11-08T05:00:04.511 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:00:04.511 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:00:04.512 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T05:00:04.512 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local state_pattern=up+replaying 2023-11-08T05:00:04.512 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local 'prev_last_update=2023-11-08 04:59:56' 2023-11-08T05:00:04.512 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ mkfname cluster1-mirror.mirror_status 2023-11-08T05:00:04.512 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ echo cluster1-mirror.mirror_status 2023-11-08T05:00:04.513 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -e 's|[/ ]|_|g' 2023-11-08T05:00:04.513 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local status_log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror.mirror_status 2023-11-08T05:00:04.513 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2023-11-08T05:00:04.544 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd: failed to query services: (13) Permission denied 2023-11-08T05:00:04.564 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local last_update state 2023-11-08T05:00:04.564 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2023-11-08T05:00:04.566 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ last_update='2023-11-08 04:59:56' 2023-11-08T05:00:04.567 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2023-11-08T05:00:04.568 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ state=up+replaying 2023-11-08T05:00:04.568 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ echo up+replaying 2023-11-08T05:00:04.569 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ grep up+replaying 2023-11-08T05:00:04.569 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:up+replaying 2023-11-08T05:00:04.570 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:00:04.570 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:00:04.571 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ snap_name=snap7 2023-11-08T05:00:04.571 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ create_snap cluster2 mirror test snap7 2023-11-08T05:00:04.571 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:00:04.571 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:00:04.571 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T05:00:04.572 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap_name=snap7 2023-11-08T05:00:04.572 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap7 --debug-rbd=20 --debug-journaler=20 2023-11-08T05:00:34.510 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_image_replay_started cluster1 mirror test 2023-11-08T05:00:34.511 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:00:34.511 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:00:34.511 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T05:00:34.511 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2023-11-08T05:00:34.511 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:00:34.512 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:00:34.512 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T05:00:34.512 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local state=started 2023-11-08T05:00:34.512 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local s 2023-11-08T05:00:34.512 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-11-08T05:00:34.512 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 1 2023-11-08T05:00:35.511 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test_image_replay_state cluster1 mirror test started 2023-11-08T05:00:35.511 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:00:35.511 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:00:35.512 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T05:00:35.512 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local test_state=started 2023-11-08T05:00:35.512 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local status_result 2023-11-08T05:00:35.512 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local current_state=stopped 2023-11-08T05:00:35.513 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2023-11-08T05:00:35.514 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_instance=cluster1 2023-11-08T05:00:35.514 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T05:00:35.514 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ grep -i state 2023-11-08T05:00:35.514 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1 2023-11-08T05:00:35.514 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance=cluster1 2023-11-08T05:00:35.515 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local loop_instance 2023-11-08T05:00:35.515 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-11-08T05:00:35.515 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sleep 0 2023-11-08T05:00:35.515 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-11-08T05:00:35.515 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ seq 0 3 2023-11-08T05:00:35.517 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T05:00:35.518 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2023-11-08T05:00:35.518 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1:0 2023-11-08T05:00:35.518 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T05:00:35.518 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance 2023-11-08T05:00:35.518 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-11-08T05:00:35.519 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local val=cluster1:0 2023-11-08T05:00:35.519 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_var_name=cluster 2023-11-08T05:00:35.519 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance_var_name=instance 2023-11-08T05:00:35.519 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cluster=cluster1 2023-11-08T05:00:35.519 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ instance=0 2023-11-08T05:00:35.520 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' 0 = cluster1:0 ']' 2023-11-08T05:00:35.520 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval cluster=cluster1 2023-11-08T05:00:35.520 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:00:35.520 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval instance=0 2023-11-08T05:00:35.520 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=0 2023-11-08T05:00:35.521 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-11-08T05:00:35.522 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local local_cluster=cluster1:0 2023-11-08T05:00:35.522 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster=cluster1 2023-11-08T05:00:35.523 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance 2023-11-08T05:00:35.523 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-11-08T05:00:35.523 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local val=cluster1:0 2023-11-08T05:00:35.523 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster_var_name=local_cluster 2023-11-08T05:00:35.523 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance_var_name=instance 2023-11-08T05:00:35.523 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:00:35.524 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=0 2023-11-08T05:00:35.524 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ '[' 0 = cluster1:0 ']' 2023-11-08T05:00:35.524 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval local_cluster=cluster1 2023-11-08T05:00:35.524 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ local_cluster=cluster1 2023-11-08T05:00:35.524 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval instance=0 2023-11-08T05:00:35.525 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ instance=0 2023-11-08T05:00:35.525 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-11-08T05:00:35.533 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-08T05:00:35.533 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-08T05:00:35.533 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-08T05:00:35.533 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2023-11-08T05:00:35.629 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-11-08T05:00:35.629 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd mirror status : get status for rbd mirror 2023-11-08T05:00:35.629 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:admin_socket: invalid command 2023-11-08T05:00:35.632 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T05:00:35.632 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2023-11-08T05:00:35.632 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1:1 2023-11-08T05:00:35.633 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T05:00:35.633 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance 2023-11-08T05:00:35.633 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-11-08T05:00:35.633 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local val=cluster1:1 2023-11-08T05:00:35.634 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_var_name=cluster 2023-11-08T05:00:35.634 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance_var_name=instance 2023-11-08T05:00:35.634 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cluster=cluster1 2023-11-08T05:00:35.634 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ instance=1 2023-11-08T05:00:35.635 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' 1 = cluster1:1 ']' 2023-11-08T05:00:35.635 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval cluster=cluster1 2023-11-08T05:00:35.635 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:00:35.636 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval instance=1 2023-11-08T05:00:35.636 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=1 2023-11-08T05:00:35.637 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-11-08T05:00:35.637 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local local_cluster=cluster1:1 2023-11-08T05:00:35.637 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster=cluster1 2023-11-08T05:00:35.637 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance 2023-11-08T05:00:35.638 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-11-08T05:00:35.638 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local val=cluster1:1 2023-11-08T05:00:35.638 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster_var_name=local_cluster 2023-11-08T05:00:35.639 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance_var_name=instance 2023-11-08T05:00:35.639 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:00:35.639 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=1 2023-11-08T05:00:35.640 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ '[' 1 = cluster1:1 ']' 2023-11-08T05:00:35.640 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval local_cluster=cluster1 2023-11-08T05:00:35.640 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ local_cluster=cluster1 2023-11-08T05:00:35.640 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval instance=1 2023-11-08T05:00:35.640 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ instance=1 2023-11-08T05:00:35.641 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-11-08T05:00:35.646 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-08T05:00:35.647 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-08T05:00:35.647 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-08T05:00:35.647 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2023-11-08T05:00:35.734 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-11-08T05:00:35.735 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd mirror status : get status for rbd mirror 2023-11-08T05:00:35.735 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:admin_socket: invalid command 2023-11-08T05:00:35.738 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T05:00:35.738 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2023-11-08T05:00:35.738 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1:2 2023-11-08T05:00:35.738 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T05:00:35.739 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance 2023-11-08T05:00:35.739 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-11-08T05:00:35.739 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local val=cluster1:2 2023-11-08T05:00:35.739 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_var_name=cluster 2023-11-08T05:00:35.739 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance_var_name=instance 2023-11-08T05:00:35.739 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cluster=cluster1 2023-11-08T05:00:35.740 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ instance=2 2023-11-08T05:00:35.740 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' 2 = cluster1:2 ']' 2023-11-08T05:00:35.740 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval cluster=cluster1 2023-11-08T05:00:35.740 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:00:35.741 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval instance=2 2023-11-08T05:00:35.741 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=2 2023-11-08T05:00:35.741 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-11-08T05:00:35.741 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local local_cluster=cluster1:2 2023-11-08T05:00:35.741 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster=cluster1 2023-11-08T05:00:35.741 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance 2023-11-08T05:00:35.742 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-11-08T05:00:35.742 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local val=cluster1:2 2023-11-08T05:00:35.742 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster_var_name=local_cluster 2023-11-08T05:00:35.742 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance_var_name=instance 2023-11-08T05:00:35.742 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:00:35.743 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=2 2023-11-08T05:00:35.743 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ '[' 2 = cluster1:2 ']' 2023-11-08T05:00:35.743 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval local_cluster=cluster1 2023-11-08T05:00:35.743 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ local_cluster=cluster1 2023-11-08T05:00:35.743 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval instance=2 2023-11-08T05:00:35.744 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ instance=2 2023-11-08T05:00:35.744 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-11-08T05:00:35.752 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-08T05:00:35.752 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-08T05:00:35.752 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-08T05:00:35.753 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2023-11-08T05:00:35.840 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-11-08T05:00:35.857 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd mirror status : get status for rbd mirror 2023-11-08T05:00:35.857 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:admin_socket: invalid command 2023-11-08T05:00:35.863 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T05:00:35.863 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2023-11-08T05:00:35.864 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1:3 2023-11-08T05:00:35.864 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T05:00:35.864 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance 2023-11-08T05:00:35.865 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ set_cluster_instance cluster1:3 cluster instance 2023-11-08T05:00:35.865 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local val=cluster1:3 2023-11-08T05:00:35.866 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_var_name=cluster 2023-11-08T05:00:35.866 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance_var_name=instance 2023-11-08T05:00:35.866 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cluster=cluster1 2023-11-08T05:00:35.867 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ instance=3 2023-11-08T05:00:35.867 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' 3 = cluster1:3 ']' 2023-11-08T05:00:35.867 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval cluster=cluster1 2023-11-08T05:00:35.868 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:00:35.868 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval instance=3 2023-11-08T05:00:35.869 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=3 2023-11-08T05:00:35.869 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ daemon_asok_file cluster1:3 cluster1 2023-11-08T05:00:35.869 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local local_cluster=cluster1:3 2023-11-08T05:00:35.870 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster=cluster1 2023-11-08T05:00:35.870 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance 2023-11-08T05:00:35.870 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2023-11-08T05:00:35.871 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local val=cluster1:3 2023-11-08T05:00:35.871 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster_var_name=local_cluster 2023-11-08T05:00:35.872 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance_var_name=instance 2023-11-08T05:00:35.872 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:00:35.872 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=3 2023-11-08T05:00:35.873 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ '[' 3 = cluster1:3 ']' 2023-11-08T05:00:35.873 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval local_cluster=cluster1 2023-11-08T05:00:35.873 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ local_cluster=cluster1 2023-11-08T05:00:35.874 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval instance=3 2023-11-08T05:00:35.874 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ instance=3 2023-11-08T05:00:35.874 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2023-11-08T05:00:35.875 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-11-08T05:00:35.875 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-11-08T05:00:35.876 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-11-08T05:00:35.876 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2023-11-08T05:00:35.955 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ return 0 2023-11-08T05:00:35.956 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ status_result=' "state": "Replaying"' 2023-11-08T05:00:35.956 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ echo ' "state": "Replaying"' 2023-11-08T05:00:35.956 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ grep -i Replaying 2023-11-08T05:00:35.957 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: "state": "Replaying" 2023-11-08T05:00:35.959 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ current_state=started 2023-11-08T05:00:35.959 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test started = started 2023-11-08T05:00:35.959 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:00:35.959 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2023-11-08T05:00:35.960 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local local_cluster=cluster1 2023-11-08T05:00:35.960 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:00:35.960 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:00:35.960 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T05:00:35.960 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' snapshot = journal ']' 2023-11-08T05:00:35.961 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' snapshot = snapshot ']' 2023-11-08T05:00:35.961 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2023-11-08T05:00:35.961 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local local_cluster=cluster1 2023-11-08T05:00:35.961 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:00:35.961 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:00:35.962 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T05:00:35.962 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ mkfname cluster2-mirror-test.status 2023-11-08T05:00:35.962 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ echo cluster2-mirror-test.status 2023-11-08T05:00:35.962 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -e 's|[/ ]|_|g' 2023-11-08T05:00:35.962 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local status_log=/tmp/tmp.Ludk8ikCAB/cluster2-mirror-test.status 2023-11-08T05:00:35.963 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ mkfname cluster1-mirror-test.status 2023-11-08T05:00:35.963 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ echo cluster1-mirror-test.status 2023-11-08T05:00:35.963 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -e 's|[/ ]|_|g' 2023-11-08T05:00:35.964 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local local_status_log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.status 2023-11-08T05:00:35.964 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ mirror_image_snapshot cluster2 mirror test 2023-11-08T05:00:35.964 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:00:35.964 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:00:35.964 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T05:00:35.965 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2023-11-08T05:00:37.509 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:Snapshot ID: 29 2023-11-08T05:00:37.531 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.Ludk8ikCAB/cluster2-mirror-test.status 2023-11-08T05:00:37.531 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:00:37.532 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:00:37.532 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T05:00:37.532 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster2-mirror-test.status 2023-11-08T05:00:37.532 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2023-11-08T05:00:37.532 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T05:00:37.584 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-11-08T05:00:37.587 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snapshot_id=29 2023-11-08T05:00:37.587 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ true 2023-11-08T05:00:37.587 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-11-08T05:00:37.588 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0.2 2023-11-08T05:00:37.788 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.status 2023-11-08T05:00:37.789 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:00:37.789 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:00:37.789 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T05:00:37.789 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.status 2023-11-08T05:00:37.790 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-11-08T05:00:37.790 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T05:00:37.830 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ true 2023-11-08T05:00:37.831 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-08T05:00:37.832 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:-:1.1: Document is empty 2023-11-08T05:00:37.832 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: 2023-11-08T05:00:37.832 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:^ 2023-11-08T05:00:37.832 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local primary_snapshot_id= 2023-11-08T05:00:37.833 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test 29 = '' 2023-11-08T05:00:37.833 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-11-08T05:00:37.833 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0.4 2023-11-08T05:00:38.233 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.status 2023-11-08T05:00:38.234 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:00:38.234 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:00:38.234 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T05:00:38.234 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.status 2023-11-08T05:00:38.235 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-11-08T05:00:38.235 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T05:00:38.276 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ true 2023-11-08T05:00:38.277 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-08T05:00:38.278 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:-:1.1: Document is empty 2023-11-08T05:00:38.278 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: 2023-11-08T05:00:38.278 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:^ 2023-11-08T05:00:38.279 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local primary_snapshot_id= 2023-11-08T05:00:38.279 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test 29 = '' 2023-11-08T05:00:38.279 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-11-08T05:00:38.279 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0.8 2023-11-08T05:00:39.079 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.status 2023-11-08T05:00:39.080 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:00:39.080 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:00:39.080 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T05:00:39.080 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.status 2023-11-08T05:00:39.081 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-11-08T05:00:39.081 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T05:00:39.124 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ true 2023-11-08T05:00:39.125 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-08T05:00:39.126 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:-:1.1: Document is empty 2023-11-08T05:00:39.127 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: 2023-11-08T05:00:39.127 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:^ 2023-11-08T05:00:39.127 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local primary_snapshot_id= 2023-11-08T05:00:39.128 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test 29 = '' 2023-11-08T05:00:39.128 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-11-08T05:00:39.128 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 1.6 2023-11-08T05:00:40.729 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.status 2023-11-08T05:00:40.729 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:00:40.729 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:00:40.729 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T05:00:40.730 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.status 2023-11-08T05:00:40.730 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-11-08T05:00:40.730 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T05:00:40.970 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-08T05:00:40.972 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local primary_snapshot_id=29 2023-11-08T05:00:40.972 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test 29 = 29 2023-11-08T05:00:40.973 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:00:40.973 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_snap_present cluster1 mirror test snap7 2023-11-08T05:00:40.973 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:00:40.973 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:00:40.973 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T05:00:40.974 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap_name=snap7 2023-11-08T05:00:40.974 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local s 2023-11-08T05:00:40.974 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-11-08T05:00:40.974 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 1 2023-11-08T05:00:41.034 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.5352050098534546) > weight (0.25) 2023-11-08T05:00:41.035 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.6516822327934398) > weight (0.25) 2023-11-08T05:00:41.035 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.6856954382011056) > weight (0.25) 2023-11-08T05:00:41.035 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.3 2023-11-08T05:00:41.036 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Sent signal 15 2023-11-08T05:00:41.036 INFO:tasks.rbd_mirror_thrash:waiting for 4 secs before reviving daemons 2023-11-08T05:00:41.037 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi087.stderr:2023-11-08T05:00:41.035+0000 7f5542608700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3 (PID: 124458) UID: 1000 2023-11-08T05:00:41.237 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi087.stderr:daemon-helper: command failed with exit status 1 2023-11-08T05:00:41.973 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 info mirror/test@snap7 2023-11-08T05:00:42.010 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:rbd image 'test': 2023-11-08T05:00:42.010 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: size 512 MiB in 128 objects 2023-11-08T05:00:42.011 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: order 22 (4 MiB objects) 2023-11-08T05:00:42.011 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: snapshot_count: 14 2023-11-08T05:00:42.011 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: id: 10afad8fe72b 2023-11-08T05:00:42.011 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: block_name_prefix: rbd_data.10afad8fe72b 2023-11-08T05:00:42.011 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: format: 2 2023-11-08T05:00:42.011 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: features: layering, exclusive-lock, non-primary 2023-11-08T05:00:42.012 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: op_features: 2023-11-08T05:00:42.012 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: flags: 2023-11-08T05:00:42.012 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: create_timestamp: Wed Nov 8 04:54:28 2023 2023-11-08T05:00:42.012 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: access_timestamp: Wed Nov 8 04:59:52 2023 2023-11-08T05:00:42.012 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: modify_timestamp: Wed Nov 8 04:54:28 2023 2023-11-08T05:00:42.012 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: protected: False 2023-11-08T05:00:42.013 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring state: enabled 2023-11-08T05:00:42.013 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring mode: snapshot 2023-11-08T05:00:42.013 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring global id: 88ced7f9-2739-4a27-ad7a-30edec05edb1 2023-11-08T05:00:42.013 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring primary: false 2023-11-08T05:00:42.015 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:00:42.015 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' -n snap6-clean ']' 2023-11-08T05:00:42.015 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ compare_image_snaps mirror test snap6-clean 2023-11-08T05:00:42.015 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:00:42.015 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T05:00:42.016 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap_name=snap6-clean 2023-11-08T05:00:42.016 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local ret=0 2023-11-08T05:00:42.016 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local rmt_export=/tmp/tmp.Ludk8ikCAB/cluster2-mirror-test.export 2023-11-08T05:00:42.016 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local loc_export=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.export 2023-11-08T05:00:42.016 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rm -f /tmp/tmp.Ludk8ikCAB/cluster2-mirror-test.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.export 2023-11-08T05:00:42.016 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 -p mirror export test@snap6-clean /tmp/tmp.Ludk8ikCAB/cluster2-mirror-test.export 2023-11-08T05:00:42.036 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.7549071612724695) > weight (0.25) 2023-11-08T05:00:42.036 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.9605674480286801) > weight (0.25) 2023-11-08T05:00:42.036 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.4929078863617954) > weight (0.25) 2023-11-08T05:00:42.036 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.3 2023-11-08T05:00:42.037 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Sent signal 15 2023-11-08T05:00:42.037 INFO:tasks.rbd_mirror_thrash:waiting for 4 secs before reviving daemons 2023-11-08T05:00:42.038 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi087.stderr:2023-11-08T05:00:42.036+0000 7f89cdf9b700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.3 (PID: 120079) UID: 1000 2023-11-08T05:00:42.238 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi087.stderr:daemon-helper: command failed with exit status 1 2023-11-08T05:00:42.616 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-11-08T05:00:42.622 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 -p mirror export test@snap6-clean /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.export 2023-11-08T05:00:43.398 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-11-08T05:00:43.401 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ cmp /tmp/tmp.Ludk8ikCAB/cluster2-mirror-test.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.export 2023-11-08T05:00:43.614 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rm -f /tmp/tmp.Ludk8ikCAB/cluster2-mirror-test.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.export 2023-11-08T05:00:43.757 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:00:43.757 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ compare_image_snaps mirror test snap7 2023-11-08T05:00:43.757 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:00:43.758 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T05:00:43.758 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap_name=snap7 2023-11-08T05:00:43.758 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local ret=0 2023-11-08T05:00:43.758 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local rmt_export=/tmp/tmp.Ludk8ikCAB/cluster2-mirror-test.export 2023-11-08T05:00:43.758 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local loc_export=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.export 2023-11-08T05:00:43.759 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rm -f /tmp/tmp.Ludk8ikCAB/cluster2-mirror-test.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.export 2023-11-08T05:00:43.759 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 -p mirror export test@snap7 /tmp/tmp.Ludk8ikCAB/cluster2-mirror-test.export 2023-11-08T05:00:44.368 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-11-08T05:00:44.371 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 -p mirror export test@snap7 /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.export 2023-11-08T05:00:45.037 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.3 2023-11-08T05:00:45.037 INFO:teuthology.orchestra.run:waiting for 600 2023-11-08T05:00:45.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-08T05:00:45.038 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.3 2023-11-08T05:00:45.038 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Restarting daemon 2023-11-08T05:00:45.039 DEBUG:teuthology.orchestra.run.smithi087:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3 2023-11-08T05:00:45.041 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Started 2023-11-08T05:00:45.042 INFO:tasks.rbd_mirror_thrash:waiting for 119 secs before thrashing 2023-11-08T05:00:45.099 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-11-08T05:00:45.102 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ cmp /tmp/tmp.Ludk8ikCAB/cluster2-mirror-test.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.export 2023-11-08T05:00:45.330 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rm -f /tmp/tmp.Ludk8ikCAB/cluster2-mirror-test.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.export 2023-11-08T05:00:45.473 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:00:45.473 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ clean_snap_name=snap7-clean 2023-11-08T05:00:45.473 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ create_snap cluster2 mirror test snap7-clean 2023-11-08T05:00:45.474 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:00:45.474 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:00:45.474 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T05:00:45.474 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap_name=snap7-clean 2023-11-08T05:00:45.474 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap7-clean --debug-rbd=20 --debug-journaler=20 2023-11-08T05:00:46.038 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.3 2023-11-08T05:00:46.039 INFO:teuthology.orchestra.run:waiting for 600 2023-11-08T05:00:46.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-08T05:00:46.039 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.3 2023-11-08T05:00:46.039 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Restarting daemon 2023-11-08T05:00:46.040 DEBUG:teuthology.orchestra.run.smithi087:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.3 2023-11-08T05:00:46.042 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Started 2023-11-08T05:00:46.043 INFO:tasks.rbd_mirror_thrash:waiting for 114 secs before thrashing 2023-11-08T05:00:46.492 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for i in `seq 1 10` 2023-11-08T05:00:46.492 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ stress_write_image cluster2 mirror test 2023-11-08T05:00:46.492 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:00:46.492 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:00:46.493 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T05:00:46.493 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-11-08T05:00:46.494 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local duration=6 2023-11-08T05:00:46.494 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ set +e 2023-11-08T05:00:46.494 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2023-11-08T05:00:46.653 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:100 IOs, 635904 bytes 2023-11-08T05:00:46.831 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:200 IOs, 43952128 bytes 2023-11-08T05:00:46.929 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:300 IOs, 53185536 bytes 2023-11-08T05:00:47.118 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:400 IOs, 113793024 bytes 2023-11-08T05:00:47.279 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:500 IOs, 149444608 bytes 2023-11-08T05:00:47.363 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:600 IOs, 168042496 bytes 2023-11-08T05:00:47.558 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:700 IOs, 204176384 bytes 2023-11-08T05:00:47.713 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:800 IOs, 249402368 bytes 2023-11-08T05:00:47.884 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:900 IOs, 297703424 bytes 2023-11-08T05:00:48.100 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1000 IOs, 361406464 bytes 2023-11-08T05:00:48.344 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1100 IOs, 443798528 bytes 2023-11-08T05:00:48.514 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1200 IOs, 485652480 bytes 2023-11-08T05:00:48.626 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1300 IOs, 500155392 bytes 2023-11-08T05:00:48.840 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1400 IOs, 570769408 bytes 2023-11-08T05:00:49.046 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1500 IOs, 634902528 bytes 2023-11-08T05:00:49.148 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1600 IOs, 646132736 bytes 2023-11-08T05:00:49.277 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1700 IOs, 668999680 bytes 2023-11-08T05:00:49.543 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1800 IOs, 750836736 bytes 2023-11-08T05:00:49.753 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1900 IOs, 830171136 bytes 2023-11-08T05:00:49.877 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2000 IOs, 831323136 bytes 2023-11-08T05:00:49.957 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2100 IOs, 840382464 bytes 2023-11-08T05:00:50.059 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2200 IOs, 855585792 bytes 2023-11-08T05:00:50.253 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2300 IOs, 899453952 bytes 2023-11-08T05:00:50.342 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2400 IOs, 907561984 bytes 2023-11-08T05:00:50.516 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2500 IOs, 966217728 bytes 2023-11-08T05:00:50.698 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2600 IOs, 997124096 bytes 2023-11-08T05:00:50.913 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2700 IOs, 1082719232 bytes 2023-11-08T05:00:51.119 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2800 IOs, 1141069824 bytes 2023-11-08T05:00:51.298 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2900 IOs, 1180745728 bytes 2023-11-08T05:00:51.414 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3000 IOs, 1199170560 bytes 2023-11-08T05:00:51.536 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3100 IOs, 1216544768 bytes 2023-11-08T05:00:51.724 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3200 IOs, 1268003840 bytes 2023-11-08T05:00:51.825 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3300 IOs, 1283487744 bytes 2023-11-08T05:00:51.908 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3400 IOs, 1298185216 bytes 2023-11-08T05:00:52.093 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3500 IOs, 1338451968 bytes 2023-11-08T05:00:52.265 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3600 IOs, 1381035008 bytes 2023-11-08T05:00:52.473 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3700 IOs, 1446641664 bytes 2023-11-08T05:00:52.498 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ error_code=124 2023-11-08T05:00:52.498 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ set -e 2023-11-08T05:00:52.499 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' 124 -eq 124 ']' 2023-11-08T05:00:52.499 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:00:52.499 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2023-11-08T05:00:52.499 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:00:52.499 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:00:52.500 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T05:00:52.500 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local state_pattern=up+replaying 2023-11-08T05:00:52.500 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local description_pattern= 2023-11-08T05:00:52.500 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local service_pattern= 2023-11-08T05:00:52.500 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-11-08T05:00:52.501 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 1 2023-11-08T05:00:53.502 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2023-11-08T05:00:53.502 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:00:53.503 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:00:53.503 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T05:00:53.503 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local state_pattern=up+replaying 2023-11-08T05:00:53.503 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local description_pattern= 2023-11-08T05:00:53.503 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local service_pattern= 2023-11-08T05:00:53.504 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ mkfname cluster1-mirror-test.mirror_status 2023-11-08T05:00:53.504 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ echo cluster1-mirror-test.mirror_status 2023-11-08T05:00:53.504 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -e 's|[/ ]|_|g' 2023-11-08T05:00:53.504 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local status_log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.mirror_status 2023-11-08T05:00:53.504 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ CEPH_ARGS= 2023-11-08T05:00:53.504 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 mirror image status mirror/test 2023-11-08T05:00:53.505 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ tee /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.mirror_status 2023-11-08T05:00:53.544 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:test: 2023-11-08T05:00:53.545 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: global_id: 88ced7f9-2739-4a27-ad7a-30edec05edb1 2023-11-08T05:00:53.545 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: state: down+unknown 2023-11-08T05:00:53.545 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: description: status not found 2023-11-08T05:00:53.545 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: last_update: 2023-11-08T05:00:53.545 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: peer_sites: 2023-11-08T05:00:53.546 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: name: cluster2 2023-11-08T05:00:53.546 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: state: up+stopped 2023-11-08T05:00:53.546 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: description: local image is primary 2023-11-08T05:00:53.546 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: last_update: 2023-11-08 05:00:26 2023-11-08T05:00:53.549 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ grep '^ state: .*up+replaying' /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.mirror_status 2023-11-08T05:00:53.550 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 1 2023-11-08T05:00:53.551 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-11-08T05:00:53.551 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 2 2023-11-08T05:00:55.552 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2023-11-08T05:00:55.553 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:00:55.553 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:00:55.553 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T05:00:55.553 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local state_pattern=up+replaying 2023-11-08T05:00:55.554 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local description_pattern= 2023-11-08T05:00:55.554 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local service_pattern= 2023-11-08T05:00:55.554 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ mkfname cluster1-mirror-test.mirror_status 2023-11-08T05:00:55.555 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ echo cluster1-mirror-test.mirror_status 2023-11-08T05:00:55.555 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -e 's|[/ ]|_|g' 2023-11-08T05:00:55.556 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local status_log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.mirror_status 2023-11-08T05:00:55.556 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ CEPH_ARGS= 2023-11-08T05:00:55.556 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 mirror image status mirror/test 2023-11-08T05:00:55.556 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ tee /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.mirror_status 2023-11-08T05:00:55.597 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:test: 2023-11-08T05:00:55.597 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: global_id: 88ced7f9-2739-4a27-ad7a-30edec05edb1 2023-11-08T05:00:55.597 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: state: down+unknown 2023-11-08T05:00:55.598 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: description: status not found 2023-11-08T05:00:55.598 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: last_update: 2023-11-08T05:00:55.598 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: peer_sites: 2023-11-08T05:00:55.598 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: name: cluster2 2023-11-08T05:00:55.599 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: state: up+stopped 2023-11-08T05:00:55.599 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: description: local image is primary 2023-11-08T05:00:55.599 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: last_update: 2023-11-08 05:00:26 2023-11-08T05:00:55.601 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ grep '^ state: .*up+replaying' /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.mirror_status 2023-11-08T05:00:55.603 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 1 2023-11-08T05:00:55.603 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-11-08T05:00:55.603 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 4 2023-11-08T05:00:59.604 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2023-11-08T05:00:59.605 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:00:59.605 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:00:59.606 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T05:00:59.606 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local state_pattern=up+replaying 2023-11-08T05:00:59.606 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local description_pattern= 2023-11-08T05:00:59.606 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local service_pattern= 2023-11-08T05:00:59.606 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ mkfname cluster1-mirror-test.mirror_status 2023-11-08T05:00:59.607 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ echo cluster1-mirror-test.mirror_status 2023-11-08T05:00:59.607 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -e 's|[/ ]|_|g' 2023-11-08T05:00:59.608 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local status_log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.mirror_status 2023-11-08T05:00:59.608 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ CEPH_ARGS= 2023-11-08T05:00:59.608 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 mirror image status mirror/test 2023-11-08T05:00:59.608 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ tee /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.mirror_status 2023-11-08T05:00:59.649 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:test: 2023-11-08T05:00:59.649 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: global_id: 88ced7f9-2739-4a27-ad7a-30edec05edb1 2023-11-08T05:00:59.649 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: state: down+unknown 2023-11-08T05:00:59.649 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: description: status not found 2023-11-08T05:00:59.650 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: last_update: 2023-11-08T05:00:59.650 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: peer_sites: 2023-11-08T05:00:59.650 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: name: cluster2 2023-11-08T05:00:59.650 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: state: up+stopped 2023-11-08T05:00:59.651 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: description: local image is primary 2023-11-08T05:00:59.651 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: last_update: 2023-11-08 05:00:56 2023-11-08T05:00:59.693 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ grep '^ state: .*up+replaying' /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.mirror_status 2023-11-08T05:00:59.694 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 1 2023-11-08T05:00:59.695 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-11-08T05:00:59.695 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 8 2023-11-08T05:01:07.696 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2023-11-08T05:01:07.697 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:01:07.697 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:01:07.697 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T05:01:07.697 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local state_pattern=up+replaying 2023-11-08T05:01:07.698 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local description_pattern= 2023-11-08T05:01:07.698 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local service_pattern= 2023-11-08T05:01:07.698 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ mkfname cluster1-mirror-test.mirror_status 2023-11-08T05:01:07.698 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ echo cluster1-mirror-test.mirror_status 2023-11-08T05:01:07.699 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -e 's|[/ ]|_|g' 2023-11-08T05:01:07.699 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local status_log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.mirror_status 2023-11-08T05:01:07.699 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ CEPH_ARGS= 2023-11-08T05:01:07.700 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 mirror image status mirror/test 2023-11-08T05:01:07.700 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ tee /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.mirror_status 2023-11-08T05:01:07.740 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:test: 2023-11-08T05:01:07.741 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: global_id: 88ced7f9-2739-4a27-ad7a-30edec05edb1 2023-11-08T05:01:07.741 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: state: down+unknown 2023-11-08T05:01:07.741 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: description: status not found 2023-11-08T05:01:07.741 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: last_update: 2023-11-08T05:01:07.741 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: peer_sites: 2023-11-08T05:01:07.742 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: name: cluster2 2023-11-08T05:01:07.742 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: state: up+stopped 2023-11-08T05:01:07.742 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: description: local image is primary 2023-11-08T05:01:07.742 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: last_update: 2023-11-08 05:00:56 2023-11-08T05:01:07.745 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ grep '^ state: .*up+replaying' /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.mirror_status 2023-11-08T05:01:07.746 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 1 2023-11-08T05:01:07.746 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-11-08T05:01:07.747 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 8 2023-11-08T05:01:15.748 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2023-11-08T05:01:15.748 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:01:15.748 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:01:15.749 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T05:01:15.749 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local state_pattern=up+replaying 2023-11-08T05:01:15.749 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local description_pattern= 2023-11-08T05:01:15.749 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local service_pattern= 2023-11-08T05:01:15.749 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ mkfname cluster1-mirror-test.mirror_status 2023-11-08T05:01:15.750 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ echo cluster1-mirror-test.mirror_status 2023-11-08T05:01:15.750 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -e 's|[/ ]|_|g' 2023-11-08T05:01:15.750 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local status_log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.mirror_status 2023-11-08T05:01:15.751 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ CEPH_ARGS= 2023-11-08T05:01:15.751 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 mirror image status mirror/test 2023-11-08T05:01:15.751 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ tee /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.mirror_status 2023-11-08T05:01:15.798 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:test: 2023-11-08T05:01:15.799 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: global_id: 88ced7f9-2739-4a27-ad7a-30edec05edb1 2023-11-08T05:01:15.799 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: state: up+replaying 2023-11-08T05:01:15.799 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1699419636,"remote_snapshot_timestamp":1699419636,"replay_state":"idle"} 2023-11-08T05:01:15.799 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: service: mirror.2 on smithi087 2023-11-08T05:01:15.799 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: last_update: 2023-11-08 05:01:13 2023-11-08T05:01:15.800 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: peer_sites: 2023-11-08T05:01:15.800 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: name: cluster2 2023-11-08T05:01:15.800 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: state: up+stopped 2023-11-08T05:01:15.800 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: description: local image is primary 2023-11-08T05:01:15.800 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: last_update: 2023-11-08 05:00:56 2023-11-08T05:01:15.803 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ grep '^ state: .*up+replaying' /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.mirror_status 2023-11-08T05:01:15.804 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: state: up+replaying 2023-11-08T05:01:15.804 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ grep '^ description: .*' /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.mirror_status 2023-11-08T05:01:15.805 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1699419636,"remote_snapshot_timestamp":1699419636,"replay_state":"idle"} 2023-11-08T05:01:15.805 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' -n '' ']' 2023-11-08T05:01:15.805 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ echo up+replaying 2023-11-08T05:01:15.806 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ grep '^up+' 2023-11-08T05:01:15.806 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:up+replaying 2023-11-08T05:01:15.807 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ grep 'service: *mirror..* on ' /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.mirror_status 2023-11-08T05:01:15.809 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: service: mirror.2 on smithi087 2023-11-08T05:01:15.810 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.mirror_status 2023-11-08T05:01:15.811 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local 'last_update=2023-11-08 05:01:13' 2023-11-08T05:01:15.811 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2023-11-08 05:01:13' 2023-11-08T05:01:15.811 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:01:15.811 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:01:15.811 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T05:01:15.812 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local state_pattern=up+replaying 2023-11-08T05:01:15.812 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local 'prev_last_update=2023-11-08 05:01:13' 2023-11-08T05:01:15.812 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ mkfname cluster1-mirror.mirror_status 2023-11-08T05:01:15.812 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ echo cluster1-mirror.mirror_status 2023-11-08T05:01:15.812 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -e 's|[/ ]|_|g' 2023-11-08T05:01:15.813 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local status_log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror.mirror_status 2023-11-08T05:01:15.813 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2023-11-08T05:01:15.842 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd: failed to query services: (13) Permission denied 2023-11-08T05:01:15.864 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local last_update state 2023-11-08T05:01:15.864 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2023-11-08T05:01:15.865 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ last_update='2023-11-08 05:01:13' 2023-11-08T05:01:15.866 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2023-11-08T05:01:15.868 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ state=up+replaying 2023-11-08T05:01:15.868 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ echo up+replaying 2023-11-08T05:01:15.868 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ grep up+replaying 2023-11-08T05:01:15.869 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:up+replaying 2023-11-08T05:01:15.870 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:01:15.870 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:01:15.870 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ snap_name=snap8 2023-11-08T05:01:15.870 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ create_snap cluster2 mirror test snap8 2023-11-08T05:01:15.870 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:01:15.871 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:01:15.871 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T05:01:15.871 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap_name=snap8 2023-11-08T05:01:15.871 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap8 --debug-rbd=20 --debug-journaler=20 2023-11-08T05:01:23.514 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_image_replay_started cluster1 mirror test 2023-11-08T05:01:23.515 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:01:23.515 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:01:23.515 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T05:01:23.515 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2023-11-08T05:01:23.516 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:01:23.516 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:01:23.516 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T05:01:23.516 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local state=started 2023-11-08T05:01:23.516 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local s 2023-11-08T05:01:23.517 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-11-08T05:01:23.517 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 1 2023-11-08T05:01:24.515 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test_image_replay_state cluster1 mirror test started 2023-11-08T05:01:24.516 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:01:24.516 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:01:24.516 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T05:01:24.516 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local test_state=started 2023-11-08T05:01:24.517 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local status_result 2023-11-08T05:01:24.517 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local current_state=stopped 2023-11-08T05:01:24.520 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2023-11-08T05:01:24.520 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ grep -i state 2023-11-08T05:01:24.520 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_instance=cluster1 2023-11-08T05:01:24.520 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T05:01:24.521 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1 2023-11-08T05:01:24.521 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance=cluster1 2023-11-08T05:01:24.521 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local loop_instance 2023-11-08T05:01:24.521 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-11-08T05:01:24.521 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sleep 0 2023-11-08T05:01:24.522 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-11-08T05:01:24.522 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ seq 0 3 2023-11-08T05:01:24.522 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T05:01:24.522 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2023-11-08T05:01:24.522 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1:0 2023-11-08T05:01:24.523 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T05:01:24.523 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance 2023-11-08T05:01:24.523 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-11-08T05:01:24.524 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local val=cluster1:0 2023-11-08T05:01:24.524 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_var_name=cluster 2023-11-08T05:01:24.524 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance_var_name=instance 2023-11-08T05:01:24.524 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cluster=cluster1 2023-11-08T05:01:24.524 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ instance=0 2023-11-08T05:01:24.525 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' 0 = cluster1:0 ']' 2023-11-08T05:01:24.525 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval cluster=cluster1 2023-11-08T05:01:24.525 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:01:24.525 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval instance=0 2023-11-08T05:01:24.525 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=0 2023-11-08T05:01:24.525 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-11-08T05:01:24.526 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local local_cluster=cluster1:0 2023-11-08T05:01:24.526 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster=cluster1 2023-11-08T05:01:24.526 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance 2023-11-08T05:01:24.526 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-11-08T05:01:24.528 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local val=cluster1:0 2023-11-08T05:01:24.528 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster_var_name=local_cluster 2023-11-08T05:01:24.528 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance_var_name=instance 2023-11-08T05:01:24.528 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:01:24.528 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=0 2023-11-08T05:01:24.529 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ '[' 0 = cluster1:0 ']' 2023-11-08T05:01:24.529 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval local_cluster=cluster1 2023-11-08T05:01:24.529 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ local_cluster=cluster1 2023-11-08T05:01:24.529 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval instance=0 2023-11-08T05:01:24.529 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ instance=0 2023-11-08T05:01:24.530 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-11-08T05:01:24.537 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-08T05:01:24.537 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-08T05:01:24.537 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-08T05:01:24.537 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2023-11-08T05:01:24.629 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-11-08T05:01:24.629 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd mirror status : get status for rbd mirror 2023-11-08T05:01:24.630 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:admin_socket: invalid command 2023-11-08T05:01:24.632 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T05:01:24.633 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2023-11-08T05:01:24.633 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1:1 2023-11-08T05:01:24.633 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T05:01:24.634 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance 2023-11-08T05:01:24.634 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-11-08T05:01:24.634 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local val=cluster1:1 2023-11-08T05:01:24.634 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_var_name=cluster 2023-11-08T05:01:24.634 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance_var_name=instance 2023-11-08T05:01:24.635 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cluster=cluster1 2023-11-08T05:01:24.635 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ instance=1 2023-11-08T05:01:24.635 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' 1 = cluster1:1 ']' 2023-11-08T05:01:24.635 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval cluster=cluster1 2023-11-08T05:01:24.636 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:01:24.636 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval instance=1 2023-11-08T05:01:24.636 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=1 2023-11-08T05:01:24.636 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-11-08T05:01:24.636 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local local_cluster=cluster1:1 2023-11-08T05:01:24.637 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster=cluster1 2023-11-08T05:01:24.637 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance 2023-11-08T05:01:24.637 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-11-08T05:01:24.637 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local val=cluster1:1 2023-11-08T05:01:24.638 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster_var_name=local_cluster 2023-11-08T05:01:24.638 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance_var_name=instance 2023-11-08T05:01:24.638 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:01:24.638 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=1 2023-11-08T05:01:24.639 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ '[' 1 = cluster1:1 ']' 2023-11-08T05:01:24.639 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval local_cluster=cluster1 2023-11-08T05:01:24.639 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ local_cluster=cluster1 2023-11-08T05:01:24.639 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval instance=1 2023-11-08T05:01:24.639 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ instance=1 2023-11-08T05:01:24.640 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-11-08T05:01:24.646 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-08T05:01:24.647 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-08T05:01:24.647 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-08T05:01:24.647 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2023-11-08T05:01:24.734 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-11-08T05:01:24.734 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd mirror status : get status for rbd mirror 2023-11-08T05:01:24.735 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:admin_socket: invalid command 2023-11-08T05:01:24.740 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T05:01:24.740 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2023-11-08T05:01:24.740 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1:2 2023-11-08T05:01:24.741 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T05:01:24.741 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance 2023-11-08T05:01:24.742 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-11-08T05:01:24.742 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local val=cluster1:2 2023-11-08T05:01:24.742 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_var_name=cluster 2023-11-08T05:01:24.743 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance_var_name=instance 2023-11-08T05:01:24.743 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cluster=cluster1 2023-11-08T05:01:24.744 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ instance=2 2023-11-08T05:01:24.744 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' 2 = cluster1:2 ']' 2023-11-08T05:01:24.744 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval cluster=cluster1 2023-11-08T05:01:24.745 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:01:24.745 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval instance=2 2023-11-08T05:01:24.746 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=2 2023-11-08T05:01:24.746 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-11-08T05:01:24.746 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local local_cluster=cluster1:2 2023-11-08T05:01:24.747 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster=cluster1 2023-11-08T05:01:24.747 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance 2023-11-08T05:01:24.747 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-11-08T05:01:24.748 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local val=cluster1:2 2023-11-08T05:01:24.748 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster_var_name=local_cluster 2023-11-08T05:01:24.749 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance_var_name=instance 2023-11-08T05:01:24.749 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:01:24.749 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=2 2023-11-08T05:01:24.750 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ '[' 2 = cluster1:2 ']' 2023-11-08T05:01:24.750 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval local_cluster=cluster1 2023-11-08T05:01:24.750 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ local_cluster=cluster1 2023-11-08T05:01:24.751 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval instance=2 2023-11-08T05:01:24.751 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ instance=2 2023-11-08T05:01:24.752 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-11-08T05:01:24.753 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-08T05:01:24.753 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-08T05:01:24.754 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-08T05:01:24.754 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2023-11-08T05:01:24.850 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ return 0 2023-11-08T05:01:24.850 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ status_result=' "state": "Replaying"' 2023-11-08T05:01:24.850 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ echo ' "state": "Replaying"' 2023-11-08T05:01:24.851 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ grep -i Replaying 2023-11-08T05:01:24.851 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: "state": "Replaying" 2023-11-08T05:01:24.853 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ current_state=started 2023-11-08T05:01:24.853 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test started = started 2023-11-08T05:01:24.854 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:01:24.854 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2023-11-08T05:01:24.854 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local local_cluster=cluster1 2023-11-08T05:01:24.854 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:01:24.855 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:01:24.855 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T05:01:24.855 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' snapshot = journal ']' 2023-11-08T05:01:24.855 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' snapshot = snapshot ']' 2023-11-08T05:01:24.856 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2023-11-08T05:01:24.856 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local local_cluster=cluster1 2023-11-08T05:01:24.856 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:01:24.856 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:01:24.857 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T05:01:24.857 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ mkfname cluster2-mirror-test.status 2023-11-08T05:01:24.857 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ echo cluster2-mirror-test.status 2023-11-08T05:01:24.858 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -e 's|[/ ]|_|g' 2023-11-08T05:01:24.858 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local status_log=/tmp/tmp.Ludk8ikCAB/cluster2-mirror-test.status 2023-11-08T05:01:24.859 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ mkfname cluster1-mirror-test.status 2023-11-08T05:01:24.859 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ echo cluster1-mirror-test.status 2023-11-08T05:01:24.859 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -e 's|[/ ]|_|g' 2023-11-08T05:01:24.861 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local local_status_log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.status 2023-11-08T05:01:24.861 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ mirror_image_snapshot cluster2 mirror test 2023-11-08T05:01:24.861 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:01:24.861 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:01:24.862 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T05:01:24.862 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2023-11-08T05:01:26.524 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:Snapshot ID: 33 2023-11-08T05:01:26.542 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.Ludk8ikCAB/cluster2-mirror-test.status 2023-11-08T05:01:26.543 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:01:26.543 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:01:26.543 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T05:01:26.543 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster2-mirror-test.status 2023-11-08T05:01:26.543 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2023-11-08T05:01:26.544 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T05:01:26.606 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-11-08T05:01:26.608 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snapshot_id=33 2023-11-08T05:01:26.608 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ true 2023-11-08T05:01:26.609 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-11-08T05:01:26.609 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0.2 2023-11-08T05:01:26.810 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.status 2023-11-08T05:01:26.810 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:01:26.810 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:01:26.811 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T05:01:26.811 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.status 2023-11-08T05:01:26.811 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-11-08T05:01:26.811 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T05:01:26.853 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ true 2023-11-08T05:01:26.853 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-08T05:01:26.854 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:-:1.1: Document is empty 2023-11-08T05:01:26.854 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: 2023-11-08T05:01:26.855 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:^ 2023-11-08T05:01:26.855 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local primary_snapshot_id= 2023-11-08T05:01:26.855 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test 33 = '' 2023-11-08T05:01:26.856 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-11-08T05:01:26.856 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0.4 2023-11-08T05:01:27.256 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.status 2023-11-08T05:01:27.256 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:01:27.257 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:01:27.257 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T05:01:27.257 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.status 2023-11-08T05:01:27.257 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-11-08T05:01:27.257 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T05:01:27.300 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ true 2023-11-08T05:01:27.301 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-08T05:01:27.302 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:-:1.1: Document is empty 2023-11-08T05:01:27.302 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: 2023-11-08T05:01:27.302 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:^ 2023-11-08T05:01:27.303 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local primary_snapshot_id= 2023-11-08T05:01:27.303 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test 33 = '' 2023-11-08T05:01:27.304 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-11-08T05:01:27.304 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0.8 2023-11-08T05:01:28.104 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.status 2023-11-08T05:01:28.104 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:01:28.104 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:01:28.105 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T05:01:28.105 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.status 2023-11-08T05:01:28.105 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-11-08T05:01:28.105 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T05:01:28.149 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ true 2023-11-08T05:01:28.150 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-08T05:01:28.151 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:-:1.1: Document is empty 2023-11-08T05:01:28.151 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: 2023-11-08T05:01:28.151 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:^ 2023-11-08T05:01:28.152 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local primary_snapshot_id= 2023-11-08T05:01:28.152 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test 33 = '' 2023-11-08T05:01:28.153 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-11-08T05:01:28.153 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 1.6 2023-11-08T05:01:29.754 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.status 2023-11-08T05:01:29.754 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:01:29.754 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:01:29.754 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T05:01:29.755 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.status 2023-11-08T05:01:29.755 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-11-08T05:01:29.755 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T05:01:29.795 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-08T05:01:29.797 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local primary_snapshot_id=33 2023-11-08T05:01:29.797 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test 33 = 33 2023-11-08T05:01:29.797 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:01:29.797 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_snap_present cluster1 mirror test snap8 2023-11-08T05:01:29.798 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:01:29.798 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:01:29.798 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T05:01:29.798 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap_name=snap8 2023-11-08T05:01:29.799 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local s 2023-11-08T05:01:29.799 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-11-08T05:01:29.799 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 1 2023-11-08T05:01:30.798 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 info mirror/test@snap8 2023-11-08T05:01:30.839 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:rbd image 'test': 2023-11-08T05:01:30.839 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: size 512 MiB in 128 objects 2023-11-08T05:01:30.839 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: order 22 (4 MiB objects) 2023-11-08T05:01:30.840 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: snapshot_count: 16 2023-11-08T05:01:30.840 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: id: 10afad8fe72b 2023-11-08T05:01:30.840 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: block_name_prefix: rbd_data.10afad8fe72b 2023-11-08T05:01:30.840 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: format: 2 2023-11-08T05:01:30.840 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: features: layering, exclusive-lock, non-primary 2023-11-08T05:01:30.841 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: op_features: 2023-11-08T05:01:30.841 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: flags: 2023-11-08T05:01:30.841 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: create_timestamp: Wed Nov 8 04:54:28 2023 2023-11-08T05:01:30.841 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: access_timestamp: Wed Nov 8 04:59:52 2023 2023-11-08T05:01:30.841 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: modify_timestamp: Wed Nov 8 04:54:28 2023 2023-11-08T05:01:30.842 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: protected: False 2023-11-08T05:01:30.842 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring state: enabled 2023-11-08T05:01:30.842 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring mode: snapshot 2023-11-08T05:01:30.842 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring global id: 88ced7f9-2739-4a27-ad7a-30edec05edb1 2023-11-08T05:01:30.843 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring primary: false 2023-11-08T05:01:30.844 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:01:30.844 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' -n snap7-clean ']' 2023-11-08T05:01:30.844 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ compare_image_snaps mirror test snap7-clean 2023-11-08T05:01:30.845 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:01:30.845 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T05:01:30.845 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap_name=snap7-clean 2023-11-08T05:01:30.845 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local ret=0 2023-11-08T05:01:30.845 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local rmt_export=/tmp/tmp.Ludk8ikCAB/cluster2-mirror-test.export 2023-11-08T05:01:30.846 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local loc_export=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.export 2023-11-08T05:01:30.846 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rm -f /tmp/tmp.Ludk8ikCAB/cluster2-mirror-test.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.export 2023-11-08T05:01:30.846 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 -p mirror export test@snap7-clean /tmp/tmp.Ludk8ikCAB/cluster2-mirror-test.export 2023-11-08T05:01:31.449 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-11-08T05:01:31.452 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 -p mirror export test@snap7-clean /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.export 2023-11-08T05:01:32.114 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-11-08T05:01:32.117 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ cmp /tmp/tmp.Ludk8ikCAB/cluster2-mirror-test.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.export 2023-11-08T05:01:32.330 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rm -f /tmp/tmp.Ludk8ikCAB/cluster2-mirror-test.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.export 2023-11-08T05:01:32.471 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:01:32.471 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ compare_image_snaps mirror test snap8 2023-11-08T05:01:32.472 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:01:32.472 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T05:01:32.472 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap_name=snap8 2023-11-08T05:01:32.472 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local ret=0 2023-11-08T05:01:32.472 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local rmt_export=/tmp/tmp.Ludk8ikCAB/cluster2-mirror-test.export 2023-11-08T05:01:32.473 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local loc_export=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.export 2023-11-08T05:01:32.473 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rm -f /tmp/tmp.Ludk8ikCAB/cluster2-mirror-test.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.export 2023-11-08T05:01:32.473 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 -p mirror export test@snap8 /tmp/tmp.Ludk8ikCAB/cluster2-mirror-test.export 2023-11-08T05:01:33.068 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-11-08T05:01:33.072 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 -p mirror export test@snap8 /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.export 2023-11-08T05:01:33.809 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-11-08T05:01:33.812 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ cmp /tmp/tmp.Ludk8ikCAB/cluster2-mirror-test.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.export 2023-11-08T05:01:34.022 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rm -f /tmp/tmp.Ludk8ikCAB/cluster2-mirror-test.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.export 2023-11-08T05:01:34.164 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:01:34.164 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ clean_snap_name=snap8-clean 2023-11-08T05:01:34.164 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ create_snap cluster2 mirror test snap8-clean 2023-11-08T05:01:34.164 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:01:34.165 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:01:34.165 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T05:01:34.165 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap_name=snap8-clean 2023-11-08T05:01:34.165 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap8-clean --debug-rbd=20 --debug-journaler=20 2023-11-08T05:01:34.504 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for i in `seq 1 10` 2023-11-08T05:01:34.504 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ stress_write_image cluster2 mirror test 2023-11-08T05:01:34.504 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:01:34.504 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:01:34.505 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T05:01:34.505 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-11-08T05:01:34.505 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local duration=10 2023-11-08T05:01:34.506 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ set +e 2023-11-08T05:01:34.506 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2023-11-08T05:01:34.787 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:100 IOs, 63325184 bytes 2023-11-08T05:01:34.973 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:200 IOs, 103553024 bytes 2023-11-08T05:01:35.083 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:300 IOs, 119805952 bytes 2023-11-08T05:01:35.248 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:400 IOs, 161898496 bytes 2023-11-08T05:01:35.368 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:500 IOs, 181460992 bytes 2023-11-08T05:01:35.536 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:600 IOs, 227425280 bytes 2023-11-08T05:01:35.676 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:700 IOs, 249093120 bytes 2023-11-08T05:01:35.794 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:800 IOs, 270384128 bytes 2023-11-08T05:01:35.995 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:900 IOs, 335100928 bytes 2023-11-08T05:01:36.158 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1000 IOs, 374076416 bytes 2023-11-08T05:01:36.294 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1100 IOs, 397598720 bytes 2023-11-08T05:01:36.427 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1200 IOs, 425616384 bytes 2023-11-08T05:01:36.560 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1300 IOs, 450855936 bytes 2023-11-08T05:01:36.706 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1400 IOs, 506156032 bytes 2023-11-08T05:01:36.893 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1500 IOs, 537966592 bytes 2023-11-08T05:01:37.034 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1600 IOs, 568105984 bytes 2023-11-08T05:01:37.170 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1700 IOs, 595838976 bytes 2023-11-08T05:01:37.327 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1800 IOs, 636362752 bytes 2023-11-08T05:01:37.470 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1900 IOs, 662709248 bytes 2023-11-08T05:01:37.606 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2000 IOs, 687117312 bytes 2023-11-08T05:01:37.842 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2100 IOs, 761850880 bytes 2023-11-08T05:01:37.915 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2200 IOs, 762468352 bytes 2023-11-08T05:01:38.252 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2300 IOs, 885496832 bytes 2023-11-08T05:01:38.394 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2400 IOs, 909184000 bytes 2023-11-08T05:01:38.547 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2500 IOs, 944093184 bytes 2023-11-08T05:01:38.672 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2600 IOs, 970478592 bytes 2023-11-08T05:01:38.792 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2700 IOs, 993121280 bytes 2023-11-08T05:01:39.004 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2800 IOs, 1062670336 bytes 2023-11-08T05:01:39.142 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2900 IOs, 1094385664 bytes 2023-11-08T05:01:39.335 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3000 IOs, 1152995328 bytes 2023-11-08T05:01:39.472 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3100 IOs, 1190710272 bytes 2023-11-08T05:01:39.699 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3200 IOs, 1255151616 bytes 2023-11-08T05:01:39.863 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3300 IOs, 1288051712 bytes 2023-11-08T05:01:40.063 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3400 IOs, 1344717824 bytes 2023-11-08T05:01:40.250 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3500 IOs, 1396307968 bytes 2023-11-08T05:01:40.421 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3600 IOs, 1433150464 bytes 2023-11-08T05:01:40.657 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3700 IOs, 1505857536 bytes 2023-11-08T05:01:40.799 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3800 IOs, 1543275520 bytes 2023-11-08T05:01:41.095 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3900 IOs, 1643770880 bytes 2023-11-08T05:01:41.233 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4000 IOs, 1673649152 bytes 2023-11-08T05:01:41.468 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4100 IOs, 1759151104 bytes 2023-11-08T05:01:41.631 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4200 IOs, 1799424000 bytes 2023-11-08T05:01:41.803 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4300 IOs, 1838815232 bytes 2023-11-08T05:01:42.133 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4400 IOs, 1956154368 bytes 2023-11-08T05:01:42.289 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4500 IOs, 1987994624 bytes 2023-11-08T05:01:42.588 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4600 IOs, 2091014144 bytes 2023-11-08T05:01:42.867 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4700 IOs, 2192187392 bytes 2023-11-08T05:01:43.022 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4800 IOs, 2229812224 bytes 2023-11-08T05:01:43.232 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4900 IOs, 2297505792 bytes 2023-11-08T05:01:43.340 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5000 IOs, 2323402752 bytes 2023-11-08T05:01:43.608 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5100 IOs, 2400706560 bytes 2023-11-08T05:01:43.696 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5200 IOs, 2401315840 bytes 2023-11-08T05:01:43.873 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5300 IOs, 2451835904 bytes 2023-11-08T05:01:43.998 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5400 IOs, 2502867968 bytes 2023-11-08T05:01:44.189 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5500 IOs, 2539593728 bytes 2023-11-08T05:01:44.308 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5600 IOs, 2543353856 bytes 2023-11-08T05:01:44.478 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5700 IOs, 2590830592 bytes 2023-11-08T05:01:44.512 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ error_code=124 2023-11-08T05:01:44.512 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ set -e 2023-11-08T05:01:44.512 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' 124 -eq 124 ']' 2023-11-08T05:01:44.513 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:01:44.513 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2023-11-08T05:01:44.513 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:01:44.513 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:01:44.513 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T05:01:44.514 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local state_pattern=up+replaying 2023-11-08T05:01:44.514 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local description_pattern= 2023-11-08T05:01:44.514 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local service_pattern= 2023-11-08T05:01:44.514 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-11-08T05:01:44.515 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 1 2023-11-08T05:01:45.513 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2023-11-08T05:01:45.514 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:01:45.514 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:01:45.514 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T05:01:45.515 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local state_pattern=up+replaying 2023-11-08T05:01:45.515 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local description_pattern= 2023-11-08T05:01:45.515 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local service_pattern= 2023-11-08T05:01:45.515 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ mkfname cluster1-mirror-test.mirror_status 2023-11-08T05:01:45.516 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ echo cluster1-mirror-test.mirror_status 2023-11-08T05:01:45.516 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -e 's|[/ ]|_|g' 2023-11-08T05:01:45.516 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local status_log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.mirror_status 2023-11-08T05:01:45.517 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ CEPH_ARGS= 2023-11-08T05:01:45.517 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 mirror image status mirror/test 2023-11-08T05:01:45.517 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ tee /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.mirror_status 2023-11-08T05:01:45.562 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:test: 2023-11-08T05:01:45.563 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: global_id: 88ced7f9-2739-4a27-ad7a-30edec05edb1 2023-11-08T05:01:45.563 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: state: up+replaying 2023-11-08T05:01:45.563 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1699419636,"remote_snapshot_timestamp":1699419636,"replay_state":"idle"} 2023-11-08T05:01:45.564 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: service: mirror.2 on smithi087 2023-11-08T05:01:45.564 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: last_update: 2023-11-08 05:01:20 2023-11-08T05:01:45.564 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: peer_sites: 2023-11-08T05:01:45.565 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: name: cluster2 2023-11-08T05:01:45.565 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: state: up+stopped 2023-11-08T05:01:45.565 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: description: local image is primary 2023-11-08T05:01:45.566 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: last_update: 2023-11-08 05:01:26 2023-11-08T05:01:45.567 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ grep '^ state: .*up+replaying' /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.mirror_status 2023-11-08T05:01:45.568 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: state: up+replaying 2023-11-08T05:01:45.569 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ grep '^ description: .*' /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.mirror_status 2023-11-08T05:01:45.569 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1699419636,"remote_snapshot_timestamp":1699419636,"replay_state":"idle"} 2023-11-08T05:01:45.570 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' -n '' ']' 2023-11-08T05:01:45.571 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ echo up+replaying 2023-11-08T05:01:45.571 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ grep '^up+' 2023-11-08T05:01:45.572 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:up+replaying 2023-11-08T05:01:45.572 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ grep 'service: *mirror..* on ' /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.mirror_status 2023-11-08T05:01:45.573 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: service: mirror.2 on smithi087 2023-11-08T05:01:45.576 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.mirror_status 2023-11-08T05:01:45.577 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local 'last_update=2023-11-08 05:01:20' 2023-11-08T05:01:45.577 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2023-11-08 05:01:20' 2023-11-08T05:01:45.577 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:01:45.578 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:01:45.578 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T05:01:45.578 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local state_pattern=up+replaying 2023-11-08T05:01:45.579 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local 'prev_last_update=2023-11-08 05:01:20' 2023-11-08T05:01:45.579 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ mkfname cluster1-mirror.mirror_status 2023-11-08T05:01:45.579 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ echo cluster1-mirror.mirror_status 2023-11-08T05:01:45.580 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -e 's|[/ ]|_|g' 2023-11-08T05:01:45.580 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local status_log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror.mirror_status 2023-11-08T05:01:45.581 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2023-11-08T05:01:45.607 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd: failed to query services: (13) Permission denied 2023-11-08T05:01:45.631 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local last_update state 2023-11-08T05:01:45.632 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2023-11-08T05:01:45.633 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ last_update='2023-11-08 05:01:20' 2023-11-08T05:01:45.634 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2023-11-08T05:01:45.635 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ state=up+replaying 2023-11-08T05:01:45.635 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ echo up+replaying 2023-11-08T05:01:45.636 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ grep up+replaying 2023-11-08T05:01:45.636 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:up+replaying 2023-11-08T05:01:45.637 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:01:45.637 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:01:45.638 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ snap_name=snap9 2023-11-08T05:01:45.638 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ create_snap cluster2 mirror test snap9 2023-11-08T05:01:45.638 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:01:45.638 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:01:45.639 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T05:01:45.639 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap_name=snap9 2023-11-08T05:01:45.639 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap9 --debug-rbd=20 --debug-journaler=20 2023-11-08T05:02:14.697 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_image_replay_started cluster1 mirror test 2023-11-08T05:02:14.698 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:02:14.698 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:02:14.698 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T05:02:14.698 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2023-11-08T05:02:14.699 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:02:14.699 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:02:14.699 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T05:02:14.699 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local state=started 2023-11-08T05:02:14.699 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local s 2023-11-08T05:02:14.700 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-11-08T05:02:14.700 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 1 2023-11-08T05:02:15.699 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test_image_replay_state cluster1 mirror test started 2023-11-08T05:02:15.699 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:02:15.699 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:02:15.699 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T05:02:15.700 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local test_state=started 2023-11-08T05:02:15.700 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local status_result 2023-11-08T05:02:15.700 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local current_state=stopped 2023-11-08T05:02:15.700 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2023-11-08T05:02:15.700 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_instance=cluster1 2023-11-08T05:02:15.701 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T05:02:15.701 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ grep -i state 2023-11-08T05:02:15.701 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1 2023-11-08T05:02:15.701 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance=cluster1 2023-11-08T05:02:15.701 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local loop_instance 2023-11-08T05:02:15.702 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-11-08T05:02:15.702 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sleep 0 2023-11-08T05:02:15.706 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-11-08T05:02:15.707 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ seq 0 3 2023-11-08T05:02:15.707 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T05:02:15.707 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2023-11-08T05:02:15.708 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1:0 2023-11-08T05:02:15.708 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T05:02:15.708 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance 2023-11-08T05:02:15.709 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-11-08T05:02:15.709 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local val=cluster1:0 2023-11-08T05:02:15.709 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_var_name=cluster 2023-11-08T05:02:15.710 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance_var_name=instance 2023-11-08T05:02:15.710 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cluster=cluster1 2023-11-08T05:02:15.710 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ instance=0 2023-11-08T05:02:15.711 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' 0 = cluster1:0 ']' 2023-11-08T05:02:15.711 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval cluster=cluster1 2023-11-08T05:02:15.711 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:02:15.712 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval instance=0 2023-11-08T05:02:15.712 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=0 2023-11-08T05:02:15.712 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-11-08T05:02:15.713 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local local_cluster=cluster1:0 2023-11-08T05:02:15.713 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster=cluster1 2023-11-08T05:02:15.713 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance 2023-11-08T05:02:15.713 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-11-08T05:02:15.713 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local val=cluster1:0 2023-11-08T05:02:15.714 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster_var_name=local_cluster 2023-11-08T05:02:15.714 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance_var_name=instance 2023-11-08T05:02:15.714 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:02:15.714 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=0 2023-11-08T05:02:15.714 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ '[' 0 = cluster1:0 ']' 2023-11-08T05:02:15.715 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval local_cluster=cluster1 2023-11-08T05:02:15.715 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ local_cluster=cluster1 2023-11-08T05:02:15.715 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval instance=0 2023-11-08T05:02:15.715 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ instance=0 2023-11-08T05:02:15.715 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-11-08T05:02:15.720 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-08T05:02:15.720 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-08T05:02:15.720 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-08T05:02:15.720 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2023-11-08T05:02:15.816 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-11-08T05:02:15.816 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd mirror status : get status for rbd mirror 2023-11-08T05:02:15.816 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:admin_socket: invalid command 2023-11-08T05:02:15.819 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T05:02:15.820 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2023-11-08T05:02:15.820 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1:1 2023-11-08T05:02:15.820 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T05:02:15.820 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance 2023-11-08T05:02:15.820 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-11-08T05:02:15.821 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local val=cluster1:1 2023-11-08T05:02:15.821 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_var_name=cluster 2023-11-08T05:02:15.821 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance_var_name=instance 2023-11-08T05:02:15.821 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cluster=cluster1 2023-11-08T05:02:15.821 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ instance=1 2023-11-08T05:02:15.821 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' 1 = cluster1:1 ']' 2023-11-08T05:02:15.822 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval cluster=cluster1 2023-11-08T05:02:15.822 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:02:15.822 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval instance=1 2023-11-08T05:02:15.822 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=1 2023-11-08T05:02:15.822 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-11-08T05:02:15.823 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local local_cluster=cluster1:1 2023-11-08T05:02:15.823 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster=cluster1 2023-11-08T05:02:15.823 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance 2023-11-08T05:02:15.823 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-11-08T05:02:15.823 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local val=cluster1:1 2023-11-08T05:02:15.824 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster_var_name=local_cluster 2023-11-08T05:02:15.824 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance_var_name=instance 2023-11-08T05:02:15.824 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:02:15.824 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=1 2023-11-08T05:02:15.824 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ '[' 1 = cluster1:1 ']' 2023-11-08T05:02:15.824 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval local_cluster=cluster1 2023-11-08T05:02:15.825 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ local_cluster=cluster1 2023-11-08T05:02:15.825 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval instance=1 2023-11-08T05:02:15.825 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ instance=1 2023-11-08T05:02:15.825 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-11-08T05:02:15.833 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-08T05:02:15.833 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-08T05:02:15.834 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-08T05:02:15.834 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2023-11-08T05:02:15.923 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-11-08T05:02:15.923 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd mirror status : get status for rbd mirror 2023-11-08T05:02:15.923 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:admin_socket: invalid command 2023-11-08T05:02:15.926 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T05:02:15.926 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2023-11-08T05:02:15.927 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1:2 2023-11-08T05:02:15.927 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T05:02:15.927 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance 2023-11-08T05:02:15.927 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-11-08T05:02:15.927 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local val=cluster1:2 2023-11-08T05:02:15.928 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_var_name=cluster 2023-11-08T05:02:15.928 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance_var_name=instance 2023-11-08T05:02:15.928 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cluster=cluster1 2023-11-08T05:02:15.928 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ instance=2 2023-11-08T05:02:15.928 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' 2 = cluster1:2 ']' 2023-11-08T05:02:15.929 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval cluster=cluster1 2023-11-08T05:02:15.929 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:02:15.929 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval instance=2 2023-11-08T05:02:15.929 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=2 2023-11-08T05:02:15.929 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-11-08T05:02:15.929 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local local_cluster=cluster1:2 2023-11-08T05:02:15.930 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster=cluster1 2023-11-08T05:02:15.930 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance 2023-11-08T05:02:15.930 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-11-08T05:02:15.930 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local val=cluster1:2 2023-11-08T05:02:15.930 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster_var_name=local_cluster 2023-11-08T05:02:15.931 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance_var_name=instance 2023-11-08T05:02:15.931 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:02:15.931 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=2 2023-11-08T05:02:15.931 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ '[' 2 = cluster1:2 ']' 2023-11-08T05:02:15.931 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval local_cluster=cluster1 2023-11-08T05:02:15.932 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ local_cluster=cluster1 2023-11-08T05:02:15.932 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval instance=2 2023-11-08T05:02:15.932 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ instance=2 2023-11-08T05:02:15.932 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-11-08T05:02:15.940 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-08T05:02:15.940 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-08T05:02:15.940 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-08T05:02:15.940 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2023-11-08T05:02:16.036 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ return 0 2023-11-08T05:02:16.037 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ status_result=' "state": "Replaying"' 2023-11-08T05:02:16.037 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ echo ' "state": "Replaying"' 2023-11-08T05:02:16.038 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ grep -i Replaying 2023-11-08T05:02:16.038 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: "state": "Replaying" 2023-11-08T05:02:16.040 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ current_state=started 2023-11-08T05:02:16.040 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test started = started 2023-11-08T05:02:16.041 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:02:16.041 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2023-11-08T05:02:16.041 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local local_cluster=cluster1 2023-11-08T05:02:16.041 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:02:16.041 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:02:16.042 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T05:02:16.042 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' snapshot = journal ']' 2023-11-08T05:02:16.042 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' snapshot = snapshot ']' 2023-11-08T05:02:16.042 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2023-11-08T05:02:16.042 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local local_cluster=cluster1 2023-11-08T05:02:16.043 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:02:16.043 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:02:16.043 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T05:02:16.043 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ mkfname cluster2-mirror-test.status 2023-11-08T05:02:16.043 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ echo cluster2-mirror-test.status 2023-11-08T05:02:16.044 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -e 's|[/ ]|_|g' 2023-11-08T05:02:16.044 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local status_log=/tmp/tmp.Ludk8ikCAB/cluster2-mirror-test.status 2023-11-08T05:02:16.044 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ mkfname cluster1-mirror-test.status 2023-11-08T05:02:16.044 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ echo cluster1-mirror-test.status 2023-11-08T05:02:16.044 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -e 's|[/ ]|_|g' 2023-11-08T05:02:16.046 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local local_status_log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.status 2023-11-08T05:02:16.046 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ mirror_image_snapshot cluster2 mirror test 2023-11-08T05:02:16.046 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:02:16.046 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:02:16.047 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T05:02:16.047 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2023-11-08T05:02:17.711 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:Snapshot ID: 37 2023-11-08T05:02:17.727 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.Ludk8ikCAB/cluster2-mirror-test.status 2023-11-08T05:02:17.727 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:02:17.728 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:02:17.728 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T05:02:17.728 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster2-mirror-test.status 2023-11-08T05:02:17.728 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2023-11-08T05:02:17.728 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T05:02:17.787 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-11-08T05:02:17.790 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snapshot_id=37 2023-11-08T05:02:17.790 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ true 2023-11-08T05:02:17.790 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-11-08T05:02:17.790 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0.2 2023-11-08T05:02:17.991 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.status 2023-11-08T05:02:17.992 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:02:17.992 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:02:17.992 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T05:02:17.992 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.status 2023-11-08T05:02:17.993 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-11-08T05:02:17.993 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T05:02:18.033 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ true 2023-11-08T05:02:18.034 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-08T05:02:18.035 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:-:1.1: Document is empty 2023-11-08T05:02:18.035 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: 2023-11-08T05:02:18.035 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:^ 2023-11-08T05:02:18.036 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local primary_snapshot_id= 2023-11-08T05:02:18.036 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test 37 = '' 2023-11-08T05:02:18.036 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-11-08T05:02:18.036 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0.4 2023-11-08T05:02:18.437 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.status 2023-11-08T05:02:18.437 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:02:18.437 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:02:18.437 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T05:02:18.438 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.status 2023-11-08T05:02:18.438 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-11-08T05:02:18.438 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T05:02:18.479 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ true 2023-11-08T05:02:18.480 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-08T05:02:18.481 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:-:1.1: Document is empty 2023-11-08T05:02:18.481 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: 2023-11-08T05:02:18.481 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:^ 2023-11-08T05:02:18.482 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local primary_snapshot_id= 2023-11-08T05:02:18.482 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test 37 = '' 2023-11-08T05:02:18.483 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-11-08T05:02:18.483 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0.8 2023-11-08T05:02:19.283 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.status 2023-11-08T05:02:19.283 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:02:19.284 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:02:19.284 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T05:02:19.284 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.status 2023-11-08T05:02:19.284 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-11-08T05:02:19.284 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T05:02:19.546 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ true 2023-11-08T05:02:19.547 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-08T05:02:19.547 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:-:1.1: Document is empty 2023-11-08T05:02:19.548 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: 2023-11-08T05:02:19.548 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:^ 2023-11-08T05:02:19.549 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local primary_snapshot_id= 2023-11-08T05:02:19.549 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test 37 = '' 2023-11-08T05:02:19.549 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-11-08T05:02:19.549 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 1.6 2023-11-08T05:02:21.149 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.status 2023-11-08T05:02:21.150 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:02:21.150 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:02:21.150 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T05:02:21.151 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.status 2023-11-08T05:02:21.151 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-11-08T05:02:21.151 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T05:02:21.192 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-08T05:02:21.195 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local primary_snapshot_id=37 2023-11-08T05:02:21.195 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test 37 = 37 2023-11-08T05:02:21.195 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:02:21.195 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_snap_present cluster1 mirror test snap9 2023-11-08T05:02:21.196 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:02:21.196 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:02:21.196 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T05:02:21.196 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap_name=snap9 2023-11-08T05:02:21.196 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local s 2023-11-08T05:02:21.197 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-11-08T05:02:21.197 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 1 2023-11-08T05:02:22.195 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 info mirror/test@snap9 2023-11-08T05:02:22.237 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:rbd image 'test': 2023-11-08T05:02:22.237 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: size 512 MiB in 128 objects 2023-11-08T05:02:22.237 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: order 22 (4 MiB objects) 2023-11-08T05:02:22.237 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: snapshot_count: 18 2023-11-08T05:02:22.238 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: id: 10afad8fe72b 2023-11-08T05:02:22.238 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: block_name_prefix: rbd_data.10afad8fe72b 2023-11-08T05:02:22.238 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: format: 2 2023-11-08T05:02:22.238 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: features: layering, exclusive-lock, non-primary 2023-11-08T05:02:22.238 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: op_features: 2023-11-08T05:02:22.239 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: flags: 2023-11-08T05:02:22.239 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: create_timestamp: Wed Nov 8 04:54:28 2023 2023-11-08T05:02:22.239 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: access_timestamp: Wed Nov 8 05:01:31 2023 2023-11-08T05:02:22.239 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: modify_timestamp: Wed Nov 8 04:54:28 2023 2023-11-08T05:02:22.239 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: protected: False 2023-11-08T05:02:22.240 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring state: enabled 2023-11-08T05:02:22.240 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring mode: snapshot 2023-11-08T05:02:22.240 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring global id: 88ced7f9-2739-4a27-ad7a-30edec05edb1 2023-11-08T05:02:22.240 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring primary: false 2023-11-08T05:02:22.241 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:02:22.242 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' -n snap8-clean ']' 2023-11-08T05:02:22.242 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ compare_image_snaps mirror test snap8-clean 2023-11-08T05:02:22.242 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:02:22.242 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T05:02:22.242 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap_name=snap8-clean 2023-11-08T05:02:22.243 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local ret=0 2023-11-08T05:02:22.243 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local rmt_export=/tmp/tmp.Ludk8ikCAB/cluster2-mirror-test.export 2023-11-08T05:02:22.243 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local loc_export=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.export 2023-11-08T05:02:22.243 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rm -f /tmp/tmp.Ludk8ikCAB/cluster2-mirror-test.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.export 2023-11-08T05:02:22.243 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 -p mirror export test@snap8-clean /tmp/tmp.Ludk8ikCAB/cluster2-mirror-test.export 2023-11-08T05:02:22.846 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-11-08T05:02:22.849 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 -p mirror export test@snap8-clean /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.export 2023-11-08T05:02:23.655 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-11-08T05:02:23.658 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ cmp /tmp/tmp.Ludk8ikCAB/cluster2-mirror-test.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.export 2023-11-08T05:02:23.870 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rm -f /tmp/tmp.Ludk8ikCAB/cluster2-mirror-test.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.export 2023-11-08T05:02:24.011 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:02:24.011 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ compare_image_snaps mirror test snap9 2023-11-08T05:02:24.011 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:02:24.011 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T05:02:24.012 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap_name=snap9 2023-11-08T05:02:24.012 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local ret=0 2023-11-08T05:02:24.012 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local rmt_export=/tmp/tmp.Ludk8ikCAB/cluster2-mirror-test.export 2023-11-08T05:02:24.012 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local loc_export=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.export 2023-11-08T05:02:24.012 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rm -f /tmp/tmp.Ludk8ikCAB/cluster2-mirror-test.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.export 2023-11-08T05:02:24.013 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 -p mirror export test@snap9 /tmp/tmp.Ludk8ikCAB/cluster2-mirror-test.export 2023-11-08T05:02:24.624 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-11-08T05:02:24.627 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 -p mirror export test@snap9 /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.export 2023-11-08T05:02:25.442 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-11-08T05:02:25.445 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ cmp /tmp/tmp.Ludk8ikCAB/cluster2-mirror-test.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.export 2023-11-08T05:02:25.659 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rm -f /tmp/tmp.Ludk8ikCAB/cluster2-mirror-test.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.export 2023-11-08T05:02:25.800 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:02:25.800 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ clean_snap_name=snap9-clean 2023-11-08T05:02:25.800 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ create_snap cluster2 mirror test snap9-clean 2023-11-08T05:02:25.800 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:02:25.800 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:02:25.801 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T05:02:25.801 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap_name=snap9-clean 2023-11-08T05:02:25.801 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap9-clean --debug-rbd=20 --debug-journaler=20 2023-11-08T05:02:26.723 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for i in `seq 1 10` 2023-11-08T05:02:26.724 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ stress_write_image cluster2 mirror test 2023-11-08T05:02:26.724 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:02:26.724 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:02:26.724 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T05:02:26.725 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-11-08T05:02:26.725 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local duration=11 2023-11-08T05:02:26.725 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ set +e 2023-11-08T05:02:26.726 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2023-11-08T05:02:26.941 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:100 IOs, 23994368 bytes 2023-11-08T05:02:27.063 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:200 IOs, 30627840 bytes 2023-11-08T05:02:27.223 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:300 IOs, 72253440 bytes 2023-11-08T05:02:27.355 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:400 IOs, 98603008 bytes 2023-11-08T05:02:27.569 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:500 IOs, 157885440 bytes 2023-11-08T05:02:27.711 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:600 IOs, 187897856 bytes 2023-11-08T05:02:27.846 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:700 IOs, 217776128 bytes 2023-11-08T05:02:28.066 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:800 IOs, 273495040 bytes 2023-11-08T05:02:28.261 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:900 IOs, 320751616 bytes 2023-11-08T05:02:28.442 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1000 IOs, 371335168 bytes 2023-11-08T05:02:28.624 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1100 IOs, 423512064 bytes 2023-11-08T05:02:28.754 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1200 IOs, 463879168 bytes 2023-11-08T05:02:28.927 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1300 IOs, 495076352 bytes 2023-11-08T05:02:29.012 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1400 IOs, 495700992 bytes 2023-11-08T05:02:29.194 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1500 IOs, 547336192 bytes 2023-11-08T05:02:29.296 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1600 IOs, 557454336 bytes 2023-11-08T05:02:29.482 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1700 IOs, 605208576 bytes 2023-11-08T05:02:29.648 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1800 IOs, 650115072 bytes 2023-11-08T05:02:29.800 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1900 IOs, 693841920 bytes 2023-11-08T05:02:29.977 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2000 IOs, 733471744 bytes 2023-11-08T05:02:30.136 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2100 IOs, 766285824 bytes 2023-11-08T05:02:30.324 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2200 IOs, 808359936 bytes 2023-11-08T05:02:30.459 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2300 IOs, 830707712 bytes 2023-11-08T05:02:30.567 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2400 IOs, 849868800 bytes 2023-11-08T05:02:30.785 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2500 IOs, 910120960 bytes 2023-11-08T05:02:30.859 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2600 IOs, 912887808 bytes 2023-11-08T05:02:31.035 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2700 IOs, 952975360 bytes 2023-11-08T05:02:31.336 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2800 IOs, 1061582848 bytes 2023-11-08T05:02:31.554 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2900 IOs, 1128955904 bytes 2023-11-08T05:02:31.726 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3000 IOs, 1162680320 bytes 2023-11-08T05:02:31.873 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3100 IOs, 1192008704 bytes 2023-11-08T05:02:32.031 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3200 IOs, 1232927744 bytes 2023-11-08T05:02:32.164 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3300 IOs, 1268572160 bytes 2023-11-08T05:02:32.302 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3400 IOs, 1287614464 bytes 2023-11-08T05:02:32.440 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3500 IOs, 1308151808 bytes 2023-11-08T05:02:32.637 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3600 IOs, 1368687616 bytes 2023-11-08T05:02:32.785 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3700 IOs, 1396512768 bytes 2023-11-08T05:02:32.902 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3800 IOs, 1415436288 bytes 2023-11-08T05:02:33.046 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3900 IOs, 1444503552 bytes 2023-11-08T05:02:33.178 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4000 IOs, 1468406784 bytes 2023-11-08T05:02:33.308 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4100 IOs, 1490751488 bytes 2023-11-08T05:02:33.437 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4200 IOs, 1514265600 bytes 2023-11-08T05:02:33.604 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4300 IOs, 1558947840 bytes 2023-11-08T05:02:33.806 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4400 IOs, 1622098944 bytes 2023-11-08T05:02:33.970 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4500 IOs, 1658167296 bytes 2023-11-08T05:02:34.092 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4600 IOs, 1683971072 bytes 2023-11-08T05:02:34.211 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4700 IOs, 1695094784 bytes 2023-11-08T05:02:34.379 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4800 IOs, 1731512320 bytes 2023-11-08T05:02:34.570 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4900 IOs, 1797468160 bytes 2023-11-08T05:02:34.723 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5000 IOs, 1819053056 bytes 2023-11-08T05:02:34.990 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5100 IOs, 1924046848 bytes 2023-11-08T05:02:35.251 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5200 IOs, 1999085568 bytes 2023-11-08T05:02:35.550 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5300 IOs, 2103308288 bytes 2023-11-08T05:02:35.643 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5400 IOs, 2103922688 bytes 2023-11-08T05:02:35.869 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5500 IOs, 2175212544 bytes 2023-11-08T05:02:36.095 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5600 IOs, 2241275904 bytes 2023-11-08T05:02:36.261 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5700 IOs, 2276947968 bytes 2023-11-08T05:02:36.532 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5800 IOs, 2365078528 bytes 2023-11-08T05:02:36.718 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5900 IOs, 2413799424 bytes 2023-11-08T05:02:36.985 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:6000 IOs, 2508552192 bytes 2023-11-08T05:02:37.075 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:6100 IOs, 2519868416 bytes 2023-11-08T05:02:37.253 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:6200 IOs, 2574280704 bytes 2023-11-08T05:02:37.369 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:6300 IOs, 2586106880 bytes 2023-11-08T05:02:37.556 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:6400 IOs, 2641695744 bytes 2023-11-08T05:02:37.718 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:6500 IOs, 2676257792 bytes 2023-11-08T05:02:37.736 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ error_code=124 2023-11-08T05:02:37.736 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ set -e 2023-11-08T05:02:37.736 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' 124 -eq 124 ']' 2023-11-08T05:02:37.737 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:02:37.737 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2023-11-08T05:02:37.737 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:02:37.737 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:02:37.737 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T05:02:37.738 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local state_pattern=up+replaying 2023-11-08T05:02:37.738 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local description_pattern= 2023-11-08T05:02:37.738 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local service_pattern= 2023-11-08T05:02:37.738 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-11-08T05:02:37.739 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 1 2023-11-08T05:02:38.738 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2023-11-08T05:02:38.739 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:02:38.739 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:02:38.739 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T05:02:38.739 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local state_pattern=up+replaying 2023-11-08T05:02:38.739 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local description_pattern= 2023-11-08T05:02:38.740 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local service_pattern= 2023-11-08T05:02:38.740 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ mkfname cluster1-mirror-test.mirror_status 2023-11-08T05:02:38.740 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ echo cluster1-mirror-test.mirror_status 2023-11-08T05:02:38.740 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -e 's|[/ ]|_|g' 2023-11-08T05:02:38.741 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local status_log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.mirror_status 2023-11-08T05:02:38.741 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ CEPH_ARGS= 2023-11-08T05:02:38.742 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 mirror image status mirror/test 2023-11-08T05:02:38.742 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ tee /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.mirror_status 2023-11-08T05:02:38.786 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:test: 2023-11-08T05:02:38.786 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: global_id: 88ced7f9-2739-4a27-ad7a-30edec05edb1 2023-11-08T05:02:38.787 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: state: up+replaying 2023-11-08T05:02:38.787 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":508704768.0,"last_snapshot_bytes":508704768,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1699419685,"remote_snapshot_timestamp":1699419685,"replay_state":"idle"} 2023-11-08T05:02:38.787 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: service: mirror.2 on smithi087 2023-11-08T05:02:38.787 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: last_update: 2023-11-08 05:02:20 2023-11-08T05:02:38.788 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: peer_sites: 2023-11-08T05:02:38.788 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: name: cluster2 2023-11-08T05:02:38.788 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: state: up+stopped 2023-11-08T05:02:38.788 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: description: local image is primary 2023-11-08T05:02:38.789 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: last_update: 2023-11-08 05:02:26 2023-11-08T05:02:38.790 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ grep '^ state: .*up+replaying' /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.mirror_status 2023-11-08T05:02:38.792 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: state: up+replaying 2023-11-08T05:02:38.792 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ grep '^ description: .*' /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.mirror_status 2023-11-08T05:02:38.793 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":508704768.0,"last_snapshot_bytes":508704768,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1699419685,"remote_snapshot_timestamp":1699419685,"replay_state":"idle"} 2023-11-08T05:02:38.794 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' -n '' ']' 2023-11-08T05:02:38.794 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ echo up+replaying 2023-11-08T05:02:38.795 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ grep '^up+' 2023-11-08T05:02:38.795 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:up+replaying 2023-11-08T05:02:38.796 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ grep 'service: *mirror..* on ' /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.mirror_status 2023-11-08T05:02:38.797 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: service: mirror.2 on smithi087 2023-11-08T05:02:38.798 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.mirror_status 2023-11-08T05:02:38.798 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local 'last_update=2023-11-08 05:02:20' 2023-11-08T05:02:38.798 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2023-11-08 05:02:20' 2023-11-08T05:02:38.798 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:02:38.799 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:02:38.799 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T05:02:38.799 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local state_pattern=up+replaying 2023-11-08T05:02:38.800 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local 'prev_last_update=2023-11-08 05:02:20' 2023-11-08T05:02:38.800 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ mkfname cluster1-mirror.mirror_status 2023-11-08T05:02:38.800 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ echo cluster1-mirror.mirror_status 2023-11-08T05:02:38.800 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -e 's|[/ ]|_|g' 2023-11-08T05:02:38.801 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local status_log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror.mirror_status 2023-11-08T05:02:38.801 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2023-11-08T05:02:38.830 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd: failed to query services: (13) Permission denied 2023-11-08T05:02:38.855 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local last_update state 2023-11-08T05:02:38.855 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2023-11-08T05:02:38.857 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ last_update='2023-11-08 05:02:20' 2023-11-08T05:02:38.857 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2023-11-08T05:02:38.859 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ state=up+replaying 2023-11-08T05:02:38.859 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ echo up+replaying 2023-11-08T05:02:38.859 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ grep up+replaying 2023-11-08T05:02:38.860 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:up+replaying 2023-11-08T05:02:38.861 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:02:38.861 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:02:38.861 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ snap_name=snap10 2023-11-08T05:02:38.861 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ create_snap cluster2 mirror test snap10 2023-11-08T05:02:38.862 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:02:38.862 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:02:38.862 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T05:02:38.862 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap_name=snap10 2023-11-08T05:02:38.862 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap10 --debug-rbd=20 --debug-journaler=20 2023-11-08T05:02:40.044 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.2867104976860321) > weight (0.25) 2023-11-08T05:02:40.044 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.6978784735316809) > weight (0.25) 2023-11-08T05:02:40.044 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.5227313740122345) > weight (0.25) 2023-11-08T05:02:40.044 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.3 with skip (0.9371770572255792) > weight (0.25) 2023-11-08T05:02:40.045 INFO:tasks.rbd_mirror_thrash:waiting for 97 secs before thrashing 2023-11-08T05:02:44.043 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.31442907287981425) > weight (0.25) 2023-11-08T05:02:44.043 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.42283657330842095) > weight (0.25) 2023-11-08T05:02:44.043 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.5612842561985125) > weight (0.25) 2023-11-08T05:02:44.044 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.3 with skip (0.6726073713372324) > weight (0.25) 2023-11-08T05:02:44.044 INFO:tasks.rbd_mirror_thrash:waiting for 72 secs before thrashing 2023-11-08T05:03:09.566 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_image_replay_started cluster1 mirror test 2023-11-08T05:03:09.567 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:03:09.567 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:03:09.567 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T05:03:09.568 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2023-11-08T05:03:09.568 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:03:09.568 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:03:09.568 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T05:03:09.568 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local state=started 2023-11-08T05:03:09.568 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local s 2023-11-08T05:03:09.569 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-11-08T05:03:09.569 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 1 2023-11-08T05:03:10.567 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test_image_replay_state cluster1 mirror test started 2023-11-08T05:03:10.567 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:03:10.568 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:03:10.568 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T05:03:10.568 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local test_state=started 2023-11-08T05:03:10.568 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local status_result 2023-11-08T05:03:10.568 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local current_state=stopped 2023-11-08T05:03:10.571 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2023-11-08T05:03:10.572 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_instance=cluster1 2023-11-08T05:03:10.572 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T05:03:10.572 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ grep -i state 2023-11-08T05:03:10.572 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1 2023-11-08T05:03:10.572 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance=cluster1 2023-11-08T05:03:10.573 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local loop_instance 2023-11-08T05:03:10.573 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-11-08T05:03:10.573 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sleep 0 2023-11-08T05:03:10.573 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-11-08T05:03:10.573 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ seq 0 3 2023-11-08T05:03:10.574 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T05:03:10.574 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2023-11-08T05:03:10.574 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1:0 2023-11-08T05:03:10.574 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T05:03:10.574 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance 2023-11-08T05:03:10.575 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-11-08T05:03:10.575 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local val=cluster1:0 2023-11-08T05:03:10.575 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_var_name=cluster 2023-11-08T05:03:10.575 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance_var_name=instance 2023-11-08T05:03:10.575 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cluster=cluster1 2023-11-08T05:03:10.576 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ instance=0 2023-11-08T05:03:10.576 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' 0 = cluster1:0 ']' 2023-11-08T05:03:10.576 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval cluster=cluster1 2023-11-08T05:03:10.576 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:03:10.576 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval instance=0 2023-11-08T05:03:10.577 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=0 2023-11-08T05:03:10.577 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-11-08T05:03:10.577 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local local_cluster=cluster1:0 2023-11-08T05:03:10.577 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster=cluster1 2023-11-08T05:03:10.577 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance 2023-11-08T05:03:10.579 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-11-08T05:03:10.579 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local val=cluster1:0 2023-11-08T05:03:10.579 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster_var_name=local_cluster 2023-11-08T05:03:10.579 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance_var_name=instance 2023-11-08T05:03:10.579 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:03:10.580 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=0 2023-11-08T05:03:10.580 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ '[' 0 = cluster1:0 ']' 2023-11-08T05:03:10.580 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval local_cluster=cluster1 2023-11-08T05:03:10.580 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ local_cluster=cluster1 2023-11-08T05:03:10.580 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval instance=0 2023-11-08T05:03:10.581 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ instance=0 2023-11-08T05:03:10.581 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-11-08T05:03:10.589 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-08T05:03:10.589 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-08T05:03:10.589 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-08T05:03:10.589 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2023-11-08T05:03:10.685 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-11-08T05:03:10.685 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd mirror status : get status for rbd mirror 2023-11-08T05:03:10.686 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:admin_socket: invalid command 2023-11-08T05:03:10.688 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T05:03:10.689 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2023-11-08T05:03:10.689 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1:1 2023-11-08T05:03:10.689 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T05:03:10.689 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance 2023-11-08T05:03:10.689 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-11-08T05:03:10.690 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local val=cluster1:1 2023-11-08T05:03:10.690 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_var_name=cluster 2023-11-08T05:03:10.690 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance_var_name=instance 2023-11-08T05:03:10.690 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cluster=cluster1 2023-11-08T05:03:10.690 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ instance=1 2023-11-08T05:03:10.691 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' 1 = cluster1:1 ']' 2023-11-08T05:03:10.691 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval cluster=cluster1 2023-11-08T05:03:10.691 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:03:10.691 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval instance=1 2023-11-08T05:03:10.691 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=1 2023-11-08T05:03:10.692 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-11-08T05:03:10.692 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local local_cluster=cluster1:1 2023-11-08T05:03:10.692 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster=cluster1 2023-11-08T05:03:10.692 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance 2023-11-08T05:03:10.692 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-11-08T05:03:10.693 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local val=cluster1:1 2023-11-08T05:03:10.693 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster_var_name=local_cluster 2023-11-08T05:03:10.693 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance_var_name=instance 2023-11-08T05:03:10.693 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:03:10.693 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=1 2023-11-08T05:03:10.694 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ '[' 1 = cluster1:1 ']' 2023-11-08T05:03:10.694 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval local_cluster=cluster1 2023-11-08T05:03:10.694 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ local_cluster=cluster1 2023-11-08T05:03:10.694 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval instance=1 2023-11-08T05:03:10.694 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ instance=1 2023-11-08T05:03:10.695 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-11-08T05:03:10.702 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-08T05:03:10.703 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-08T05:03:10.703 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-08T05:03:10.703 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2023-11-08T05:03:10.791 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-11-08T05:03:10.791 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd mirror status : get status for rbd mirror 2023-11-08T05:03:10.791 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:admin_socket: invalid command 2023-11-08T05:03:10.794 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T05:03:10.794 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2023-11-08T05:03:10.795 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1:2 2023-11-08T05:03:10.795 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T05:03:10.795 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance 2023-11-08T05:03:10.795 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-11-08T05:03:10.795 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local val=cluster1:2 2023-11-08T05:03:10.796 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_var_name=cluster 2023-11-08T05:03:10.796 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance_var_name=instance 2023-11-08T05:03:10.796 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cluster=cluster1 2023-11-08T05:03:10.796 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ instance=2 2023-11-08T05:03:10.796 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' 2 = cluster1:2 ']' 2023-11-08T05:03:10.797 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval cluster=cluster1 2023-11-08T05:03:10.797 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:03:10.797 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval instance=2 2023-11-08T05:03:10.797 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=2 2023-11-08T05:03:10.797 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-11-08T05:03:10.798 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local local_cluster=cluster1:2 2023-11-08T05:03:10.798 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster=cluster1 2023-11-08T05:03:10.798 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance 2023-11-08T05:03:10.798 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-11-08T05:03:10.798 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local val=cluster1:2 2023-11-08T05:03:10.799 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster_var_name=local_cluster 2023-11-08T05:03:10.799 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance_var_name=instance 2023-11-08T05:03:10.799 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:03:10.799 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=2 2023-11-08T05:03:10.799 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ '[' 2 = cluster1:2 ']' 2023-11-08T05:03:10.800 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval local_cluster=cluster1 2023-11-08T05:03:10.800 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ local_cluster=cluster1 2023-11-08T05:03:10.800 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval instance=2 2023-11-08T05:03:10.800 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ instance=2 2023-11-08T05:03:10.800 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-11-08T05:03:10.808 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-08T05:03:10.808 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-08T05:03:10.808 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-08T05:03:10.808 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2023-11-08T05:03:10.906 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ return 0 2023-11-08T05:03:10.906 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ status_result=' "state": "Replaying"' 2023-11-08T05:03:10.907 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ echo ' "state": "Replaying"' 2023-11-08T05:03:10.907 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ grep -i Replaying 2023-11-08T05:03:10.907 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: "state": "Replaying" 2023-11-08T05:03:10.909 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ current_state=started 2023-11-08T05:03:10.910 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test started = started 2023-11-08T05:03:10.910 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:03:10.910 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2023-11-08T05:03:10.910 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local local_cluster=cluster1 2023-11-08T05:03:10.911 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:03:10.911 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:03:10.911 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T05:03:10.911 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' snapshot = journal ']' 2023-11-08T05:03:10.911 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' snapshot = snapshot ']' 2023-11-08T05:03:10.912 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2023-11-08T05:03:10.912 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local local_cluster=cluster1 2023-11-08T05:03:10.912 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:03:10.912 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:03:10.912 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T05:03:10.913 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ mkfname cluster2-mirror-test.status 2023-11-08T05:03:10.913 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ echo cluster2-mirror-test.status 2023-11-08T05:03:10.913 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -e 's|[/ ]|_|g' 2023-11-08T05:03:10.913 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local status_log=/tmp/tmp.Ludk8ikCAB/cluster2-mirror-test.status 2023-11-08T05:03:10.913 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ mkfname cluster1-mirror-test.status 2023-11-08T05:03:10.914 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ echo cluster1-mirror-test.status 2023-11-08T05:03:10.914 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -e 's|[/ ]|_|g' 2023-11-08T05:03:10.915 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local local_status_log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.status 2023-11-08T05:03:10.915 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ mirror_image_snapshot cluster2 mirror test 2023-11-08T05:03:10.915 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:03:10.915 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:03:10.916 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T05:03:10.916 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2023-11-08T05:03:12.575 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:Snapshot ID: 41 2023-11-08T05:03:12.599 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.Ludk8ikCAB/cluster2-mirror-test.status 2023-11-08T05:03:12.599 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:03:12.599 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:03:12.599 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T05:03:12.600 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster2-mirror-test.status 2023-11-08T05:03:12.600 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2023-11-08T05:03:12.600 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T05:03:12.652 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-11-08T05:03:12.654 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snapshot_id=41 2023-11-08T05:03:12.654 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ true 2023-11-08T05:03:12.655 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-11-08T05:03:12.655 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0.2 2023-11-08T05:03:12.856 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.status 2023-11-08T05:03:12.856 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:03:12.856 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:03:12.857 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T05:03:12.857 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.status 2023-11-08T05:03:12.857 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-11-08T05:03:12.857 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T05:03:12.901 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ true 2023-11-08T05:03:12.902 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-08T05:03:12.902 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:-:1.1: Document is empty 2023-11-08T05:03:12.902 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: 2023-11-08T05:03:12.903 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:^ 2023-11-08T05:03:12.903 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local primary_snapshot_id= 2023-11-08T05:03:12.904 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test 41 = '' 2023-11-08T05:03:12.904 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-11-08T05:03:12.904 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0.4 2023-11-08T05:03:13.304 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.status 2023-11-08T05:03:13.304 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:03:13.305 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:03:13.305 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T05:03:13.305 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.status 2023-11-08T05:03:13.305 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-11-08T05:03:13.306 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T05:03:13.346 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ true 2023-11-08T05:03:13.347 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-08T05:03:13.348 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:-:1.1: Document is empty 2023-11-08T05:03:13.348 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: 2023-11-08T05:03:13.348 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:^ 2023-11-08T05:03:13.349 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local primary_snapshot_id= 2023-11-08T05:03:13.349 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test 41 = '' 2023-11-08T05:03:13.349 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-11-08T05:03:13.349 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0.8 2023-11-08T05:03:14.150 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.status 2023-11-08T05:03:14.150 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:03:14.151 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:03:14.151 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T05:03:14.151 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.status 2023-11-08T05:03:14.152 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-11-08T05:03:14.152 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T05:03:14.197 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ true 2023-11-08T05:03:14.198 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-08T05:03:14.199 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:-:1.1: Document is empty 2023-11-08T05:03:14.199 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: 2023-11-08T05:03:14.200 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:^ 2023-11-08T05:03:14.201 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local primary_snapshot_id= 2023-11-08T05:03:14.201 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test 41 = '' 2023-11-08T05:03:14.201 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-11-08T05:03:14.202 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 1.6 2023-11-08T05:03:15.801 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.status 2023-11-08T05:03:15.801 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:03:15.802 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:03:15.802 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T05:03:15.802 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.status 2023-11-08T05:03:15.802 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-11-08T05:03:15.802 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T05:03:15.847 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-08T05:03:15.850 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local primary_snapshot_id=41 2023-11-08T05:03:15.850 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test 41 = 41 2023-11-08T05:03:15.850 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:03:15.850 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_snap_present cluster1 mirror test snap10 2023-11-08T05:03:15.851 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:03:15.851 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:03:15.851 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T05:03:15.851 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap_name=snap10 2023-11-08T05:03:15.851 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local s 2023-11-08T05:03:15.852 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-11-08T05:03:15.852 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 1 2023-11-08T05:03:16.850 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 info mirror/test@snap10 2023-11-08T05:03:16.894 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:rbd image 'test': 2023-11-08T05:03:16.894 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: size 512 MiB in 128 objects 2023-11-08T05:03:16.894 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: order 22 (4 MiB objects) 2023-11-08T05:03:16.894 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: snapshot_count: 20 2023-11-08T05:03:16.894 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: id: 10afad8fe72b 2023-11-08T05:03:16.895 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: block_name_prefix: rbd_data.10afad8fe72b 2023-11-08T05:03:16.895 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: format: 2 2023-11-08T05:03:16.895 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: features: layering, exclusive-lock, non-primary 2023-11-08T05:03:16.895 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: op_features: 2023-11-08T05:03:16.895 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: flags: 2023-11-08T05:03:16.895 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: create_timestamp: Wed Nov 8 04:54:28 2023 2023-11-08T05:03:16.896 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: access_timestamp: Wed Nov 8 05:01:31 2023 2023-11-08T05:03:16.896 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: modify_timestamp: Wed Nov 8 04:54:28 2023 2023-11-08T05:03:16.896 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: protected: False 2023-11-08T05:03:16.896 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring state: enabled 2023-11-08T05:03:16.896 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring mode: snapshot 2023-11-08T05:03:16.897 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring global id: 88ced7f9-2739-4a27-ad7a-30edec05edb1 2023-11-08T05:03:16.897 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring primary: false 2023-11-08T05:03:16.898 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:03:16.898 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' -n snap9-clean ']' 2023-11-08T05:03:16.899 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ compare_image_snaps mirror test snap9-clean 2023-11-08T05:03:16.899 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:03:16.899 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T05:03:16.899 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap_name=snap9-clean 2023-11-08T05:03:16.899 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local ret=0 2023-11-08T05:03:16.900 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local rmt_export=/tmp/tmp.Ludk8ikCAB/cluster2-mirror-test.export 2023-11-08T05:03:16.900 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local loc_export=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.export 2023-11-08T05:03:16.900 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rm -f /tmp/tmp.Ludk8ikCAB/cluster2-mirror-test.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.export 2023-11-08T05:03:16.900 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 -p mirror export test@snap9-clean /tmp/tmp.Ludk8ikCAB/cluster2-mirror-test.export 2023-11-08T05:03:17.506 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-11-08T05:03:17.509 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 -p mirror export test@snap9-clean /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.export 2023-11-08T05:03:18.276 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-11-08T05:03:18.279 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ cmp /tmp/tmp.Ludk8ikCAB/cluster2-mirror-test.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.export 2023-11-08T05:03:18.492 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rm -f /tmp/tmp.Ludk8ikCAB/cluster2-mirror-test.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.export 2023-11-08T05:03:18.634 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:03:18.634 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ compare_image_snaps mirror test snap10 2023-11-08T05:03:18.634 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:03:18.635 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T05:03:18.635 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap_name=snap10 2023-11-08T05:03:18.635 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local ret=0 2023-11-08T05:03:18.635 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local rmt_export=/tmp/tmp.Ludk8ikCAB/cluster2-mirror-test.export 2023-11-08T05:03:18.635 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local loc_export=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.export 2023-11-08T05:03:18.636 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rm -f /tmp/tmp.Ludk8ikCAB/cluster2-mirror-test.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.export 2023-11-08T05:03:18.636 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 -p mirror export test@snap10 /tmp/tmp.Ludk8ikCAB/cluster2-mirror-test.export 2023-11-08T05:03:19.259 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-11-08T05:03:19.262 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 -p mirror export test@snap10 /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.export 2023-11-08T05:03:20.040 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-11-08T05:03:20.043 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ cmp /tmp/tmp.Ludk8ikCAB/cluster2-mirror-test.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.export 2023-11-08T05:03:20.258 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rm -f /tmp/tmp.Ludk8ikCAB/cluster2-mirror-test.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.export 2023-11-08T05:03:20.401 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:03:20.401 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ clean_snap_name=snap10-clean 2023-11-08T05:03:20.401 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ create_snap cluster2 mirror test snap10-clean 2023-11-08T05:03:20.401 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:03:20.401 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:03:20.402 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T05:03:20.402 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap_name=snap10-clean 2023-11-08T05:03:20.402 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap10-clean --debug-rbd=20 --debug-journaler=20 2023-11-08T05:03:20.703 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_image_replay_started cluster1 mirror test 2023-11-08T05:03:20.703 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:03:20.703 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:03:20.704 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T05:03:20.704 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2023-11-08T05:03:20.704 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:03:20.704 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:03:20.704 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T05:03:20.705 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local state=started 2023-11-08T05:03:20.705 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local s 2023-11-08T05:03:20.705 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-11-08T05:03:20.705 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 1 2023-11-08T05:03:21.705 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test_image_replay_state cluster1 mirror test started 2023-11-08T05:03:21.705 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:03:21.706 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:03:21.706 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T05:03:21.706 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local test_state=started 2023-11-08T05:03:21.706 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local status_result 2023-11-08T05:03:21.707 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local current_state=stopped 2023-11-08T05:03:21.707 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2023-11-08T05:03:21.707 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_instance=cluster1 2023-11-08T05:03:21.707 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T05:03:21.707 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ grep -i state 2023-11-08T05:03:21.708 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1 2023-11-08T05:03:21.708 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance=cluster1 2023-11-08T05:03:21.708 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local loop_instance 2023-11-08T05:03:21.708 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-11-08T05:03:21.708 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sleep 0 2023-11-08T05:03:21.709 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-11-08T05:03:21.712 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ seq 0 3 2023-11-08T05:03:21.712 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T05:03:21.712 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2023-11-08T05:03:21.712 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1:0 2023-11-08T05:03:21.713 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T05:03:21.713 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance 2023-11-08T05:03:21.713 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-11-08T05:03:21.713 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local val=cluster1:0 2023-11-08T05:03:21.713 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_var_name=cluster 2023-11-08T05:03:21.714 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance_var_name=instance 2023-11-08T05:03:21.714 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cluster=cluster1 2023-11-08T05:03:21.714 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ instance=0 2023-11-08T05:03:21.714 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' 0 = cluster1:0 ']' 2023-11-08T05:03:21.714 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval cluster=cluster1 2023-11-08T05:03:21.715 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:03:21.715 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval instance=0 2023-11-08T05:03:21.715 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=0 2023-11-08T05:03:21.715 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-11-08T05:03:21.715 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local local_cluster=cluster1:0 2023-11-08T05:03:21.716 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster=cluster1 2023-11-08T05:03:21.716 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance 2023-11-08T05:03:21.716 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-11-08T05:03:21.716 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local val=cluster1:0 2023-11-08T05:03:21.716 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster_var_name=local_cluster 2023-11-08T05:03:21.716 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance_var_name=instance 2023-11-08T05:03:21.717 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:03:21.717 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=0 2023-11-08T05:03:21.717 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ '[' 0 = cluster1:0 ']' 2023-11-08T05:03:21.717 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval local_cluster=cluster1 2023-11-08T05:03:21.717 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ local_cluster=cluster1 2023-11-08T05:03:21.718 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval instance=0 2023-11-08T05:03:21.718 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ instance=0 2023-11-08T05:03:21.718 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-11-08T05:03:21.725 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-08T05:03:21.725 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-08T05:03:21.725 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-08T05:03:21.725 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2023-11-08T05:03:21.818 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-11-08T05:03:21.818 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd mirror status : get status for rbd mirror 2023-11-08T05:03:21.819 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:admin_socket: invalid command 2023-11-08T05:03:21.821 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T05:03:21.822 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2023-11-08T05:03:21.822 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1:1 2023-11-08T05:03:21.822 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T05:03:21.822 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance 2023-11-08T05:03:21.822 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-11-08T05:03:21.823 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local val=cluster1:1 2023-11-08T05:03:21.823 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_var_name=cluster 2023-11-08T05:03:21.823 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance_var_name=instance 2023-11-08T05:03:21.823 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cluster=cluster1 2023-11-08T05:03:21.823 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ instance=1 2023-11-08T05:03:21.824 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' 1 = cluster1:1 ']' 2023-11-08T05:03:21.824 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval cluster=cluster1 2023-11-08T05:03:21.824 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:03:21.824 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval instance=1 2023-11-08T05:03:21.825 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=1 2023-11-08T05:03:21.825 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-11-08T05:03:21.825 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local local_cluster=cluster1:1 2023-11-08T05:03:21.825 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster=cluster1 2023-11-08T05:03:21.825 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance 2023-11-08T05:03:21.826 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-11-08T05:03:21.826 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local val=cluster1:1 2023-11-08T05:03:21.826 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster_var_name=local_cluster 2023-11-08T05:03:21.826 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance_var_name=instance 2023-11-08T05:03:21.826 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:03:21.827 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=1 2023-11-08T05:03:21.827 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ '[' 1 = cluster1:1 ']' 2023-11-08T05:03:21.827 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval local_cluster=cluster1 2023-11-08T05:03:21.827 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ local_cluster=cluster1 2023-11-08T05:03:21.828 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval instance=1 2023-11-08T05:03:21.828 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ instance=1 2023-11-08T05:03:21.828 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-11-08T05:03:21.835 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-08T05:03:21.835 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-08T05:03:21.836 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-08T05:03:21.836 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2023-11-08T05:03:21.924 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-11-08T05:03:21.925 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd mirror status : get status for rbd mirror 2023-11-08T05:03:21.925 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:admin_socket: invalid command 2023-11-08T05:03:21.928 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T05:03:21.928 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2023-11-08T05:03:21.928 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1:2 2023-11-08T05:03:21.928 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T05:03:21.929 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance 2023-11-08T05:03:21.929 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-11-08T05:03:21.929 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local val=cluster1:2 2023-11-08T05:03:21.929 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_var_name=cluster 2023-11-08T05:03:21.929 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance_var_name=instance 2023-11-08T05:03:21.930 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cluster=cluster1 2023-11-08T05:03:21.930 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ instance=2 2023-11-08T05:03:21.930 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' 2 = cluster1:2 ']' 2023-11-08T05:03:21.930 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval cluster=cluster1 2023-11-08T05:03:21.930 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:03:21.931 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval instance=2 2023-11-08T05:03:21.931 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=2 2023-11-08T05:03:21.931 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-11-08T05:03:21.931 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local local_cluster=cluster1:2 2023-11-08T05:03:21.931 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster=cluster1 2023-11-08T05:03:21.931 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance 2023-11-08T05:03:21.932 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-11-08T05:03:21.932 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local val=cluster1:2 2023-11-08T05:03:21.932 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster_var_name=local_cluster 2023-11-08T05:03:21.932 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance_var_name=instance 2023-11-08T05:03:21.932 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:03:21.933 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=2 2023-11-08T05:03:21.933 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ '[' 2 = cluster1:2 ']' 2023-11-08T05:03:21.933 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval local_cluster=cluster1 2023-11-08T05:03:21.933 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ local_cluster=cluster1 2023-11-08T05:03:21.933 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval instance=2 2023-11-08T05:03:21.934 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ instance=2 2023-11-08T05:03:21.934 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-11-08T05:03:21.942 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-08T05:03:21.943 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-08T05:03:21.943 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-08T05:03:21.943 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2023-11-08T05:03:22.042 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ return 0 2023-11-08T05:03:22.043 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ status_result=' "state": "Replaying"' 2023-11-08T05:03:22.043 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ echo ' "state": "Replaying"' 2023-11-08T05:03:22.044 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ grep -i Replaying 2023-11-08T05:03:22.044 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: "state": "Replaying" 2023-11-08T05:03:22.046 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ current_state=started 2023-11-08T05:03:22.047 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test started = started 2023-11-08T05:03:22.047 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:03:22.047 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2023-11-08T05:03:22.047 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local local_cluster=cluster1 2023-11-08T05:03:22.048 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:03:22.048 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:03:22.048 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T05:03:22.048 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' snapshot = journal ']' 2023-11-08T05:03:22.049 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' snapshot = snapshot ']' 2023-11-08T05:03:22.049 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2023-11-08T05:03:22.049 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local local_cluster=cluster1 2023-11-08T05:03:22.049 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:03:22.050 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:03:22.050 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T05:03:22.050 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ mkfname cluster2-mirror-test.status 2023-11-08T05:03:22.051 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ echo cluster2-mirror-test.status 2023-11-08T05:03:22.051 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -e 's|[/ ]|_|g' 2023-11-08T05:03:22.051 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local status_log=/tmp/tmp.Ludk8ikCAB/cluster2-mirror-test.status 2023-11-08T05:03:22.052 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ mkfname cluster1-mirror-test.status 2023-11-08T05:03:22.052 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ echo cluster1-mirror-test.status 2023-11-08T05:03:22.052 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -e 's|[/ ]|_|g' 2023-11-08T05:03:22.054 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local local_status_log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.status 2023-11-08T05:03:22.054 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ mirror_image_snapshot cluster2 mirror test 2023-11-08T05:03:22.054 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:03:22.055 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:03:22.055 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T05:03:22.055 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2023-11-08T05:03:23.566 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:Snapshot ID: 44 2023-11-08T05:03:23.579 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.Ludk8ikCAB/cluster2-mirror-test.status 2023-11-08T05:03:23.580 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:03:23.580 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:03:23.580 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T05:03:23.580 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster2-mirror-test.status 2023-11-08T05:03:23.580 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2023-11-08T05:03:23.581 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T05:03:23.635 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-11-08T05:03:23.638 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snapshot_id=44 2023-11-08T05:03:23.638 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ true 2023-11-08T05:03:23.638 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-11-08T05:03:23.638 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0.2 2023-11-08T05:03:23.839 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.status 2023-11-08T05:03:23.839 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:03:23.840 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:03:23.840 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T05:03:23.840 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.status 2023-11-08T05:03:23.840 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-11-08T05:03:23.840 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T05:03:23.883 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ true 2023-11-08T05:03:23.883 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-08T05:03:23.884 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:-:1.1: Document is empty 2023-11-08T05:03:23.884 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: 2023-11-08T05:03:23.885 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:^ 2023-11-08T05:03:23.885 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local primary_snapshot_id= 2023-11-08T05:03:23.886 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test 44 = '' 2023-11-08T05:03:23.886 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-11-08T05:03:23.886 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0.4 2023-11-08T05:03:24.286 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.status 2023-11-08T05:03:24.287 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:03:24.287 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:03:24.287 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T05:03:24.287 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.status 2023-11-08T05:03:24.287 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-11-08T05:03:24.288 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T05:03:24.335 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ true 2023-11-08T05:03:24.335 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-08T05:03:24.336 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:-:1.1: Document is empty 2023-11-08T05:03:24.337 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: 2023-11-08T05:03:24.337 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:^ 2023-11-08T05:03:24.338 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local primary_snapshot_id= 2023-11-08T05:03:24.338 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test 44 = '' 2023-11-08T05:03:24.338 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-11-08T05:03:24.338 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0.8 2023-11-08T05:03:25.138 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.status 2023-11-08T05:03:25.138 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:03:25.139 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:03:25.139 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T05:03:25.139 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.status 2023-11-08T05:03:25.139 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-11-08T05:03:25.140 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T05:03:25.188 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-08T05:03:25.190 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local primary_snapshot_id=44 2023-11-08T05:03:25.191 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test 44 = 44 2023-11-08T05:03:25.191 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:03:25.191 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_snap_present cluster1 mirror test snap10-clean 2023-11-08T05:03:25.191 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:03:25.191 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:03:25.192 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T05:03:25.192 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap_name=snap10-clean 2023-11-08T05:03:25.192 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local s 2023-11-08T05:03:25.192 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-11-08T05:03:25.192 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 1 2023-11-08T05:03:26.191 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 info mirror/test@snap10-clean 2023-11-08T05:03:26.233 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:rbd image 'test': 2023-11-08T05:03:26.234 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: size 512 MiB in 128 objects 2023-11-08T05:03:26.234 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: order 22 (4 MiB objects) 2023-11-08T05:03:26.234 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: snapshot_count: 21 2023-11-08T05:03:26.234 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: id: 10afad8fe72b 2023-11-08T05:03:26.234 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: block_name_prefix: rbd_data.10afad8fe72b 2023-11-08T05:03:26.235 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: format: 2 2023-11-08T05:03:26.235 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: features: layering, exclusive-lock, non-primary 2023-11-08T05:03:26.235 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: op_features: 2023-11-08T05:03:26.235 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: flags: 2023-11-08T05:03:26.235 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: create_timestamp: Wed Nov 8 04:54:28 2023 2023-11-08T05:03:26.236 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: access_timestamp: Wed Nov 8 05:03:17 2023 2023-11-08T05:03:26.236 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: modify_timestamp: Wed Nov 8 04:54:28 2023 2023-11-08T05:03:26.236 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: protected: False 2023-11-08T05:03:26.236 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring state: enabled 2023-11-08T05:03:26.237 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring mode: snapshot 2023-11-08T05:03:26.237 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring global id: 88ced7f9-2739-4a27-ad7a-30edec05edb1 2023-11-08T05:03:26.237 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring primary: false 2023-11-08T05:03:26.238 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:03:26.239 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ seq 1 10 2023-11-08T05:03:26.239 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for i in `seq 1 10` 2023-11-08T05:03:26.239 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ snap_name=snap1 2023-11-08T05:03:26.239 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ compare_image_snaps mirror test snap1 2023-11-08T05:03:26.240 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:03:26.240 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T05:03:26.240 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap_name=snap1 2023-11-08T05:03:26.240 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local ret=0 2023-11-08T05:03:26.240 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local rmt_export=/tmp/tmp.Ludk8ikCAB/cluster2-mirror-test.export 2023-11-08T05:03:26.241 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local loc_export=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.export 2023-11-08T05:03:26.241 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rm -f /tmp/tmp.Ludk8ikCAB/cluster2-mirror-test.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.export 2023-11-08T05:03:26.241 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 -p mirror export test@snap1 /tmp/tmp.Ludk8ikCAB/cluster2-mirror-test.export 2023-11-08T05:03:26.993 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-11-08T05:03:26.997 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 -p mirror export test@snap1 /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.export 2023-11-08T05:03:27.716 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-11-08T05:03:27.721 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ cmp /tmp/tmp.Ludk8ikCAB/cluster2-mirror-test.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.export 2023-11-08T05:03:27.932 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rm -f /tmp/tmp.Ludk8ikCAB/cluster2-mirror-test.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.export 2023-11-08T05:03:28.071 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:03:28.071 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ snap_name=snap1-clean 2023-11-08T05:03:28.071 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ compare_image_snaps mirror test snap1-clean 2023-11-08T05:03:28.072 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:03:28.072 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T05:03:28.072 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap_name=snap1-clean 2023-11-08T05:03:28.072 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local ret=0 2023-11-08T05:03:28.072 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local rmt_export=/tmp/tmp.Ludk8ikCAB/cluster2-mirror-test.export 2023-11-08T05:03:28.073 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local loc_export=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.export 2023-11-08T05:03:28.073 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rm -f /tmp/tmp.Ludk8ikCAB/cluster2-mirror-test.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.export 2023-11-08T05:03:28.073 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 -p mirror export test@snap1-clean /tmp/tmp.Ludk8ikCAB/cluster2-mirror-test.export 2023-11-08T05:03:28.620 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-11-08T05:03:28.624 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 -p mirror export test@snap1-clean /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.export 2023-11-08T05:03:29.370 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-11-08T05:03:29.374 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ cmp /tmp/tmp.Ludk8ikCAB/cluster2-mirror-test.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.export 2023-11-08T05:03:29.588 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rm -f /tmp/tmp.Ludk8ikCAB/cluster2-mirror-test.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.export 2023-11-08T05:03:29.728 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:03:29.729 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for i in `seq 1 10` 2023-11-08T05:03:29.729 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ snap_name=snap2 2023-11-08T05:03:29.729 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ compare_image_snaps mirror test snap2 2023-11-08T05:03:29.729 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:03:29.730 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T05:03:29.730 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap_name=snap2 2023-11-08T05:03:29.730 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local ret=0 2023-11-08T05:03:29.730 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local rmt_export=/tmp/tmp.Ludk8ikCAB/cluster2-mirror-test.export 2023-11-08T05:03:29.730 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local loc_export=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.export 2023-11-08T05:03:29.731 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rm -f /tmp/tmp.Ludk8ikCAB/cluster2-mirror-test.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.export 2023-11-08T05:03:29.731 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 -p mirror export test@snap2 /tmp/tmp.Ludk8ikCAB/cluster2-mirror-test.export 2023-11-08T05:03:30.364 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-11-08T05:03:30.367 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 -p mirror export test@snap2 /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.export 2023-11-08T05:03:31.084 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-11-08T05:03:31.089 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ cmp /tmp/tmp.Ludk8ikCAB/cluster2-mirror-test.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.export 2023-11-08T05:03:31.304 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rm -f /tmp/tmp.Ludk8ikCAB/cluster2-mirror-test.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.export 2023-11-08T05:03:31.448 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:03:31.448 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ snap_name=snap2-clean 2023-11-08T05:03:31.448 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ compare_image_snaps mirror test snap2-clean 2023-11-08T05:03:31.449 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:03:31.449 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T05:03:31.449 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap_name=snap2-clean 2023-11-08T05:03:31.449 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local ret=0 2023-11-08T05:03:31.450 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local rmt_export=/tmp/tmp.Ludk8ikCAB/cluster2-mirror-test.export 2023-11-08T05:03:31.450 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local loc_export=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.export 2023-11-08T05:03:31.450 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rm -f /tmp/tmp.Ludk8ikCAB/cluster2-mirror-test.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.export 2023-11-08T05:03:31.450 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 -p mirror export test@snap2-clean /tmp/tmp.Ludk8ikCAB/cluster2-mirror-test.export 2023-11-08T05:03:32.048 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-11-08T05:03:32.051 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 -p mirror export test@snap2-clean /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.export 2023-11-08T05:03:32.853 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-11-08T05:03:32.856 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ cmp /tmp/tmp.Ludk8ikCAB/cluster2-mirror-test.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.export 2023-11-08T05:03:33.067 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rm -f /tmp/tmp.Ludk8ikCAB/cluster2-mirror-test.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.export 2023-11-08T05:03:33.215 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:03:33.216 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for i in `seq 1 10` 2023-11-08T05:03:33.216 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ snap_name=snap3 2023-11-08T05:03:33.216 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ compare_image_snaps mirror test snap3 2023-11-08T05:03:33.216 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:03:33.217 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T05:03:33.217 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap_name=snap3 2023-11-08T05:03:33.217 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local ret=0 2023-11-08T05:03:33.217 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local rmt_export=/tmp/tmp.Ludk8ikCAB/cluster2-mirror-test.export 2023-11-08T05:03:33.217 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local loc_export=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.export 2023-11-08T05:03:33.218 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rm -f /tmp/tmp.Ludk8ikCAB/cluster2-mirror-test.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.export 2023-11-08T05:03:33.218 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 -p mirror export test@snap3 /tmp/tmp.Ludk8ikCAB/cluster2-mirror-test.export 2023-11-08T05:03:33.823 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-11-08T05:03:33.826 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 -p mirror export test@snap3 /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.export 2023-11-08T05:03:34.639 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-11-08T05:03:34.642 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ cmp /tmp/tmp.Ludk8ikCAB/cluster2-mirror-test.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.export 2023-11-08T05:03:34.856 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rm -f /tmp/tmp.Ludk8ikCAB/cluster2-mirror-test.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.export 2023-11-08T05:03:34.999 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:03:34.999 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ snap_name=snap3-clean 2023-11-08T05:03:35.000 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ compare_image_snaps mirror test snap3-clean 2023-11-08T05:03:35.000 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:03:35.000 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T05:03:35.000 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap_name=snap3-clean 2023-11-08T05:03:35.001 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local ret=0 2023-11-08T05:03:35.001 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local rmt_export=/tmp/tmp.Ludk8ikCAB/cluster2-mirror-test.export 2023-11-08T05:03:35.001 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local loc_export=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.export 2023-11-08T05:03:35.001 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rm -f /tmp/tmp.Ludk8ikCAB/cluster2-mirror-test.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.export 2023-11-08T05:03:35.001 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 -p mirror export test@snap3-clean /tmp/tmp.Ludk8ikCAB/cluster2-mirror-test.export 2023-11-08T05:03:35.645 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-11-08T05:03:35.648 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 -p mirror export test@snap3-clean /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.export 2023-11-08T05:03:36.445 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-11-08T05:03:36.449 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ cmp /tmp/tmp.Ludk8ikCAB/cluster2-mirror-test.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.export 2023-11-08T05:03:36.660 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rm -f /tmp/tmp.Ludk8ikCAB/cluster2-mirror-test.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.export 2023-11-08T05:03:36.804 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:03:36.804 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for i in `seq 1 10` 2023-11-08T05:03:36.804 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ snap_name=snap4 2023-11-08T05:03:36.805 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ compare_image_snaps mirror test snap4 2023-11-08T05:03:36.805 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:03:36.805 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T05:03:36.805 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap_name=snap4 2023-11-08T05:03:36.805 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local ret=0 2023-11-08T05:03:36.806 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local rmt_export=/tmp/tmp.Ludk8ikCAB/cluster2-mirror-test.export 2023-11-08T05:03:36.806 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local loc_export=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.export 2023-11-08T05:03:36.806 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rm -f /tmp/tmp.Ludk8ikCAB/cluster2-mirror-test.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.export 2023-11-08T05:03:36.806 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 -p mirror export test@snap4 /tmp/tmp.Ludk8ikCAB/cluster2-mirror-test.export 2023-11-08T05:03:37.401 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-11-08T05:03:37.405 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 -p mirror export test@snap4 /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.export 2023-11-08T05:03:38.182 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-11-08T05:03:38.186 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ cmp /tmp/tmp.Ludk8ikCAB/cluster2-mirror-test.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.export 2023-11-08T05:03:38.402 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rm -f /tmp/tmp.Ludk8ikCAB/cluster2-mirror-test.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.export 2023-11-08T05:03:38.550 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:03:38.550 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ snap_name=snap4-clean 2023-11-08T05:03:38.550 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ compare_image_snaps mirror test snap4-clean 2023-11-08T05:03:38.551 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:03:38.551 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T05:03:38.551 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap_name=snap4-clean 2023-11-08T05:03:38.551 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local ret=0 2023-11-08T05:03:38.551 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local rmt_export=/tmp/tmp.Ludk8ikCAB/cluster2-mirror-test.export 2023-11-08T05:03:38.552 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local loc_export=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.export 2023-11-08T05:03:38.552 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rm -f /tmp/tmp.Ludk8ikCAB/cluster2-mirror-test.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.export 2023-11-08T05:03:38.552 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 -p mirror export test@snap4-clean /tmp/tmp.Ludk8ikCAB/cluster2-mirror-test.export 2023-11-08T05:03:39.150 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-11-08T05:03:39.153 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 -p mirror export test@snap4-clean /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.export 2023-11-08T05:03:39.955 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-11-08T05:03:39.958 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ cmp /tmp/tmp.Ludk8ikCAB/cluster2-mirror-test.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.export 2023-11-08T05:03:40.171 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rm -f /tmp/tmp.Ludk8ikCAB/cluster2-mirror-test.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.export 2023-11-08T05:03:40.317 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:03:40.317 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for i in `seq 1 10` 2023-11-08T05:03:40.317 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ snap_name=snap5 2023-11-08T05:03:40.317 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ compare_image_snaps mirror test snap5 2023-11-08T05:03:40.318 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:03:40.318 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T05:03:40.318 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap_name=snap5 2023-11-08T05:03:40.318 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local ret=0 2023-11-08T05:03:40.318 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local rmt_export=/tmp/tmp.Ludk8ikCAB/cluster2-mirror-test.export 2023-11-08T05:03:40.319 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local loc_export=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.export 2023-11-08T05:03:40.319 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rm -f /tmp/tmp.Ludk8ikCAB/cluster2-mirror-test.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.export 2023-11-08T05:03:40.319 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 -p mirror export test@snap5 /tmp/tmp.Ludk8ikCAB/cluster2-mirror-test.export 2023-11-08T05:03:40.913 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-11-08T05:03:40.916 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 -p mirror export test@snap5 /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.export 2023-11-08T05:03:41.678 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-11-08T05:03:41.681 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ cmp /tmp/tmp.Ludk8ikCAB/cluster2-mirror-test.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.export 2023-11-08T05:03:41.895 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rm -f /tmp/tmp.Ludk8ikCAB/cluster2-mirror-test.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.export 2023-11-08T05:03:42.038 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:03:42.038 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ snap_name=snap5-clean 2023-11-08T05:03:42.038 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ compare_image_snaps mirror test snap5-clean 2023-11-08T05:03:42.038 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:03:42.039 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T05:03:42.039 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap_name=snap5-clean 2023-11-08T05:03:42.039 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local ret=0 2023-11-08T05:03:42.039 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local rmt_export=/tmp/tmp.Ludk8ikCAB/cluster2-mirror-test.export 2023-11-08T05:03:42.039 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local loc_export=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.export 2023-11-08T05:03:42.040 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rm -f /tmp/tmp.Ludk8ikCAB/cluster2-mirror-test.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.export 2023-11-08T05:03:42.040 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 -p mirror export test@snap5-clean /tmp/tmp.Ludk8ikCAB/cluster2-mirror-test.export 2023-11-08T05:03:42.658 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-11-08T05:03:42.661 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 -p mirror export test@snap5-clean /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.export 2023-11-08T05:03:43.447 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-11-08T05:03:43.451 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ cmp /tmp/tmp.Ludk8ikCAB/cluster2-mirror-test.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.export 2023-11-08T05:03:43.667 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rm -f /tmp/tmp.Ludk8ikCAB/cluster2-mirror-test.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.export 2023-11-08T05:03:43.811 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:03:43.812 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for i in `seq 1 10` 2023-11-08T05:03:43.812 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ snap_name=snap6 2023-11-08T05:03:43.812 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ compare_image_snaps mirror test snap6 2023-11-08T05:03:43.813 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:03:43.813 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T05:03:43.813 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap_name=snap6 2023-11-08T05:03:43.813 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local ret=0 2023-11-08T05:03:43.813 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local rmt_export=/tmp/tmp.Ludk8ikCAB/cluster2-mirror-test.export 2023-11-08T05:03:43.814 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local loc_export=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.export 2023-11-08T05:03:43.814 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rm -f /tmp/tmp.Ludk8ikCAB/cluster2-mirror-test.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.export 2023-11-08T05:03:43.814 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 -p mirror export test@snap6 /tmp/tmp.Ludk8ikCAB/cluster2-mirror-test.export 2023-11-08T05:03:44.416 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-11-08T05:03:44.419 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 -p mirror export test@snap6 /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.export 2023-11-08T05:03:45.164 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-11-08T05:03:45.167 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ cmp /tmp/tmp.Ludk8ikCAB/cluster2-mirror-test.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.export 2023-11-08T05:03:45.383 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rm -f /tmp/tmp.Ludk8ikCAB/cluster2-mirror-test.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.export 2023-11-08T05:03:45.530 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:03:45.530 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ snap_name=snap6-clean 2023-11-08T05:03:45.531 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ compare_image_snaps mirror test snap6-clean 2023-11-08T05:03:45.531 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:03:45.531 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T05:03:45.531 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap_name=snap6-clean 2023-11-08T05:03:45.531 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local ret=0 2023-11-08T05:03:45.532 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local rmt_export=/tmp/tmp.Ludk8ikCAB/cluster2-mirror-test.export 2023-11-08T05:03:45.532 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local loc_export=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.export 2023-11-08T05:03:45.532 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rm -f /tmp/tmp.Ludk8ikCAB/cluster2-mirror-test.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.export 2023-11-08T05:03:45.532 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 -p mirror export test@snap6-clean /tmp/tmp.Ludk8ikCAB/cluster2-mirror-test.export 2023-11-08T05:03:46.140 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-11-08T05:03:46.144 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 -p mirror export test@snap6-clean /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.export 2023-11-08T05:03:46.813 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-11-08T05:03:46.816 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ cmp /tmp/tmp.Ludk8ikCAB/cluster2-mirror-test.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.export 2023-11-08T05:03:47.030 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rm -f /tmp/tmp.Ludk8ikCAB/cluster2-mirror-test.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.export 2023-11-08T05:03:47.175 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:03:47.175 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for i in `seq 1 10` 2023-11-08T05:03:47.175 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ snap_name=snap7 2023-11-08T05:03:47.175 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ compare_image_snaps mirror test snap7 2023-11-08T05:03:47.176 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:03:47.176 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T05:03:47.176 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap_name=snap7 2023-11-08T05:03:47.176 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local ret=0 2023-11-08T05:03:47.177 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local rmt_export=/tmp/tmp.Ludk8ikCAB/cluster2-mirror-test.export 2023-11-08T05:03:47.177 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local loc_export=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.export 2023-11-08T05:03:47.177 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rm -f /tmp/tmp.Ludk8ikCAB/cluster2-mirror-test.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.export 2023-11-08T05:03:47.177 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 -p mirror export test@snap7 /tmp/tmp.Ludk8ikCAB/cluster2-mirror-test.export 2023-11-08T05:03:47.799 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-11-08T05:03:47.802 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 -p mirror export test@snap7 /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.export 2023-11-08T05:03:48.629 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-11-08T05:03:48.632 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ cmp /tmp/tmp.Ludk8ikCAB/cluster2-mirror-test.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.export 2023-11-08T05:03:48.844 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rm -f /tmp/tmp.Ludk8ikCAB/cluster2-mirror-test.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.export 2023-11-08T05:03:48.988 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:03:48.988 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ snap_name=snap7-clean 2023-11-08T05:03:48.989 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ compare_image_snaps mirror test snap7-clean 2023-11-08T05:03:48.989 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:03:48.989 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T05:03:48.989 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap_name=snap7-clean 2023-11-08T05:03:48.989 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local ret=0 2023-11-08T05:03:48.990 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local rmt_export=/tmp/tmp.Ludk8ikCAB/cluster2-mirror-test.export 2023-11-08T05:03:48.990 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local loc_export=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.export 2023-11-08T05:03:48.990 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rm -f /tmp/tmp.Ludk8ikCAB/cluster2-mirror-test.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.export 2023-11-08T05:03:48.990 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 -p mirror export test@snap7-clean /tmp/tmp.Ludk8ikCAB/cluster2-mirror-test.export 2023-11-08T05:03:49.597 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-11-08T05:03:49.600 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 -p mirror export test@snap7-clean /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.export 2023-11-08T05:03:50.409 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-11-08T05:03:50.412 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ cmp /tmp/tmp.Ludk8ikCAB/cluster2-mirror-test.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.export 2023-11-08T05:03:50.626 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rm -f /tmp/tmp.Ludk8ikCAB/cluster2-mirror-test.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.export 2023-11-08T05:03:50.770 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:03:50.770 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for i in `seq 1 10` 2023-11-08T05:03:50.770 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ snap_name=snap8 2023-11-08T05:03:50.770 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ compare_image_snaps mirror test snap8 2023-11-08T05:03:50.771 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:03:50.771 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T05:03:50.771 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap_name=snap8 2023-11-08T05:03:50.771 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local ret=0 2023-11-08T05:03:50.772 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local rmt_export=/tmp/tmp.Ludk8ikCAB/cluster2-mirror-test.export 2023-11-08T05:03:50.772 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local loc_export=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.export 2023-11-08T05:03:50.772 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rm -f /tmp/tmp.Ludk8ikCAB/cluster2-mirror-test.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.export 2023-11-08T05:03:50.772 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 -p mirror export test@snap8 /tmp/tmp.Ludk8ikCAB/cluster2-mirror-test.export 2023-11-08T05:03:51.380 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-11-08T05:03:51.383 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 -p mirror export test@snap8 /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.export 2023-11-08T05:03:52.147 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-11-08T05:03:52.151 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ cmp /tmp/tmp.Ludk8ikCAB/cluster2-mirror-test.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.export 2023-11-08T05:03:52.367 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rm -f /tmp/tmp.Ludk8ikCAB/cluster2-mirror-test.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.export 2023-11-08T05:03:52.512 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:03:52.512 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ snap_name=snap8-clean 2023-11-08T05:03:52.512 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ compare_image_snaps mirror test snap8-clean 2023-11-08T05:03:52.512 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:03:52.513 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T05:03:52.513 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap_name=snap8-clean 2023-11-08T05:03:52.513 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local ret=0 2023-11-08T05:03:52.513 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local rmt_export=/tmp/tmp.Ludk8ikCAB/cluster2-mirror-test.export 2023-11-08T05:03:52.513 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local loc_export=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.export 2023-11-08T05:03:52.513 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rm -f /tmp/tmp.Ludk8ikCAB/cluster2-mirror-test.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.export 2023-11-08T05:03:52.514 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 -p mirror export test@snap8-clean /tmp/tmp.Ludk8ikCAB/cluster2-mirror-test.export 2023-11-08T05:03:53.320 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-11-08T05:03:53.323 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 -p mirror export test@snap8-clean /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.export 2023-11-08T05:03:54.079 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-11-08T05:03:54.082 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ cmp /tmp/tmp.Ludk8ikCAB/cluster2-mirror-test.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.export 2023-11-08T05:03:54.297 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rm -f /tmp/tmp.Ludk8ikCAB/cluster2-mirror-test.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.export 2023-11-08T05:03:54.443 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:03:54.443 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for i in `seq 1 10` 2023-11-08T05:03:54.443 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ snap_name=snap9 2023-11-08T05:03:54.443 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ compare_image_snaps mirror test snap9 2023-11-08T05:03:54.444 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:03:54.444 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T05:03:54.444 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap_name=snap9 2023-11-08T05:03:54.444 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local ret=0 2023-11-08T05:03:54.444 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local rmt_export=/tmp/tmp.Ludk8ikCAB/cluster2-mirror-test.export 2023-11-08T05:03:54.445 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local loc_export=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.export 2023-11-08T05:03:54.445 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rm -f /tmp/tmp.Ludk8ikCAB/cluster2-mirror-test.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.export 2023-11-08T05:03:54.445 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 -p mirror export test@snap9 /tmp/tmp.Ludk8ikCAB/cluster2-mirror-test.export 2023-11-08T05:03:55.038 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-11-08T05:03:55.042 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 -p mirror export test@snap9 /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.export 2023-11-08T05:03:55.837 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-11-08T05:03:55.840 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ cmp /tmp/tmp.Ludk8ikCAB/cluster2-mirror-test.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.export 2023-11-08T05:03:56.045 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.9660480745988163) > weight (0.25) 2023-11-08T05:03:56.045 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.27416190100435434) > weight (0.25) 2023-11-08T05:03:56.046 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.7181529652810966) > weight (0.25) 2023-11-08T05:03:56.046 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.3 with skip (0.49883761450966346) > weight (0.25) 2023-11-08T05:03:56.046 INFO:tasks.rbd_mirror_thrash:waiting for 78 secs before thrashing 2023-11-08T05:03:56.052 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rm -f /tmp/tmp.Ludk8ikCAB/cluster2-mirror-test.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.export 2023-11-08T05:03:56.200 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:03:56.201 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ snap_name=snap9-clean 2023-11-08T05:03:56.201 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ compare_image_snaps mirror test snap9-clean 2023-11-08T05:03:56.201 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:03:56.201 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T05:03:56.202 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap_name=snap9-clean 2023-11-08T05:03:56.202 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local ret=0 2023-11-08T05:03:56.202 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local rmt_export=/tmp/tmp.Ludk8ikCAB/cluster2-mirror-test.export 2023-11-08T05:03:56.202 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local loc_export=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.export 2023-11-08T05:03:56.202 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rm -f /tmp/tmp.Ludk8ikCAB/cluster2-mirror-test.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.export 2023-11-08T05:03:56.203 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 -p mirror export test@snap9-clean /tmp/tmp.Ludk8ikCAB/cluster2-mirror-test.export 2023-11-08T05:03:56.819 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-11-08T05:03:56.822 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 -p mirror export test@snap9-clean /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.export 2023-11-08T05:03:57.648 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-11-08T05:03:57.651 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ cmp /tmp/tmp.Ludk8ikCAB/cluster2-mirror-test.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.export 2023-11-08T05:03:57.865 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rm -f /tmp/tmp.Ludk8ikCAB/cluster2-mirror-test.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.export 2023-11-08T05:03:58.009 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:03:58.010 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for i in `seq 1 10` 2023-11-08T05:03:58.010 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ snap_name=snap10 2023-11-08T05:03:58.010 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ compare_image_snaps mirror test snap10 2023-11-08T05:03:58.010 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:03:58.010 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T05:03:58.011 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap_name=snap10 2023-11-08T05:03:58.011 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local ret=0 2023-11-08T05:03:58.011 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local rmt_export=/tmp/tmp.Ludk8ikCAB/cluster2-mirror-test.export 2023-11-08T05:03:58.011 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local loc_export=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.export 2023-11-08T05:03:58.011 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rm -f /tmp/tmp.Ludk8ikCAB/cluster2-mirror-test.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.export 2023-11-08T05:03:58.012 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 -p mirror export test@snap10 /tmp/tmp.Ludk8ikCAB/cluster2-mirror-test.export 2023-11-08T05:03:58.615 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-11-08T05:03:58.618 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 -p mirror export test@snap10 /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.export 2023-11-08T05:03:59.288 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-11-08T05:03:59.291 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ cmp /tmp/tmp.Ludk8ikCAB/cluster2-mirror-test.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.export 2023-11-08T05:03:59.503 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rm -f /tmp/tmp.Ludk8ikCAB/cluster2-mirror-test.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.export 2023-11-08T05:03:59.651 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:03:59.652 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ snap_name=snap10-clean 2023-11-08T05:03:59.652 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ compare_image_snaps mirror test snap10-clean 2023-11-08T05:03:59.652 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:03:59.652 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T05:03:59.653 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap_name=snap10-clean 2023-11-08T05:03:59.653 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local ret=0 2023-11-08T05:03:59.653 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local rmt_export=/tmp/tmp.Ludk8ikCAB/cluster2-mirror-test.export 2023-11-08T05:03:59.653 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local loc_export=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.export 2023-11-08T05:03:59.653 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rm -f /tmp/tmp.Ludk8ikCAB/cluster2-mirror-test.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.export 2023-11-08T05:03:59.654 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 -p mirror export test@snap10-clean /tmp/tmp.Ludk8ikCAB/cluster2-mirror-test.export 2023-11-08T05:04:00.296 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-11-08T05:04:00.300 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 -p mirror export test@snap10-clean /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.export 2023-11-08T05:04:01.081 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-11-08T05:04:01.084 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ cmp /tmp/tmp.Ludk8ikCAB/cluster2-mirror-test.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.export 2023-11-08T05:04:01.301 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rm -f /tmp/tmp.Ludk8ikCAB/cluster2-mirror-test.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-test.export 2023-11-08T05:04:01.448 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:04:01.448 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ seq 1 10 2023-11-08T05:04:01.449 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for i in `seq 1 10` 2023-11-08T05:04:01.450 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ snap_name=snap1 2023-11-08T05:04:01.450 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ remove_snapshot cluster2 mirror test snap1 2023-11-08T05:04:01.450 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:04:01.450 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:04:01.451 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T05:04:01.451 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap=snap1 2023-11-08T05:04:01.451 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap1 2023-11-08T05:04:01.708 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Removing snap: 100% complete...done. 2023-11-08T05:04:01.725 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ snap_name=snap1-clean 2023-11-08T05:04:01.725 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ remove_snapshot cluster2 mirror test snap1-clean 2023-11-08T05:04:01.726 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:04:01.726 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:04:01.726 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T05:04:01.726 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap=snap1-clean 2023-11-08T05:04:01.726 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap1-clean 2023-11-08T05:04:02.722 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Removing snap: 100% complete...done. 2023-11-08T05:04:02.739 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for i in `seq 1 10` 2023-11-08T05:04:02.739 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ snap_name=snap2 2023-11-08T05:04:02.739 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ remove_snapshot cluster2 mirror test snap2 2023-11-08T05:04:02.740 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:04:02.740 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:04:02.740 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T05:04:02.740 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap=snap2 2023-11-08T05:04:02.740 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap2 2023-11-08T05:04:03.723 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Removing snap: 100% complete...done. 2023-11-08T05:04:03.745 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ snap_name=snap2-clean 2023-11-08T05:04:03.745 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ remove_snapshot cluster2 mirror test snap2-clean 2023-11-08T05:04:03.746 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:04:03.746 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:04:03.746 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T05:04:03.746 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap=snap2-clean 2023-11-08T05:04:03.747 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap2-clean 2023-11-08T05:04:04.731 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Removing snap: 100% complete...done. 2023-11-08T05:04:04.754 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for i in `seq 1 10` 2023-11-08T05:04:04.755 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ snap_name=snap3 2023-11-08T05:04:04.755 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ remove_snapshot cluster2 mirror test snap3 2023-11-08T05:04:04.755 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:04:04.755 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:04:04.755 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T05:04:04.756 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap=snap3 2023-11-08T05:04:04.756 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap3 2023-11-08T05:04:05.731 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Removing snap: 100% complete...done. 2023-11-08T05:04:05.743 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ snap_name=snap3-clean 2023-11-08T05:04:05.744 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ remove_snapshot cluster2 mirror test snap3-clean 2023-11-08T05:04:05.744 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:04:05.744 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:04:05.744 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T05:04:05.745 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap=snap3-clean 2023-11-08T05:04:05.745 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap3-clean 2023-11-08T05:04:06.734 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Removing snap: 100% complete...done. 2023-11-08T05:04:06.748 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for i in `seq 1 10` 2023-11-08T05:04:06.748 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ snap_name=snap4 2023-11-08T05:04:06.749 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ remove_snapshot cluster2 mirror test snap4 2023-11-08T05:04:06.749 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:04:06.749 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:04:06.749 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T05:04:06.749 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap=snap4 2023-11-08T05:04:06.750 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap4 2023-11-08T05:04:07.735 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Removing snap: 100% complete...done. 2023-11-08T05:04:07.750 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ snap_name=snap4-clean 2023-11-08T05:04:07.750 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ remove_snapshot cluster2 mirror test snap4-clean 2023-11-08T05:04:07.750 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:04:07.751 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:04:07.751 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T05:04:07.751 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap=snap4-clean 2023-11-08T05:04:07.751 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap4-clean 2023-11-08T05:04:08.745 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Removing snap: 100% complete...done. 2023-11-08T05:04:08.773 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for i in `seq 1 10` 2023-11-08T05:04:08.773 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ snap_name=snap5 2023-11-08T05:04:08.773 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ remove_snapshot cluster2 mirror test snap5 2023-11-08T05:04:08.773 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:04:08.774 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:04:08.774 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T05:04:08.774 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap=snap5 2023-11-08T05:04:08.774 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap5 2023-11-08T05:04:09.745 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Removing snap: 100% complete...done. 2023-11-08T05:04:09.758 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ snap_name=snap5-clean 2023-11-08T05:04:09.759 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ remove_snapshot cluster2 mirror test snap5-clean 2023-11-08T05:04:09.759 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:04:09.759 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:04:09.759 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T05:04:09.760 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap=snap5-clean 2023-11-08T05:04:09.760 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap5-clean 2023-11-08T05:04:10.755 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Removing snap: 100% complete...done. 2023-11-08T05:04:10.775 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for i in `seq 1 10` 2023-11-08T05:04:10.775 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ snap_name=snap6 2023-11-08T05:04:10.776 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ remove_snapshot cluster2 mirror test snap6 2023-11-08T05:04:10.776 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:04:10.777 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:04:10.777 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T05:04:10.777 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap=snap6 2023-11-08T05:04:10.778 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap6 2023-11-08T05:04:11.763 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Removing snap: 100% complete...done. 2023-11-08T05:04:11.773 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ snap_name=snap6-clean 2023-11-08T05:04:11.773 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ remove_snapshot cluster2 mirror test snap6-clean 2023-11-08T05:04:11.774 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:04:11.774 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:04:11.774 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T05:04:11.774 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap=snap6-clean 2023-11-08T05:04:11.775 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap6-clean 2023-11-08T05:04:12.602 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Removing snap: 100% complete...done. 2023-11-08T05:04:12.617 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for i in `seq 1 10` 2023-11-08T05:04:12.618 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ snap_name=snap7 2023-11-08T05:04:12.618 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ remove_snapshot cluster2 mirror test snap7 2023-11-08T05:04:12.618 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:04:12.618 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:04:12.619 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T05:04:12.619 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap=snap7 2023-11-08T05:04:12.619 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap7 2023-11-08T05:04:13.586 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Removing snap: 100% complete...done. 2023-11-08T05:04:13.601 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ snap_name=snap7-clean 2023-11-08T05:04:13.601 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ remove_snapshot cluster2 mirror test snap7-clean 2023-11-08T05:04:13.601 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:04:13.602 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:04:13.602 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T05:04:13.602 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap=snap7-clean 2023-11-08T05:04:13.602 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap7-clean 2023-11-08T05:04:14.600 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Removing snap: 100% complete...done. 2023-11-08T05:04:14.624 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for i in `seq 1 10` 2023-11-08T05:04:14.624 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ snap_name=snap8 2023-11-08T05:04:14.624 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ remove_snapshot cluster2 mirror test snap8 2023-11-08T05:04:14.624 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:04:14.624 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:04:14.625 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T05:04:14.625 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap=snap8 2023-11-08T05:04:14.625 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap8 2023-11-08T05:04:15.600 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Removing snap: 100% complete...done. 2023-11-08T05:04:15.620 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ snap_name=snap8-clean 2023-11-08T05:04:15.620 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ remove_snapshot cluster2 mirror test snap8-clean 2023-11-08T05:04:15.620 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:04:15.621 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:04:15.621 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T05:04:15.621 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap=snap8-clean 2023-11-08T05:04:15.621 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap8-clean 2023-11-08T05:04:16.603 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Removing snap: 100% complete...done. 2023-11-08T05:04:16.623 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for i in `seq 1 10` 2023-11-08T05:04:16.624 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ snap_name=snap9 2023-11-08T05:04:16.624 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ remove_snapshot cluster2 mirror test snap9 2023-11-08T05:04:16.624 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:04:16.624 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:04:16.624 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T05:04:16.624 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap=snap9 2023-11-08T05:04:16.625 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap9 2023-11-08T05:04:17.045 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.0 2023-11-08T05:04:17.046 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Sent signal 15 2023-11-08T05:04:17.046 INFO:tasks.rbd_mirror_thrash:waiting for 1 secs before reviving daemons 2023-11-08T05:04:17.047 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi087.stderr:2023-11-08T05:04:17.045+0000 7f43deb08700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 125031) UID: 1000 2023-11-08T05:04:17.247 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi087.stderr:daemon-helper: command failed with exit status 1 2023-11-08T05:04:17.608 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Removing snap: 100% complete...done. 2023-11-08T05:04:17.625 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ snap_name=snap9-clean 2023-11-08T05:04:17.625 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ remove_snapshot cluster2 mirror test snap9-clean 2023-11-08T05:04:17.625 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:04:17.625 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:04:17.626 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T05:04:17.626 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap=snap9-clean 2023-11-08T05:04:17.626 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap9-clean 2023-11-08T05:04:18.047 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.0 2023-11-08T05:04:18.048 INFO:teuthology.orchestra.run:waiting for 600 2023-11-08T05:04:18.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-08T05:04:18.048 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.0 2023-11-08T05:04:18.049 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2023-11-08T05:04:18.049 DEBUG:teuthology.orchestra.run.smithi087:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2023-11-08T05:04:18.052 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2023-11-08T05:04:18.052 INFO:tasks.rbd_mirror_thrash:waiting for 67 secs before thrashing 2023-11-08T05:04:18.613 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Removing snap: 100% complete...done. 2023-11-08T05:04:18.629 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for i in `seq 1 10` 2023-11-08T05:04:18.630 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ snap_name=snap10 2023-11-08T05:04:18.630 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ remove_snapshot cluster2 mirror test snap10 2023-11-08T05:04:18.630 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:04:18.630 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:04:18.630 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T05:04:18.631 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap=snap10 2023-11-08T05:04:18.631 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap10 2023-11-08T05:04:19.618 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Removing snap: 100% complete...done. 2023-11-08T05:04:19.629 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ snap_name=snap10-clean 2023-11-08T05:04:19.629 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ remove_snapshot cluster2 mirror test snap10-clean 2023-11-08T05:04:19.629 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:04:19.630 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:04:19.630 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T05:04:19.630 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap=snap10-clean 2023-11-08T05:04:19.630 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap10-clean 2023-11-08T05:04:20.622 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Removing snap: 100% complete...done. 2023-11-08T05:04:20.634 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ remove_image_retry cluster2 mirror test 2023-11-08T05:04:20.634 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:04:20.634 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:04:20.634 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T05:04:20.635 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0 1 2 4 8 16 32 2023-11-08T05:04:20.635 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0 2023-11-08T05:04:20.635 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ remove_image cluster2 mirror test 2023-11-08T05:04:20.636 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:04:20.636 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:04:20.636 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T05:04:20.636 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster=cluster2 snap purge mirror/test 2023-11-08T05:04:20.674 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster=cluster2 rm mirror/test 2023-11-08T05:04:20.781 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi087.stderr:2023-11-08T05:04:20.779+0000 7f9a55a64700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55bc742ce340 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-11-08T05:04:20.781 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi087.stderr:2023-11-08T05:04:20.779+0000 7f9a5ba70700 -1 rbd::mirror::ImageReplayer: 0x55bc70244780 [3/88ced7f9-2739-4a27-ad7a-30edec05edb1] operator(): start failed: (85) Interrupted system call should be restarted 2023-11-08T05:04:21.825 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2023-11-08T05:04:21.829 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:04:21.829 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_image_present cluster1 mirror test deleted 2023-11-08T05:04:21.830 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:04:21.830 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:04:21.830 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T05:04:21.830 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local state=deleted 2023-11-08T05:04:21.830 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image_id= 2023-11-08T05:04:21.831 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local s 2023-11-08T05:04:21.831 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test -n '' 2023-11-08T05:04:21.831 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ get_image_id cluster1 mirror test 2023-11-08T05:04:21.831 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1 2023-11-08T05:04:21.831 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local pool=mirror 2023-11-08T05:04:21.832 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local image=test 2023-11-08T05:04:21.832 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ rbd --cluster=cluster1 info mirror/test 2023-11-08T05:04:21.832 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-11-08T05:04:21.855 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd: error opening image test: (2) No such file or directory 2023-11-08T05:04:21.858 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image_id= 2023-11-08T05:04:21.858 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-11-08T05:04:21.858 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0.1 2023-11-08T05:04:21.960 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test_image_present cluster1 mirror test deleted '' 2023-11-08T05:04:21.960 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:04:21.960 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:04:21.960 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=test 2023-11-08T05:04:21.961 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local test_state=deleted 2023-11-08T05:04:21.961 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image_id= 2023-11-08T05:04:21.961 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local current_state=deleted 2023-11-08T05:04:21.961 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local current_image_id 2023-11-08T05:04:21.961 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ get_image_id cluster1 mirror test 2023-11-08T05:04:21.962 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1 2023-11-08T05:04:21.962 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local pool=mirror 2023-11-08T05:04:21.962 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local image=test 2023-11-08T05:04:21.962 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ rbd --cluster=cluster1 info mirror/test 2023-11-08T05:04:21.962 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-11-08T05:04:21.987 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd: error opening image test: (2) No such file or directory 2023-11-08T05:04:21.989 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ current_image_id= 2023-11-08T05:04:21.990 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test -n '' 2023-11-08T05:04:21.990 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test deleted = deleted 2023-11-08T05:04:21.990 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:04:21.990 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ testlog 'TEST: create many images' 2023-11-08T05:04:21.991 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ tee -a /tmp/tmp.Ludk8ikCAB/rbd-mirror.test.log 2023-11-08T05:04:21.991 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ date '+%F %T' 2023-11-08T05:04:21.993 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ echo 2023-11-08 05:04:21 TEST: create many images 2023-11-08T05:04:21.993 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:2023-11-08 05:04:21 TEST: create many images 2023-11-08T05:04:21.993 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ snap_name=snap 2023-11-08T05:04:21.993 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ seq 1 50 2023-11-08T05:04:21.994 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-11-08T05:04:21.994 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image=image_1 2023-11-08T05:04:21.994 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ create_image_and_enable_mirror cluster2 mirror image_1 snapshot 128M 2023-11-08T05:04:21.994 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:04:21.994 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:04:21.995 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:04:21.995 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:04:21.995 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_1 2023-11-08T05:04:21.995 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:04:21.995 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local mode=snapshot 2023-11-08T05:04:21.996 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' -n snapshot ']' 2023-11-08T05:04:21.996 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:04:21.996 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ create_image cluster2 mirror image_1 128M 2023-11-08T05:04:21.996 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:04:21.996 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:04:21.996 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:04:21.997 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:04:21.997 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_1 2023-11-08T05:04:21.997 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:04:21.997 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local size=128 2023-11-08T05:04:21.997 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' -n 128M ']' 2023-11-08T05:04:21.998 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ size=128M 2023-11-08T05:04:21.998 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:04:21.998 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_1 2023-11-08T05:04:22.030 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' image = image ']' 2023-11-08T05:04:22.030 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ enable_mirror cluster2 mirror image_1 snapshot 2023-11-08T05:04:22.030 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:04:22.030 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:04:22.031 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_1 2023-11-08T05:04:22.031 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local mode=snapshot 2023-11-08T05:04:22.031 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_1 snapshot 2023-11-08T05:04:22.639 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:Mirroring enabled 2023-11-08T05:04:22.645 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster=cluster2 info mirror/image_1 2023-11-08T05:04:22.676 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:rbd image 'image_1': 2023-11-08T05:04:22.677 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: size 128 MiB in 32 objects 2023-11-08T05:04:22.677 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: order 22 (4 MiB objects) 2023-11-08T05:04:22.677 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: snapshot_count: 1 2023-11-08T05:04:22.677 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: id: 1259e254ed55 2023-11-08T05:04:22.677 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: block_name_prefix: rbd_data.1259e254ed55 2023-11-08T05:04:22.678 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: format: 2 2023-11-08T05:04:22.678 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: features: layering, exclusive-lock 2023-11-08T05:04:22.678 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: op_features: 2023-11-08T05:04:22.678 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: flags: 2023-11-08T05:04:22.678 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: create_timestamp: Wed Nov 8 05:04:22 2023 2023-11-08T05:04:22.679 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: access_timestamp: Wed Nov 8 05:04:22 2023 2023-11-08T05:04:22.679 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: modify_timestamp: Wed Nov 8 05:04:22 2023 2023-11-08T05:04:22.679 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring state: enabled 2023-11-08T05:04:22.679 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring mode: snapshot 2023-11-08T05:04:22.679 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring global id: 8f09f37c-3f8d-4d10-ad86-50b1dda83090 2023-11-08T05:04:22.680 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring primary: true 2023-11-08T05:04:22.680 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' -n '' ']' 2023-11-08T05:04:22.680 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ stress_write_image cluster2 mirror image_1 2023-11-08T05:04:22.681 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:04:22.681 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:04:22.681 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_1 2023-11-08T05:04:22.681 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-11-08T05:04:22.682 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local duration=10 2023-11-08T05:04:22.682 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ set +e 2023-11-08T05:04:22.682 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_1 --debug-rbd=20 --debug-journaler=20 2023-11-08T05:04:22.852 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:100 IOs, 21980160 bytes 2023-11-08T05:04:23.073 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:200 IOs, 100517888 bytes 2023-11-08T05:04:23.178 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:300 IOs, 105854976 bytes 2023-11-08T05:04:23.307 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:400 IOs, 121780224 bytes 2023-11-08T05:04:23.487 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:500 IOs, 164108288 bytes 2023-11-08T05:04:23.658 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:600 IOs, 219457536 bytes 2023-11-08T05:04:23.846 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:700 IOs, 263236608 bytes 2023-11-08T05:04:23.985 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:800 IOs, 286732288 bytes 2023-11-08T05:04:24.207 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:900 IOs, 367073280 bytes 2023-11-08T05:04:24.357 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1000 IOs, 392161280 bytes 2023-11-08T05:04:24.503 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1100 IOs, 421139456 bytes 2023-11-08T05:04:24.702 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1200 IOs, 471941120 bytes 2023-11-08T05:04:24.805 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1300 IOs, 487051264 bytes 2023-11-08T05:04:25.011 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1400 IOs, 554641408 bytes 2023-11-08T05:04:25.170 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1500 IOs, 586118144 bytes 2023-11-08T05:04:25.277 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1600 IOs, 599915520 bytes 2023-11-08T05:04:25.447 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1700 IOs, 655990784 bytes 2023-11-08T05:04:25.714 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1800 IOs, 732229632 bytes 2023-11-08T05:04:25.848 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1900 IOs, 775681024 bytes 2023-11-08T05:04:26.064 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2000 IOs, 831286272 bytes 2023-11-08T05:04:26.188 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2100 IOs, 848232448 bytes 2023-11-08T05:04:26.366 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2200 IOs, 891742208 bytes 2023-11-08T05:04:26.457 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2300 IOs, 911753216 bytes 2023-11-08T05:04:26.610 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2400 IOs, 931085312 bytes 2023-11-08T05:04:26.789 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2500 IOs, 968002560 bytes 2023-11-08T05:04:26.968 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2600 IOs, 1011200000 bytes 2023-11-08T05:04:27.165 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2700 IOs, 1060295680 bytes 2023-11-08T05:04:27.319 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2800 IOs, 1096295424 bytes 2023-11-08T05:04:27.411 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2900 IOs, 1099334656 bytes 2023-11-08T05:04:27.516 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3000 IOs, 1112442880 bytes 2023-11-08T05:04:27.641 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3100 IOs, 1124371456 bytes 2023-11-08T05:04:27.773 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3200 IOs, 1139412992 bytes 2023-11-08T05:04:27.980 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3300 IOs, 1199964160 bytes 2023-11-08T05:04:28.101 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3400 IOs, 1215976448 bytes 2023-11-08T05:04:28.255 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3500 IOs, 1250077696 bytes 2023-11-08T05:04:28.374 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3600 IOs, 1275355136 bytes 2023-11-08T05:04:28.471 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3700 IOs, 1281244160 bytes 2023-11-08T05:04:28.622 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3800 IOs, 1318677504 bytes 2023-11-08T05:04:28.766 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3900 IOs, 1346113536 bytes 2023-11-08T05:04:28.915 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4000 IOs, 1377681408 bytes 2023-11-08T05:04:29.084 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4100 IOs, 1424253952 bytes 2023-11-08T05:04:29.202 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4200 IOs, 1442167808 bytes 2023-11-08T05:04:29.376 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4300 IOs, 1483641856 bytes 2023-11-08T05:04:29.573 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4400 IOs, 1534558208 bytes 2023-11-08T05:04:29.749 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4500 IOs, 1572491264 bytes 2023-11-08T05:04:29.896 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4600 IOs, 1610222592 bytes 2023-11-08T05:04:30.034 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4700 IOs, 1630401536 bytes 2023-11-08T05:04:30.182 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4800 IOs, 1672662016 bytes 2023-11-08T05:04:30.318 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4900 IOs, 1679649792 bytes 2023-11-08T05:04:30.491 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5000 IOs, 1725528064 bytes 2023-11-08T05:04:30.675 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5100 IOs, 1785174016 bytes 2023-11-08T05:04:30.788 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5200 IOs, 1786501120 bytes 2023-11-08T05:04:30.954 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5300 IOs, 1828617216 bytes 2023-11-08T05:04:31.219 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5400 IOs, 1945740288 bytes 2023-11-08T05:04:31.454 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5500 IOs, 2003152896 bytes 2023-11-08T05:04:31.614 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5600 IOs, 2041772032 bytes 2023-11-08T05:04:31.796 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5700 IOs, 2093669376 bytes 2023-11-08T05:04:31.950 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5800 IOs, 2121126912 bytes 2023-11-08T05:04:32.134 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5900 IOs, 2173302784 bytes 2023-11-08T05:04:32.269 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:6000 IOs, 2184951808 bytes 2023-11-08T05:04:32.450 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:6100 IOs, 2232206336 bytes 2023-11-08T05:04:32.630 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:6200 IOs, 2269025280 bytes 2023-11-08T05:04:32.687 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ error_code=124 2023-11-08T05:04:32.687 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ set -e 2023-11-08T05:04:32.688 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' 124 -eq 124 ']' 2023-11-08T05:04:32.688 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:04:32.688 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-11-08T05:04:32.688 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image=image_2 2023-11-08T05:04:32.688 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ create_image_and_enable_mirror cluster2 mirror image_2 snapshot 128M 2023-11-08T05:04:32.689 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:04:32.689 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:04:32.689 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:04:32.689 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:04:32.689 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_2 2023-11-08T05:04:32.690 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:04:32.690 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local mode=snapshot 2023-11-08T05:04:32.690 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' -n snapshot ']' 2023-11-08T05:04:32.690 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:04:32.690 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ create_image cluster2 mirror image_2 128M 2023-11-08T05:04:32.691 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:04:32.691 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:04:32.691 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:04:32.691 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:04:32.691 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_2 2023-11-08T05:04:32.691 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:04:32.692 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local size=128 2023-11-08T05:04:32.692 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' -n 128M ']' 2023-11-08T05:04:32.692 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ size=128M 2023-11-08T05:04:32.692 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:04:32.692 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_2 2023-11-08T05:04:32.727 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' image = image ']' 2023-11-08T05:04:32.727 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ enable_mirror cluster2 mirror image_2 snapshot 2023-11-08T05:04:32.727 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:04:32.728 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:04:32.728 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_2 2023-11-08T05:04:32.728 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local mode=snapshot 2023-11-08T05:04:32.728 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_2 snapshot 2023-11-08T05:04:33.745 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:Mirroring enabled 2023-11-08T05:04:33.755 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster=cluster2 info mirror/image_2 2023-11-08T05:04:33.787 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:rbd image 'image_2': 2023-11-08T05:04:33.787 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: size 128 MiB in 32 objects 2023-11-08T05:04:33.788 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: order 22 (4 MiB objects) 2023-11-08T05:04:33.788 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: snapshot_count: 1 2023-11-08T05:04:33.789 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: id: 12656da150a9 2023-11-08T05:04:33.789 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: block_name_prefix: rbd_data.12656da150a9 2023-11-08T05:04:33.789 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: format: 2 2023-11-08T05:04:33.790 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: features: layering, exclusive-lock 2023-11-08T05:04:33.790 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: op_features: 2023-11-08T05:04:33.791 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: flags: 2023-11-08T05:04:33.791 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: create_timestamp: Wed Nov 8 05:04:32 2023 2023-11-08T05:04:33.791 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: access_timestamp: Wed Nov 8 05:04:32 2023 2023-11-08T05:04:33.792 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: modify_timestamp: Wed Nov 8 05:04:32 2023 2023-11-08T05:04:33.793 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring state: enabled 2023-11-08T05:04:33.793 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring mode: snapshot 2023-11-08T05:04:33.793 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring global id: c9cc5896-9129-4803-9473-1d795c399c55 2023-11-08T05:04:33.793 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring primary: true 2023-11-08T05:04:33.795 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' -n '' ']' 2023-11-08T05:04:33.795 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ stress_write_image cluster2 mirror image_2 2023-11-08T05:04:33.796 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:04:33.796 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:04:33.796 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_2 2023-11-08T05:04:33.796 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-11-08T05:04:33.797 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local duration=8 2023-11-08T05:04:33.797 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ set +e 2023-11-08T05:04:33.797 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_2 --debug-rbd=20 --debug-journaler=20 2023-11-08T05:04:33.944 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:100 IOs, 27244544 bytes 2023-11-08T05:04:34.124 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:200 IOs, 69339136 bytes 2023-11-08T05:04:34.274 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:300 IOs, 101545984 bytes 2023-11-08T05:04:34.460 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:400 IOs, 158291968 bytes 2023-11-08T05:04:34.601 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:500 IOs, 185662464 bytes 2023-11-08T05:04:34.771 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:600 IOs, 226957312 bytes 2023-11-08T05:04:34.861 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:700 IOs, 239276032 bytes 2023-11-08T05:04:34.959 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:800 IOs, 244620288 bytes 2023-11-08T05:04:35.138 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:900 IOs, 283453440 bytes 2023-11-08T05:04:35.330 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1000 IOs, 333554688 bytes 2023-11-08T05:04:35.468 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1100 IOs, 360288256 bytes 2023-11-08T05:04:35.619 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1200 IOs, 393239552 bytes 2023-11-08T05:04:35.773 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1300 IOs, 433208320 bytes 2023-11-08T05:04:35.893 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1400 IOs, 452081664 bytes 2023-11-08T05:04:36.097 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1500 IOs, 508868608 bytes 2023-11-08T05:04:36.283 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1600 IOs, 548814848 bytes 2023-11-08T05:04:36.366 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1700 IOs, 549427200 bytes 2023-11-08T05:04:36.610 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1800 IOs, 620352512 bytes 2023-11-08T05:04:36.827 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1900 IOs, 686358528 bytes 2023-11-08T05:04:36.968 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2000 IOs, 710828032 bytes 2023-11-08T05:04:37.106 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2100 IOs, 732511232 bytes 2023-11-08T05:04:37.240 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2200 IOs, 755056640 bytes 2023-11-08T05:04:37.552 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2300 IOs, 859463680 bytes 2023-11-08T05:04:37.751 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2400 IOs, 910608384 bytes 2023-11-08T05:04:37.891 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2500 IOs, 933763072 bytes 2023-11-08T05:04:38.069 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2600 IOs, 974169088 bytes 2023-11-08T05:04:38.225 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2700 IOs, 1004091392 bytes 2023-11-08T05:04:38.386 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2800 IOs, 1039175680 bytes 2023-11-08T05:04:38.533 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2900 IOs, 1068500992 bytes 2023-11-08T05:04:38.640 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3000 IOs, 1076826112 bytes 2023-11-08T05:04:38.886 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3100 IOs, 1150824448 bytes 2023-11-08T05:04:38.979 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3200 IOs, 1151427584 bytes 2023-11-08T05:04:39.166 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3300 IOs, 1229026304 bytes 2023-11-08T05:04:39.405 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3400 IOs, 1274268672 bytes 2023-11-08T05:04:39.662 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3500 IOs, 1348764672 bytes 2023-11-08T05:04:39.837 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3600 IOs, 1398849536 bytes 2023-11-08T05:04:40.146 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3700 IOs, 1496415232 bytes 2023-11-08T05:04:40.244 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3800 IOs, 1497041920 bytes 2023-11-08T05:04:40.432 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3900 IOs, 1538568192 bytes 2023-11-08T05:04:40.613 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4000 IOs, 1588977664 bytes 2023-11-08T05:04:40.882 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4100 IOs, 1685292032 bytes 2023-11-08T05:04:41.017 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4200 IOs, 1714712576 bytes 2023-11-08T05:04:41.168 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4300 IOs, 1751397376 bytes 2023-11-08T05:04:41.318 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4400 IOs, 1783390208 bytes 2023-11-08T05:04:41.475 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4500 IOs, 1810603008 bytes 2023-11-08T05:04:41.591 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4600 IOs, 1825676288 bytes 2023-11-08T05:04:41.765 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4700 IOs, 1873193984 bytes 2023-11-08T05:04:41.799 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ error_code=124 2023-11-08T05:04:41.799 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ set -e 2023-11-08T05:04:41.799 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' 124 -eq 124 ']' 2023-11-08T05:04:41.800 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:04:41.800 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-11-08T05:04:41.800 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image=image_3 2023-11-08T05:04:41.800 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ create_image_and_enable_mirror cluster2 mirror image_3 snapshot 128M 2023-11-08T05:04:41.800 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:04:41.801 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:04:41.801 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:04:41.801 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:04:41.801 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_3 2023-11-08T05:04:41.801 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:04:41.802 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local mode=snapshot 2023-11-08T05:04:41.802 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' -n snapshot ']' 2023-11-08T05:04:41.802 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:04:41.802 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ create_image cluster2 mirror image_3 128M 2023-11-08T05:04:41.802 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:04:41.803 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:04:41.803 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:04:41.803 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:04:41.803 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_3 2023-11-08T05:04:41.803 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:04:41.804 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local size=128 2023-11-08T05:04:41.804 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' -n 128M ']' 2023-11-08T05:04:41.804 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ size=128M 2023-11-08T05:04:41.804 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:04:41.804 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_3 2023-11-08T05:04:41.879 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' image = image ']' 2023-11-08T05:04:41.880 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ enable_mirror cluster2 mirror image_3 snapshot 2023-11-08T05:04:41.880 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:04:41.880 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:04:41.880 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_3 2023-11-08T05:04:41.880 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local mode=snapshot 2023-11-08T05:04:41.880 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_3 snapshot 2023-11-08T05:04:42.758 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:Mirroring enabled 2023-11-08T05:04:42.765 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster=cluster2 info mirror/image_3 2023-11-08T05:04:42.796 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:rbd image 'image_3': 2023-11-08T05:04:42.796 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: size 128 MiB in 32 objects 2023-11-08T05:04:42.797 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: order 22 (4 MiB objects) 2023-11-08T05:04:42.797 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: snapshot_count: 1 2023-11-08T05:04:42.797 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: id: 127156fe8e2 2023-11-08T05:04:42.797 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: block_name_prefix: rbd_data.127156fe8e2 2023-11-08T05:04:42.797 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: format: 2 2023-11-08T05:04:42.798 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: features: layering, exclusive-lock 2023-11-08T05:04:42.798 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: op_features: 2023-11-08T05:04:42.798 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: flags: 2023-11-08T05:04:42.798 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: create_timestamp: Wed Nov 8 05:04:41 2023 2023-11-08T05:04:42.798 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: access_timestamp: Wed Nov 8 05:04:41 2023 2023-11-08T05:04:42.799 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: modify_timestamp: Wed Nov 8 05:04:41 2023 2023-11-08T05:04:42.799 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring state: enabled 2023-11-08T05:04:42.799 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring mode: snapshot 2023-11-08T05:04:42.799 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring global id: 64454ccd-8f11-4a7c-ba33-492d579d2c3b 2023-11-08T05:04:42.799 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring primary: true 2023-11-08T05:04:42.800 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' -n '' ']' 2023-11-08T05:04:42.801 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ stress_write_image cluster2 mirror image_3 2023-11-08T05:04:42.801 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:04:42.801 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:04:42.801 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_3 2023-11-08T05:04:42.801 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-11-08T05:04:42.802 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local duration=10 2023-11-08T05:04:42.802 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ set +e 2023-11-08T05:04:42.802 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_3 --debug-rbd=20 --debug-journaler=20 2023-11-08T05:04:42.960 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:100 IOs, 19295232 bytes 2023-11-08T05:04:43.075 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:200 IOs, 36514816 bytes 2023-11-08T05:04:43.251 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:300 IOs, 81940480 bytes 2023-11-08T05:04:43.494 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:400 IOs, 153096192 bytes 2023-11-08T05:04:43.761 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:500 IOs, 244719616 bytes 2023-11-08T05:04:43.952 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:600 IOs, 296564736 bytes 2023-11-08T05:04:44.033 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:700 IOs, 298435584 bytes 2023-11-08T05:04:44.130 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:800 IOs, 299070464 bytes 2023-11-08T05:04:44.358 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:900 IOs, 373278720 bytes 2023-11-08T05:04:44.458 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1000 IOs, 383772672 bytes 2023-11-08T05:04:44.619 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1100 IOs, 420440064 bytes 2023-11-08T05:04:44.784 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1200 IOs, 458007552 bytes 2023-11-08T05:04:44.894 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1300 IOs, 473003008 bytes 2023-11-08T05:04:45.032 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1400 IOs, 505659392 bytes 2023-11-08T05:04:45.150 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1500 IOs, 519323648 bytes 2023-11-08T05:04:45.272 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1600 IOs, 538818560 bytes 2023-11-08T05:04:45.398 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1700 IOs, 555034624 bytes 2023-11-08T05:04:45.671 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1800 IOs, 649329664 bytes 2023-11-08T05:04:45.823 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1900 IOs, 683281408 bytes 2023-11-08T05:04:45.948 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2000 IOs, 703391744 bytes 2023-11-08T05:04:46.083 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2100 IOs, 728095744 bytes 2023-11-08T05:04:46.195 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2200 IOs, 752276480 bytes 2023-11-08T05:04:46.386 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2300 IOs, 789217280 bytes 2023-11-08T05:04:46.641 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2400 IOs, 878778368 bytes 2023-11-08T05:04:46.750 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2500 IOs, 923679744 bytes 2023-11-08T05:04:46.997 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2600 IOs, 969020416 bytes 2023-11-08T05:04:47.109 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2700 IOs, 978419712 bytes 2023-11-08T05:04:47.299 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2800 IOs, 1036085248 bytes 2023-11-08T05:04:47.401 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2900 IOs, 1043046400 bytes 2023-11-08T05:04:47.562 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3000 IOs, 1082866688 bytes 2023-11-08T05:04:47.721 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3100 IOs, 1118270464 bytes 2023-11-08T05:04:47.890 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3200 IOs, 1165051904 bytes 2023-11-08T05:04:48.203 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3300 IOs, 1279194112 bytes 2023-11-08T05:04:48.416 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3400 IOs, 1341954048 bytes 2023-11-08T05:04:48.606 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3500 IOs, 1391060992 bytes 2023-11-08T05:04:48.760 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3600 IOs, 1425843200 bytes 2023-11-08T05:04:49.082 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3700 IOs, 1552572416 bytes 2023-11-08T05:04:49.344 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3800 IOs, 1618806784 bytes 2023-11-08T05:04:49.471 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3900 IOs, 1639499776 bytes 2023-11-08T05:04:49.698 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4000 IOs, 1710656512 bytes 2023-11-08T05:04:49.990 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4100 IOs, 1816681472 bytes 2023-11-08T05:04:50.140 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4200 IOs, 1846642688 bytes 2023-11-08T05:04:50.285 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4300 IOs, 1872067584 bytes 2023-11-08T05:04:50.466 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4400 IOs, 1922163712 bytes 2023-11-08T05:04:50.763 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4500 IOs, 2026820608 bytes 2023-11-08T05:04:50.979 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4600 IOs, 2087141376 bytes 2023-11-08T05:04:51.159 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4700 IOs, 2131987456 bytes 2023-11-08T05:04:51.346 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4800 IOs, 2179608576 bytes 2023-11-08T05:04:51.489 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4900 IOs, 2224146432 bytes 2023-11-08T05:04:51.660 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5000 IOs, 2260853760 bytes 2023-11-08T05:04:51.913 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5100 IOs, 2330186752 bytes 2023-11-08T05:04:52.068 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5200 IOs, 2346694656 bytes 2023-11-08T05:04:52.154 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5300 IOs, 2347298816 bytes 2023-11-08T05:04:52.291 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5400 IOs, 2362266624 bytes 2023-11-08T05:04:52.428 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5500 IOs, 2403585024 bytes 2023-11-08T05:04:52.653 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5600 IOs, 2442647552 bytes 2023-11-08T05:04:52.787 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5700 IOs, 2467521536 bytes 2023-11-08T05:04:52.808 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ error_code=124 2023-11-08T05:04:52.808 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ set -e 2023-11-08T05:04:52.808 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' 124 -eq 124 ']' 2023-11-08T05:04:52.808 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:04:52.808 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-11-08T05:04:52.809 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image=image_4 2023-11-08T05:04:52.809 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ create_image_and_enable_mirror cluster2 mirror image_4 snapshot 128M 2023-11-08T05:04:52.809 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:04:52.809 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:04:52.809 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:04:52.809 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:04:52.810 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_4 2023-11-08T05:04:52.810 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:04:52.810 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local mode=snapshot 2023-11-08T05:04:52.810 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' -n snapshot ']' 2023-11-08T05:04:52.810 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:04:52.811 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ create_image cluster2 mirror image_4 128M 2023-11-08T05:04:52.811 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:04:52.811 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:04:52.811 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:04:52.811 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:04:52.811 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_4 2023-11-08T05:04:52.812 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:04:52.812 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local size=128 2023-11-08T05:04:52.812 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' -n 128M ']' 2023-11-08T05:04:52.812 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ size=128M 2023-11-08T05:04:52.812 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:04:52.813 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_4 2023-11-08T05:04:52.847 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' image = image ']' 2023-11-08T05:04:52.847 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ enable_mirror cluster2 mirror image_4 snapshot 2023-11-08T05:04:52.848 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:04:52.848 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:04:52.848 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_4 2023-11-08T05:04:52.848 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local mode=snapshot 2023-11-08T05:04:52.848 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_4 snapshot 2023-11-08T05:04:53.700 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:Mirroring enabled 2023-11-08T05:04:53.710 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster=cluster2 info mirror/image_4 2023-11-08T05:04:53.743 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:rbd image 'image_4': 2023-11-08T05:04:53.744 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: size 128 MiB in 32 objects 2023-11-08T05:04:53.744 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: order 22 (4 MiB objects) 2023-11-08T05:04:53.744 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: snapshot_count: 1 2023-11-08T05:04:53.744 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: id: 127d21f2679e 2023-11-08T05:04:53.744 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: block_name_prefix: rbd_data.127d21f2679e 2023-11-08T05:04:53.745 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: format: 2 2023-11-08T05:04:53.745 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: features: layering, exclusive-lock 2023-11-08T05:04:53.745 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: op_features: 2023-11-08T05:04:53.745 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: flags: 2023-11-08T05:04:53.745 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: create_timestamp: Wed Nov 8 05:04:52 2023 2023-11-08T05:04:53.746 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: access_timestamp: Wed Nov 8 05:04:52 2023 2023-11-08T05:04:53.746 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: modify_timestamp: Wed Nov 8 05:04:52 2023 2023-11-08T05:04:53.746 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring state: enabled 2023-11-08T05:04:53.746 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring mode: snapshot 2023-11-08T05:04:53.746 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring global id: 93d1c29c-aa4a-4563-b1cd-30608ea2b43f 2023-11-08T05:04:53.747 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring primary: true 2023-11-08T05:04:53.747 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' -n '' ']' 2023-11-08T05:04:53.748 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ stress_write_image cluster2 mirror image_4 2023-11-08T05:04:53.748 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:04:53.748 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:04:53.748 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_4 2023-11-08T05:04:53.748 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-11-08T05:04:53.749 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local duration=10 2023-11-08T05:04:53.749 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ set +e 2023-11-08T05:04:53.749 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_4 --debug-rbd=20 --debug-journaler=20 2023-11-08T05:04:53.950 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:100 IOs, 45565952 bytes 2023-11-08T05:04:54.167 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:200 IOs, 108916736 bytes 2023-11-08T05:04:54.271 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:300 IOs, 127520768 bytes 2023-11-08T05:04:54.424 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:400 IOs, 156680192 bytes 2023-11-08T05:04:54.579 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:500 IOs, 186970112 bytes 2023-11-08T05:04:54.816 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:600 IOs, 261898240 bytes 2023-11-08T05:04:54.978 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:700 IOs, 291490816 bytes 2023-11-08T05:04:55.149 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:800 IOs, 348024832 bytes 2023-11-08T05:04:55.290 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:900 IOs, 374782976 bytes 2023-11-08T05:04:55.620 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1000 IOs, 479198208 bytes 2023-11-08T05:04:55.734 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1100 IOs, 493283328 bytes 2023-11-08T05:04:55.923 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1200 IOs, 545034240 bytes 2023-11-08T05:04:56.041 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1300 IOs, 579164160 bytes 2023-11-08T05:04:56.328 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1400 IOs, 653053952 bytes 2023-11-08T05:04:56.465 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1500 IOs, 680555520 bytes 2023-11-08T05:04:56.610 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1600 IOs, 708745216 bytes 2023-11-08T05:04:56.831 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1700 IOs, 784493568 bytes 2023-11-08T05:04:56.988 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1800 IOs, 806531072 bytes 2023-11-08T05:04:57.238 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1900 IOs, 886926336 bytes 2023-11-08T05:04:57.428 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2000 IOs, 949092352 bytes 2023-11-08T05:04:57.552 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2100 IOs, 956559360 bytes 2023-11-08T05:04:57.703 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2200 IOs, 985738240 bytes 2023-11-08T05:04:57.814 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2300 IOs, 997877760 bytes 2023-11-08T05:04:58.061 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2400 IOs, 1073251328 bytes 2023-11-08T05:04:58.262 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2500 IOs, 1117483008 bytes 2023-11-08T05:04:58.479 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2600 IOs, 1177435136 bytes 2023-11-08T05:04:58.598 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2700 IOs, 1185236992 bytes 2023-11-08T05:04:58.803 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2800 IOs, 1244956672 bytes 2023-11-08T05:04:59.138 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2900 IOs, 1364157440 bytes 2023-11-08T05:04:59.358 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3000 IOs, 1430444032 bytes 2023-11-08T05:04:59.546 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3100 IOs, 1477607424 bytes 2023-11-08T05:04:59.683 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3200 IOs, 1502264320 bytes 2023-11-08T05:04:59.883 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3300 IOs, 1553271808 bytes 2023-11-08T05:05:00.125 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3400 IOs, 1626419200 bytes 2023-11-08T05:05:00.289 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3500 IOs, 1666513920 bytes 2023-11-08T05:05:00.431 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3600 IOs, 1687596032 bytes 2023-11-08T05:05:00.618 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3700 IOs, 1757096960 bytes 2023-11-08T05:05:00.762 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3800 IOs, 1773536256 bytes 2023-11-08T05:05:01.014 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3900 IOs, 1849926656 bytes 2023-11-08T05:05:01.190 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4000 IOs, 1895598080 bytes 2023-11-08T05:05:01.369 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4100 IOs, 1935776768 bytes 2023-11-08T05:05:01.529 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4200 IOs, 1974959104 bytes 2023-11-08T05:05:01.632 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4300 IOs, 1984215040 bytes 2023-11-08T05:05:01.812 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4400 IOs, 2036064256 bytes 2023-11-08T05:05:01.976 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4500 IOs, 2076538880 bytes 2023-11-08T05:05:02.186 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4600 IOs, 2132930560 bytes 2023-11-08T05:05:02.364 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4700 IOs, 2179108864 bytes 2023-11-08T05:05:02.497 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4800 IOs, 2221987840 bytes 2023-11-08T05:05:02.699 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4900 IOs, 2253638656 bytes 2023-11-08T05:05:02.918 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5000 IOs, 2322027520 bytes 2023-11-08T05:05:03.171 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5100 IOs, 2397559808 bytes 2023-11-08T05:05:03.359 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5200 IOs, 2447576064 bytes 2023-11-08T05:05:03.563 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5300 IOs, 2494136320 bytes 2023-11-08T05:05:03.755 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ error_code=124 2023-11-08T05:05:03.755 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ set -e 2023-11-08T05:05:03.755 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' 124 -eq 124 ']' 2023-11-08T05:05:03.756 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:05:03.756 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-11-08T05:05:03.756 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image=image_5 2023-11-08T05:05:03.756 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ create_image_and_enable_mirror cluster2 mirror image_5 snapshot 128M 2023-11-08T05:05:03.756 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:05:03.757 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:05:03.757 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:05:03.757 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:05:03.757 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_5 2023-11-08T05:05:03.757 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:05:03.758 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local mode=snapshot 2023-11-08T05:05:03.758 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' -n snapshot ']' 2023-11-08T05:05:03.758 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:05:03.758 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ create_image cluster2 mirror image_5 128M 2023-11-08T05:05:03.758 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:05:03.759 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:05:03.759 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:05:03.759 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:05:03.759 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_5 2023-11-08T05:05:03.759 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:05:03.760 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local size=128 2023-11-08T05:05:03.760 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' -n 128M ']' 2023-11-08T05:05:03.760 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ size=128M 2023-11-08T05:05:03.760 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:05:03.760 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_5 2023-11-08T05:05:03.793 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' image = image ']' 2023-11-08T05:05:03.793 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ enable_mirror cluster2 mirror image_5 snapshot 2023-11-08T05:05:03.793 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:05:03.793 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:05:03.794 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_5 2023-11-08T05:05:03.794 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local mode=snapshot 2023-11-08T05:05:03.794 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_5 snapshot 2023-11-08T05:05:04.780 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:Mirroring enabled 2023-11-08T05:05:04.788 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster=cluster2 info mirror/image_5 2023-11-08T05:05:04.823 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:rbd image 'image_5': 2023-11-08T05:05:04.823 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: size 128 MiB in 32 objects 2023-11-08T05:05:04.823 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: order 22 (4 MiB objects) 2023-11-08T05:05:04.824 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: snapshot_count: 1 2023-11-08T05:05:04.824 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: id: 1289d692a1a 2023-11-08T05:05:04.824 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: block_name_prefix: rbd_data.1289d692a1a 2023-11-08T05:05:04.824 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: format: 2 2023-11-08T05:05:04.824 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: features: layering, exclusive-lock 2023-11-08T05:05:04.825 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: op_features: 2023-11-08T05:05:04.825 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: flags: 2023-11-08T05:05:04.825 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: create_timestamp: Wed Nov 8 05:05:03 2023 2023-11-08T05:05:04.825 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: access_timestamp: Wed Nov 8 05:05:03 2023 2023-11-08T05:05:04.825 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: modify_timestamp: Wed Nov 8 05:05:03 2023 2023-11-08T05:05:04.826 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring state: enabled 2023-11-08T05:05:04.826 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring mode: snapshot 2023-11-08T05:05:04.826 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring global id: c80b0fb2-a1fa-458e-baf2-277b0fc2657c 2023-11-08T05:05:04.826 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring primary: true 2023-11-08T05:05:04.827 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' -n '' ']' 2023-11-08T05:05:04.827 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ stress_write_image cluster2 mirror image_5 2023-11-08T05:05:04.828 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:05:04.828 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:05:04.828 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_5 2023-11-08T05:05:04.828 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-11-08T05:05:04.829 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local duration=9 2023-11-08T05:05:04.829 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ set +e 2023-11-08T05:05:04.829 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_5 --debug-rbd=20 --debug-journaler=20 2023-11-08T05:05:05.063 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:100 IOs, 68744192 bytes 2023-11-08T05:05:05.167 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:200 IOs, 92262400 bytes 2023-11-08T05:05:05.434 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:300 IOs, 117671936 bytes 2023-11-08T05:05:05.577 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:400 IOs, 148332544 bytes 2023-11-08T05:05:05.853 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:500 IOs, 247661568 bytes 2023-11-08T05:05:05.958 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:600 IOs, 255221760 bytes 2023-11-08T05:05:06.261 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:700 IOs, 366359552 bytes 2023-11-08T05:05:06.501 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:800 IOs, 463482880 bytes 2023-11-08T05:05:06.686 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:900 IOs, 496276480 bytes 2023-11-08T05:05:06.840 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1000 IOs, 542785536 bytes 2023-11-08T05:05:07.051 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1100 IOs, 583808000 bytes 2023-11-08T05:05:07.169 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1200 IOs, 597154816 bytes 2023-11-08T05:05:07.309 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1300 IOs, 624844800 bytes 2023-11-08T05:05:07.620 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1400 IOs, 729680896 bytes 2023-11-08T05:05:07.776 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1500 IOs, 762553344 bytes 2023-11-08T05:05:07.953 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1600 IOs, 802714624 bytes 2023-11-08T05:05:08.041 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1700 IOs, 814316544 bytes 2023-11-08T05:05:08.161 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1800 IOs, 822277120 bytes 2023-11-08T05:05:08.364 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1900 IOs, 878971904 bytes 2023-11-08T05:05:08.551 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2000 IOs, 922333184 bytes 2023-11-08T05:05:08.656 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2100 IOs, 938219520 bytes 2023-11-08T05:05:08.919 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2200 IOs, 1029016576 bytes 2023-11-08T05:05:09.047 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2300 IOs, 1045958656 bytes 2023-11-08T05:05:09.327 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2400 IOs, 1145978880 bytes 2023-11-08T05:05:09.537 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2500 IOs, 1208441856 bytes 2023-11-08T05:05:09.641 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2600 IOs, 1209051136 bytes 2023-11-08T05:05:09.783 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2700 IOs, 1233303552 bytes 2023-11-08T05:05:10.019 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2800 IOs, 1320175616 bytes 2023-11-08T05:05:10.284 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2900 IOs, 1384702976 bytes 2023-11-08T05:05:10.489 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3000 IOs, 1447920640 bytes 2023-11-08T05:05:10.743 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3100 IOs, 1532686336 bytes 2023-11-08T05:05:10.921 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3200 IOs, 1581551616 bytes 2023-11-08T05:05:11.081 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3300 IOs, 1611280384 bytes 2023-11-08T05:05:11.260 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3400 IOs, 1659623424 bytes 2023-11-08T05:05:11.392 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3500 IOs, 1685665792 bytes 2023-11-08T05:05:11.503 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3600 IOs, 1700817920 bytes 2023-11-08T05:05:11.607 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3700 IOs, 1717664768 bytes 2023-11-08T05:05:11.879 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3800 IOs, 1809616896 bytes 2023-11-08T05:05:12.060 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3900 IOs, 1850297344 bytes 2023-11-08T05:05:12.145 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4000 IOs, 1855240192 bytes 2023-11-08T05:05:12.335 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4100 IOs, 1911527424 bytes 2023-11-08T05:05:12.509 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4200 IOs, 1945131008 bytes 2023-11-08T05:05:12.686 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4300 IOs, 1987398656 bytes 2023-11-08T05:05:12.896 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4400 IOs, 2044305408 bytes 2023-11-08T05:05:13.008 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4500 IOs, 2056977408 bytes 2023-11-08T05:05:13.276 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4600 IOs, 2155013120 bytes 2023-11-08T05:05:13.407 INFO:tasks.ceph.cluster2.osd.0.smithi087.stderr:2023-11-08T05:05:13.405+0000 7f1ac5683700 -1 reset not still connected to 0x55f9e3216410 2023-11-08T05:05:13.411 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4700 IOs, 2171123712 bytes 2023-11-08T05:05:13.704 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4800 IOs, 2262861824 bytes 2023-11-08T05:05:13.802 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4900 IOs, 2267270144 bytes 2023-11-08T05:05:13.838 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ error_code=124 2023-11-08T05:05:13.838 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ set -e 2023-11-08T05:05:13.839 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' 124 -eq 124 ']' 2023-11-08T05:05:13.839 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:05:13.839 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-11-08T05:05:13.839 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image=image_6 2023-11-08T05:05:13.839 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ create_image_and_enable_mirror cluster2 mirror image_6 snapshot 128M 2023-11-08T05:05:13.840 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:05:13.840 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:05:13.840 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:05:13.840 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:05:13.840 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_6 2023-11-08T05:05:13.841 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:05:13.841 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local mode=snapshot 2023-11-08T05:05:13.841 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' -n snapshot ']' 2023-11-08T05:05:13.841 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:05:13.842 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ create_image cluster2 mirror image_6 128M 2023-11-08T05:05:13.842 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:05:13.842 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:05:13.842 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:05:13.842 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:05:13.843 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_6 2023-11-08T05:05:13.843 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:05:13.843 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local size=128 2023-11-08T05:05:13.843 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' -n 128M ']' 2023-11-08T05:05:13.843 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ size=128M 2023-11-08T05:05:13.844 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:05:13.844 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_6 2023-11-08T05:05:13.878 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' image = image ']' 2023-11-08T05:05:13.878 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ enable_mirror cluster2 mirror image_6 snapshot 2023-11-08T05:05:13.878 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:05:13.879 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:05:13.879 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_6 2023-11-08T05:05:13.879 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local mode=snapshot 2023-11-08T05:05:13.879 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_6 snapshot 2023-11-08T05:05:14.047 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.0 2023-11-08T05:05:14.048 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Sent signal 15 2023-11-08T05:05:14.048 INFO:tasks.rbd_mirror_thrash:waiting for 8 secs before reviving daemons 2023-11-08T05:05:14.049 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi087.stderr:2023-11-08T05:05:14.047+0000 7fd7d4fd8700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 (PID: 126223) UID: 1000 2023-11-08T05:05:14.249 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi087.stderr:daemon-helper: command failed with exit status 1 2023-11-08T05:05:14.785 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:Mirroring enabled 2023-11-08T05:05:14.792 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster=cluster2 info mirror/image_6 2023-11-08T05:05:14.822 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:rbd image 'image_6': 2023-11-08T05:05:14.822 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: size 128 MiB in 32 objects 2023-11-08T05:05:14.822 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: order 22 (4 MiB objects) 2023-11-08T05:05:14.823 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: snapshot_count: 1 2023-11-08T05:05:14.823 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: id: 1296507baf9d 2023-11-08T05:05:14.823 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: block_name_prefix: rbd_data.1296507baf9d 2023-11-08T05:05:14.823 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: format: 2 2023-11-08T05:05:14.823 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: features: layering, exclusive-lock 2023-11-08T05:05:14.824 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: op_features: 2023-11-08T05:05:14.824 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: flags: 2023-11-08T05:05:14.824 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: create_timestamp: Wed Nov 8 05:05:13 2023 2023-11-08T05:05:14.824 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: access_timestamp: Wed Nov 8 05:05:13 2023 2023-11-08T05:05:14.824 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: modify_timestamp: Wed Nov 8 05:05:13 2023 2023-11-08T05:05:14.824 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring state: enabled 2023-11-08T05:05:14.825 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring mode: snapshot 2023-11-08T05:05:14.825 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring global id: acb6ea51-6256-49db-9edb-d84ff9a099a2 2023-11-08T05:05:14.825 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring primary: true 2023-11-08T05:05:14.826 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' -n '' ']' 2023-11-08T05:05:14.826 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ stress_write_image cluster2 mirror image_6 2023-11-08T05:05:14.826 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:05:14.826 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:05:14.827 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_6 2023-11-08T05:05:14.827 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-11-08T05:05:14.827 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local duration=7 2023-11-08T05:05:14.828 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ set +e 2023-11-08T05:05:14.828 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_6 --debug-rbd=20 --debug-journaler=20 2023-11-08T05:05:14.989 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:100 IOs, 16710656 bytes 2023-11-08T05:05:15.115 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:200 IOs, 38155264 bytes 2023-11-08T05:05:15.211 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:300 IOs, 47969280 bytes 2023-11-08T05:05:15.331 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:400 IOs, 67185664 bytes 2023-11-08T05:05:15.746 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:500 IOs, 126502912 bytes 2023-11-08T05:05:15.923 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:600 IOs, 191705088 bytes 2023-11-08T05:05:16.106 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:700 IOs, 222454784 bytes 2023-11-08T05:05:16.210 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:800 IOs, 234218496 bytes 2023-11-08T05:05:16.423 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:900 IOs, 291577856 bytes 2023-11-08T05:05:16.599 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1000 IOs, 357732352 bytes 2023-11-08T05:05:16.929 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1100 IOs, 468012032 bytes 2023-11-08T05:05:17.214 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1200 IOs, 562936832 bytes 2023-11-08T05:05:17.343 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1300 IOs, 568903680 bytes 2023-11-08T05:05:17.585 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1400 IOs, 651218944 bytes 2023-11-08T05:05:17.751 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1500 IOs, 700811264 bytes 2023-11-08T05:05:17.993 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1600 IOs, 777686016 bytes 2023-11-08T05:05:18.172 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1700 IOs, 822837248 bytes 2023-11-08T05:05:18.339 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1800 IOs, 865599488 bytes 2023-11-08T05:05:18.455 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1900 IOs, 875002880 bytes 2023-11-08T05:05:18.670 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2000 IOs, 938949632 bytes 2023-11-08T05:05:18.780 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2100 IOs, 950643712 bytes 2023-11-08T05:05:18.958 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2200 IOs, 987864064 bytes 2023-11-08T05:05:19.230 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2300 IOs, 1079284736 bytes 2023-11-08T05:05:19.410 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2400 IOs, 1126023168 bytes 2023-11-08T05:05:19.534 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2500 IOs, 1148705792 bytes 2023-11-08T05:05:19.643 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2600 IOs, 1158045696 bytes 2023-11-08T05:05:19.798 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2700 IOs, 1193876480 bytes 2023-11-08T05:05:19.990 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2800 IOs, 1249587200 bytes 2023-11-08T05:05:20.104 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2900 IOs, 1265761280 bytes 2023-11-08T05:05:20.277 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3000 IOs, 1311007744 bytes 2023-11-08T05:05:20.445 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3100 IOs, 1373631488 bytes 2023-11-08T05:05:20.685 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3200 IOs, 1428331520 bytes 2023-11-08T05:05:20.804 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3300 IOs, 1452605440 bytes 2023-11-08T05:05:20.966 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3400 IOs, 1507952640 bytes 2023-11-08T05:05:21.200 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3500 IOs, 1570115584 bytes 2023-11-08T05:05:21.407 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3600 IOs, 1622134784 bytes 2023-11-08T05:05:21.567 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3700 IOs, 1651047424 bytes 2023-11-08T05:05:21.667 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3800 IOs, 1651657728 bytes 2023-11-08T05:05:21.749 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3900 IOs, 1652263936 bytes 2023-11-08T05:05:21.832 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ error_code=124 2023-11-08T05:05:21.833 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ set -e 2023-11-08T05:05:21.833 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' 124 -eq 124 ']' 2023-11-08T05:05:21.833 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:05:21.833 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-11-08T05:05:21.833 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image=image_7 2023-11-08T05:05:21.834 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ create_image_and_enable_mirror cluster2 mirror image_7 snapshot 128M 2023-11-08T05:05:21.834 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:05:21.834 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:05:21.834 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:05:21.835 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:05:21.835 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_7 2023-11-08T05:05:21.835 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:05:21.835 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local mode=snapshot 2023-11-08T05:05:21.835 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' -n snapshot ']' 2023-11-08T05:05:21.835 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:05:21.836 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ create_image cluster2 mirror image_7 128M 2023-11-08T05:05:21.836 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:05:21.836 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:05:21.836 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:05:21.836 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:05:21.837 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_7 2023-11-08T05:05:21.837 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:05:21.837 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local size=128 2023-11-08T05:05:21.837 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' -n 128M ']' 2023-11-08T05:05:21.837 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ size=128M 2023-11-08T05:05:21.838 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:05:21.838 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_7 2023-11-08T05:05:21.871 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' image = image ']' 2023-11-08T05:05:21.871 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ enable_mirror cluster2 mirror image_7 snapshot 2023-11-08T05:05:21.871 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:05:21.872 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:05:21.872 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_7 2023-11-08T05:05:21.872 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local mode=snapshot 2023-11-08T05:05:21.872 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_7 snapshot 2023-11-08T05:05:22.050 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.0 2023-11-08T05:05:22.050 INFO:teuthology.orchestra.run:waiting for 600 2023-11-08T05:05:22.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-08T05:05:22.051 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.0 2023-11-08T05:05:22.051 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Restarting daemon 2023-11-08T05:05:22.052 DEBUG:teuthology.orchestra.run.smithi087:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 2023-11-08T05:05:22.054 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Started 2023-11-08T05:05:22.055 INFO:tasks.rbd_mirror_thrash:waiting for 113 secs before thrashing 2023-11-08T05:05:22.820 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:Mirroring enabled 2023-11-08T05:05:22.826 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster=cluster2 info mirror/image_7 2023-11-08T05:05:22.856 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:rbd image 'image_7': 2023-11-08T05:05:22.857 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: size 128 MiB in 32 objects 2023-11-08T05:05:22.857 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: order 22 (4 MiB objects) 2023-11-08T05:05:22.857 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: snapshot_count: 1 2023-11-08T05:05:22.857 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: id: 12a263f30a01 2023-11-08T05:05:22.858 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: block_name_prefix: rbd_data.12a263f30a01 2023-11-08T05:05:22.858 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: format: 2 2023-11-08T05:05:22.858 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: features: layering, exclusive-lock 2023-11-08T05:05:22.858 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: op_features: 2023-11-08T05:05:22.858 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: flags: 2023-11-08T05:05:22.858 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: create_timestamp: Wed Nov 8 05:05:21 2023 2023-11-08T05:05:22.859 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: access_timestamp: Wed Nov 8 05:05:21 2023 2023-11-08T05:05:22.859 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: modify_timestamp: Wed Nov 8 05:05:21 2023 2023-11-08T05:05:22.859 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring state: enabled 2023-11-08T05:05:22.859 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring mode: snapshot 2023-11-08T05:05:22.859 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring global id: baa8ce06-b6bc-450a-80c9-52c7757d4aa3 2023-11-08T05:05:22.860 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring primary: true 2023-11-08T05:05:22.860 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' -n '' ']' 2023-11-08T05:05:22.860 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ stress_write_image cluster2 mirror image_7 2023-11-08T05:05:22.861 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:05:22.861 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:05:22.861 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_7 2023-11-08T05:05:22.861 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-11-08T05:05:22.862 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local duration=12 2023-11-08T05:05:22.862 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ set +e 2023-11-08T05:05:22.862 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_7 --debug-rbd=20 --debug-journaler=20 2023-11-08T05:05:23.032 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:100 IOs, 29532160 bytes 2023-11-08T05:05:23.151 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:200 IOs, 49144832 bytes 2023-11-08T05:05:23.375 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:300 IOs, 118279168 bytes 2023-11-08T05:05:23.527 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:400 IOs, 178491392 bytes 2023-11-08T05:05:23.735 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:500 IOs, 225739776 bytes 2023-11-08T05:05:23.929 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:600 IOs, 283719680 bytes 2023-11-08T05:05:24.111 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:700 IOs, 325796864 bytes 2023-11-08T05:05:24.294 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:800 IOs, 383653888 bytes 2023-11-08T05:05:24.439 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:900 IOs, 409878528 bytes 2023-11-08T05:05:24.601 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1000 IOs, 452183040 bytes 2023-11-08T05:05:24.695 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1100 IOs, 497778688 bytes 2023-11-08T05:05:24.890 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1200 IOs, 516028416 bytes 2023-11-08T05:05:25.041 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1300 IOs, 550410240 bytes 2023-11-08T05:05:25.053 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.3675071603918514) > weight (0.25) 2023-11-08T05:05:25.053 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.9877627699057452) > weight (0.25) 2023-11-08T05:05:25.053 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.7940191033470191) > weight (0.25) 2023-11-08T05:05:25.053 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.3 with skip (0.716526067002451) > weight (0.25) 2023-11-08T05:05:25.054 INFO:tasks.rbd_mirror_thrash:waiting for 103 secs before thrashing 2023-11-08T05:05:25.157 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1400 IOs, 572214272 bytes 2023-11-08T05:05:25.377 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1500 IOs, 625233920 bytes 2023-11-08T05:05:25.458 INFO:tasks.ceph.cluster2.osd.1.smithi087.stderr:2023-11-08T05:05:25.455+0000 7fbb7a4fc700 -1 reset not still connected to 0x5576fdb3e680 2023-11-08T05:05:25.528 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1600 IOs, 656216064 bytes 2023-11-08T05:05:25.709 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1700 IOs, 699503616 bytes 2023-11-08T05:05:25.803 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1800 IOs, 712823808 bytes 2023-11-08T05:05:26.118 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1900 IOs, 830078976 bytes 2023-11-08T05:05:26.245 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2000 IOs, 850013184 bytes 2023-11-08T05:05:26.340 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2100 IOs, 856783872 bytes 2023-11-08T05:05:26.468 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2200 IOs, 893713408 bytes 2023-11-08T05:05:26.563 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2300 IOs, 894355456 bytes 2023-11-08T05:05:26.759 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2400 IOs, 957452288 bytes 2023-11-08T05:05:27.033 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2500 IOs, 1051471872 bytes 2023-11-08T05:05:27.130 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2600 IOs, 1059757056 bytes 2023-11-08T05:05:27.299 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2700 IOs, 1104167936 bytes 2023-11-08T05:05:27.396 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2800 IOs, 1104772096 bytes 2023-11-08T05:05:27.533 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2900 IOs, 1124623360 bytes 2023-11-08T05:05:27.730 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3000 IOs, 1181933568 bytes 2023-11-08T05:05:27.933 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3100 IOs, 1242628096 bytes 2023-11-08T05:05:28.108 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3200 IOs, 1301662720 bytes 2023-11-08T05:05:28.335 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3300 IOs, 1356935168 bytes 2023-11-08T05:05:28.610 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3400 IOs, 1366646784 bytes 2023-11-08T05:05:28.731 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3500 IOs, 1385177088 bytes 2023-11-08T05:05:28.997 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3600 IOs, 1478413312 bytes 2023-11-08T05:05:29.127 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3700 IOs, 1501350912 bytes 2023-11-08T05:05:29.280 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3800 IOs, 1532286976 bytes 2023-11-08T05:05:29.451 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3900 IOs, 1576544256 bytes 2023-11-08T05:05:29.630 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4000 IOs, 1656797184 bytes 2023-11-08T05:05:29.850 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4100 IOs, 1698387968 bytes 2023-11-08T05:05:30.025 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4200 IOs, 1742422016 bytes 2023-11-08T05:05:30.204 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4300 IOs, 1791849472 bytes 2023-11-08T05:05:30.357 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4400 IOs, 1826600960 bytes 2023-11-08T05:05:30.475 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4500 IOs, 1845447680 bytes 2023-11-08T05:05:30.669 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4600 IOs, 1888940032 bytes 2023-11-08T05:05:30.834 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4700 IOs, 1956145152 bytes 2023-11-08T05:05:31.044 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4800 IOs, 1993095168 bytes 2023-11-08T05:05:31.216 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4900 IOs, 2033846272 bytes 2023-11-08T05:05:31.366 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5000 IOs, 2057896960 bytes 2023-11-08T05:05:31.673 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5100 IOs, 2108909568 bytes 2023-11-08T05:05:31.868 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5200 IOs, 2146613248 bytes 2023-11-08T05:05:31.999 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5300 IOs, 2175638528 bytes 2023-11-08T05:05:32.286 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5400 IOs, 2272031744 bytes 2023-11-08T05:05:32.586 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5500 IOs, 2393723904 bytes 2023-11-08T05:05:32.735 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5600 IOs, 2411003904 bytes 2023-11-08T05:05:32.847 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5700 IOs, 2429635584 bytes 2023-11-08T05:05:32.979 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5800 IOs, 2448476160 bytes 2023-11-08T05:05:33.096 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5900 IOs, 2465697792 bytes 2023-11-08T05:05:33.354 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:6000 IOs, 2557728768 bytes 2023-11-08T05:05:33.580 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:6100 IOs, 2632905728 bytes 2023-11-08T05:05:33.750 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:6200 IOs, 2672555008 bytes 2023-11-08T05:05:33.986 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:6300 IOs, 2760296448 bytes 2023-11-08T05:05:34.268 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:6400 IOs, 2840435712 bytes 2023-11-08T05:05:34.531 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:6500 IOs, 2923689984 bytes 2023-11-08T05:05:34.631 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:6600 IOs, 2935686144 bytes 2023-11-08T05:05:34.848 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:6700 IOs, 3001077760 bytes 2023-11-08T05:05:34.867 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ error_code=124 2023-11-08T05:05:34.867 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ set -e 2023-11-08T05:05:34.867 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' 124 -eq 124 ']' 2023-11-08T05:05:34.868 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:05:34.868 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-11-08T05:05:34.868 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image=image_8 2023-11-08T05:05:34.868 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ create_image_and_enable_mirror cluster2 mirror image_8 snapshot 128M 2023-11-08T05:05:34.868 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:05:34.869 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:05:34.869 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:05:34.869 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:05:34.869 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_8 2023-11-08T05:05:34.869 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:05:34.870 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local mode=snapshot 2023-11-08T05:05:34.870 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' -n snapshot ']' 2023-11-08T05:05:34.870 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:05:34.870 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ create_image cluster2 mirror image_8 128M 2023-11-08T05:05:34.871 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:05:34.871 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:05:34.871 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:05:34.871 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:05:34.871 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_8 2023-11-08T05:05:34.872 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:05:34.872 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local size=128 2023-11-08T05:05:34.872 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' -n 128M ']' 2023-11-08T05:05:34.872 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ size=128M 2023-11-08T05:05:34.872 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:05:34.873 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_8 2023-11-08T05:05:34.905 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' image = image ']' 2023-11-08T05:05:34.905 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ enable_mirror cluster2 mirror image_8 snapshot 2023-11-08T05:05:34.905 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:05:34.906 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:05:34.906 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_8 2023-11-08T05:05:34.906 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local mode=snapshot 2023-11-08T05:05:34.906 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_8 snapshot 2023-11-08T05:05:35.933 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:Mirroring enabled 2023-11-08T05:05:35.941 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster=cluster2 info mirror/image_8 2023-11-08T05:05:35.975 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:rbd image 'image_8': 2023-11-08T05:05:35.975 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: size 128 MiB in 32 objects 2023-11-08T05:05:35.975 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: order 22 (4 MiB objects) 2023-11-08T05:05:35.975 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: snapshot_count: 1 2023-11-08T05:05:35.976 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: id: 12b8ef1af47a 2023-11-08T05:05:35.976 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: block_name_prefix: rbd_data.12b8ef1af47a 2023-11-08T05:05:35.976 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: format: 2 2023-11-08T05:05:35.976 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: features: layering, exclusive-lock 2023-11-08T05:05:35.976 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: op_features: 2023-11-08T05:05:35.977 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: flags: 2023-11-08T05:05:35.977 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: create_timestamp: Wed Nov 8 05:05:34 2023 2023-11-08T05:05:35.977 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: access_timestamp: Wed Nov 8 05:05:34 2023 2023-11-08T05:05:35.977 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: modify_timestamp: Wed Nov 8 05:05:34 2023 2023-11-08T05:05:35.977 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring state: enabled 2023-11-08T05:05:35.978 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring mode: snapshot 2023-11-08T05:05:35.978 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring global id: c78a25c8-e255-457e-8ce9-bb570e06b394 2023-11-08T05:05:35.978 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring primary: true 2023-11-08T05:05:35.979 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' -n '' ']' 2023-11-08T05:05:35.979 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ stress_write_image cluster2 mirror image_8 2023-11-08T05:05:35.979 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:05:35.980 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:05:35.980 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_8 2023-11-08T05:05:35.980 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-11-08T05:05:35.981 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local duration=13 2023-11-08T05:05:35.981 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ set +e 2023-11-08T05:05:35.981 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_8 --debug-rbd=20 --debug-journaler=20 2023-11-08T05:05:36.223 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:100 IOs, 64082944 bytes 2023-11-08T05:05:36.387 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:200 IOs, 89479168 bytes 2023-11-08T05:05:36.513 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:300 IOs, 119443456 bytes 2023-11-08T05:05:36.732 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:400 IOs, 189575168 bytes 2023-11-08T05:05:36.960 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:500 IOs, 263349248 bytes 2023-11-08T05:05:37.125 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:600 IOs, 302429184 bytes 2023-11-08T05:05:37.377 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:700 IOs, 383373312 bytes 2023-11-08T05:05:37.583 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:800 IOs, 436582400 bytes 2023-11-08T05:05:37.741 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:900 IOs, 469327872 bytes 2023-11-08T05:05:37.891 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1000 IOs, 505485312 bytes 2023-11-08T05:05:38.001 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1100 IOs, 520893440 bytes 2023-11-08T05:05:38.218 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1200 IOs, 584602624 bytes 2023-11-08T05:05:38.438 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1300 IOs, 655841280 bytes 2023-11-08T05:05:38.576 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1400 IOs, 677045248 bytes 2023-11-08T05:05:38.666 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1500 IOs, 678934528 bytes 2023-11-08T05:05:38.783 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1600 IOs, 689892352 bytes 2023-11-08T05:05:39.023 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1700 IOs, 760459264 bytes 2023-11-08T05:05:39.176 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1800 IOs, 804704256 bytes 2023-11-08T05:05:39.391 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1900 IOs, 845827072 bytes 2023-11-08T05:05:39.526 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2000 IOs, 878048256 bytes 2023-11-08T05:05:39.711 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2100 IOs, 917325824 bytes 2023-11-08T05:05:39.865 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2200 IOs, 942918656 bytes 2023-11-08T05:05:40.178 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2300 IOs, 1048347648 bytes 2023-11-08T05:05:40.359 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2400 IOs, 1090848768 bytes 2023-11-08T05:05:40.479 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2500 IOs, 1104547840 bytes 2023-11-08T05:05:40.729 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2600 IOs, 1179147264 bytes 2023-11-08T05:05:40.817 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2700 IOs, 1179772928 bytes 2023-11-08T05:05:40.997 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2800 IOs, 1220285440 bytes 2023-11-08T05:05:41.214 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2900 IOs, 1299946496 bytes 2023-11-08T05:05:41.361 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3000 IOs, 1318986752 bytes 2023-11-08T05:05:41.551 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3100 IOs, 1358953472 bytes 2023-11-08T05:05:41.713 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3200 IOs, 1402360832 bytes 2023-11-08T05:05:41.876 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3300 IOs, 1439486976 bytes 2023-11-08T05:05:42.052 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3400 IOs, 1476380672 bytes 2023-11-08T05:05:42.262 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3500 IOs, 1533094912 bytes 2023-11-08T05:05:42.428 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3600 IOs, 1573287936 bytes 2023-11-08T05:05:42.652 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3700 IOs, 1662511104 bytes 2023-11-08T05:05:42.875 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3800 IOs, 1716486144 bytes 2023-11-08T05:05:43.094 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3900 IOs, 1783076864 bytes 2023-11-08T05:05:43.268 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4000 IOs, 1822248960 bytes 2023-11-08T05:05:43.363 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4100 IOs, 1828984832 bytes 2023-11-08T05:05:43.550 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4200 IOs, 1880864768 bytes 2023-11-08T05:05:43.729 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4300 IOs, 1943988224 bytes 2023-11-08T05:05:43.958 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4400 IOs, 1981169664 bytes 2023-11-08T05:05:44.077 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4500 IOs, 1991700480 bytes 2023-11-08T05:05:44.280 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4600 IOs, 2040679424 bytes 2023-11-08T05:05:44.381 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4700 IOs, 2050418688 bytes 2023-11-08T05:05:44.525 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4800 IOs, 2090318848 bytes 2023-11-08T05:05:44.703 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4900 IOs, 2124446720 bytes 2023-11-08T05:05:44.908 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5000 IOs, 2170307584 bytes 2023-11-08T05:05:45.089 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5100 IOs, 2208828416 bytes 2023-11-08T05:05:45.213 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5200 IOs, 2226262016 bytes 2023-11-08T05:05:45.451 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5300 IOs, 2305364992 bytes 2023-11-08T05:05:45.567 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5400 IOs, 2318936064 bytes 2023-11-08T05:05:45.712 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5500 IOs, 2342810624 bytes 2023-11-08T05:05:45.828 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5600 IOs, 2361831424 bytes 2023-11-08T05:05:45.968 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5700 IOs, 2390150144 bytes 2023-11-08T05:05:46.172 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5800 IOs, 2446498816 bytes 2023-11-08T05:05:46.302 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5900 IOs, 2469958656 bytes 2023-11-08T05:05:46.444 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:6000 IOs, 2488210432 bytes 2023-11-08T05:05:46.582 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:6100 IOs, 2526036992 bytes 2023-11-08T05:05:46.775 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:6200 IOs, 2562906112 bytes 2023-11-08T05:05:46.987 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:6300 IOs, 2629369856 bytes 2023-11-08T05:05:47.180 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:6400 IOs, 2683715584 bytes 2023-11-08T05:05:47.329 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:6500 IOs, 2711206912 bytes 2023-11-08T05:05:47.476 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:6600 IOs, 2737152000 bytes 2023-11-08T05:05:47.611 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:6700 IOs, 2761504768 bytes 2023-11-08T05:05:47.855 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:6800 IOs, 2839930880 bytes 2023-11-08T05:05:48.039 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:6900 IOs, 2882640896 bytes 2023-11-08T05:05:48.230 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:7000 IOs, 2929956864 bytes 2023-11-08T05:05:48.322 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:7100 IOs, 2930591744 bytes 2023-11-08T05:05:48.480 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:7200 IOs, 2971731968 bytes 2023-11-08T05:05:48.632 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:7300 IOs, 3018693632 bytes 2023-11-08T05:05:48.837 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:7400 IOs, 3060727808 bytes 2023-11-08T05:05:48.986 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ error_code=124 2023-11-08T05:05:48.986 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ set -e 2023-11-08T05:05:48.986 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' 124 -eq 124 ']' 2023-11-08T05:05:48.987 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:05:48.987 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-11-08T05:05:48.987 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image=image_9 2023-11-08T05:05:48.987 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ create_image_and_enable_mirror cluster2 mirror image_9 snapshot 128M 2023-11-08T05:05:48.988 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:05:48.988 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:05:48.988 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:05:48.988 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:05:48.988 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_9 2023-11-08T05:05:48.989 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:05:48.989 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local mode=snapshot 2023-11-08T05:05:48.989 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' -n snapshot ']' 2023-11-08T05:05:48.989 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:05:48.989 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ create_image cluster2 mirror image_9 128M 2023-11-08T05:05:48.989 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:05:48.990 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:05:48.990 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:05:48.990 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:05:48.990 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_9 2023-11-08T05:05:48.990 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:05:48.991 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local size=128 2023-11-08T05:05:48.991 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' -n 128M ']' 2023-11-08T05:05:48.991 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ size=128M 2023-11-08T05:05:48.991 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:05:48.991 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_9 2023-11-08T05:05:49.026 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' image = image ']' 2023-11-08T05:05:49.026 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ enable_mirror cluster2 mirror image_9 snapshot 2023-11-08T05:05:49.026 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:05:49.026 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:05:49.027 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_9 2023-11-08T05:05:49.027 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local mode=snapshot 2023-11-08T05:05:49.027 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_9 snapshot 2023-11-08T05:05:50.009 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:Mirroring enabled 2023-11-08T05:05:50.016 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster=cluster2 info mirror/image_9 2023-11-08T05:05:50.049 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:rbd image 'image_9': 2023-11-08T05:05:50.049 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: size 128 MiB in 32 objects 2023-11-08T05:05:50.049 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: order 22 (4 MiB objects) 2023-11-08T05:05:50.049 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: snapshot_count: 1 2023-11-08T05:05:50.050 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: id: 12c4f5cbb18a 2023-11-08T05:05:50.050 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: block_name_prefix: rbd_data.12c4f5cbb18a 2023-11-08T05:05:50.050 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: format: 2 2023-11-08T05:05:50.050 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: features: layering, exclusive-lock 2023-11-08T05:05:50.050 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: op_features: 2023-11-08T05:05:50.051 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: flags: 2023-11-08T05:05:50.051 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: create_timestamp: Wed Nov 8 05:05:49 2023 2023-11-08T05:05:50.051 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: access_timestamp: Wed Nov 8 05:05:49 2023 2023-11-08T05:05:50.051 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: modify_timestamp: Wed Nov 8 05:05:49 2023 2023-11-08T05:05:50.051 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring state: enabled 2023-11-08T05:05:50.052 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring mode: snapshot 2023-11-08T05:05:50.052 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring global id: bd37ef4f-d302-424b-952f-0d71e891061b 2023-11-08T05:05:50.052 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring primary: true 2023-11-08T05:05:50.053 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' -n '' ']' 2023-11-08T05:05:50.053 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ stress_write_image cluster2 mirror image_9 2023-11-08T05:05:50.053 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:05:50.053 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:05:50.054 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_9 2023-11-08T05:05:50.054 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-11-08T05:05:50.054 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local duration=13 2023-11-08T05:05:50.055 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ set +e 2023-11-08T05:05:50.055 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_9 --debug-rbd=20 --debug-journaler=20 2023-11-08T05:05:50.289 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:100 IOs, 59123712 bytes 2023-11-08T05:05:50.429 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:200 IOs, 88835072 bytes 2023-11-08T05:05:50.569 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:300 IOs, 128228352 bytes 2023-11-08T05:05:50.738 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:400 IOs, 168364032 bytes 2023-11-08T05:05:50.950 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:500 IOs, 230506496 bytes 2023-11-08T05:05:51.131 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:600 IOs, 269648896 bytes 2023-11-08T05:05:51.315 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:700 IOs, 322653184 bytes 2023-11-08T05:05:51.507 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:800 IOs, 363464704 bytes 2023-11-08T05:05:51.697 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:900 IOs, 408213504 bytes 2023-11-08T05:05:51.882 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1000 IOs, 451756032 bytes 2023-11-08T05:05:52.113 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1100 IOs, 519506944 bytes 2023-11-08T05:05:52.216 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1200 IOs, 528275456 bytes 2023-11-08T05:05:52.380 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1300 IOs, 564903936 bytes 2023-11-08T05:05:52.656 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1400 IOs, 660880384 bytes 2023-11-08T05:05:52.780 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1500 IOs, 679176192 bytes 2023-11-08T05:05:52.957 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1600 IOs, 723771392 bytes 2023-11-08T05:05:53.079 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1700 IOs, 741432320 bytes 2023-11-08T05:05:53.242 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1800 IOs, 777066496 bytes 2023-11-08T05:05:53.346 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1900 IOs, 784719872 bytes 2023-11-08T05:05:53.576 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2000 IOs, 834150400 bytes 2023-11-08T05:05:53.708 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2100 IOs, 855458816 bytes 2023-11-08T05:05:53.864 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2200 IOs, 886003712 bytes 2023-11-08T05:05:54.058 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2300 IOs, 956677120 bytes 2023-11-08T05:05:54.255 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2400 IOs, 987729920 bytes 2023-11-08T05:05:54.496 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2500 IOs, 1064192000 bytes 2023-11-08T05:05:54.623 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2600 IOs, 1081013248 bytes 2023-11-08T05:05:54.881 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2700 IOs, 1164247040 bytes 2023-11-08T05:05:55.047 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2800 IOs, 1202054144 bytes 2023-11-08T05:05:55.301 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2900 IOs, 1298332672 bytes 2023-11-08T05:05:55.477 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3000 IOs, 1339623424 bytes 2023-11-08T05:05:55.689 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3100 IOs, 1402675200 bytes 2023-11-08T05:05:55.855 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3200 IOs, 1440742400 bytes 2023-11-08T05:05:56.201 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3300 IOs, 1569008640 bytes 2023-11-08T05:05:56.464 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3400 IOs, 1656922112 bytes 2023-11-08T05:05:56.609 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3500 IOs, 1705031680 bytes 2023-11-08T05:05:56.790 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3600 IOs, 1733871616 bytes 2023-11-08T05:05:56.893 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3700 IOs, 1742327808 bytes 2023-11-08T05:05:57.044 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3800 IOs, 1777957888 bytes 2023-11-08T05:05:57.201 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3900 IOs, 1811305472 bytes 2023-11-08T05:05:57.368 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4000 IOs, 1843634176 bytes 2023-11-08T05:05:57.687 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4100 IOs, 1959900160 bytes 2023-11-08T05:05:57.772 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4200 IOs, 1960520704 bytes 2023-11-08T05:05:58.003 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4300 IOs, 2037053440 bytes 2023-11-08T05:05:58.273 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4400 IOs, 2136925184 bytes 2023-11-08T05:05:58.489 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4500 IOs, 2179889152 bytes 2023-11-08T05:05:58.642 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4600 IOs, 2209333248 bytes 2023-11-08T05:05:58.744 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4700 IOs, 2216020992 bytes 2023-11-08T05:05:58.971 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4800 IOs, 2310267904 bytes 2023-11-08T05:05:59.125 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4900 IOs, 2310870016 bytes 2023-11-08T05:05:59.336 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5000 IOs, 2373807104 bytes 2023-11-08T05:05:59.518 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5100 IOs, 2417420288 bytes 2023-11-08T05:05:59.620 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5200 IOs, 2450896896 bytes 2023-11-08T05:05:59.840 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5300 IOs, 2491308032 bytes 2023-11-08T05:06:00.049 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5400 IOs, 2563675136 bytes 2023-11-08T05:06:00.275 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5500 IOs, 2612343808 bytes 2023-11-08T05:06:00.432 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5600 IOs, 2644487168 bytes 2023-11-08T05:06:00.559 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5700 IOs, 2673410048 bytes 2023-11-08T05:06:00.876 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5800 IOs, 2789991424 bytes 2023-11-08T05:06:01.086 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5900 IOs, 2855963648 bytes 2023-11-08T05:06:01.292 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:6000 IOs, 2897313792 bytes 2023-11-08T05:06:01.481 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:6100 IOs, 2961413120 bytes 2023-11-08T05:06:01.669 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:6200 IOs, 2990842880 bytes 2023-11-08T05:06:01.832 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:6300 IOs, 3027456000 bytes 2023-11-08T05:06:02.020 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:6400 IOs, 3074063360 bytes 2023-11-08T05:06:02.173 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:6500 IOs, 3124908032 bytes 2023-11-08T05:06:02.364 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:6600 IOs, 3147653120 bytes 2023-11-08T05:06:02.578 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:6700 IOs, 3209172992 bytes 2023-11-08T05:06:02.729 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:6800 IOs, 3237006336 bytes 2023-11-08T05:06:03.039 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:6900 IOs, 3348622336 bytes 2023-11-08T05:06:03.060 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ error_code=124 2023-11-08T05:06:03.060 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ set -e 2023-11-08T05:06:03.060 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' 124 -eq 124 ']' 2023-11-08T05:06:03.060 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:06:03.061 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-11-08T05:06:03.061 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image=image_10 2023-11-08T05:06:03.061 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ create_image_and_enable_mirror cluster2 mirror image_10 snapshot 128M 2023-11-08T05:06:03.061 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:06:03.061 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:06:03.062 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:06:03.062 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:06:03.062 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_10 2023-11-08T05:06:03.062 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:06:03.063 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local mode=snapshot 2023-11-08T05:06:03.063 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' -n snapshot ']' 2023-11-08T05:06:03.063 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:06:03.063 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ create_image cluster2 mirror image_10 128M 2023-11-08T05:06:03.063 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:06:03.063 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:06:03.064 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:06:03.064 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:06:03.064 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_10 2023-11-08T05:06:03.064 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:06:03.064 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local size=128 2023-11-08T05:06:03.065 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' -n 128M ']' 2023-11-08T05:06:03.065 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ size=128M 2023-11-08T05:06:03.065 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:06:03.065 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_10 2023-11-08T05:06:03.099 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' image = image ']' 2023-11-08T05:06:03.099 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ enable_mirror cluster2 mirror image_10 snapshot 2023-11-08T05:06:03.099 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:06:03.100 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:06:03.100 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_10 2023-11-08T05:06:03.100 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local mode=snapshot 2023-11-08T05:06:03.101 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_10 snapshot 2023-11-08T05:06:03.758 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:Mirroring enabled 2023-11-08T05:06:03.766 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster=cluster2 info mirror/image_10 2023-11-08T05:06:03.798 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:rbd image 'image_10': 2023-11-08T05:06:03.799 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: size 128 MiB in 32 objects 2023-11-08T05:06:03.799 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: order 22 (4 MiB objects) 2023-11-08T05:06:03.799 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: snapshot_count: 1 2023-11-08T05:06:03.799 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: id: 12d01b8d32d3 2023-11-08T05:06:03.800 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: block_name_prefix: rbd_data.12d01b8d32d3 2023-11-08T05:06:03.800 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: format: 2 2023-11-08T05:06:03.800 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: features: layering, exclusive-lock 2023-11-08T05:06:03.800 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: op_features: 2023-11-08T05:06:03.800 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: flags: 2023-11-08T05:06:03.801 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: create_timestamp: Wed Nov 8 05:06:03 2023 2023-11-08T05:06:03.801 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: access_timestamp: Wed Nov 8 05:06:03 2023 2023-11-08T05:06:03.801 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: modify_timestamp: Wed Nov 8 05:06:03 2023 2023-11-08T05:06:03.801 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring state: enabled 2023-11-08T05:06:03.801 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring mode: snapshot 2023-11-08T05:06:03.802 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring global id: ab5153f3-f7f8-4e7b-ad36-9e427596786a 2023-11-08T05:06:03.802 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring primary: true 2023-11-08T05:06:03.803 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' -n '' ']' 2023-11-08T05:06:03.803 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ stress_write_image cluster2 mirror image_10 2023-11-08T05:06:03.803 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:06:03.804 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:06:03.804 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_10 2023-11-08T05:06:03.804 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-11-08T05:06:03.804 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local duration=5 2023-11-08T05:06:03.804 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ set +e 2023-11-08T05:06:03.805 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_10 --debug-rbd=20 --debug-journaler=20 2023-11-08T05:06:04.021 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:100 IOs, 49751040 bytes 2023-11-08T05:06:04.125 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:200 IOs, 63290368 bytes 2023-11-08T05:06:04.206 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:300 IOs, 67769344 bytes 2023-11-08T05:06:04.394 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:400 IOs, 121538560 bytes 2023-11-08T05:06:04.557 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:500 IOs, 163822592 bytes 2023-11-08T05:06:04.736 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:600 IOs, 205904896 bytes 2023-11-08T05:06:04.899 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:700 IOs, 251689984 bytes 2023-11-08T05:06:05.017 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:800 IOs, 266458112 bytes 2023-11-08T05:06:05.161 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:900 IOs, 294755328 bytes 2023-11-08T05:06:05.310 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1000 IOs, 327684096 bytes 2023-11-08T05:06:05.574 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1100 IOs, 405881856 bytes 2023-11-08T05:06:05.742 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1200 IOs, 444737536 bytes 2023-11-08T05:06:06.007 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1300 IOs, 526469120 bytes 2023-11-08T05:06:06.155 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1400 IOs, 544873472 bytes 2023-11-08T05:06:06.250 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1500 IOs, 547045376 bytes 2023-11-08T05:06:06.522 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1600 IOs, 640338944 bytes 2023-11-08T05:06:06.811 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1700 IOs, 762386432 bytes 2023-11-08T05:06:06.994 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1800 IOs, 794653696 bytes 2023-11-08T05:06:07.146 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1900 IOs, 828663808 bytes 2023-11-08T05:06:07.326 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2000 IOs, 893027328 bytes 2023-11-08T05:06:07.600 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2100 IOs, 962265088 bytes 2023-11-08T05:06:07.773 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2200 IOs, 1005574144 bytes 2023-11-08T05:06:08.086 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2300 IOs, 1112609792 bytes 2023-11-08T05:06:08.288 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2400 IOs, 1178732544 bytes 2023-11-08T05:06:08.490 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2500 IOs, 1256916992 bytes 2023-11-08T05:06:08.758 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2600 IOs, 1317846016 bytes 2023-11-08T05:06:08.809 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ error_code=124 2023-11-08T05:06:08.809 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ set -e 2023-11-08T05:06:08.809 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' 124 -eq 124 ']' 2023-11-08T05:06:08.810 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:06:08.810 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-11-08T05:06:08.810 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image=image_11 2023-11-08T05:06:08.810 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ create_image_and_enable_mirror cluster2 mirror image_11 snapshot 128M 2023-11-08T05:06:08.810 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:06:08.811 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:06:08.811 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:06:08.811 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:06:08.811 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_11 2023-11-08T05:06:08.811 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:06:08.812 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local mode=snapshot 2023-11-08T05:06:08.812 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' -n snapshot ']' 2023-11-08T05:06:08.812 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:06:08.812 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ create_image cluster2 mirror image_11 128M 2023-11-08T05:06:08.812 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:06:08.812 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:06:08.813 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:06:08.813 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:06:08.813 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_11 2023-11-08T05:06:08.813 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:06:08.813 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local size=128 2023-11-08T05:06:08.814 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' -n 128M ']' 2023-11-08T05:06:08.814 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ size=128M 2023-11-08T05:06:08.814 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:06:08.814 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_11 2023-11-08T05:06:08.848 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' image = image ']' 2023-11-08T05:06:08.849 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ enable_mirror cluster2 mirror image_11 snapshot 2023-11-08T05:06:08.849 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:06:08.849 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:06:08.849 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_11 2023-11-08T05:06:08.849 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local mode=snapshot 2023-11-08T05:06:08.850 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_11 snapshot 2023-11-08T05:06:09.777 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:Mirroring enabled 2023-11-08T05:06:09.784 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster=cluster2 info mirror/image_11 2023-11-08T05:06:09.814 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:rbd image 'image_11': 2023-11-08T05:06:09.815 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: size 128 MiB in 32 objects 2023-11-08T05:06:09.815 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: order 22 (4 MiB objects) 2023-11-08T05:06:09.815 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: snapshot_count: 1 2023-11-08T05:06:09.815 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: id: 12dc1d72722c 2023-11-08T05:06:09.815 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: block_name_prefix: rbd_data.12dc1d72722c 2023-11-08T05:06:09.816 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: format: 2 2023-11-08T05:06:09.816 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: features: layering, exclusive-lock 2023-11-08T05:06:09.816 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: op_features: 2023-11-08T05:06:09.816 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: flags: 2023-11-08T05:06:09.816 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: create_timestamp: Wed Nov 8 05:06:08 2023 2023-11-08T05:06:09.817 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: access_timestamp: Wed Nov 8 05:06:08 2023 2023-11-08T05:06:09.817 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: modify_timestamp: Wed Nov 8 05:06:08 2023 2023-11-08T05:06:09.817 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring state: enabled 2023-11-08T05:06:09.817 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring mode: snapshot 2023-11-08T05:06:09.817 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring global id: b31ed973-6d38-4d25-adab-1da0bc236902 2023-11-08T05:06:09.818 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring primary: true 2023-11-08T05:06:09.818 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' -n '' ']' 2023-11-08T05:06:09.819 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ stress_write_image cluster2 mirror image_11 2023-11-08T05:06:09.819 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:06:09.819 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:06:09.819 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_11 2023-11-08T05:06:09.819 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-11-08T05:06:09.820 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local duration=9 2023-11-08T05:06:09.820 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ set +e 2023-11-08T05:06:09.820 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_11 --debug-rbd=20 --debug-journaler=20 2023-11-08T05:06:10.004 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:100 IOs, 24996864 bytes 2023-11-08T05:06:10.154 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:200 IOs, 60451840 bytes 2023-11-08T05:06:10.253 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:300 IOs, 83139584 bytes 2023-11-08T05:06:10.379 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:400 IOs, 83766272 bytes 2023-11-08T05:06:10.515 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:500 IOs, 104594432 bytes 2023-11-08T05:06:10.747 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:600 IOs, 185544704 bytes 2023-11-08T05:06:10.987 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:700 IOs, 252160000 bytes 2023-11-08T05:06:11.145 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:800 IOs, 277683200 bytes 2023-11-08T05:06:11.237 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:900 IOs, 278298624 bytes 2023-11-08T05:06:11.525 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1000 IOs, 378008576 bytes 2023-11-08T05:06:11.717 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1100 IOs, 438795264 bytes 2023-11-08T05:06:11.866 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1200 IOs, 447680512 bytes 2023-11-08T05:06:12.025 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1300 IOs, 484390912 bytes 2023-11-08T05:06:12.192 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1400 IOs, 524133376 bytes 2023-11-08T05:06:12.344 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1500 IOs, 544940032 bytes 2023-11-08T05:06:12.544 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1600 IOs, 594792448 bytes 2023-11-08T05:06:12.805 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1700 IOs, 686831616 bytes 2023-11-08T05:06:13.097 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1800 IOs, 769523712 bytes 2023-11-08T05:06:13.239 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1900 IOs, 795729920 bytes 2023-11-08T05:06:13.364 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2000 IOs, 807414784 bytes 2023-11-08T05:06:13.513 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2100 IOs, 835315712 bytes 2023-11-08T05:06:13.639 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2200 IOs, 842738688 bytes 2023-11-08T05:06:13.796 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2300 IOs, 871147520 bytes 2023-11-08T05:06:13.891 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2400 IOs, 871778304 bytes 2023-11-08T05:06:13.995 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2500 IOs, 872403968 bytes 2023-11-08T05:06:14.142 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2600 IOs, 901740544 bytes 2023-11-08T05:06:14.308 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2700 IOs, 924827648 bytes 2023-11-08T05:06:14.449 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2800 IOs, 945254400 bytes 2023-11-08T05:06:14.677 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2900 IOs, 1008829440 bytes 2023-11-08T05:06:14.817 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3000 IOs, 1029245952 bytes 2023-11-08T05:06:14.973 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3100 IOs, 1074701312 bytes 2023-11-08T05:06:15.156 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3200 IOs, 1109644288 bytes 2023-11-08T05:06:15.318 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3300 IOs, 1128774656 bytes 2023-11-08T05:06:15.538 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3400 IOs, 1194262528 bytes 2023-11-08T05:06:15.699 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3500 IOs, 1238359040 bytes 2023-11-08T05:06:15.869 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3600 IOs, 1285601280 bytes 2023-11-08T05:06:16.116 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3700 IOs, 1342289920 bytes 2023-11-08T05:06:16.263 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3800 IOs, 1368029184 bytes 2023-11-08T05:06:16.463 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3900 IOs, 1422727168 bytes 2023-11-08T05:06:16.743 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4000 IOs, 1510773760 bytes 2023-11-08T05:06:16.929 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4100 IOs, 1551125504 bytes 2023-11-08T05:06:17.121 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4200 IOs, 1596756992 bytes 2023-11-08T05:06:17.243 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4300 IOs, 1607910400 bytes 2023-11-08T05:06:17.405 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4400 IOs, 1638623232 bytes 2023-11-08T05:06:17.649 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4500 IOs, 1726435328 bytes 2023-11-08T05:06:17.836 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4600 IOs, 1759326208 bytes 2023-11-08T05:06:18.015 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4700 IOs, 1800648704 bytes 2023-11-08T05:06:18.134 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4800 IOs, 1814105088 bytes 2023-11-08T05:06:18.415 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4900 IOs, 1902552064 bytes 2023-11-08T05:06:18.544 INFO:tasks.ceph.cluster2.osd.0.smithi087.stderr:2023-11-08T05:06:18.542+0000 7f1ac5683700 -1 reset not still connected to 0x55f9f32048f0 2023-11-08T05:06:18.583 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5000 IOs, 1934053376 bytes 2023-11-08T05:06:18.710 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5100 IOs, 1952472064 bytes 2023-11-08T05:06:18.825 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ error_code=124 2023-11-08T05:06:18.826 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ set -e 2023-11-08T05:06:18.826 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' 124 -eq 124 ']' 2023-11-08T05:06:18.826 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:06:18.827 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-11-08T05:06:18.827 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image=image_12 2023-11-08T05:06:18.827 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ create_image_and_enable_mirror cluster2 mirror image_12 snapshot 128M 2023-11-08T05:06:18.827 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:06:18.827 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:06:18.828 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:06:18.828 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:06:18.828 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_12 2023-11-08T05:06:18.828 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:06:18.828 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local mode=snapshot 2023-11-08T05:06:18.828 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' -n snapshot ']' 2023-11-08T05:06:18.829 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:06:18.829 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ create_image cluster2 mirror image_12 128M 2023-11-08T05:06:18.829 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:06:18.829 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:06:18.829 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:06:18.830 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:06:18.830 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_12 2023-11-08T05:06:18.830 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:06:18.830 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local size=128 2023-11-08T05:06:18.830 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' -n 128M ']' 2023-11-08T05:06:18.831 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ size=128M 2023-11-08T05:06:18.831 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:06:18.831 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_12 2023-11-08T05:06:18.865 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' image = image ']' 2023-11-08T05:06:18.865 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ enable_mirror cluster2 mirror image_12 snapshot 2023-11-08T05:06:18.865 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:06:18.866 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:06:18.866 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_12 2023-11-08T05:06:18.866 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local mode=snapshot 2023-11-08T05:06:18.866 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_12 snapshot 2023-11-08T05:06:19.820 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:Mirroring enabled 2023-11-08T05:06:19.829 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster=cluster2 info mirror/image_12 2023-11-08T05:06:19.863 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:rbd image 'image_12': 2023-11-08T05:06:19.863 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: size 128 MiB in 32 objects 2023-11-08T05:06:19.863 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: order 22 (4 MiB objects) 2023-11-08T05:06:19.864 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: snapshot_count: 1 2023-11-08T05:06:19.864 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: id: 12e866d9a42c 2023-11-08T05:06:19.864 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: block_name_prefix: rbd_data.12e866d9a42c 2023-11-08T05:06:19.864 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: format: 2 2023-11-08T05:06:19.865 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: features: layering, exclusive-lock 2023-11-08T05:06:19.865 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: op_features: 2023-11-08T05:06:19.865 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: flags: 2023-11-08T05:06:19.865 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: create_timestamp: Wed Nov 8 05:06:18 2023 2023-11-08T05:06:19.865 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: access_timestamp: Wed Nov 8 05:06:18 2023 2023-11-08T05:06:19.866 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: modify_timestamp: Wed Nov 8 05:06:18 2023 2023-11-08T05:06:19.866 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring state: enabled 2023-11-08T05:06:19.866 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring mode: snapshot 2023-11-08T05:06:19.866 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring global id: c28fe69d-285b-4d38-a8a1-5a5367bf07ce 2023-11-08T05:06:19.866 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring primary: true 2023-11-08T05:06:19.867 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' -n '' ']' 2023-11-08T05:06:19.867 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ stress_write_image cluster2 mirror image_12 2023-11-08T05:06:19.868 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:06:19.868 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:06:19.868 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_12 2023-11-08T05:06:19.868 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-11-08T05:06:19.869 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local duration=11 2023-11-08T05:06:19.869 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ set +e 2023-11-08T05:06:19.869 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_12 --debug-rbd=20 --debug-journaler=20 2023-11-08T05:06:20.014 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:100 IOs, 10792960 bytes 2023-11-08T05:06:20.229 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:200 IOs, 72733696 bytes 2023-11-08T05:06:20.363 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:300 IOs, 96624640 bytes 2023-11-08T05:06:20.534 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:400 IOs, 147170304 bytes 2023-11-08T05:06:20.746 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:500 IOs, 208793600 bytes 2023-11-08T05:06:20.889 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:600 IOs, 241853440 bytes 2023-11-08T05:06:21.141 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:700 IOs, 319851520 bytes 2023-11-08T05:06:21.429 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:800 IOs, 414640128 bytes 2023-11-08T05:06:21.615 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:900 IOs, 470594560 bytes 2023-11-08T05:06:21.735 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1000 IOs, 482258944 bytes 2023-11-08T05:06:21.890 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1100 IOs, 505835520 bytes 2023-11-08T05:06:22.077 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1200 IOs, 556154880 bytes 2023-11-08T05:06:22.228 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1300 IOs, 586974208 bytes 2023-11-08T05:06:22.335 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1400 IOs, 603416576 bytes 2023-11-08T05:06:22.512 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1500 IOs, 621895680 bytes 2023-11-08T05:06:22.698 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1600 IOs, 670491648 bytes 2023-11-08T05:06:22.928 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1700 IOs, 734834688 bytes 2023-11-08T05:06:23.119 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1800 IOs, 784617472 bytes 2023-11-08T05:06:23.230 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1900 IOs, 791792640 bytes 2023-11-08T05:06:23.437 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2000 IOs, 851578880 bytes 2023-11-08T05:06:23.578 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2100 IOs, 881772544 bytes 2023-11-08T05:06:23.783 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2200 IOs, 942971904 bytes 2023-11-08T05:06:23.974 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2300 IOs, 987456512 bytes 2023-11-08T05:06:24.299 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2400 IOs, 1094025216 bytes 2023-11-08T05:06:24.492 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2500 IOs, 1141677056 bytes 2023-11-08T05:06:24.620 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2600 IOs, 1155389440 bytes 2023-11-08T05:06:24.779 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2700 IOs, 1185550336 bytes 2023-11-08T05:06:24.997 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2800 IOs, 1251775488 bytes 2023-11-08T05:06:25.130 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2900 IOs, 1265282048 bytes 2023-11-08T05:06:25.282 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3000 IOs, 1295187968 bytes 2023-11-08T05:06:25.480 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3100 IOs, 1344035840 bytes 2023-11-08T05:06:25.638 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3200 IOs, 1376066560 bytes 2023-11-08T05:06:25.781 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3300 IOs, 1398512640 bytes 2023-11-08T05:06:25.975 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3400 IOs, 1447669760 bytes 2023-11-08T05:06:26.190 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3500 IOs, 1505887232 bytes 2023-11-08T05:06:26.354 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3600 IOs, 1534038016 bytes 2023-11-08T05:06:26.496 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3700 IOs, 1557677056 bytes 2023-11-08T05:06:26.701 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3800 IOs, 1618817024 bytes 2023-11-08T05:06:26.887 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3900 IOs, 1674115072 bytes 2023-11-08T05:06:27.200 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4000 IOs, 1774076928 bytes 2023-11-08T05:06:27.303 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4100 IOs, 1775774720 bytes 2023-11-08T05:06:27.460 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4200 IOs, 1806951424 bytes 2023-11-08T05:06:27.609 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4300 IOs, 1828953088 bytes 2023-11-08T05:06:27.762 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4400 IOs, 1855757312 bytes 2023-11-08T05:06:27.953 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4500 IOs, 1894173696 bytes 2023-11-08T05:06:28.136 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4600 IOs, 1932561408 bytes 2023-11-08T05:06:28.314 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4700 IOs, 1984933888 bytes 2023-11-08T05:06:28.495 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4800 IOs, 2026275840 bytes 2023-11-08T05:06:28.656 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4900 IOs, 2055027712 bytes 2023-11-08T05:06:28.841 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5000 IOs, 2096719872 bytes 2023-11-08T05:06:28.970 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5100 IOs, 2121885696 bytes 2023-11-08T05:06:29.110 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5200 IOs, 2144176128 bytes 2023-11-08T05:06:29.286 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5300 IOs, 2188975104 bytes 2023-11-08T05:06:29.524 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5400 IOs, 2259012608 bytes 2023-11-08T05:06:29.704 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5500 IOs, 2297430016 bytes 2023-11-08T05:06:29.899 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5600 IOs, 2349775872 bytes 2023-11-08T05:06:30.028 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5700 IOs, 2362589184 bytes 2023-11-08T05:06:30.169 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5800 IOs, 2384490496 bytes 2023-11-08T05:06:30.302 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5900 IOs, 2395495424 bytes 2023-11-08T05:06:30.546 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:6000 IOs, 2468452352 bytes 2023-11-08T05:06:30.651 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:6100 IOs, 2471058432 bytes 2023-11-08T05:06:30.791 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:6200 IOs, 2492749824 bytes 2023-11-08T05:06:30.875 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ error_code=124 2023-11-08T05:06:30.876 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ set -e 2023-11-08T05:06:30.876 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' 124 -eq 124 ']' 2023-11-08T05:06:30.876 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:06:30.876 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-11-08T05:06:30.876 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image=image_13 2023-11-08T05:06:30.877 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ create_image_and_enable_mirror cluster2 mirror image_13 snapshot 128M 2023-11-08T05:06:30.877 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:06:30.877 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:06:30.877 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:06:30.877 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:06:30.878 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_13 2023-11-08T05:06:30.878 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:06:30.878 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local mode=snapshot 2023-11-08T05:06:30.878 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' -n snapshot ']' 2023-11-08T05:06:30.878 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:06:30.879 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ create_image cluster2 mirror image_13 128M 2023-11-08T05:06:30.879 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:06:30.879 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:06:30.879 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:06:30.879 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:06:30.880 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_13 2023-11-08T05:06:30.880 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:06:30.880 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local size=128 2023-11-08T05:06:30.880 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' -n 128M ']' 2023-11-08T05:06:30.880 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ size=128M 2023-11-08T05:06:30.881 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:06:30.881 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_13 2023-11-08T05:06:30.914 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' image = image ']' 2023-11-08T05:06:30.914 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ enable_mirror cluster2 mirror image_13 snapshot 2023-11-08T05:06:30.915 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:06:30.915 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:06:30.915 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_13 2023-11-08T05:06:30.915 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local mode=snapshot 2023-11-08T05:06:30.915 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_13 snapshot 2023-11-08T05:06:31.913 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:Mirroring enabled 2023-11-08T05:06:31.920 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster=cluster2 info mirror/image_13 2023-11-08T05:06:31.953 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:rbd image 'image_13': 2023-11-08T05:06:31.953 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: size 128 MiB in 32 objects 2023-11-08T05:06:31.953 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: order 22 (4 MiB objects) 2023-11-08T05:06:31.953 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: snapshot_count: 1 2023-11-08T05:06:31.954 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: id: 12f4d8f0f1ac 2023-11-08T05:06:31.954 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: block_name_prefix: rbd_data.12f4d8f0f1ac 2023-11-08T05:06:31.954 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: format: 2 2023-11-08T05:06:31.954 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: features: layering, exclusive-lock 2023-11-08T05:06:31.955 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: op_features: 2023-11-08T05:06:31.955 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: flags: 2023-11-08T05:06:31.955 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: create_timestamp: Wed Nov 8 05:06:30 2023 2023-11-08T05:06:31.955 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: access_timestamp: Wed Nov 8 05:06:30 2023 2023-11-08T05:06:31.955 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: modify_timestamp: Wed Nov 8 05:06:30 2023 2023-11-08T05:06:31.956 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring state: enabled 2023-11-08T05:06:31.956 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring mode: snapshot 2023-11-08T05:06:31.956 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring global id: cb42c52c-8a81-4992-b8fb-d273572391a5 2023-11-08T05:06:31.956 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring primary: true 2023-11-08T05:06:31.957 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' -n '' ']' 2023-11-08T05:06:31.957 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ stress_write_image cluster2 mirror image_13 2023-11-08T05:06:31.957 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:06:31.958 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:06:31.958 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_13 2023-11-08T05:06:31.958 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-11-08T05:06:31.959 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local duration=11 2023-11-08T05:06:31.959 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ set +e 2023-11-08T05:06:31.959 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_13 --debug-rbd=20 --debug-journaler=20 2023-11-08T05:06:32.232 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:100 IOs, 69899264 bytes 2023-11-08T05:06:32.408 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:200 IOs, 110197760 bytes 2023-11-08T05:06:32.528 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:300 IOs, 121331712 bytes 2023-11-08T05:06:32.673 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:400 IOs, 150409216 bytes 2023-11-08T05:06:32.905 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:500 IOs, 228129792 bytes 2023-11-08T05:06:33.120 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:600 IOs, 283393024 bytes 2023-11-08T05:06:33.308 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:700 IOs, 326623232 bytes 2023-11-08T05:06:33.509 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:800 IOs, 375510016 bytes 2023-11-08T05:06:33.693 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:900 IOs, 432023552 bytes 2023-11-08T05:06:33.832 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1000 IOs, 474431488 bytes 2023-11-08T05:06:34.054 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1100 IOs, 517489664 bytes 2023-11-08T05:06:34.287 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1200 IOs, 581121024 bytes 2023-11-08T05:06:34.513 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1300 IOs, 642065408 bytes 2023-11-08T05:06:34.712 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1400 IOs, 691482624 bytes 2023-11-08T05:06:34.910 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1500 IOs, 748460032 bytes 2023-11-08T05:06:35.019 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1600 IOs, 749077504 bytes 2023-11-08T05:06:35.186 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1700 IOs, 787895296 bytes 2023-11-08T05:06:35.380 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1800 IOs, 855648256 bytes 2023-11-08T05:06:35.577 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1900 IOs, 884584448 bytes 2023-11-08T05:06:35.736 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2000 IOs, 919066624 bytes 2023-11-08T05:06:35.936 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2100 IOs, 976556032 bytes 2023-11-08T05:06:36.143 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2200 IOs, 1029871616 bytes 2023-11-08T05:06:36.321 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2300 IOs, 1073823744 bytes 2023-11-08T05:06:36.557 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2400 IOs, 1135385600 bytes 2023-11-08T05:06:36.732 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2500 IOs, 1180414976 bytes 2023-11-08T05:06:36.891 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2600 IOs, 1214914560 bytes 2023-11-08T05:06:37.102 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2700 IOs, 1272128512 bytes 2023-11-08T05:06:37.281 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2800 IOs, 1311469568 bytes 2023-11-08T05:06:37.386 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2900 IOs, 1312070656 bytes 2023-11-08T05:06:37.622 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3000 IOs, 1387125760 bytes 2023-11-08T05:06:37.834 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3100 IOs, 1442327552 bytes 2023-11-08T05:06:37.986 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3200 IOs, 1474769920 bytes 2023-11-08T05:06:38.123 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3300 IOs, 1495706624 bytes 2023-11-08T05:06:38.250 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3400 IOs, 1509212160 bytes 2023-11-08T05:06:38.386 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3500 IOs, 1536104448 bytes 2023-11-08T05:06:38.644 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3600 IOs, 1610083328 bytes 2023-11-08T05:06:38.807 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3700 IOs, 1654677504 bytes 2023-11-08T05:06:38.956 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3800 IOs, 1663314944 bytes 2023-11-08T05:06:39.121 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3900 IOs, 1696951296 bytes 2023-11-08T05:06:39.272 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4000 IOs, 1728149504 bytes 2023-11-08T05:06:39.466 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4100 IOs, 1777488896 bytes 2023-11-08T05:06:39.630 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4200 IOs, 1811746816 bytes 2023-11-08T05:06:39.799 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4300 IOs, 1843005440 bytes 2023-11-08T05:06:39.902 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4400 IOs, 1843599360 bytes 2023-11-08T05:06:40.084 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4500 IOs, 1883300864 bytes 2023-11-08T05:06:40.217 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4600 IOs, 1916589056 bytes 2023-11-08T05:06:40.378 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4700 IOs, 1936834560 bytes 2023-11-08T05:06:40.514 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4800 IOs, 1967055872 bytes 2023-11-08T05:06:40.634 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4900 IOs, 1977315328 bytes 2023-11-08T05:06:40.776 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5000 IOs, 1996957696 bytes 2023-11-08T05:06:41.053 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5100 IOs, 2092178432 bytes 2023-11-08T05:06:41.340 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5200 IOs, 2177727488 bytes 2023-11-08T05:06:41.511 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5300 IOs, 2217695232 bytes 2023-11-08T05:06:41.679 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5400 IOs, 2257996800 bytes 2023-11-08T05:06:41.823 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5500 IOs, 2280232960 bytes 2023-11-08T05:06:42.126 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5600 IOs, 2385056768 bytes 2023-11-08T05:06:42.233 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5700 IOs, 2387924992 bytes 2023-11-08T05:06:42.340 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5800 IOs, 2392338432 bytes 2023-11-08T05:06:42.534 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5900 IOs, 2434637824 bytes 2023-11-08T05:06:42.642 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:6000 IOs, 2442627072 bytes 2023-11-08T05:06:42.840 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:6100 IOs, 2502474752 bytes 2023-11-08T05:06:42.963 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ error_code=124 2023-11-08T05:06:42.964 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ set -e 2023-11-08T05:06:42.964 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' 124 -eq 124 ']' 2023-11-08T05:06:42.964 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:06:42.964 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-11-08T05:06:42.964 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image=image_14 2023-11-08T05:06:42.965 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ create_image_and_enable_mirror cluster2 mirror image_14 snapshot 128M 2023-11-08T05:06:42.965 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:06:42.965 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:06:42.965 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:06:42.965 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:06:42.966 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_14 2023-11-08T05:06:42.966 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:06:42.966 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local mode=snapshot 2023-11-08T05:06:42.966 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' -n snapshot ']' 2023-11-08T05:06:42.967 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:06:42.967 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ create_image cluster2 mirror image_14 128M 2023-11-08T05:06:42.967 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:06:42.967 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:06:42.967 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:06:42.968 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:06:42.968 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_14 2023-11-08T05:06:42.968 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:06:42.968 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local size=128 2023-11-08T05:06:42.968 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' -n 128M ']' 2023-11-08T05:06:42.969 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ size=128M 2023-11-08T05:06:42.969 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:06:42.969 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_14 2023-11-08T05:06:43.008 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' image = image ']' 2023-11-08T05:06:43.008 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ enable_mirror cluster2 mirror image_14 snapshot 2023-11-08T05:06:43.008 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:06:43.008 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:06:43.008 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_14 2023-11-08T05:06:43.009 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local mode=snapshot 2023-11-08T05:06:43.009 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_14 snapshot 2023-11-08T05:06:44.001 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:Mirroring enabled 2023-11-08T05:06:44.008 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster=cluster2 info mirror/image_14 2023-11-08T05:06:44.042 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:rbd image 'image_14': 2023-11-08T05:06:44.042 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: size 128 MiB in 32 objects 2023-11-08T05:06:44.042 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: order 22 (4 MiB objects) 2023-11-08T05:06:44.042 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: snapshot_count: 1 2023-11-08T05:06:44.043 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: id: 1300977192e5 2023-11-08T05:06:44.043 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: block_name_prefix: rbd_data.1300977192e5 2023-11-08T05:06:44.043 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: format: 2 2023-11-08T05:06:44.043 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: features: layering, exclusive-lock 2023-11-08T05:06:44.043 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: op_features: 2023-11-08T05:06:44.044 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: flags: 2023-11-08T05:06:44.044 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: create_timestamp: Wed Nov 8 05:06:43 2023 2023-11-08T05:06:44.044 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: access_timestamp: Wed Nov 8 05:06:43 2023 2023-11-08T05:06:44.044 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: modify_timestamp: Wed Nov 8 05:06:43 2023 2023-11-08T05:06:44.044 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring state: enabled 2023-11-08T05:06:44.045 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring mode: snapshot 2023-11-08T05:06:44.045 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring global id: 2e7c6dbb-61d7-4f4a-9510-adb4bfb2cb02 2023-11-08T05:06:44.045 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring primary: true 2023-11-08T05:06:44.046 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' -n '' ']' 2023-11-08T05:06:44.046 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ stress_write_image cluster2 mirror image_14 2023-11-08T05:06:44.047 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:06:44.047 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:06:44.047 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_14 2023-11-08T05:06:44.047 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-11-08T05:06:44.047 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local duration=13 2023-11-08T05:06:44.048 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ set +e 2023-11-08T05:06:44.048 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_14 --debug-rbd=20 --debug-journaler=20 2023-11-08T05:06:44.216 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:100 IOs, 22974464 bytes 2023-11-08T05:06:44.337 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:200 IOs, 40985600 bytes 2023-11-08T05:06:44.518 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:300 IOs, 90906624 bytes 2023-11-08T05:06:44.662 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:400 IOs, 113022976 bytes 2023-11-08T05:06:44.825 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:500 IOs, 146434048 bytes 2023-11-08T05:06:44.977 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:600 IOs, 171916288 bytes 2023-11-08T05:06:45.193 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:700 IOs, 234011648 bytes 2023-11-08T05:06:45.353 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:800 IOs, 262663168 bytes 2023-11-08T05:06:45.476 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:900 IOs, 277416960 bytes 2023-11-08T05:06:45.577 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1000 IOs, 278054912 bytes 2023-11-08T05:06:45.715 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1100 IOs, 304901120 bytes 2023-11-08T05:06:45.927 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1200 IOs, 358354944 bytes 2023-11-08T05:06:46.142 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1300 IOs, 420128768 bytes 2023-11-08T05:06:46.366 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1400 IOs, 479514624 bytes 2023-11-08T05:06:46.558 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1500 IOs, 525989888 bytes 2023-11-08T05:06:46.715 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1600 IOs, 556569600 bytes 2023-11-08T05:06:46.828 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1700 IOs, 576013312 bytes 2023-11-08T05:06:47.032 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1800 IOs, 629708800 bytes 2023-11-08T05:06:47.243 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1900 IOs, 676325376 bytes 2023-11-08T05:06:47.386 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2000 IOs, 692560896 bytes 2023-11-08T05:06:47.577 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2100 IOs, 740615168 bytes 2023-11-08T05:06:47.681 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2200 IOs, 741197824 bytes 2023-11-08T05:06:47.963 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2300 IOs, 831449088 bytes 2023-11-08T05:06:48.184 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2400 IOs, 890141696 bytes 2023-11-08T05:06:48.399 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2500 IOs, 954128384 bytes 2023-11-08T05:06:48.521 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2600 IOs, 962356224 bytes 2023-11-08T05:06:48.665 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2700 IOs, 1011289088 bytes 2023-11-08T05:06:48.889 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2800 IOs, 1046935552 bytes 2023-11-08T05:06:49.034 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2900 IOs, 1065645056 bytes 2023-11-08T05:06:49.182 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3000 IOs, 1090264064 bytes 2023-11-08T05:06:49.406 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3100 IOs, 1154692096 bytes 2023-11-08T05:06:49.637 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3200 IOs, 1228912640 bytes 2023-11-08T05:06:49.745 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3300 IOs, 1235773440 bytes 2023-11-08T05:06:49.933 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3400 IOs, 1276450816 bytes 2023-11-08T05:06:50.133 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3500 IOs, 1329084416 bytes 2023-11-08T05:06:50.288 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3600 IOs, 1356221440 bytes 2023-11-08T05:06:50.483 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3700 IOs, 1411126272 bytes 2023-11-08T05:06:50.790 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3800 IOs, 1511606272 bytes 2023-11-08T05:06:50.890 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3900 IOs, 1512229888 bytes 2023-11-08T05:06:51.062 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4000 IOs, 1542193152 bytes 2023-11-08T05:06:51.206 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4100 IOs, 1559539712 bytes 2023-11-08T05:06:51.313 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4200 IOs, 1560162304 bytes 2023-11-08T05:06:51.545 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4300 IOs, 1628576768 bytes 2023-11-08T05:06:51.692 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4400 IOs, 1652390912 bytes 2023-11-08T05:06:51.886 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4500 IOs, 1694010368 bytes 2023-11-08T05:06:52.039 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4600 IOs, 1725881344 bytes 2023-11-08T05:06:52.197 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4700 IOs, 1752667136 bytes 2023-11-08T05:06:52.373 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4800 IOs, 1791072256 bytes 2023-11-08T05:06:52.534 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4900 IOs, 1841923072 bytes 2023-11-08T05:06:52.736 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5000 IOs, 1869996032 bytes 2023-11-08T05:06:52.946 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5100 IOs, 1933097984 bytes 2023-11-08T05:06:53.047 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5200 IOs, 1940582400 bytes 2023-11-08T05:06:53.182 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5300 IOs, 1956402176 bytes 2023-11-08T05:06:53.429 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5400 IOs, 2041974784 bytes 2023-11-08T05:06:53.638 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5500 IOs, 2083270656 bytes 2023-11-08T05:06:53.805 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5600 IOs, 2115130368 bytes 2023-11-08T05:06:53.922 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5700 IOs, 2128418816 bytes 2023-11-08T05:06:54.155 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5800 IOs, 2186194944 bytes 2023-11-08T05:06:54.450 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5900 IOs, 2280856576 bytes 2023-11-08T05:06:54.621 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:6000 IOs, 2321125376 bytes 2023-11-08T05:06:54.948 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:6100 IOs, 2389438464 bytes 2023-11-08T05:06:55.059 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:6200 IOs, 2397393920 bytes 2023-11-08T05:06:55.166 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:6300 IOs, 2398016512 bytes 2023-11-08T05:06:55.395 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:6400 IOs, 2457871360 bytes 2023-11-08T05:06:55.531 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:6500 IOs, 2477172736 bytes 2023-11-08T05:06:55.785 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:6600 IOs, 2563002368 bytes 2023-11-08T05:06:55.887 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:6700 IOs, 2563617792 bytes 2023-11-08T05:06:56.169 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:6800 IOs, 2655689728 bytes 2023-11-08T05:06:56.334 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:6900 IOs, 2691363840 bytes 2023-11-08T05:06:56.542 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:7000 IOs, 2743388160 bytes 2023-11-08T05:06:56.717 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:7100 IOs, 2784642048 bytes 2023-11-08T05:06:56.818 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:7200 IOs, 2785264640 bytes 2023-11-08T05:06:56.996 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:7300 IOs, 2826512384 bytes 2023-11-08T05:06:57.054 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ error_code=124 2023-11-08T05:06:57.055 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ set -e 2023-11-08T05:06:57.055 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' 124 -eq 124 ']' 2023-11-08T05:06:57.055 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:06:57.055 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-11-08T05:06:57.056 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image=image_15 2023-11-08T05:06:57.056 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ create_image_and_enable_mirror cluster2 mirror image_15 snapshot 128M 2023-11-08T05:06:57.056 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:06:57.056 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:06:57.056 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:06:57.056 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:06:57.057 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_15 2023-11-08T05:06:57.057 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:06:57.057 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local mode=snapshot 2023-11-08T05:06:57.057 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' -n snapshot ']' 2023-11-08T05:06:57.057 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:06:57.058 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ create_image cluster2 mirror image_15 128M 2023-11-08T05:06:57.058 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:06:57.058 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:06:57.058 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:06:57.058 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:06:57.059 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_15 2023-11-08T05:06:57.059 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:06:57.059 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local size=128 2023-11-08T05:06:57.059 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' -n 128M ']' 2023-11-08T05:06:57.059 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ size=128M 2023-11-08T05:06:57.060 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:06:57.060 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_15 2023-11-08T05:06:57.095 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' image = image ']' 2023-11-08T05:06:57.095 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ enable_mirror cluster2 mirror image_15 snapshot 2023-11-08T05:06:57.095 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:06:57.095 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:06:57.096 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_15 2023-11-08T05:06:57.096 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local mode=snapshot 2023-11-08T05:06:57.096 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_15 snapshot 2023-11-08T05:06:58.120 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:Mirroring enabled 2023-11-08T05:06:58.126 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster=cluster2 info mirror/image_15 2023-11-08T05:06:58.157 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:rbd image 'image_15': 2023-11-08T05:06:58.157 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: size 128 MiB in 32 objects 2023-11-08T05:06:58.157 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: order 22 (4 MiB objects) 2023-11-08T05:06:58.157 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: snapshot_count: 1 2023-11-08T05:06:58.158 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: id: 130c8ecbaaf0 2023-11-08T05:06:58.158 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: block_name_prefix: rbd_data.130c8ecbaaf0 2023-11-08T05:06:58.158 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: format: 2 2023-11-08T05:06:58.158 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: features: layering, exclusive-lock 2023-11-08T05:06:58.158 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: op_features: 2023-11-08T05:06:58.159 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: flags: 2023-11-08T05:06:58.159 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: create_timestamp: Wed Nov 8 05:06:57 2023 2023-11-08T05:06:58.159 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: access_timestamp: Wed Nov 8 05:06:57 2023 2023-11-08T05:06:58.159 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: modify_timestamp: Wed Nov 8 05:06:57 2023 2023-11-08T05:06:58.159 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring state: enabled 2023-11-08T05:06:58.160 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring mode: snapshot 2023-11-08T05:06:58.160 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring global id: c5f2bed1-9f45-45f4-aaf6-7a50884f15f8 2023-11-08T05:06:58.160 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring primary: true 2023-11-08T05:06:58.161 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' -n '' ']' 2023-11-08T05:06:58.161 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ stress_write_image cluster2 mirror image_15 2023-11-08T05:06:58.162 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:06:58.162 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:06:58.162 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_15 2023-11-08T05:06:58.162 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-11-08T05:06:58.162 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local duration=8 2023-11-08T05:06:58.163 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ set +e 2023-11-08T05:06:58.163 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_15 --debug-rbd=20 --debug-journaler=20 2023-11-08T05:06:58.412 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:100 IOs, 62723072 bytes 2023-11-08T05:06:58.651 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:200 IOs, 135525376 bytes 2023-11-08T05:06:58.752 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:300 IOs, 136152064 bytes 2023-11-08T05:06:58.941 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:400 IOs, 180511744 bytes 2023-11-08T05:06:59.170 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:500 IOs, 250923008 bytes 2023-11-08T05:06:59.272 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:600 IOs, 252477440 bytes 2023-11-08T05:06:59.522 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:700 IOs, 326862848 bytes 2023-11-08T05:06:59.645 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:800 IOs, 345211904 bytes 2023-11-08T05:06:59.876 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:900 IOs, 409901056 bytes 2023-11-08T05:07:00.057 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1000 IOs, 456276992 bytes 2023-11-08T05:07:00.185 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1100 IOs, 462746624 bytes 2023-11-08T05:07:00.331 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1200 IOs, 485262336 bytes 2023-11-08T05:07:00.445 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1300 IOs, 488751104 bytes 2023-11-08T05:07:00.650 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1400 IOs, 545205248 bytes 2023-11-08T05:07:00.818 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1500 IOs, 576295936 bytes 2023-11-08T05:07:00.962 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1600 IOs, 597021696 bytes 2023-11-08T05:07:01.108 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1700 IOs, 621413376 bytes 2023-11-08T05:07:01.304 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1800 IOs, 676007936 bytes 2023-11-08T05:07:01.450 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1900 IOs, 703763456 bytes 2023-11-08T05:07:01.588 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2000 IOs, 720071680 bytes 2023-11-08T05:07:01.774 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2100 IOs, 764960768 bytes 2023-11-08T05:07:01.933 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2200 IOs, 795663360 bytes 2023-11-08T05:07:02.142 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2300 IOs, 845762560 bytes 2023-11-08T05:07:02.241 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2400 IOs, 846383104 bytes 2023-11-08T05:07:02.451 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2500 IOs, 896128000 bytes 2023-11-08T05:07:02.658 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2600 IOs, 948685824 bytes 2023-11-08T05:07:02.819 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2700 IOs, 982135808 bytes 2023-11-08T05:07:02.944 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2800 IOs, 993058816 bytes 2023-11-08T05:07:03.144 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2900 IOs, 1039060992 bytes 2023-11-08T05:07:03.329 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3000 IOs, 1083352064 bytes 2023-11-08T05:07:03.496 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3100 IOs, 1135005696 bytes 2023-11-08T05:07:03.653 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3200 IOs, 1144730624 bytes 2023-11-08T05:07:03.841 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3300 IOs, 1186254848 bytes 2023-11-08T05:07:04.065 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3400 IOs, 1256145920 bytes 2023-11-08T05:07:04.199 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3500 IOs, 1276587008 bytes 2023-11-08T05:07:04.457 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3600 IOs, 1368243200 bytes 2023-11-08T05:07:04.683 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3700 IOs, 1415342080 bytes 2023-11-08T05:07:04.830 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3800 IOs, 1442592768 bytes 2023-11-08T05:07:04.991 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3900 IOs, 1477069824 bytes 2023-11-08T05:07:05.173 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4000 IOs, 1510249472 bytes 2023-11-08T05:07:05.362 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4100 IOs, 1549884416 bytes 2023-11-08T05:07:05.536 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4200 IOs, 1591908352 bytes 2023-11-08T05:07:05.709 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4300 IOs, 1626093568 bytes 2023-11-08T05:07:05.804 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4400 IOs, 1626710016 bytes 2023-11-08T05:07:05.919 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4500 IOs, 1640368128 bytes 2023-11-08T05:07:06.059 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4600 IOs, 1660104704 bytes 2023-11-08T05:07:06.170 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ error_code=124 2023-11-08T05:07:06.170 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ set -e 2023-11-08T05:07:06.170 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' 124 -eq 124 ']' 2023-11-08T05:07:06.171 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:07:06.171 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-11-08T05:07:06.171 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image=image_16 2023-11-08T05:07:06.171 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ create_image_and_enable_mirror cluster2 mirror image_16 snapshot 128M 2023-11-08T05:07:06.171 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:07:06.172 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:07:06.172 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:07:06.172 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:07:06.172 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_16 2023-11-08T05:07:06.172 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:07:06.173 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local mode=snapshot 2023-11-08T05:07:06.173 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' -n snapshot ']' 2023-11-08T05:07:06.173 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:07:06.173 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ create_image cluster2 mirror image_16 128M 2023-11-08T05:07:06.173 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:07:06.174 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:07:06.174 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:07:06.174 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:07:06.174 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_16 2023-11-08T05:07:06.174 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:07:06.175 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local size=128 2023-11-08T05:07:06.175 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' -n 128M ']' 2023-11-08T05:07:06.175 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ size=128M 2023-11-08T05:07:06.175 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:07:06.175 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_16 2023-11-08T05:07:06.209 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' image = image ']' 2023-11-08T05:07:06.210 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ enable_mirror cluster2 mirror image_16 snapshot 2023-11-08T05:07:06.210 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:07:06.210 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:07:06.210 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_16 2023-11-08T05:07:06.211 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local mode=snapshot 2023-11-08T05:07:06.211 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_16 snapshot 2023-11-08T05:07:07.192 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:Mirroring enabled 2023-11-08T05:07:07.200 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster=cluster2 info mirror/image_16 2023-11-08T05:07:07.233 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:rbd image 'image_16': 2023-11-08T05:07:07.234 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: size 128 MiB in 32 objects 2023-11-08T05:07:07.234 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: order 22 (4 MiB objects) 2023-11-08T05:07:07.234 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: snapshot_count: 1 2023-11-08T05:07:07.234 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: id: 13184f242c4 2023-11-08T05:07:07.234 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: block_name_prefix: rbd_data.13184f242c4 2023-11-08T05:07:07.235 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: format: 2 2023-11-08T05:07:07.235 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: features: layering, exclusive-lock 2023-11-08T05:07:07.235 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: op_features: 2023-11-08T05:07:07.235 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: flags: 2023-11-08T05:07:07.235 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: create_timestamp: Wed Nov 8 05:07:06 2023 2023-11-08T05:07:07.236 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: access_timestamp: Wed Nov 8 05:07:06 2023 2023-11-08T05:07:07.236 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: modify_timestamp: Wed Nov 8 05:07:06 2023 2023-11-08T05:07:07.236 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring state: enabled 2023-11-08T05:07:07.236 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring mode: snapshot 2023-11-08T05:07:07.236 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring global id: 6852ffd8-136b-481a-b662-fdde8fec627d 2023-11-08T05:07:07.237 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring primary: true 2023-11-08T05:07:07.238 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' -n '' ']' 2023-11-08T05:07:07.238 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ stress_write_image cluster2 mirror image_16 2023-11-08T05:07:07.238 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:07:07.238 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:07:07.238 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_16 2023-11-08T05:07:07.239 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-11-08T05:07:07.239 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local duration=12 2023-11-08T05:07:07.240 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ set +e 2023-11-08T05:07:07.240 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_16 --debug-rbd=20 --debug-journaler=20 2023-11-08T05:07:07.543 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:100 IOs, 106908672 bytes 2023-11-08T05:07:07.734 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:200 IOs, 137111552 bytes 2023-11-08T05:07:07.845 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:300 IOs, 151306240 bytes 2023-11-08T05:07:07.978 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:400 IOs, 176501760 bytes 2023-11-08T05:07:08.054 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.3805290633302375) > weight (0.25) 2023-11-08T05:07:08.055 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.36122398835047054) > weight (0.25) 2023-11-08T05:07:08.055 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.9240112666485778) > weight (0.25) 2023-11-08T05:07:08.055 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.3 with skip (0.3615904020695331) > weight (0.25) 2023-11-08T05:07:08.056 INFO:tasks.rbd_mirror_thrash:waiting for 101 secs before thrashing 2023-11-08T05:07:08.165 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:500 IOs, 220265472 bytes 2023-11-08T05:07:08.356 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:600 IOs, 264992768 bytes 2023-11-08T05:07:08.488 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:700 IOs, 282739712 bytes 2023-11-08T05:07:08.695 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:800 IOs, 342493184 bytes 2023-11-08T05:07:08.847 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:900 IOs, 381801472 bytes 2023-11-08T05:07:09.014 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1000 IOs, 408375296 bytes 2023-11-08T05:07:09.302 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1100 IOs, 495171584 bytes 2023-11-08T05:07:09.447 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1200 IOs, 516730880 bytes 2023-11-08T05:07:09.659 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1300 IOs, 580242432 bytes 2023-11-08T05:07:09.874 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1400 IOs, 639644672 bytes 2023-11-08T05:07:10.030 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1500 IOs, 671639552 bytes 2023-11-08T05:07:10.288 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1600 IOs, 748077056 bytes 2023-11-08T05:07:10.437 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1700 IOs, 781399040 bytes 2023-11-08T05:07:10.673 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1800 IOs, 856076288 bytes 2023-11-08T05:07:10.846 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1900 IOs, 896140288 bytes 2023-11-08T05:07:11.005 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2000 IOs, 916161536 bytes 2023-11-08T05:07:11.278 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2100 IOs, 1014241280 bytes 2023-11-08T05:07:11.410 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2200 IOs, 1027065856 bytes 2023-11-08T05:07:11.539 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2300 IOs, 1050856448 bytes 2023-11-08T05:07:11.682 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2400 IOs, 1073184768 bytes 2023-11-08T05:07:11.821 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2500 IOs, 1113518080 bytes 2023-11-08T05:07:12.050 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2600 IOs, 1156121600 bytes 2023-11-08T05:07:12.236 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2700 IOs, 1206772736 bytes 2023-11-08T05:07:12.429 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2800 IOs, 1254375424 bytes 2023-11-08T05:07:12.605 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2900 IOs, 1277983744 bytes 2023-11-08T05:07:12.823 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3000 IOs, 1336350720 bytes 2023-11-08T05:07:13.012 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3100 IOs, 1383785472 bytes 2023-11-08T05:07:13.261 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3200 IOs, 1457926144 bytes 2023-11-08T05:07:13.441 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3300 IOs, 1490884608 bytes 2023-11-08T05:07:13.570 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3400 IOs, 1502249984 bytes 2023-11-08T05:07:13.725 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3500 IOs, 1538037760 bytes 2023-11-08T05:07:13.976 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3600 IOs, 1616890880 bytes 2023-11-08T05:07:14.200 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3700 IOs, 1676013568 bytes 2023-11-08T05:07:14.310 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3800 IOs, 1685341184 bytes 2023-11-08T05:07:14.507 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3900 IOs, 1737951232 bytes 2023-11-08T05:07:14.667 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4000 IOs, 1772771328 bytes 2023-11-08T05:07:14.792 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4100 IOs, 1797518336 bytes 2023-11-08T05:07:15.048 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4200 IOs, 1880625152 bytes 2023-11-08T05:07:15.056 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.9883226973839571) > weight (0.25) 2023-11-08T05:07:15.056 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.1 2023-11-08T05:07:15.056 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Sent signal 15 2023-11-08T05:07:15.057 INFO:tasks.rbd_mirror_thrash:waiting for 5 secs before reviving daemons 2023-11-08T05:07:15.057 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi087.stderr:2023-11-08T05:07:15.056+0000 7f9a5ca72700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 (PID: 120142) UID: 1000 2023-11-08T05:07:15.241 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4300 IOs, 1915592704 bytes 2023-11-08T05:07:15.422 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4400 IOs, 1958927360 bytes 2023-11-08T05:07:15.458 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi087.stderr:daemon-helper: command failed with exit status 1 2023-11-08T05:07:15.528 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4500 IOs, 1964982272 bytes 2023-11-08T05:07:15.646 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4600 IOs, 1974320128 bytes 2023-11-08T05:07:15.783 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4700 IOs, 1989714944 bytes 2023-11-08T05:07:15.998 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4800 IOs, 2058469376 bytes 2023-11-08T05:07:16.172 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4900 IOs, 2085898240 bytes 2023-11-08T05:07:16.436 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5000 IOs, 2160935936 bytes 2023-11-08T05:07:16.611 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5100 IOs, 2198062080 bytes 2023-11-08T05:07:16.742 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5200 IOs, 2214192128 bytes 2023-11-08T05:07:16.870 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5300 IOs, 2232531968 bytes 2023-11-08T05:07:17.054 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5400 IOs, 2279819264 bytes 2023-11-08T05:07:17.324 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5500 IOs, 2354868224 bytes 2023-11-08T05:07:17.506 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5600 IOs, 2395788288 bytes 2023-11-08T05:07:17.745 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5700 IOs, 2489763840 bytes 2023-11-08T05:07:17.882 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5800 IOs, 2509894656 bytes 2023-11-08T05:07:18.110 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5900 IOs, 2564924416 bytes 2023-11-08T05:07:18.239 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:6000 IOs, 2575546368 bytes 2023-11-08T05:07:18.349 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:6100 IOs, 2584188928 bytes 2023-11-08T05:07:18.494 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:6200 IOs, 2599730176 bytes 2023-11-08T05:07:18.829 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:6300 IOs, 2717658112 bytes 2023-11-08T05:07:18.989 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:6400 IOs, 2753103872 bytes 2023-11-08T05:07:19.245 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ error_code=124 2023-11-08T05:07:19.246 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ set -e 2023-11-08T05:07:19.246 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' 124 -eq 124 ']' 2023-11-08T05:07:19.246 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:07:19.246 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-11-08T05:07:19.246 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image=image_17 2023-11-08T05:07:19.247 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ create_image_and_enable_mirror cluster2 mirror image_17 snapshot 128M 2023-11-08T05:07:19.247 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:07:19.247 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:07:19.247 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:07:19.248 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:07:19.248 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_17 2023-11-08T05:07:19.248 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:07:19.248 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local mode=snapshot 2023-11-08T05:07:19.248 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' -n snapshot ']' 2023-11-08T05:07:19.248 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:07:19.249 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ create_image cluster2 mirror image_17 128M 2023-11-08T05:07:19.249 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:07:19.249 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:07:19.249 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:07:19.249 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:07:19.250 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_17 2023-11-08T05:07:19.250 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:07:19.250 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local size=128 2023-11-08T05:07:19.250 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' -n 128M ']' 2023-11-08T05:07:19.250 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ size=128M 2023-11-08T05:07:19.251 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:07:19.251 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_17 2023-11-08T05:07:19.282 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' image = image ']' 2023-11-08T05:07:19.283 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ enable_mirror cluster2 mirror image_17 snapshot 2023-11-08T05:07:19.283 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:07:19.283 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:07:19.283 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_17 2023-11-08T05:07:19.283 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local mode=snapshot 2023-11-08T05:07:19.284 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_17 snapshot 2023-11-08T05:07:20.058 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.1 2023-11-08T05:07:20.058 INFO:teuthology.orchestra.run:waiting for 600 2023-11-08T05:07:20.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-08T05:07:20.059 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.1 2023-11-08T05:07:20.059 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Restarting daemon 2023-11-08T05:07:20.060 DEBUG:teuthology.orchestra.run.smithi087:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 2023-11-08T05:07:20.062 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Started 2023-11-08T05:07:20.063 INFO:tasks.rbd_mirror_thrash:waiting for 77 secs before thrashing 2023-11-08T05:07:20.320 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:Mirroring enabled 2023-11-08T05:07:20.326 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster=cluster2 info mirror/image_17 2023-11-08T05:07:20.356 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:rbd image 'image_17': 2023-11-08T05:07:20.356 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: size 128 MiB in 32 objects 2023-11-08T05:07:20.357 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: order 22 (4 MiB objects) 2023-11-08T05:07:20.357 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: snapshot_count: 1 2023-11-08T05:07:20.357 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: id: 13247f4183e1 2023-11-08T05:07:20.357 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: block_name_prefix: rbd_data.13247f4183e1 2023-11-08T05:07:20.357 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: format: 2 2023-11-08T05:07:20.358 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: features: layering, exclusive-lock 2023-11-08T05:07:20.358 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: op_features: 2023-11-08T05:07:20.358 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: flags: 2023-11-08T05:07:20.358 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: create_timestamp: Wed Nov 8 05:07:19 2023 2023-11-08T05:07:20.358 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: access_timestamp: Wed Nov 8 05:07:19 2023 2023-11-08T05:07:20.359 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: modify_timestamp: Wed Nov 8 05:07:19 2023 2023-11-08T05:07:20.359 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring state: enabled 2023-11-08T05:07:20.359 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring mode: snapshot 2023-11-08T05:07:20.359 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring global id: 7b62298f-cd5f-402a-b9f9-d2211585565d 2023-11-08T05:07:20.359 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring primary: true 2023-11-08T05:07:20.360 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' -n '' ']' 2023-11-08T05:07:20.361 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ stress_write_image cluster2 mirror image_17 2023-11-08T05:07:20.361 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:07:20.361 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:07:20.361 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_17 2023-11-08T05:07:20.361 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-11-08T05:07:20.361 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local duration=14 2023-11-08T05:07:20.362 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ set +e 2023-11-08T05:07:20.362 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_17 --debug-rbd=20 --debug-journaler=20 2023-11-08T05:07:20.504 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:100 IOs, 8717312 bytes 2023-11-08T05:07:20.650 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:200 IOs, 28528640 bytes 2023-11-08T05:07:20.787 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:300 IOs, 51033088 bytes 2023-11-08T05:07:20.992 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:400 IOs, 101734400 bytes 2023-11-08T05:07:21.259 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:500 IOs, 186747904 bytes 2023-11-08T05:07:21.562 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:600 IOs, 291116032 bytes 2023-11-08T05:07:21.722 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:700 IOs, 338799616 bytes 2023-11-08T05:07:21.973 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:800 IOs, 407491584 bytes 2023-11-08T05:07:22.212 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:900 IOs, 475244544 bytes 2023-11-08T05:07:22.427 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1000 IOs, 528464896 bytes 2023-11-08T05:07:22.668 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1100 IOs, 601458688 bytes 2023-11-08T05:07:22.945 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1200 IOs, 688804864 bytes 2023-11-08T05:07:23.238 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1300 IOs, 782928896 bytes 2023-11-08T05:07:23.463 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1400 IOs, 850319360 bytes 2023-11-08T05:07:23.682 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1500 IOs, 904323072 bytes 2023-11-08T05:07:23.861 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1600 IOs, 945595392 bytes 2023-11-08T05:07:24.095 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1700 IOs, 1026788352 bytes 2023-11-08T05:07:24.325 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1800 IOs, 1078787072 bytes 2023-11-08T05:07:24.537 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1900 IOs, 1148693504 bytes 2023-11-08T05:07:24.773 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2000 IOs, 1204599808 bytes 2023-11-08T05:07:24.913 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2100 IOs, 1222121472 bytes 2023-11-08T05:07:25.116 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2200 IOs, 1285528576 bytes 2023-11-08T05:07:25.371 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2300 IOs, 1353809920 bytes 2023-11-08T05:07:25.510 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2400 IOs, 1373553664 bytes 2023-11-08T05:07:25.717 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2500 IOs, 1433589760 bytes 2023-11-08T05:07:25.810 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2600 IOs, 1434206208 bytes 2023-11-08T05:07:25.900 INFO:tasks.ceph.cluster2.osd.0.smithi087.stderr:2023-11-08T05:07:25.898+0000 7f1ac5683700 -1 reset not still connected to 0x55f9ff47a680 2023-11-08T05:07:26.014 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2700 IOs, 1490563072 bytes 2023-11-08T05:07:26.296 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2800 IOs, 1572720640 bytes 2023-11-08T05:07:26.510 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2900 IOs, 1633346560 bytes 2023-11-08T05:07:26.686 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3000 IOs, 1682103296 bytes 2023-11-08T05:07:26.861 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3100 IOs, 1718568960 bytes 2023-11-08T05:07:26.975 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3200 IOs, 1722449920 bytes 2023-11-08T05:07:27.183 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3300 IOs, 1779633152 bytes 2023-11-08T05:07:27.470 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3400 IOs, 1876463616 bytes 2023-11-08T05:07:27.618 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3500 IOs, 1901192192 bytes 2023-11-08T05:07:27.756 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3600 IOs, 1919010816 bytes 2023-11-08T05:07:27.929 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3700 IOs, 1959779328 bytes 2023-11-08T05:07:28.102 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3800 IOs, 1993971712 bytes 2023-11-08T05:07:28.356 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3900 IOs, 2076179456 bytes 2023-11-08T05:07:28.540 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4000 IOs, 2115974144 bytes 2023-11-08T05:07:28.826 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4100 IOs, 2176355328 bytes 2023-11-08T05:07:29.034 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4200 IOs, 2227291136 bytes 2023-11-08T05:07:29.218 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4300 IOs, 2288162816 bytes 2023-11-08T05:07:29.483 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4400 IOs, 2346157056 bytes 2023-11-08T05:07:29.674 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4500 IOs, 2401068032 bytes 2023-11-08T05:07:29.873 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4600 IOs, 2433592320 bytes 2023-11-08T05:07:30.005 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4700 IOs, 2468578304 bytes 2023-11-08T05:07:30.161 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4800 IOs, 2488066048 bytes 2023-11-08T05:07:30.411 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4900 IOs, 2541939712 bytes 2023-11-08T05:07:30.559 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5000 IOs, 2565188608 bytes 2023-11-08T05:07:30.734 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5100 IOs, 2604688384 bytes 2023-11-08T05:07:30.857 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5200 IOs, 2619152384 bytes 2023-11-08T05:07:31.004 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5300 IOs, 2636388352 bytes 2023-11-08T05:07:31.226 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5400 IOs, 2692884480 bytes 2023-11-08T05:07:31.438 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5500 IOs, 2750220288 bytes 2023-11-08T05:07:31.641 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5600 IOs, 2808218624 bytes 2023-11-08T05:07:31.822 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5700 IOs, 2849243136 bytes 2023-11-08T05:07:31.991 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5800 IOs, 2885637120 bytes 2023-11-08T05:07:32.156 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5900 IOs, 2918069248 bytes 2023-11-08T05:07:32.326 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:6000 IOs, 2953559040 bytes 2023-11-08T05:07:32.478 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:6100 IOs, 2982078464 bytes 2023-11-08T05:07:32.599 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:6200 IOs, 2993033216 bytes 2023-11-08T05:07:32.860 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:6300 IOs, 3078312960 bytes 2023-11-08T05:07:32.982 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:6400 IOs, 3096755200 bytes 2023-11-08T05:07:33.196 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:6500 IOs, 3137971200 bytes 2023-11-08T05:07:33.451 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:6600 IOs, 3226248192 bytes 2023-11-08T05:07:33.600 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:6700 IOs, 3252014080 bytes 2023-11-08T05:07:33.738 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:6800 IOs, 3269972992 bytes 2023-11-08T05:07:33.950 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:6900 IOs, 3326592000 bytes 2023-11-08T05:07:34.229 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:7000 IOs, 3410247680 bytes 2023-11-08T05:07:34.373 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ error_code=124 2023-11-08T05:07:34.374 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ set -e 2023-11-08T05:07:34.374 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' 124 -eq 124 ']' 2023-11-08T05:07:34.374 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:07:34.375 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-11-08T05:07:34.375 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image=image_18 2023-11-08T05:07:34.375 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ create_image_and_enable_mirror cluster2 mirror image_18 snapshot 128M 2023-11-08T05:07:34.375 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:07:34.375 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:07:34.376 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:07:34.376 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:07:34.376 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_18 2023-11-08T05:07:34.377 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:07:34.377 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local mode=snapshot 2023-11-08T05:07:34.377 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' -n snapshot ']' 2023-11-08T05:07:34.377 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:07:34.377 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ create_image cluster2 mirror image_18 128M 2023-11-08T05:07:34.378 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:07:34.378 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:07:34.378 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:07:34.378 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:07:34.378 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_18 2023-11-08T05:07:34.379 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:07:34.379 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local size=128 2023-11-08T05:07:34.379 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' -n 128M ']' 2023-11-08T05:07:34.379 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ size=128M 2023-11-08T05:07:34.379 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:07:34.380 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_18 2023-11-08T05:07:34.618 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' image = image ']' 2023-11-08T05:07:34.619 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ enable_mirror cluster2 mirror image_18 snapshot 2023-11-08T05:07:34.619 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:07:34.619 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:07:34.619 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_18 2023-11-08T05:07:34.619 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local mode=snapshot 2023-11-08T05:07:34.620 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_18 snapshot 2023-11-08T05:07:35.451 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:Mirroring enabled 2023-11-08T05:07:35.459 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster=cluster2 info mirror/image_18 2023-11-08T05:07:35.492 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:rbd image 'image_18': 2023-11-08T05:07:35.492 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: size 128 MiB in 32 objects 2023-11-08T05:07:35.493 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: order 22 (4 MiB objects) 2023-11-08T05:07:35.493 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: snapshot_count: 1 2023-11-08T05:07:35.493 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: id: 133774ae646f 2023-11-08T05:07:35.493 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: block_name_prefix: rbd_data.133774ae646f 2023-11-08T05:07:35.494 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: format: 2 2023-11-08T05:07:35.494 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: features: layering, exclusive-lock 2023-11-08T05:07:35.494 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: op_features: 2023-11-08T05:07:35.495 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: flags: 2023-11-08T05:07:35.495 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: create_timestamp: Wed Nov 8 05:07:34 2023 2023-11-08T05:07:35.495 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: access_timestamp: Wed Nov 8 05:07:34 2023 2023-11-08T05:07:35.496 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: modify_timestamp: Wed Nov 8 05:07:34 2023 2023-11-08T05:07:35.496 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring state: enabled 2023-11-08T05:07:35.496 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring mode: snapshot 2023-11-08T05:07:35.496 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring global id: 2645822b-c23b-4b5d-bdd0-8114af333323 2023-11-08T05:07:35.497 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring primary: true 2023-11-08T05:07:35.498 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' -n '' ']' 2023-11-08T05:07:35.499 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ stress_write_image cluster2 mirror image_18 2023-11-08T05:07:35.499 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:07:35.499 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:07:35.499 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_18 2023-11-08T05:07:35.499 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-11-08T05:07:35.500 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local duration=10 2023-11-08T05:07:35.500 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ set +e 2023-11-08T05:07:35.500 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_18 --debug-rbd=20 --debug-journaler=20 2023-11-08T05:07:35.674 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:100 IOs, 30702592 bytes 2023-11-08T05:07:35.793 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:200 IOs, 45623296 bytes 2023-11-08T05:07:35.951 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:300 IOs, 78429184 bytes 2023-11-08T05:07:36.083 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:400 IOs, 100449280 bytes 2023-11-08T05:07:36.287 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:500 IOs, 151905280 bytes 2023-11-08T05:07:36.475 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:600 IOs, 201093120 bytes 2023-11-08T05:07:36.713 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:700 IOs, 283488256 bytes 2023-11-08T05:07:36.877 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:800 IOs, 316597248 bytes 2023-11-08T05:07:37.024 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:900 IOs, 339237888 bytes 2023-11-08T05:07:37.142 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1000 IOs, 368336896 bytes 2023-11-08T05:07:37.341 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1100 IOs, 410288128 bytes 2023-11-08T05:07:37.509 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1200 IOs, 454879232 bytes 2023-11-08T05:07:37.909 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1300 IOs, 592486400 bytes 2023-11-08T05:07:38.040 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1400 IOs, 624719872 bytes 2023-11-08T05:07:38.186 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1500 IOs, 641320960 bytes 2023-11-08T05:07:38.293 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1600 IOs, 646328320 bytes 2023-11-08T05:07:38.524 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1700 IOs, 721977344 bytes 2023-11-08T05:07:38.753 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1800 IOs, 772998144 bytes 2023-11-08T05:07:38.854 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1900 IOs, 778460160 bytes 2023-11-08T05:07:39.009 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2000 IOs, 813136896 bytes 2023-11-08T05:07:39.135 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2100 IOs, 831191040 bytes 2023-11-08T05:07:39.348 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2200 IOs, 898288640 bytes 2023-11-08T05:07:39.510 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2300 IOs, 927551488 bytes 2023-11-08T05:07:39.649 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2400 IOs, 943659008 bytes 2023-11-08T05:07:39.841 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2500 IOs, 993700864 bytes 2023-11-08T05:07:40.014 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2600 IOs, 1034844160 bytes 2023-11-08T05:07:40.216 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2700 IOs, 1100122112 bytes 2023-11-08T05:07:40.507 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2800 IOs, 1200242688 bytes 2023-11-08T05:07:40.639 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2900 IOs, 1227121664 bytes 2023-11-08T05:07:40.768 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3000 IOs, 1227736064 bytes 2023-11-08T05:07:40.972 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3100 IOs, 1281963008 bytes 2023-11-08T05:07:41.157 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3200 IOs, 1334091776 bytes 2023-11-08T05:07:41.257 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3300 IOs, 1366654976 bytes 2023-11-08T05:07:41.477 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3400 IOs, 1401371648 bytes 2023-11-08T05:07:41.747 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3500 IOs, 1499797504 bytes 2023-11-08T05:07:41.942 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3600 IOs, 1538885632 bytes 2023-11-08T05:07:42.165 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3700 IOs, 1610445824 bytes 2023-11-08T05:07:42.415 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3800 IOs, 1673733120 bytes 2023-11-08T05:07:42.538 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3900 IOs, 1688444928 bytes 2023-11-08T05:07:42.635 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4000 IOs, 1689054208 bytes 2023-11-08T05:07:42.834 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4100 IOs, 1737986048 bytes 2023-11-08T05:07:43.044 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4200 IOs, 1795861504 bytes 2023-11-08T05:07:43.266 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4300 IOs, 1860308992 bytes 2023-11-08T05:07:43.395 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4400 IOs, 1878608896 bytes 2023-11-08T05:07:43.612 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4500 IOs, 1938648064 bytes 2023-11-08T05:07:43.881 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4600 IOs, 2011771904 bytes 2023-11-08T05:07:44.030 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4700 IOs, 2040873984 bytes 2023-11-08T05:07:44.212 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4800 IOs, 2094599168 bytes 2023-11-08T05:07:44.354 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4900 IOs, 2112515072 bytes 2023-11-08T05:07:44.494 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5000 IOs, 2127466496 bytes 2023-11-08T05:07:44.741 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5100 IOs, 2196570112 bytes 2023-11-08T05:07:44.877 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5200 IOs, 2226881536 bytes 2023-11-08T05:07:45.077 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5300 IOs, 2270562304 bytes 2023-11-08T05:07:45.177 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5400 IOs, 2271159296 bytes 2023-11-08T05:07:45.369 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5500 IOs, 2318854144 bytes 2023-11-08T05:07:45.509 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ error_code=124 2023-11-08T05:07:45.509 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ set -e 2023-11-08T05:07:45.509 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' 124 -eq 124 ']' 2023-11-08T05:07:45.510 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:07:45.510 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-11-08T05:07:45.510 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image=image_19 2023-11-08T05:07:45.510 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ create_image_and_enable_mirror cluster2 mirror image_19 snapshot 128M 2023-11-08T05:07:45.510 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:07:45.511 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:07:45.511 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:07:45.511 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:07:45.511 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_19 2023-11-08T05:07:45.511 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:07:45.512 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local mode=snapshot 2023-11-08T05:07:45.512 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' -n snapshot ']' 2023-11-08T05:07:45.512 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:07:45.512 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ create_image cluster2 mirror image_19 128M 2023-11-08T05:07:45.512 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:07:45.513 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:07:45.513 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:07:45.513 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:07:45.513 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_19 2023-11-08T05:07:45.513 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:07:45.514 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local size=128 2023-11-08T05:07:45.514 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' -n 128M ']' 2023-11-08T05:07:45.514 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ size=128M 2023-11-08T05:07:45.514 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:07:45.514 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_19 2023-11-08T05:07:45.548 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' image = image ']' 2023-11-08T05:07:45.549 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ enable_mirror cluster2 mirror image_19 snapshot 2023-11-08T05:07:45.549 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:07:45.549 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:07:45.549 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_19 2023-11-08T05:07:45.549 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local mode=snapshot 2023-11-08T05:07:45.550 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_19 snapshot 2023-11-08T05:07:46.550 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:Mirroring enabled 2023-11-08T05:07:46.558 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster=cluster2 info mirror/image_19 2023-11-08T05:07:46.591 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:rbd image 'image_19': 2023-11-08T05:07:46.592 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: size 128 MiB in 32 objects 2023-11-08T05:07:46.592 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: order 22 (4 MiB objects) 2023-11-08T05:07:46.592 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: snapshot_count: 1 2023-11-08T05:07:46.593 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: id: 13425827c1d9 2023-11-08T05:07:46.593 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: block_name_prefix: rbd_data.13425827c1d9 2023-11-08T05:07:46.593 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: format: 2 2023-11-08T05:07:46.593 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: features: layering, exclusive-lock 2023-11-08T05:07:46.593 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: op_features: 2023-11-08T05:07:46.594 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: flags: 2023-11-08T05:07:46.594 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: create_timestamp: Wed Nov 8 05:07:45 2023 2023-11-08T05:07:46.594 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: access_timestamp: Wed Nov 8 05:07:45 2023 2023-11-08T05:07:46.594 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: modify_timestamp: Wed Nov 8 05:07:45 2023 2023-11-08T05:07:46.595 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring state: enabled 2023-11-08T05:07:46.595 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring mode: snapshot 2023-11-08T05:07:46.595 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring global id: 783a1447-f0d7-471d-9452-7cb8dece9780 2023-11-08T05:07:46.595 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring primary: true 2023-11-08T05:07:46.596 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' -n '' ']' 2023-11-08T05:07:46.597 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ stress_write_image cluster2 mirror image_19 2023-11-08T05:07:46.597 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:07:46.597 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:07:46.597 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_19 2023-11-08T05:07:46.597 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-11-08T05:07:46.597 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local duration=8 2023-11-08T05:07:46.598 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ set +e 2023-11-08T05:07:46.598 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_19 --debug-rbd=20 --debug-journaler=20 2023-11-08T05:07:46.757 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:100 IOs, 14454784 bytes 2023-11-08T05:07:46.938 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:200 IOs, 54416384 bytes 2023-11-08T05:07:47.112 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:300 IOs, 87135232 bytes 2023-11-08T05:07:47.239 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:400 IOs, 100774912 bytes 2023-11-08T05:07:47.421 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:500 IOs, 141936640 bytes 2023-11-08T05:07:47.636 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:600 IOs, 200048640 bytes 2023-11-08T05:07:47.954 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:700 IOs, 307490816 bytes 2023-11-08T05:07:48.101 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:800 IOs, 338636800 bytes 2023-11-08T05:07:48.297 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:900 IOs, 380397568 bytes 2023-11-08T05:07:48.398 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1000 IOs, 383379456 bytes 2023-11-08T05:07:48.649 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1100 IOs, 470185984 bytes 2023-11-08T05:07:48.852 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1200 IOs, 525339648 bytes 2023-11-08T05:07:48.998 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1300 IOs, 534819840 bytes 2023-11-08T05:07:49.092 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1400 IOs, 535414784 bytes 2023-11-08T05:07:49.253 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1500 IOs, 563603456 bytes 2023-11-08T05:07:49.270 INFO:tasks.ceph.cluster2.osd.1.smithi087.stderr:2023-11-08T05:07:49.269+0000 7fbb7a4fc700 -1 reset not still connected to 0x55770f770dd0 2023-11-08T05:07:49.363 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1600 IOs, 584142848 bytes 2023-11-08T05:07:49.527 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1700 IOs, 596739072 bytes 2023-11-08T05:07:49.722 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1800 IOs, 642091008 bytes 2023-11-08T05:07:49.821 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1900 IOs, 643320832 bytes 2023-11-08T05:07:50.039 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2000 IOs, 697457664 bytes 2023-11-08T05:07:50.212 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2100 IOs, 732399616 bytes 2023-11-08T05:07:50.316 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2200 IOs, 733040640 bytes 2023-11-08T05:07:50.511 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2300 IOs, 781841408 bytes 2023-11-08T05:07:50.782 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2400 IOs, 865381376 bytes 2023-11-08T05:07:50.979 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2500 IOs, 913885184 bytes 2023-11-08T05:07:51.185 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2600 IOs, 969229312 bytes 2023-11-08T05:07:51.320 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2700 IOs, 1012348928 bytes 2023-11-08T05:07:51.479 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2800 IOs, 1035808768 bytes 2023-11-08T05:07:51.716 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2900 IOs, 1090031616 bytes 2023-11-08T05:07:51.978 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3000 IOs, 1166267392 bytes 2023-11-08T05:07:52.163 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3100 IOs, 1206740992 bytes 2023-11-08T05:07:52.321 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3200 IOs, 1231969280 bytes 2023-11-08T05:07:52.589 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3300 IOs, 1313657856 bytes 2023-11-08T05:07:52.735 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3400 IOs, 1335171072 bytes 2023-11-08T05:07:52.862 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3500 IOs, 1347340288 bytes 2023-11-08T05:07:53.037 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3600 IOs, 1387601920 bytes 2023-11-08T05:07:53.227 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3700 IOs, 1429629952 bytes 2023-11-08T05:07:53.509 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3800 IOs, 1521672192 bytes 2023-11-08T05:07:53.714 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3900 IOs, 1575011328 bytes 2023-11-08T05:07:53.886 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4000 IOs, 1612417024 bytes 2023-11-08T05:07:54.093 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4100 IOs, 1661666304 bytes 2023-11-08T05:07:54.234 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4200 IOs, 1678992384 bytes 2023-11-08T05:07:54.412 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4300 IOs, 1714537472 bytes 2023-11-08T05:07:54.602 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ error_code=124 2023-11-08T05:07:54.602 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ set -e 2023-11-08T05:07:54.603 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' 124 -eq 124 ']' 2023-11-08T05:07:54.603 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:07:54.603 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-11-08T05:07:54.603 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image=image_20 2023-11-08T05:07:54.603 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ create_image_and_enable_mirror cluster2 mirror image_20 snapshot 128M 2023-11-08T05:07:54.604 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:07:54.604 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:07:54.604 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:07:54.604 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:07:54.604 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_20 2023-11-08T05:07:54.605 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:07:54.605 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local mode=snapshot 2023-11-08T05:07:54.605 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' -n snapshot ']' 2023-11-08T05:07:54.605 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:07:54.605 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ create_image cluster2 mirror image_20 128M 2023-11-08T05:07:54.606 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:07:54.606 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:07:54.606 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:07:54.606 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:07:54.606 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_20 2023-11-08T05:07:54.607 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:07:54.607 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local size=128 2023-11-08T05:07:54.607 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' -n 128M ']' 2023-11-08T05:07:54.607 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ size=128M 2023-11-08T05:07:54.607 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:07:54.608 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_20 2023-11-08T05:07:54.644 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' image = image ']' 2023-11-08T05:07:54.645 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ enable_mirror cluster2 mirror image_20 snapshot 2023-11-08T05:07:54.645 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:07:54.645 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:07:54.646 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_20 2023-11-08T05:07:54.646 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local mode=snapshot 2023-11-08T05:07:54.646 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_20 snapshot 2023-11-08T05:07:54.868 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:Mirroring enabled 2023-11-08T05:07:54.877 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster=cluster2 info mirror/image_20 2023-11-08T05:07:54.905 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:rbd image 'image_20': 2023-11-08T05:07:54.905 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: size 128 MiB in 32 objects 2023-11-08T05:07:54.905 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: order 22 (4 MiB objects) 2023-11-08T05:07:54.905 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: snapshot_count: 1 2023-11-08T05:07:54.906 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: id: 134bcef0ea41 2023-11-08T05:07:54.906 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: block_name_prefix: rbd_data.134bcef0ea41 2023-11-08T05:07:54.906 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: format: 2 2023-11-08T05:07:54.906 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: features: layering, exclusive-lock 2023-11-08T05:07:54.906 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: op_features: 2023-11-08T05:07:54.907 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: flags: 2023-11-08T05:07:54.907 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: create_timestamp: Wed Nov 8 05:07:54 2023 2023-11-08T05:07:54.907 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: access_timestamp: Wed Nov 8 05:07:54 2023 2023-11-08T05:07:54.907 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: modify_timestamp: Wed Nov 8 05:07:54 2023 2023-11-08T05:07:54.907 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring state: enabled 2023-11-08T05:07:54.908 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring mode: snapshot 2023-11-08T05:07:54.908 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring global id: d1ee9201-2afc-4853-a96a-0fbcc08cbfe5 2023-11-08T05:07:54.908 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring primary: true 2023-11-08T05:07:54.909 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' -n '' ']' 2023-11-08T05:07:54.909 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ stress_write_image cluster2 mirror image_20 2023-11-08T05:07:54.910 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:07:54.910 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:07:54.910 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_20 2023-11-08T05:07:54.910 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-11-08T05:07:54.910 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local duration=9 2023-11-08T05:07:54.911 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ set +e 2023-11-08T05:07:54.911 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_20 --debug-rbd=20 --debug-journaler=20 2023-11-08T05:07:55.090 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:100 IOs, 24930304 bytes 2023-11-08T05:07:55.249 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:200 IOs, 58293248 bytes 2023-11-08T05:07:55.364 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:300 IOs, 70546432 bytes 2023-11-08T05:07:55.540 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:400 IOs, 110683136 bytes 2023-11-08T05:07:55.769 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:500 IOs, 189685760 bytes 2023-11-08T05:07:56.057 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:600 IOs, 272146432 bytes 2023-11-08T05:07:56.190 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:700 IOs, 285275136 bytes 2023-11-08T05:07:56.348 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:800 IOs, 316547072 bytes 2023-11-08T05:07:56.499 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:900 IOs, 353675264 bytes 2023-11-08T05:07:56.616 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1000 IOs, 359596032 bytes 2023-11-08T05:07:56.725 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1100 IOs, 368432128 bytes 2023-11-08T05:07:56.896 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1200 IOs, 396226560 bytes 2023-11-08T05:07:57.055 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1300 IOs, 423625728 bytes 2023-11-08T05:07:57.355 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1400 IOs, 532954112 bytes 2023-11-08T05:07:57.543 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1500 IOs, 578791424 bytes 2023-11-08T05:07:57.688 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1600 IOs, 607400960 bytes 2023-11-08T05:07:57.783 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1700 IOs, 607998976 bytes 2023-11-08T05:07:57.962 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1800 IOs, 646148096 bytes 2023-11-08T05:07:58.130 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1900 IOs, 683252736 bytes 2023-11-08T05:07:58.276 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2000 IOs, 714762240 bytes 2023-11-08T05:07:58.475 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2100 IOs, 767311872 bytes 2023-11-08T05:07:58.648 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2200 IOs, 800359424 bytes 2023-11-08T05:07:58.808 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2300 IOs, 843560960 bytes 2023-11-08T05:07:59.043 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2400 IOs, 904022016 bytes 2023-11-08T05:07:59.351 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2500 IOs, 1006973952 bytes 2023-11-08T05:07:59.607 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2600 IOs, 1085622272 bytes 2023-11-08T05:07:59.762 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2700 IOs, 1111110656 bytes 2023-11-08T05:07:59.907 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2800 IOs, 1135130624 bytes 2023-11-08T05:08:00.043 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2900 IOs, 1154114560 bytes 2023-11-08T05:08:00.205 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3000 IOs, 1180019712 bytes 2023-11-08T05:08:00.342 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3100 IOs, 1198518272 bytes 2023-11-08T05:08:00.559 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3200 IOs, 1262242816 bytes 2023-11-08T05:08:00.671 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3300 IOs, 1266141184 bytes 2023-11-08T05:08:00.769 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3400 IOs, 1266777088 bytes 2023-11-08T05:08:00.942 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3500 IOs, 1304693760 bytes 2023-11-08T05:08:01.111 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3600 IOs, 1340443648 bytes 2023-11-08T05:08:01.253 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3700 IOs, 1363424256 bytes 2023-11-08T05:08:01.375 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3800 IOs, 1387670528 bytes 2023-11-08T05:08:01.584 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3900 IOs, 1416717312 bytes 2023-11-08T05:08:01.755 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4000 IOs, 1452627968 bytes 2023-11-08T05:08:01.901 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4100 IOs, 1469505536 bytes 2023-11-08T05:08:02.024 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4200 IOs, 1504023552 bytes 2023-11-08T05:08:02.280 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4300 IOs, 1570335744 bytes 2023-11-08T05:08:02.529 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4400 IOs, 1627882496 bytes 2023-11-08T05:08:02.718 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4500 IOs, 1674098688 bytes 2023-11-08T05:08:02.868 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4600 IOs, 1691049984 bytes 2023-11-08T05:08:03.025 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4700 IOs, 1715581952 bytes 2023-11-08T05:08:03.223 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4800 IOs, 1758896128 bytes 2023-11-08T05:08:03.482 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4900 IOs, 1816771584 bytes 2023-11-08T05:08:03.652 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5000 IOs, 1850501120 bytes 2023-11-08T05:08:03.870 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5100 IOs, 1909026816 bytes 2023-11-08T05:08:03.914 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ error_code=124 2023-11-08T05:08:03.914 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ set -e 2023-11-08T05:08:03.915 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' 124 -eq 124 ']' 2023-11-08T05:08:03.915 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:08:03.915 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-11-08T05:08:03.915 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image=image_21 2023-11-08T05:08:03.915 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ create_image_and_enable_mirror cluster2 mirror image_21 snapshot 128M 2023-11-08T05:08:03.915 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:08:03.916 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:08:03.916 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:08:03.916 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:08:03.916 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_21 2023-11-08T05:08:03.916 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:08:03.917 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local mode=snapshot 2023-11-08T05:08:03.917 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' -n snapshot ']' 2023-11-08T05:08:03.917 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:08:03.918 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ create_image cluster2 mirror image_21 128M 2023-11-08T05:08:03.918 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:08:03.918 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:08:03.918 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:08:03.918 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:08:03.919 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_21 2023-11-08T05:08:03.919 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:08:03.919 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local size=128 2023-11-08T05:08:03.919 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' -n 128M ']' 2023-11-08T05:08:03.919 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ size=128M 2023-11-08T05:08:03.920 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:08:03.920 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_21 2023-11-08T05:08:03.957 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' image = image ']' 2023-11-08T05:08:03.958 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ enable_mirror cluster2 mirror image_21 snapshot 2023-11-08T05:08:03.958 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:08:03.958 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:08:03.958 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_21 2023-11-08T05:08:03.959 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local mode=snapshot 2023-11-08T05:08:03.959 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_21 snapshot 2023-11-08T05:08:04.958 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:Mirroring enabled 2023-11-08T05:08:04.964 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster=cluster2 info mirror/image_21 2023-11-08T05:08:04.996 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:rbd image 'image_21': 2023-11-08T05:08:04.997 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: size 128 MiB in 32 objects 2023-11-08T05:08:04.997 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: order 22 (4 MiB objects) 2023-11-08T05:08:04.997 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: snapshot_count: 1 2023-11-08T05:08:04.998 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: id: 1353e39e8ebe 2023-11-08T05:08:04.998 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: block_name_prefix: rbd_data.1353e39e8ebe 2023-11-08T05:08:04.998 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: format: 2 2023-11-08T05:08:04.998 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: features: layering, exclusive-lock 2023-11-08T05:08:04.998 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: op_features: 2023-11-08T05:08:04.999 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: flags: 2023-11-08T05:08:04.999 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: create_timestamp: Wed Nov 8 05:08:03 2023 2023-11-08T05:08:04.999 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: access_timestamp: Wed Nov 8 05:08:03 2023 2023-11-08T05:08:04.999 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: modify_timestamp: Wed Nov 8 05:08:03 2023 2023-11-08T05:08:04.999 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring state: enabled 2023-11-08T05:08:05.000 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring mode: snapshot 2023-11-08T05:08:05.000 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring global id: 1d89653a-15af-4c63-92d4-196f96656cd3 2023-11-08T05:08:05.000 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring primary: true 2023-11-08T05:08:05.001 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' -n '' ']' 2023-11-08T05:08:05.001 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ stress_write_image cluster2 mirror image_21 2023-11-08T05:08:05.002 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:08:05.002 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:08:05.002 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_21 2023-11-08T05:08:05.002 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-11-08T05:08:05.003 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local duration=11 2023-11-08T05:08:05.003 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ set +e 2023-11-08T05:08:05.003 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_21 --debug-rbd=20 --debug-journaler=20 2023-11-08T05:08:05.235 INFO:tasks.ceph.cluster2.osd.2.smithi087.stderr:2023-11-08T05:08:05.232+0000 7fa243317700 -1 reset not still connected to 0x5613a4e37930 2023-11-08T05:08:05.235 INFO:tasks.ceph.cluster2.osd.2.smithi087.stderr:2023-11-08T05:08:05.232+0000 7fa243317700 -1 reset not still connected to 0x5613a5009ee0 2023-11-08T05:08:05.235 INFO:tasks.ceph.cluster2.osd.2.smithi087.stderr:2023-11-08T05:08:05.232+0000 7fa243317700 -1 reset not still connected to 0x5613a50edba0 2023-11-08T05:08:05.236 INFO:tasks.ceph.cluster2.osd.2.smithi087.stderr:2023-11-08T05:08:05.232+0000 7fa243317700 -1 reset not still connected to 0x5613a5dfdc70 2023-11-08T05:08:05.252 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:100 IOs, 61028352 bytes 2023-11-08T05:08:05.420 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:200 IOs, 108342272 bytes 2023-11-08T05:08:05.598 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:300 IOs, 140446720 bytes 2023-11-08T05:08:05.744 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:400 IOs, 167436288 bytes 2023-11-08T05:08:05.856 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:500 IOs, 179904512 bytes 2023-11-08T05:08:06.046 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:600 IOs, 227943424 bytes 2023-11-08T05:08:06.259 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:700 IOs, 278912000 bytes 2023-11-08T05:08:06.444 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:800 IOs, 320937984 bytes 2023-11-08T05:08:06.620 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:900 IOs, 357031936 bytes 2023-11-08T05:08:06.808 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1000 IOs, 403416064 bytes 2023-11-08T05:08:06.965 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1100 IOs, 435150848 bytes 2023-11-08T05:08:07.105 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1200 IOs, 452251648 bytes 2023-11-08T05:08:07.317 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1300 IOs, 512840704 bytes 2023-11-08T05:08:07.423 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1400 IOs, 513468416 bytes 2023-11-08T05:08:07.733 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1500 IOs, 626540544 bytes 2023-11-08T05:08:07.944 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1600 IOs, 680423424 bytes 2023-11-08T05:08:08.115 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1700 IOs, 715195392 bytes 2023-11-08T05:08:08.298 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1800 IOs, 761314304 bytes 2023-11-08T05:08:08.429 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1900 IOs, 775580672 bytes 2023-11-08T05:08:08.571 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2000 IOs, 797687808 bytes 2023-11-08T05:08:08.748 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2100 IOs, 840282112 bytes 2023-11-08T05:08:09.006 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2200 IOs, 925619200 bytes 2023-11-08T05:08:09.172 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2300 IOs, 975880192 bytes 2023-11-08T05:08:09.354 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2400 IOs, 997903360 bytes 2023-11-08T05:08:09.492 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2500 IOs, 1011231744 bytes 2023-11-08T05:08:09.677 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2600 IOs, 1052612608 bytes 2023-11-08T05:08:09.899 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2700 IOs, 1112338432 bytes 2023-11-08T05:08:10.141 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2800 IOs, 1177456640 bytes 2023-11-08T05:08:10.277 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2900 IOs, 1192624128 bytes 2023-11-08T05:08:10.539 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3000 IOs, 1273254912 bytes 2023-11-08T05:08:10.682 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3100 IOs, 1295676416 bytes 2023-11-08T05:08:10.817 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3200 IOs, 1316748288 bytes 2023-11-08T05:08:11.000 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3300 IOs, 1357469696 bytes 2023-11-08T05:08:11.194 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3400 IOs, 1405073408 bytes 2023-11-08T05:08:11.461 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3500 IOs, 1488930816 bytes 2023-11-08T05:08:11.610 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3600 IOs, 1511739392 bytes 2023-11-08T05:08:11.872 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3700 IOs, 1595896832 bytes 2023-11-08T05:08:12.078 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3800 IOs, 1645174784 bytes 2023-11-08T05:08:12.233 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3900 IOs, 1674957824 bytes 2023-11-08T05:08:12.470 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4000 IOs, 1732727808 bytes 2023-11-08T05:08:12.639 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4100 IOs, 1760718848 bytes 2023-11-08T05:08:12.739 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4200 IOs, 1761349632 bytes 2023-11-08T05:08:12.893 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4300 IOs, 1786225664 bytes 2023-11-08T05:08:13.062 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4400 IOs, 1819188224 bytes 2023-11-08T05:08:13.232 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4500 IOs, 1894823936 bytes 2023-11-08T05:08:13.461 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4600 IOs, 1927064576 bytes 2023-11-08T05:08:13.615 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4700 IOs, 1948573696 bytes 2023-11-08T05:08:13.823 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4800 IOs, 2008159232 bytes 2023-11-08T05:08:14.137 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4900 IOs, 2136328192 bytes 2023-11-08T05:08:14.412 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5000 IOs, 2224394240 bytes 2023-11-08T05:08:14.515 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5100 IOs, 2226573312 bytes 2023-11-08T05:08:14.680 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5200 IOs, 2252670976 bytes 2023-11-08T05:08:14.983 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5300 IOs, 2355878912 bytes 2023-11-08T05:08:15.136 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5400 IOs, 2382306304 bytes 2023-11-08T05:08:15.288 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5500 IOs, 2403294208 bytes 2023-11-08T05:08:15.400 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5600 IOs, 2403913728 bytes 2023-11-08T05:08:15.557 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5700 IOs, 2433579008 bytes 2023-11-08T05:08:15.803 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5800 IOs, 2505675776 bytes 2023-11-08T05:08:15.995 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5900 IOs, 2549186560 bytes 2023-11-08T05:08:16.008 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ error_code=124 2023-11-08T05:08:16.008 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ set -e 2023-11-08T05:08:16.009 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' 124 -eq 124 ']' 2023-11-08T05:08:16.009 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:08:16.009 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-11-08T05:08:16.009 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image=image_22 2023-11-08T05:08:16.009 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ create_image_and_enable_mirror cluster2 mirror image_22 snapshot 128M 2023-11-08T05:08:16.010 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:08:16.010 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:08:16.010 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:08:16.010 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:08:16.010 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_22 2023-11-08T05:08:16.011 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:08:16.011 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local mode=snapshot 2023-11-08T05:08:16.011 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' -n snapshot ']' 2023-11-08T05:08:16.011 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:08:16.011 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ create_image cluster2 mirror image_22 128M 2023-11-08T05:08:16.011 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:08:16.012 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:08:16.012 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:08:16.012 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:08:16.012 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_22 2023-11-08T05:08:16.012 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:08:16.013 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local size=128 2023-11-08T05:08:16.013 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' -n 128M ']' 2023-11-08T05:08:16.013 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ size=128M 2023-11-08T05:08:16.013 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:08:16.013 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_22 2023-11-08T05:08:16.046 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' image = image ']' 2023-11-08T05:08:16.046 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ enable_mirror cluster2 mirror image_22 snapshot 2023-11-08T05:08:16.047 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:08:16.047 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:08:16.047 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_22 2023-11-08T05:08:16.047 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local mode=snapshot 2023-11-08T05:08:16.047 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_22 snapshot 2023-11-08T05:08:17.044 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:Mirroring enabled 2023-11-08T05:08:17.050 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster=cluster2 info mirror/image_22 2023-11-08T05:08:17.082 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:rbd image 'image_22': 2023-11-08T05:08:17.082 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: size 128 MiB in 32 objects 2023-11-08T05:08:17.082 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: order 22 (4 MiB objects) 2023-11-08T05:08:17.083 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: snapshot_count: 1 2023-11-08T05:08:17.083 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: id: 135da5c12a5c 2023-11-08T05:08:17.083 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: block_name_prefix: rbd_data.135da5c12a5c 2023-11-08T05:08:17.083 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: format: 2 2023-11-08T05:08:17.084 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: features: layering, exclusive-lock 2023-11-08T05:08:17.084 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: op_features: 2023-11-08T05:08:17.084 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: flags: 2023-11-08T05:08:17.084 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: create_timestamp: Wed Nov 8 05:08:16 2023 2023-11-08T05:08:17.084 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: access_timestamp: Wed Nov 8 05:08:16 2023 2023-11-08T05:08:17.084 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: modify_timestamp: Wed Nov 8 05:08:16 2023 2023-11-08T05:08:17.085 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring state: enabled 2023-11-08T05:08:17.085 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring mode: snapshot 2023-11-08T05:08:17.085 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring global id: b8cf32bf-f325-43e1-a374-ee759365a5a1 2023-11-08T05:08:17.085 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring primary: true 2023-11-08T05:08:17.086 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' -n '' ']' 2023-11-08T05:08:17.087 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ stress_write_image cluster2 mirror image_22 2023-11-08T05:08:17.087 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:08:17.087 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:08:17.087 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_22 2023-11-08T05:08:17.087 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-11-08T05:08:17.087 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local duration=10 2023-11-08T05:08:17.088 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ set +e 2023-11-08T05:08:17.088 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_22 --debug-rbd=20 --debug-journaler=20 2023-11-08T05:08:17.310 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:100 IOs, 63497216 bytes 2023-11-08T05:08:17.480 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:200 IOs, 100596736 bytes 2023-11-08T05:08:17.664 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:300 IOs, 136481792 bytes 2023-11-08T05:08:17.840 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:400 IOs, 190526464 bytes 2023-11-08T05:08:17.992 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:500 IOs, 193999872 bytes 2023-11-08T05:08:18.124 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:600 IOs, 208614400 bytes 2023-11-08T05:08:18.311 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:700 IOs, 253703168 bytes 2023-11-08T05:08:18.455 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:800 IOs, 280038400 bytes 2023-11-08T05:08:18.558 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:900 IOs, 280671232 bytes 2023-11-08T05:08:18.678 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1000 IOs, 291874816 bytes 2023-11-08T05:08:18.843 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1100 IOs, 318545920 bytes 2023-11-08T05:08:19.040 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1200 IOs, 365185024 bytes 2023-11-08T05:08:19.167 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1300 IOs, 382333952 bytes 2023-11-08T05:08:19.441 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1400 IOs, 469335040 bytes 2023-11-08T05:08:19.670 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1500 IOs, 535156736 bytes 2023-11-08T05:08:19.858 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1600 IOs, 580731904 bytes 2023-11-08T05:08:20.035 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1700 IOs, 614103040 bytes 2023-11-08T05:08:20.190 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1800 IOs, 646717440 bytes 2023-11-08T05:08:20.432 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1900 IOs, 714597376 bytes 2023-11-08T05:08:20.573 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2000 IOs, 736953344 bytes 2023-11-08T05:08:20.678 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2100 IOs, 737570816 bytes 2023-11-08T05:08:20.850 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2200 IOs, 769827840 bytes 2023-11-08T05:08:21.093 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2300 IOs, 837488640 bytes 2023-11-08T05:08:21.244 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2400 IOs, 862409728 bytes 2023-11-08T05:08:21.466 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2500 IOs, 917005312 bytes 2023-11-08T05:08:21.604 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2600 IOs, 934821888 bytes 2023-11-08T05:08:21.813 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2700 IOs, 1003538432 bytes 2023-11-08T05:08:22.044 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2800 IOs, 1054020608 bytes 2023-11-08T05:08:22.238 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2900 IOs, 1099485184 bytes 2023-11-08T05:08:22.417 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3000 IOs, 1138521088 bytes 2023-11-08T05:08:22.584 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3100 IOs, 1172081664 bytes 2023-11-08T05:08:22.817 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3200 IOs, 1228195840 bytes 2023-11-08T05:08:22.973 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3300 IOs, 1277589504 bytes 2023-11-08T05:08:23.115 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3400 IOs, 1278191616 bytes 2023-11-08T05:08:23.336 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3500 IOs, 1331093504 bytes 2023-11-08T05:08:23.445 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3600 IOs, 1351365632 bytes 2023-11-08T05:08:23.693 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3700 IOs, 1417023488 bytes 2023-11-08T05:08:23.929 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3800 IOs, 1479581696 bytes 2023-11-08T05:08:24.095 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3900 IOs, 1507467264 bytes 2023-11-08T05:08:24.297 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4000 IOs, 1553707008 bytes 2023-11-08T05:08:24.448 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4100 IOs, 1577473024 bytes 2023-11-08T05:08:24.604 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4200 IOs, 1603083264 bytes 2023-11-08T05:08:24.828 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4300 IOs, 1665733632 bytes 2023-11-08T05:08:25.096 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4400 IOs, 1744116736 bytes 2023-11-08T05:08:25.235 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4500 IOs, 1762046976 bytes 2023-11-08T05:08:25.380 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4600 IOs, 1785066496 bytes 2023-11-08T05:08:25.522 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4700 IOs, 1807222784 bytes 2023-11-08T05:08:25.685 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4800 IOs, 1832710144 bytes 2023-11-08T05:08:25.869 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4900 IOs, 1870778368 bytes 2023-11-08T05:08:26.036 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5000 IOs, 1901606912 bytes 2023-11-08T05:08:26.169 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5100 IOs, 1920673792 bytes 2023-11-08T05:08:26.403 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5200 IOs, 1986274304 bytes 2023-11-08T05:08:26.543 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5300 IOs, 2006403072 bytes 2023-11-08T05:08:26.697 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5400 IOs, 2037088256 bytes 2023-11-08T05:08:26.861 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5500 IOs, 2062573568 bytes 2023-11-08T05:08:26.995 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5600 IOs, 2081369088 bytes 2023-11-08T05:08:27.093 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ error_code=124 2023-11-08T05:08:27.093 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ set -e 2023-11-08T05:08:27.094 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' 124 -eq 124 ']' 2023-11-08T05:08:27.094 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:08:27.094 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-11-08T05:08:27.094 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image=image_23 2023-11-08T05:08:27.094 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ create_image_and_enable_mirror cluster2 mirror image_23 snapshot 128M 2023-11-08T05:08:27.095 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:08:27.095 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:08:27.095 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:08:27.095 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:08:27.095 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_23 2023-11-08T05:08:27.096 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:08:27.096 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local mode=snapshot 2023-11-08T05:08:27.096 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' -n snapshot ']' 2023-11-08T05:08:27.096 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:08:27.097 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ create_image cluster2 mirror image_23 128M 2023-11-08T05:08:27.097 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:08:27.097 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:08:27.097 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:08:27.097 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:08:27.097 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_23 2023-11-08T05:08:27.098 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:08:27.098 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local size=128 2023-11-08T05:08:27.098 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' -n 128M ']' 2023-11-08T05:08:27.098 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ size=128M 2023-11-08T05:08:27.099 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:08:27.099 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_23 2023-11-08T05:08:27.133 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' image = image ']' 2023-11-08T05:08:27.133 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ enable_mirror cluster2 mirror image_23 snapshot 2023-11-08T05:08:27.133 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:08:27.134 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:08:27.134 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_23 2023-11-08T05:08:27.134 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local mode=snapshot 2023-11-08T05:08:27.134 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_23 snapshot 2023-11-08T05:08:28.091 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:Mirroring enabled 2023-11-08T05:08:28.099 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster=cluster2 info mirror/image_23 2023-11-08T05:08:28.132 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:rbd image 'image_23': 2023-11-08T05:08:28.133 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: size 128 MiB in 32 objects 2023-11-08T05:08:28.133 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: order 22 (4 MiB objects) 2023-11-08T05:08:28.133 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: snapshot_count: 1 2023-11-08T05:08:28.133 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: id: 13671e40786d 2023-11-08T05:08:28.133 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: block_name_prefix: rbd_data.13671e40786d 2023-11-08T05:08:28.134 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: format: 2 2023-11-08T05:08:28.134 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: features: layering, exclusive-lock 2023-11-08T05:08:28.134 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: op_features: 2023-11-08T05:08:28.134 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: flags: 2023-11-08T05:08:28.135 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: create_timestamp: Wed Nov 8 05:08:27 2023 2023-11-08T05:08:28.135 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: access_timestamp: Wed Nov 8 05:08:27 2023 2023-11-08T05:08:28.135 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: modify_timestamp: Wed Nov 8 05:08:27 2023 2023-11-08T05:08:28.135 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring state: enabled 2023-11-08T05:08:28.135 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring mode: snapshot 2023-11-08T05:08:28.136 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring global id: b0c3eb50-dd81-436f-b795-2235c773f092 2023-11-08T05:08:28.136 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring primary: true 2023-11-08T05:08:28.137 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' -n '' ']' 2023-11-08T05:08:28.137 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ stress_write_image cluster2 mirror image_23 2023-11-08T05:08:28.137 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:08:28.137 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:08:28.138 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_23 2023-11-08T05:08:28.138 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-11-08T05:08:28.138 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local duration=5 2023-11-08T05:08:28.138 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ set +e 2023-11-08T05:08:28.138 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_23 --debug-rbd=20 --debug-journaler=20 2023-11-08T05:08:28.312 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:100 IOs, 21448704 bytes 2023-11-08T05:08:28.585 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:200 IOs, 111579136 bytes 2023-11-08T05:08:28.832 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:300 IOs, 182728704 bytes 2023-11-08T05:08:28.980 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:400 IOs, 207547392 bytes 2023-11-08T05:08:29.110 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:500 IOs, 220608512 bytes 2023-11-08T05:08:29.278 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:600 IOs, 264162304 bytes 2023-11-08T05:08:29.459 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:700 IOs, 295640064 bytes 2023-11-08T05:08:29.627 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:800 IOs, 335150080 bytes 2023-11-08T05:08:29.918 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:900 IOs, 445029376 bytes 2023-11-08T05:08:30.127 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1000 IOs, 491401216 bytes 2023-11-08T05:08:30.351 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1100 IOs, 551162880 bytes 2023-11-08T05:08:30.537 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1200 IOs, 600016896 bytes 2023-11-08T05:08:30.646 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1300 IOs, 600643584 bytes 2023-11-08T05:08:30.782 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1400 IOs, 624348160 bytes 2023-11-08T05:08:30.995 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1500 IOs, 673516544 bytes 2023-11-08T05:08:31.152 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1600 IOs, 700665856 bytes 2023-11-08T05:08:31.372 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1700 IOs, 769946624 bytes 2023-11-08T05:08:31.651 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1800 IOs, 844426240 bytes 2023-11-08T05:08:31.886 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1900 IOs, 929029120 bytes 2023-11-08T05:08:32.039 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2000 IOs, 941416448 bytes 2023-11-08T05:08:32.259 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2100 IOs, 1002600448 bytes 2023-11-08T05:08:32.423 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2200 IOs, 1024325632 bytes 2023-11-08T05:08:32.560 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2300 IOs, 1041372160 bytes 2023-11-08T05:08:32.687 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2400 IOs, 1055668224 bytes 2023-11-08T05:08:32.853 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2500 IOs, 1081364480 bytes 2023-11-08T05:08:33.100 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2600 IOs, 1151933440 bytes 2023-11-08T05:08:33.143 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ error_code=124 2023-11-08T05:08:33.143 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ set -e 2023-11-08T05:08:33.144 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' 124 -eq 124 ']' 2023-11-08T05:08:33.144 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:08:33.144 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-11-08T05:08:33.144 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image=image_24 2023-11-08T05:08:33.144 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ create_image_and_enable_mirror cluster2 mirror image_24 snapshot 128M 2023-11-08T05:08:33.145 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:08:33.145 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:08:33.145 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:08:33.145 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:08:33.145 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_24 2023-11-08T05:08:33.146 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:08:33.146 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local mode=snapshot 2023-11-08T05:08:33.146 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' -n snapshot ']' 2023-11-08T05:08:33.146 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:08:33.146 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ create_image cluster2 mirror image_24 128M 2023-11-08T05:08:33.147 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:08:33.147 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:08:33.147 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:08:33.147 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:08:33.147 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_24 2023-11-08T05:08:33.148 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:08:33.148 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local size=128 2023-11-08T05:08:33.148 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' -n 128M ']' 2023-11-08T05:08:33.148 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ size=128M 2023-11-08T05:08:33.148 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:08:33.149 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_24 2023-11-08T05:08:33.185 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' image = image ']' 2023-11-08T05:08:33.186 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ enable_mirror cluster2 mirror image_24 snapshot 2023-11-08T05:08:33.186 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:08:33.186 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:08:33.186 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_24 2023-11-08T05:08:33.186 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local mode=snapshot 2023-11-08T05:08:33.187 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_24 snapshot 2023-11-08T05:08:34.152 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:Mirroring enabled 2023-11-08T05:08:34.160 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster=cluster2 info mirror/image_24 2023-11-08T05:08:34.192 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:rbd image 'image_24': 2023-11-08T05:08:34.193 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: size 128 MiB in 32 objects 2023-11-08T05:08:34.193 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: order 22 (4 MiB objects) 2023-11-08T05:08:34.193 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: snapshot_count: 1 2023-11-08T05:08:34.193 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: id: 137061ad947c 2023-11-08T05:08:34.194 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: block_name_prefix: rbd_data.137061ad947c 2023-11-08T05:08:34.194 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: format: 2 2023-11-08T05:08:34.194 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: features: layering, exclusive-lock 2023-11-08T05:08:34.194 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: op_features: 2023-11-08T05:08:34.195 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: flags: 2023-11-08T05:08:34.195 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: create_timestamp: Wed Nov 8 05:08:33 2023 2023-11-08T05:08:34.195 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: access_timestamp: Wed Nov 8 05:08:33 2023 2023-11-08T05:08:34.195 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: modify_timestamp: Wed Nov 8 05:08:33 2023 2023-11-08T05:08:34.195 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring state: enabled 2023-11-08T05:08:34.196 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring mode: snapshot 2023-11-08T05:08:34.196 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring global id: 4bf12ca9-9a70-496b-8c0e-da3173089291 2023-11-08T05:08:34.196 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring primary: true 2023-11-08T05:08:34.197 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' -n '' ']' 2023-11-08T05:08:34.197 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ stress_write_image cluster2 mirror image_24 2023-11-08T05:08:34.198 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:08:34.198 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:08:34.198 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_24 2023-11-08T05:08:34.198 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-11-08T05:08:34.199 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local duration=12 2023-11-08T05:08:34.199 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ set +e 2023-11-08T05:08:34.199 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_24 --debug-rbd=20 --debug-journaler=20 2023-11-08T05:08:34.412 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:100 IOs, 41975808 bytes 2023-11-08T05:08:34.581 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:200 IOs, 116709376 bytes 2023-11-08T05:08:34.793 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:300 IOs, 139448320 bytes 2023-11-08T05:08:34.967 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:400 IOs, 175872000 bytes 2023-11-08T05:08:35.141 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:500 IOs, 208498688 bytes 2023-11-08T05:08:35.383 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:600 IOs, 297032704 bytes 2023-11-08T05:08:35.654 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:700 IOs, 363677696 bytes 2023-11-08T05:08:35.806 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:800 IOs, 389724160 bytes 2023-11-08T05:08:36.001 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:900 IOs, 436488192 bytes 2023-11-08T05:08:36.148 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1000 IOs, 457696256 bytes 2023-11-08T05:08:36.304 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1100 IOs, 489156608 bytes 2023-11-08T05:08:36.511 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1200 IOs, 549320704 bytes 2023-11-08T05:08:36.747 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1300 IOs, 615094272 bytes 2023-11-08T05:08:37.010 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1400 IOs, 700681216 bytes 2023-11-08T05:08:37.064 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.5003133273232343) > weight (0.25) 2023-11-08T05:08:37.064 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.38579040128697784) > weight (0.25) 2023-11-08T05:08:37.065 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.4360114031925726) > weight (0.25) 2023-11-08T05:08:37.065 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.3 2023-11-08T05:08:37.065 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Sent signal 15 2023-11-08T05:08:37.066 INFO:tasks.rbd_mirror_thrash:waiting for 1 secs before reviving daemons 2023-11-08T05:08:37.066 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi087.stderr:2023-11-08T05:08:37.064+0000 7f66b4b61700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3 (PID: 127050) UID: 1000 2023-11-08T05:08:37.159 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1500 IOs, 718758912 bytes 2023-11-08T05:08:37.192 INFO:tasks.ceph.cluster2.osd.2.smithi087.stderr:2023-11-08T05:08:37.190+0000 7fa243317700 -1 reset not still connected to 0x5613a456cea0 2023-11-08T05:08:37.193 INFO:tasks.ceph.cluster2.osd.2.smithi087.stderr:2023-11-08T05:08:37.190+0000 7fa243317700 -1 reset not still connected to 0x5613a4631860 2023-11-08T05:08:37.193 INFO:tasks.ceph.cluster2.osd.2.smithi087.stderr:2023-11-08T05:08:37.190+0000 7fa243317700 -1 reset not still connected to 0x5613b889e4e0 2023-11-08T05:08:37.289 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1600 IOs, 721430528 bytes 2023-11-08T05:08:37.466 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi087.stderr:daemon-helper: command failed with exit status 1 2023-11-08T05:08:37.597 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1700 IOs, 821805056 bytes 2023-11-08T05:08:37.848 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1800 IOs, 896459776 bytes 2023-11-08T05:08:38.067 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.3 2023-11-08T05:08:38.067 INFO:teuthology.orchestra.run:waiting for 600 2023-11-08T05:08:38.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-08T05:08:38.068 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.3 2023-11-08T05:08:38.068 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Restarting daemon 2023-11-08T05:08:38.068 DEBUG:teuthology.orchestra.run.smithi087:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3 2023-11-08T05:08:38.071 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Started 2023-11-08T05:08:38.071 INFO:tasks.rbd_mirror_thrash:waiting for 117 secs before thrashing 2023-11-08T05:08:38.084 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1900 IOs, 959438848 bytes 2023-11-08T05:08:38.240 INFO:tasks.ceph.cluster2.osd.2.smithi087.stderr:2023-11-08T05:08:38.237+0000 7fa243317700 -1 reset not still connected to 0x5613a5ed1380 2023-11-08T05:08:38.240 INFO:tasks.ceph.cluster2.osd.2.smithi087.stderr:2023-11-08T05:08:38.237+0000 7fa243317700 -1 reset not still connected to 0x5613b824e820 2023-11-08T05:08:38.240 INFO:tasks.ceph.cluster2.osd.2.smithi087.stderr:2023-11-08T05:08:38.237+0000 7fa243317700 -1 reset not still connected to 0x5613bdabe820 2023-11-08T05:08:38.367 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2000 IOs, 1035743232 bytes 2023-11-08T05:08:38.586 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2100 IOs, 1087955968 bytes 2023-11-08T05:08:38.757 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2200 IOs, 1119345664 bytes 2023-11-08T05:08:38.867 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2300 IOs, 1119950848 bytes 2023-11-08T05:08:38.985 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2400 IOs, 1129208832 bytes 2023-11-08T05:08:39.248 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2500 IOs, 1210912768 bytes 2023-11-08T05:08:39.384 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2600 IOs, 1227102208 bytes 2023-11-08T05:08:39.628 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2700 IOs, 1295361024 bytes 2023-11-08T05:08:39.832 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2800 IOs, 1341773824 bytes 2023-11-08T05:08:39.980 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2900 IOs, 1365268480 bytes 2023-11-08T05:08:40.086 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3000 IOs, 1366432768 bytes 2023-11-08T05:08:40.241 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3100 IOs, 1392789504 bytes 2023-11-08T05:08:40.394 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3200 IOs, 1415370752 bytes 2023-11-08T05:08:40.544 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3300 IOs, 1439461376 bytes 2023-11-08T05:08:40.709 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3400 IOs, 1471990784 bytes 2023-11-08T05:08:40.851 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3500 IOs, 1484658688 bytes 2023-11-08T05:08:41.066 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3600 IOs, 1544830976 bytes 2023-11-08T05:08:41.273 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3700 IOs, 1614180352 bytes 2023-11-08T05:08:41.681 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3800 IOs, 1749158912 bytes 2023-11-08T05:08:41.939 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3900 IOs, 1819200512 bytes 2023-11-08T05:08:42.147 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4000 IOs, 1865592832 bytes 2023-11-08T05:08:42.272 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4100 IOs, 1870269440 bytes 2023-11-08T05:08:42.414 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4200 IOs, 1891501056 bytes 2023-11-08T05:08:42.590 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4300 IOs, 1922696192 bytes 2023-11-08T05:08:42.781 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4400 IOs, 1972622336 bytes 2023-11-08T05:08:43.069 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4500 IOs, 2068856832 bytes 2023-11-08T05:08:43.300 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4600 IOs, 2126285824 bytes 2023-11-08T05:08:43.621 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4700 IOs, 2240462848 bytes 2023-11-08T05:08:43.912 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4800 IOs, 2341792768 bytes 2023-11-08T05:08:44.223 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4900 IOs, 2445875200 bytes 2023-11-08T05:08:44.385 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5000 IOs, 2466939904 bytes 2023-11-08T05:08:44.622 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5100 IOs, 2537997312 bytes 2023-11-08T05:08:44.792 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5200 IOs, 2570489856 bytes 2023-11-08T05:08:44.940 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5300 IOs, 2589399040 bytes 2023-11-08T05:08:45.157 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5400 IOs, 2648682496 bytes 2023-11-08T05:08:45.336 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5500 IOs, 2682975232 bytes 2023-11-08T05:08:45.449 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5600 IOs, 2684075008 bytes 2023-11-08T05:08:45.605 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5700 IOs, 2717464576 bytes 2023-11-08T05:08:45.741 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5800 IOs, 2733868032 bytes 2023-11-08T05:08:45.941 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5900 IOs, 2780733440 bytes 2023-11-08T05:08:46.125 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:6000 IOs, 2839405568 bytes 2023-11-08T05:08:46.204 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ error_code=124 2023-11-08T05:08:46.204 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ set -e 2023-11-08T05:08:46.204 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' 124 -eq 124 ']' 2023-11-08T05:08:46.204 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:08:46.205 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-11-08T05:08:46.205 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image=image_25 2023-11-08T05:08:46.205 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ create_image_and_enable_mirror cluster2 mirror image_25 snapshot 128M 2023-11-08T05:08:46.205 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:08:46.205 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:08:46.206 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:08:46.206 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:08:46.206 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_25 2023-11-08T05:08:46.206 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:08:46.207 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local mode=snapshot 2023-11-08T05:08:46.207 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' -n snapshot ']' 2023-11-08T05:08:46.207 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:08:46.207 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ create_image cluster2 mirror image_25 128M 2023-11-08T05:08:46.207 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:08:46.207 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:08:46.208 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:08:46.208 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:08:46.208 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_25 2023-11-08T05:08:46.208 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:08:46.208 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local size=128 2023-11-08T05:08:46.209 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' -n 128M ']' 2023-11-08T05:08:46.209 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ size=128M 2023-11-08T05:08:46.209 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:08:46.209 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_25 2023-11-08T05:08:46.244 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' image = image ']' 2023-11-08T05:08:46.244 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ enable_mirror cluster2 mirror image_25 snapshot 2023-11-08T05:08:46.244 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:08:46.245 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:08:46.245 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_25 2023-11-08T05:08:46.245 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local mode=snapshot 2023-11-08T05:08:46.245 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_25 snapshot 2023-11-08T05:08:46.319 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:Mirroring enabled 2023-11-08T05:08:46.328 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster=cluster2 info mirror/image_25 2023-11-08T05:08:46.357 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:rbd image 'image_25': 2023-11-08T05:08:46.358 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: size 128 MiB in 32 objects 2023-11-08T05:08:46.358 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: order 22 (4 MiB objects) 2023-11-08T05:08:46.358 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: snapshot_count: 1 2023-11-08T05:08:46.359 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: id: 13869ad91be5 2023-11-08T05:08:46.359 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: block_name_prefix: rbd_data.13869ad91be5 2023-11-08T05:08:46.359 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: format: 2 2023-11-08T05:08:46.359 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: features: layering, exclusive-lock 2023-11-08T05:08:46.360 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: op_features: 2023-11-08T05:08:46.360 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: flags: 2023-11-08T05:08:46.360 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: create_timestamp: Wed Nov 8 05:08:46 2023 2023-11-08T05:08:46.360 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: access_timestamp: Wed Nov 8 05:08:46 2023 2023-11-08T05:08:46.360 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: modify_timestamp: Wed Nov 8 05:08:46 2023 2023-11-08T05:08:46.361 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring state: enabled 2023-11-08T05:08:46.361 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring mode: snapshot 2023-11-08T05:08:46.361 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring global id: 4f72c9b1-e028-4538-ade8-ae09b2818aa9 2023-11-08T05:08:46.361 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring primary: true 2023-11-08T05:08:46.362 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' -n '' ']' 2023-11-08T05:08:46.362 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ stress_write_image cluster2 mirror image_25 2023-11-08T05:08:46.363 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:08:46.363 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:08:46.363 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_25 2023-11-08T05:08:46.363 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-11-08T05:08:46.363 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local duration=8 2023-11-08T05:08:46.364 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ set +e 2023-11-08T05:08:46.364 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_25 --debug-rbd=20 --debug-journaler=20 2023-11-08T05:08:46.519 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:100 IOs, 20648960 bytes 2023-11-08T05:08:46.714 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:200 IOs, 68697088 bytes 2023-11-08T05:08:46.884 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:300 IOs, 105276416 bytes 2023-11-08T05:08:47.072 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:400 IOs, 153010176 bytes 2023-11-08T05:08:47.220 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:500 IOs, 175855616 bytes 2023-11-08T05:08:47.529 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:600 IOs, 283025408 bytes 2023-11-08T05:08:47.663 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:700 IOs, 306299904 bytes 2023-11-08T05:08:47.894 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:800 IOs, 378055680 bytes 2023-11-08T05:08:48.082 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:900 IOs, 430041088 bytes 2023-11-08T05:08:48.362 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1000 IOs, 514382848 bytes 2023-11-08T05:08:48.511 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1100 IOs, 532161536 bytes 2023-11-08T05:08:48.682 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1200 IOs, 557366272 bytes 2023-11-08T05:08:48.862 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1300 IOs, 599346176 bytes 2023-11-08T05:08:49.020 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1400 IOs, 630103040 bytes 2023-11-08T05:08:49.057 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.44111559536162237) > weight (0.25) 2023-11-08T05:08:49.057 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.1 2023-11-08T05:08:49.058 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Sent signal 15 2023-11-08T05:08:49.058 INFO:tasks.rbd_mirror_thrash:waiting for 0 secs before reviving daemons 2023-11-08T05:08:49.058 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.1 2023-11-08T05:08:49.058 INFO:teuthology.orchestra.run:waiting for 600 2023-11-08T05:08:49.059 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi087.stderr:2023-11-08T05:08:49.056+0000 7f9b8690d700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 (PID: 123979) UID: 1000 2023-11-08T05:08:49.292 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1500 IOs, 712857600 bytes 2023-11-08T05:08:49.459 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi087.stderr:daemon-helper: command failed with exit status 1 2023-11-08T05:08:49.462 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1600 IOs, 753166336 bytes 2023-11-08T05:08:49.663 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1700 IOs, 804748288 bytes 2023-11-08T05:08:49.838 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1800 IOs, 842359808 bytes 2023-11-08T05:08:50.073 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1900 IOs, 933026816 bytes 2023-11-08T05:08:50.368 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2000 IOs, 997443584 bytes 2023-11-08T05:08:50.611 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2100 IOs, 1038889984 bytes 2023-11-08T05:08:50.764 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2200 IOs, 1063716864 bytes 2023-11-08T05:08:50.938 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2300 IOs, 1099065344 bytes 2023-11-08T05:08:51.207 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2400 IOs, 1179833344 bytes 2023-11-08T05:08:51.352 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2500 IOs, 1199758336 bytes 2023-11-08T05:08:51.481 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2600 IOs, 1217347584 bytes 2023-11-08T05:08:51.647 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2700 IOs, 1248589824 bytes 2023-11-08T05:08:51.810 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2800 IOs, 1275693056 bytes 2023-11-08T05:08:52.028 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2900 IOs, 1333722112 bytes 2023-11-08T05:08:52.169 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3000 IOs, 1355565056 bytes 2023-11-08T05:08:52.303 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3100 IOs, 1371751424 bytes 2023-11-08T05:08:52.461 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3200 IOs, 1403005952 bytes 2023-11-08T05:08:52.620 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3300 IOs, 1430188032 bytes 2023-11-08T05:08:52.734 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3400 IOs, 1430801408 bytes 2023-11-08T05:08:52.870 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3500 IOs, 1462382592 bytes 2023-11-08T05:08:53.140 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3600 IOs, 1534411776 bytes 2023-11-08T05:08:53.282 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3700 IOs, 1553752064 bytes 2023-11-08T05:08:53.429 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3800 IOs, 1577289728 bytes 2023-11-08T05:08:53.734 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3900 IOs, 1682329600 bytes 2023-11-08T05:08:53.855 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4000 IOs, 1719142400 bytes 2023-11-08T05:08:54.121 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4100 IOs, 1775516672 bytes 2023-11-08T05:08:54.260 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4200 IOs, 1795172352 bytes 2023-11-08T05:08:54.370 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ error_code=124 2023-11-08T05:08:54.370 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ set -e 2023-11-08T05:08:54.370 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' 124 -eq 124 ']' 2023-11-08T05:08:54.370 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:08:54.371 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-11-08T05:08:54.371 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image=image_26 2023-11-08T05:08:54.371 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ create_image_and_enable_mirror cluster2 mirror image_26 snapshot 128M 2023-11-08T05:08:54.371 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:08:54.371 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:08:54.372 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:08:54.372 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:08:54.372 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_26 2023-11-08T05:08:54.372 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:08:54.372 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local mode=snapshot 2023-11-08T05:08:54.373 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' -n snapshot ']' 2023-11-08T05:08:54.373 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:08:54.373 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ create_image cluster2 mirror image_26 128M 2023-11-08T05:08:54.373 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:08:54.373 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:08:54.373 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:08:54.374 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:08:54.374 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_26 2023-11-08T05:08:54.374 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:08:54.374 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local size=128 2023-11-08T05:08:54.374 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' -n 128M ']' 2023-11-08T05:08:54.375 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ size=128M 2023-11-08T05:08:54.375 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:08:54.375 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_26 2023-11-08T05:08:54.411 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' image = image ']' 2023-11-08T05:08:54.412 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ enable_mirror cluster2 mirror image_26 snapshot 2023-11-08T05:08:54.412 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:08:54.412 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:08:54.412 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_26 2023-11-08T05:08:54.412 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local mode=snapshot 2023-11-08T05:08:54.413 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_26 snapshot 2023-11-08T05:08:55.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-08T05:08:55.161 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.1 2023-11-08T05:08:55.161 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Restarting daemon 2023-11-08T05:08:55.161 DEBUG:teuthology.orchestra.run.smithi087:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 2023-11-08T05:08:55.165 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Started 2023-11-08T05:08:55.165 INFO:tasks.rbd_mirror_thrash:waiting for 93 secs before thrashing 2023-11-08T05:08:55.446 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:Mirroring enabled 2023-11-08T05:08:55.452 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster=cluster2 info mirror/image_26 2023-11-08T05:08:55.481 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:rbd image 'image_26': 2023-11-08T05:08:55.482 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: size 128 MiB in 32 objects 2023-11-08T05:08:55.482 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: order 22 (4 MiB objects) 2023-11-08T05:08:55.482 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: snapshot_count: 1 2023-11-08T05:08:55.483 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: id: 138f94e40bde 2023-11-08T05:08:55.483 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: block_name_prefix: rbd_data.138f94e40bde 2023-11-08T05:08:55.483 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: format: 2 2023-11-08T05:08:55.483 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: features: layering, exclusive-lock 2023-11-08T05:08:55.484 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: op_features: 2023-11-08T05:08:55.484 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: flags: 2023-11-08T05:08:55.484 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: create_timestamp: Wed Nov 8 05:08:54 2023 2023-11-08T05:08:55.485 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: access_timestamp: Wed Nov 8 05:08:54 2023 2023-11-08T05:08:55.485 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: modify_timestamp: Wed Nov 8 05:08:54 2023 2023-11-08T05:08:55.485 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring state: enabled 2023-11-08T05:08:55.485 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring mode: snapshot 2023-11-08T05:08:55.486 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring global id: fff00620-df68-4ab5-8679-1bb60d01c79a 2023-11-08T05:08:55.486 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring primary: true 2023-11-08T05:08:55.487 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' -n '' ']' 2023-11-08T05:08:55.487 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ stress_write_image cluster2 mirror image_26 2023-11-08T05:08:55.487 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:08:55.488 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:08:55.488 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_26 2023-11-08T05:08:55.488 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-11-08T05:08:55.488 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local duration=5 2023-11-08T05:08:55.488 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ set +e 2023-11-08T05:08:55.489 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_26 --debug-rbd=20 --debug-journaler=20 2023-11-08T05:08:55.773 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:100 IOs, 85472256 bytes 2023-11-08T05:08:55.900 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:200 IOs, 105991168 bytes 2023-11-08T05:08:56.120 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:300 IOs, 148840448 bytes 2023-11-08T05:08:56.250 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:400 IOs, 164050944 bytes 2023-11-08T05:08:56.466 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:500 IOs, 234150912 bytes 2023-11-08T05:08:56.784 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:600 IOs, 346753024 bytes 2023-11-08T05:08:56.993 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:700 IOs, 416875520 bytes 2023-11-08T05:08:57.345 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:800 IOs, 506778624 bytes 2023-11-08T05:08:57.509 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:900 IOs, 541581312 bytes 2023-11-08T05:08:57.683 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1000 IOs, 584084480 bytes 2023-11-08T05:08:57.895 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1100 IOs, 664408064 bytes 2023-11-08T05:08:58.211 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1200 IOs, 743287808 bytes 2023-11-08T05:08:58.349 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1300 IOs, 776254464 bytes 2023-11-08T05:08:58.490 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1400 IOs, 776852480 bytes 2023-11-08T05:08:58.612 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1500 IOs, 787139584 bytes 2023-11-08T05:08:58.786 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1600 IOs, 845584384 bytes 2023-11-08T05:08:58.930 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1700 IOs, 851336192 bytes 2023-11-08T05:08:59.109 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1800 IOs, 886277120 bytes 2023-11-08T05:08:59.235 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1900 IOs, 900009984 bytes 2023-11-08T05:08:59.386 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2000 IOs, 925442048 bytes 2023-11-08T05:08:59.517 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2100 IOs, 933651456 bytes 2023-11-08T05:08:59.650 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2200 IOs, 943763456 bytes 2023-11-08T05:08:59.908 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2300 IOs, 1021392896 bytes 2023-11-08T05:09:00.138 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2400 IOs, 1087307776 bytes 2023-11-08T05:09:00.282 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2500 IOs, 1104544768 bytes 2023-11-08T05:09:00.441 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2600 IOs, 1138007040 bytes 2023-11-08T05:09:00.517 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ error_code=124 2023-11-08T05:09:00.518 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ set -e 2023-11-08T05:09:00.518 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' 124 -eq 124 ']' 2023-11-08T05:09:00.518 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:09:00.518 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-11-08T05:09:00.519 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image=image_27 2023-11-08T05:09:00.519 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ create_image_and_enable_mirror cluster2 mirror image_27 snapshot 128M 2023-11-08T05:09:00.519 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:09:00.519 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:09:00.519 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:09:00.520 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:09:00.520 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_27 2023-11-08T05:09:00.520 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:09:00.520 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local mode=snapshot 2023-11-08T05:09:00.520 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' -n snapshot ']' 2023-11-08T05:09:00.521 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:09:00.521 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ create_image cluster2 mirror image_27 128M 2023-11-08T05:09:00.521 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:09:00.521 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:09:00.521 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:09:00.521 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:09:00.522 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_27 2023-11-08T05:09:00.522 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:09:00.522 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local size=128 2023-11-08T05:09:00.522 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' -n 128M ']' 2023-11-08T05:09:00.522 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ size=128M 2023-11-08T05:09:00.523 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:09:00.523 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_27 2023-11-08T05:09:00.557 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' image = image ']' 2023-11-08T05:09:00.557 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ enable_mirror cluster2 mirror image_27 snapshot 2023-11-08T05:09:00.558 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:09:00.558 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:09:00.558 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_27 2023-11-08T05:09:00.558 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local mode=snapshot 2023-11-08T05:09:00.558 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_27 snapshot 2023-11-08T05:09:01.510 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:Mirroring enabled 2023-11-08T05:09:01.518 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster=cluster2 info mirror/image_27 2023-11-08T05:09:01.550 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:rbd image 'image_27': 2023-11-08T05:09:01.551 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: size 128 MiB in 32 objects 2023-11-08T05:09:01.551 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: order 22 (4 MiB objects) 2023-11-08T05:09:01.551 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: snapshot_count: 1 2023-11-08T05:09:01.552 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: id: 139f7cf4478a 2023-11-08T05:09:01.552 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: block_name_prefix: rbd_data.139f7cf4478a 2023-11-08T05:09:01.552 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: format: 2 2023-11-08T05:09:01.552 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: features: layering, exclusive-lock 2023-11-08T05:09:01.552 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: op_features: 2023-11-08T05:09:01.553 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: flags: 2023-11-08T05:09:01.553 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: create_timestamp: Wed Nov 8 05:09:00 2023 2023-11-08T05:09:01.553 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: access_timestamp: Wed Nov 8 05:09:00 2023 2023-11-08T05:09:01.553 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: modify_timestamp: Wed Nov 8 05:09:00 2023 2023-11-08T05:09:01.553 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring state: enabled 2023-11-08T05:09:01.553 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring mode: snapshot 2023-11-08T05:09:01.554 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring global id: ebbdc03f-1275-4b8e-b5a6-94b379cd1440 2023-11-08T05:09:01.554 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring primary: true 2023-11-08T05:09:01.555 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' -n '' ']' 2023-11-08T05:09:01.555 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ stress_write_image cluster2 mirror image_27 2023-11-08T05:09:01.555 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:09:01.555 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:09:01.556 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_27 2023-11-08T05:09:01.556 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-11-08T05:09:01.556 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local duration=14 2023-11-08T05:09:01.556 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ set +e 2023-11-08T05:09:01.556 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_27 --debug-rbd=20 --debug-journaler=20 2023-11-08T05:09:01.775 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:100 IOs, 36173824 bytes 2023-11-08T05:09:01.943 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:200 IOs, 68116480 bytes 2023-11-08T05:09:02.048 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:300 IOs, 68729856 bytes 2023-11-08T05:09:02.205 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:400 IOs, 103195648 bytes 2023-11-08T05:09:02.412 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:500 IOs, 166337536 bytes 2023-11-08T05:09:02.681 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:600 IOs, 222555136 bytes 2023-11-08T05:09:02.887 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:700 IOs, 291007488 bytes 2023-11-08T05:09:02.994 INFO:tasks.ceph.cluster2.osd.0.smithi087.stderr:2023-11-08T05:09:02.992+0000 7f1ac5683700 -1 reset not still connected to 0x55fa0fe4f5f0 2023-11-08T05:09:03.114 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:800 IOs, 345512960 bytes 2023-11-08T05:09:03.335 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:900 IOs, 407222272 bytes 2023-11-08T05:09:03.516 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1000 IOs, 451456000 bytes 2023-11-08T05:09:03.699 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1100 IOs, 491579392 bytes 2023-11-08T05:09:03.795 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1200 IOs, 492219392 bytes 2023-11-08T05:09:03.920 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1300 IOs, 517064704 bytes 2023-11-08T05:09:04.183 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1400 IOs, 586735616 bytes 2023-11-08T05:09:04.385 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1500 IOs, 635543552 bytes 2023-11-08T05:09:04.517 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1600 IOs, 654066688 bytes 2023-11-08T05:09:04.631 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1700 IOs, 674985984 bytes 2023-11-08T05:09:04.789 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1800 IOs, 685648896 bytes 2023-11-08T05:09:04.972 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1900 IOs, 736796672 bytes 2023-11-08T05:09:05.127 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2000 IOs, 761024512 bytes 2023-11-08T05:09:05.402 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2100 IOs, 841385984 bytes 2023-11-08T05:09:05.615 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2200 IOs, 892355584 bytes 2023-11-08T05:09:05.811 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2300 IOs, 968227840 bytes 2023-11-08T05:09:06.027 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2400 IOs, 1003545600 bytes 2023-11-08T05:09:06.245 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2500 IOs, 1058035712 bytes 2023-11-08T05:09:06.509 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2600 IOs, 1132804096 bytes 2023-11-08T05:09:06.750 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2700 IOs, 1202116608 bytes 2023-11-08T05:09:06.899 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2800 IOs, 1230203904 bytes 2023-11-08T05:09:07.034 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2900 IOs, 1267820544 bytes 2023-11-08T05:09:07.213 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3000 IOs, 1285976064 bytes 2023-11-08T05:09:07.495 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3100 IOs, 1375710208 bytes 2023-11-08T05:09:07.734 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3200 IOs, 1457086464 bytes 2023-11-08T05:09:07.914 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3300 IOs, 1481832448 bytes 2023-11-08T05:09:08.040 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3400 IOs, 1497845760 bytes 2023-11-08T05:09:08.246 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3500 IOs, 1549481984 bytes 2023-11-08T05:09:08.444 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3600 IOs, 1593216000 bytes 2023-11-08T05:09:08.629 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3700 IOs, 1636046848 bytes 2023-11-08T05:09:08.878 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3800 IOs, 1706275840 bytes 2023-11-08T05:09:09.232 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3900 IOs, 1835602944 bytes 2023-11-08T05:09:09.404 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4000 IOs, 1886127104 bytes 2023-11-08T05:09:09.604 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4100 IOs, 1929060352 bytes 2023-11-08T05:09:09.786 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4200 IOs, 1967623168 bytes 2023-11-08T05:09:10.081 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4300 IOs, 2070062080 bytes 2023-11-08T05:09:10.232 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4400 IOs, 2092328960 bytes 2023-11-08T05:09:10.415 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4500 IOs, 2123527168 bytes 2023-11-08T05:09:10.558 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4600 IOs, 2144568320 bytes 2023-11-08T05:09:10.824 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4700 IOs, 2226851840 bytes 2023-11-08T05:09:11.011 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4800 IOs, 2262901760 bytes 2023-11-08T05:09:11.215 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4900 IOs, 2340685824 bytes 2023-11-08T05:09:11.408 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5000 IOs, 2397665280 bytes 2023-11-08T05:09:11.682 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5100 IOs, 2478169088 bytes 2023-11-08T05:09:11.887 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5200 IOs, 2499320832 bytes 2023-11-08T05:09:12.042 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5300 IOs, 2527548416 bytes 2023-11-08T05:09:12.235 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5400 IOs, 2569814016 bytes 2023-11-08T05:09:12.380 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5500 IOs, 2607126528 bytes 2023-11-08T05:09:12.577 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5600 IOs, 2643640320 bytes 2023-11-08T05:09:12.714 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5700 IOs, 2670831616 bytes 2023-11-08T05:09:12.851 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5800 IOs, 2685546496 bytes 2023-11-08T05:09:13.067 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5900 IOs, 2741293056 bytes 2023-11-08T05:09:13.231 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:6000 IOs, 2773808128 bytes 2023-11-08T05:09:13.413 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:6100 IOs, 2807597056 bytes 2023-11-08T05:09:13.689 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:6200 IOs, 2897034240 bytes 2023-11-08T05:09:13.854 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:6300 IOs, 2927594496 bytes 2023-11-08T05:09:14.085 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:6400 IOs, 2992590848 bytes 2023-11-08T05:09:14.360 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:6500 IOs, 3074802688 bytes 2023-11-08T05:09:14.527 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:6600 IOs, 3108989952 bytes 2023-11-08T05:09:14.751 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:6700 IOs, 3173566464 bytes 2023-11-08T05:09:14.955 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:6800 IOs, 3221480448 bytes 2023-11-08T05:09:15.080 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:6900 IOs, 3254962176 bytes 2023-11-08T05:09:15.332 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:7000 IOs, 3304741888 bytes 2023-11-08T05:09:15.464 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:7100 IOs, 3321384960 bytes 2023-11-08T05:09:15.562 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ error_code=124 2023-11-08T05:09:15.562 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ set -e 2023-11-08T05:09:15.563 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' 124 -eq 124 ']' 2023-11-08T05:09:15.563 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:09:15.563 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-11-08T05:09:15.563 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image=image_28 2023-11-08T05:09:15.563 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ create_image_and_enable_mirror cluster2 mirror image_28 snapshot 128M 2023-11-08T05:09:15.564 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:09:15.564 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:09:15.564 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:09:15.564 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:09:15.564 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_28 2023-11-08T05:09:15.565 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:09:15.565 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local mode=snapshot 2023-11-08T05:09:15.565 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' -n snapshot ']' 2023-11-08T05:09:15.565 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:09:15.565 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ create_image cluster2 mirror image_28 128M 2023-11-08T05:09:15.566 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:09:15.566 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:09:15.566 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:09:15.566 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:09:15.566 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_28 2023-11-08T05:09:15.567 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:09:15.567 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local size=128 2023-11-08T05:09:15.567 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' -n 128M ']' 2023-11-08T05:09:15.567 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ size=128M 2023-11-08T05:09:15.567 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:09:15.567 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_28 2023-11-08T05:09:15.603 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' image = image ']' 2023-11-08T05:09:15.603 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ enable_mirror cluster2 mirror image_28 snapshot 2023-11-08T05:09:15.603 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:09:15.604 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:09:15.604 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_28 2023-11-08T05:09:15.604 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local mode=snapshot 2023-11-08T05:09:15.604 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_28 snapshot 2023-11-08T05:09:15.996 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:Mirroring enabled 2023-11-08T05:09:16.003 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster=cluster2 info mirror/image_28 2023-11-08T05:09:16.034 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:rbd image 'image_28': 2023-11-08T05:09:16.034 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: size 128 MiB in 32 objects 2023-11-08T05:09:16.034 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: order 22 (4 MiB objects) 2023-11-08T05:09:16.034 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: snapshot_count: 1 2023-11-08T05:09:16.035 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: id: 13a891f0d031 2023-11-08T05:09:16.035 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: block_name_prefix: rbd_data.13a891f0d031 2023-11-08T05:09:16.035 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: format: 2 2023-11-08T05:09:16.035 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: features: layering, exclusive-lock 2023-11-08T05:09:16.035 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: op_features: 2023-11-08T05:09:16.036 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: flags: 2023-11-08T05:09:16.036 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: create_timestamp: Wed Nov 8 05:09:15 2023 2023-11-08T05:09:16.036 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: access_timestamp: Wed Nov 8 05:09:15 2023 2023-11-08T05:09:16.036 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: modify_timestamp: Wed Nov 8 05:09:15 2023 2023-11-08T05:09:16.036 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring state: enabled 2023-11-08T05:09:16.037 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring mode: snapshot 2023-11-08T05:09:16.037 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring global id: 06978a7d-19c8-42c0-8df8-1e3a18982290 2023-11-08T05:09:16.037 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring primary: true 2023-11-08T05:09:16.038 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' -n '' ']' 2023-11-08T05:09:16.038 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ stress_write_image cluster2 mirror image_28 2023-11-08T05:09:16.038 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:09:16.039 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:09:16.039 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_28 2023-11-08T05:09:16.039 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-11-08T05:09:16.039 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local duration=12 2023-11-08T05:09:16.039 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ set +e 2023-11-08T05:09:16.039 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_28 --debug-rbd=20 --debug-journaler=20 2023-11-08T05:09:16.238 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:100 IOs, 46439424 bytes 2023-11-08T05:09:16.460 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:200 IOs, 98002944 bytes 2023-11-08T05:09:16.656 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:300 IOs, 145548288 bytes 2023-11-08T05:09:16.855 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:400 IOs, 193442816 bytes 2023-11-08T05:09:17.053 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:500 IOs, 245280768 bytes 2023-11-08T05:09:17.285 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:600 IOs, 306561024 bytes 2023-11-08T05:09:17.442 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:700 IOs, 347485184 bytes 2023-11-08T05:09:17.640 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:800 IOs, 380399616 bytes 2023-11-08T05:09:17.796 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:900 IOs, 413195264 bytes 2023-11-08T05:09:18.002 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1000 IOs, 466726912 bytes 2023-11-08T05:09:18.152 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1100 IOs, 503476224 bytes 2023-11-08T05:09:18.358 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1200 IOs, 539310080 bytes 2023-11-08T05:09:18.484 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1300 IOs, 549325824 bytes 2023-11-08T05:09:18.632 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1400 IOs, 574358528 bytes 2023-11-08T05:09:18.744 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1500 IOs, 574996480 bytes 2023-11-08T05:09:18.929 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1600 IOs, 620079104 bytes 2023-11-08T05:09:19.164 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1700 IOs, 682464256 bytes 2023-11-08T05:09:19.280 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1800 IOs, 687780864 bytes 2023-11-08T05:09:19.470 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1900 IOs, 733652992 bytes 2023-11-08T05:09:19.625 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2000 IOs, 762901504 bytes 2023-11-08T05:09:19.748 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2100 IOs, 763513856 bytes 2023-11-08T05:09:19.946 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2200 IOs, 812701696 bytes 2023-11-08T05:09:20.095 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2300 IOs, 832116736 bytes 2023-11-08T05:09:20.242 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2400 IOs, 848476160 bytes 2023-11-08T05:09:20.428 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2500 IOs, 888324096 bytes 2023-11-08T05:09:20.656 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2600 IOs, 946518016 bytes 2023-11-08T05:09:20.955 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2700 IOs, 1022226432 bytes 2023-11-08T05:09:21.173 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2800 IOs, 1079258112 bytes 2023-11-08T05:09:21.364 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2900 IOs, 1126232064 bytes 2023-11-08T05:09:21.581 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3000 IOs, 1185464320 bytes 2023-11-08T05:09:21.780 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3100 IOs, 1232783360 bytes 2023-11-08T05:09:22.075 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3200 IOs, 1323152384 bytes 2023-11-08T05:09:22.315 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3300 IOs, 1386719232 bytes 2023-11-08T05:09:22.555 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3400 IOs, 1456988160 bytes 2023-11-08T05:09:22.740 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3500 IOs, 1497704448 bytes 2023-11-08T05:09:22.907 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3600 IOs, 1532919808 bytes 2023-11-08T05:09:23.047 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3700 IOs, 1553851392 bytes 2023-11-08T05:09:23.223 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3800 IOs, 1589460992 bytes 2023-11-08T05:09:23.470 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3900 IOs, 1659703296 bytes 2023-11-08T05:09:23.619 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4000 IOs, 1679364096 bytes 2023-11-08T05:09:23.819 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4100 IOs, 1728122880 bytes 2023-11-08T05:09:24.118 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4200 IOs, 1819292672 bytes 2023-11-08T05:09:24.270 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4300 IOs, 1845198848 bytes 2023-11-08T05:09:24.458 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4400 IOs, 1885301760 bytes 2023-11-08T05:09:24.721 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4500 IOs, 1986170880 bytes 2023-11-08T05:09:24.944 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4600 IOs, 2026142720 bytes 2023-11-08T05:09:25.205 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4700 IOs, 2103745536 bytes 2023-11-08T05:09:25.425 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4800 IOs, 2156656640 bytes 2023-11-08T05:09:25.671 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4900 IOs, 2231100416 bytes 2023-11-08T05:09:25.871 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5000 IOs, 2264190976 bytes 2023-11-08T05:09:25.987 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5100 IOs, 2277657600 bytes 2023-11-08T05:09:26.144 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5200 IOs, 2301582336 bytes 2023-11-08T05:09:26.312 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5300 IOs, 2334592000 bytes 2023-11-08T05:09:26.434 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5400 IOs, 2335926272 bytes 2023-11-08T05:09:26.600 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5500 IOs, 2366002176 bytes 2023-11-08T05:09:26.755 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5600 IOs, 2386823168 bytes 2023-11-08T05:09:27.005 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5700 IOs, 2459290624 bytes 2023-11-08T05:09:27.185 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5800 IOs, 2492418048 bytes 2023-11-08T05:09:27.415 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5900 IOs, 2552973312 bytes 2023-11-08T05:09:27.638 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:6000 IOs, 2610656256 bytes 2023-11-08T05:09:27.793 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:6100 IOs, 2635682816 bytes 2023-11-08T05:09:27.948 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:6200 IOs, 2659518464 bytes 2023-11-08T05:09:28.048 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ error_code=124 2023-11-08T05:09:28.048 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ set -e 2023-11-08T05:09:28.048 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' 124 -eq 124 ']' 2023-11-08T05:09:28.048 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:09:28.049 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-11-08T05:09:28.049 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image=image_29 2023-11-08T05:09:28.049 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ create_image_and_enable_mirror cluster2 mirror image_29 snapshot 128M 2023-11-08T05:09:28.049 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:09:28.049 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:09:28.050 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:09:28.050 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:09:28.050 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_29 2023-11-08T05:09:28.050 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:09:28.050 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local mode=snapshot 2023-11-08T05:09:28.051 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' -n snapshot ']' 2023-11-08T05:09:28.051 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:09:28.051 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ create_image cluster2 mirror image_29 128M 2023-11-08T05:09:28.051 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:09:28.051 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:09:28.051 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:09:28.052 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:09:28.052 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_29 2023-11-08T05:09:28.052 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:09:28.052 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local size=128 2023-11-08T05:09:28.052 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' -n 128M ']' 2023-11-08T05:09:28.053 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ size=128M 2023-11-08T05:09:28.053 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:09:28.053 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_29 2023-11-08T05:09:28.088 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' image = image ']' 2023-11-08T05:09:28.088 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ enable_mirror cluster2 mirror image_29 snapshot 2023-11-08T05:09:28.088 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:09:28.088 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:09:28.089 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_29 2023-11-08T05:09:28.089 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local mode=snapshot 2023-11-08T05:09:28.089 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_29 snapshot 2023-11-08T05:09:29.096 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:Mirroring enabled 2023-11-08T05:09:29.103 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster=cluster2 info mirror/image_29 2023-11-08T05:09:29.136 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:rbd image 'image_29': 2023-11-08T05:09:29.137 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: size 128 MiB in 32 objects 2023-11-08T05:09:29.137 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: order 22 (4 MiB objects) 2023-11-08T05:09:29.137 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: snapshot_count: 1 2023-11-08T05:09:29.137 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: id: 13b1c1e10f67 2023-11-08T05:09:29.137 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: block_name_prefix: rbd_data.13b1c1e10f67 2023-11-08T05:09:29.138 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: format: 2 2023-11-08T05:09:29.138 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: features: layering, exclusive-lock 2023-11-08T05:09:29.138 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: op_features: 2023-11-08T05:09:29.138 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: flags: 2023-11-08T05:09:29.139 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: create_timestamp: Wed Nov 8 05:09:28 2023 2023-11-08T05:09:29.139 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: access_timestamp: Wed Nov 8 05:09:28 2023 2023-11-08T05:09:29.139 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: modify_timestamp: Wed Nov 8 05:09:28 2023 2023-11-08T05:09:29.139 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring state: enabled 2023-11-08T05:09:29.139 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring mode: snapshot 2023-11-08T05:09:29.140 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring global id: 224b796b-9a1e-47f0-a771-84babb70a923 2023-11-08T05:09:29.140 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring primary: true 2023-11-08T05:09:29.141 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' -n '' ']' 2023-11-08T05:09:29.141 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ stress_write_image cluster2 mirror image_29 2023-11-08T05:09:29.141 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:09:29.141 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:09:29.142 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_29 2023-11-08T05:09:29.142 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-11-08T05:09:29.142 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local duration=14 2023-11-08T05:09:29.142 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ set +e 2023-11-08T05:09:29.143 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_29 --debug-rbd=20 --debug-journaler=20 2023-11-08T05:09:29.352 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:100 IOs, 29351936 bytes 2023-11-08T05:09:29.571 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:200 IOs, 87645184 bytes 2023-11-08T05:09:29.757 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:300 IOs, 132969472 bytes 2023-11-08T05:09:30.122 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:400 IOs, 260265984 bytes 2023-11-08T05:09:30.335 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:500 IOs, 337640448 bytes 2023-11-08T05:09:30.482 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:600 IOs, 338268160 bytes 2023-11-08T05:09:30.706 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:700 IOs, 370527232 bytes 2023-11-08T05:09:30.931 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:800 IOs, 431703040 bytes 2023-11-08T05:09:31.129 INFO:tasks.ceph.cluster2.osd.0.smithi087.stderr:2023-11-08T05:09:31.127+0000 7f1ac5683700 -1 reset not still connected to 0x55f9f36d9790 2023-11-08T05:09:31.266 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:900 IOs, 542520320 bytes 2023-11-08T05:09:31.530 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1000 IOs, 618689536 bytes 2023-11-08T05:09:31.539 INFO:tasks.ceph.cluster1.osd.2.smithi077.stderr:2023-11-08T05:09:31.536+0000 7f9900abf700 -1 reset not still connected to 0x555e97cd8680 2023-11-08T05:09:31.771 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1100 IOs, 690980864 bytes 2023-11-08T05:09:31.989 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1200 IOs, 736643072 bytes 2023-11-08T05:09:32.184 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1300 IOs, 773581824 bytes 2023-11-08T05:09:32.348 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1400 IOs, 805213184 bytes 2023-11-08T05:09:32.581 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1500 IOs, 869092352 bytes 2023-11-08T05:09:32.746 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1600 IOs, 901468160 bytes 2023-11-08T05:09:32.870 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1700 IOs, 907708416 bytes 2023-11-08T05:09:33.050 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1800 IOs, 944973824 bytes 2023-11-08T05:09:33.218 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1900 IOs, 994321408 bytes 2023-11-08T05:09:33.454 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2000 IOs, 1045064704 bytes 2023-11-08T05:09:33.581 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2100 IOs, 1053042688 bytes 2023-11-08T05:09:33.741 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2200 IOs, 1080276992 bytes 2023-11-08T05:09:33.862 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2300 IOs, 1090990080 bytes 2023-11-08T05:09:34.118 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2400 IOs, 1167734784 bytes 2023-11-08T05:09:34.263 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2500 IOs, 1186520064 bytes 2023-11-08T05:09:34.453 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2600 IOs, 1233152000 bytes 2023-11-08T05:09:34.628 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2700 IOs, 1265741824 bytes 2023-11-08T05:09:34.780 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2800 IOs, 1289070592 bytes 2023-11-08T05:09:34.935 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2900 IOs, 1313012736 bytes 2023-11-08T05:09:35.285 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3000 IOs, 1447192576 bytes 2023-11-08T05:09:35.504 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3100 IOs, 1491149824 bytes 2023-11-08T05:09:35.741 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3200 IOs, 1574746112 bytes 2023-11-08T05:09:35.873 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3300 IOs, 1575353344 bytes 2023-11-08T05:09:36.017 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3400 IOs, 1589793792 bytes 2023-11-08T05:09:36.168 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3500 IOs, 1607300096 bytes 2023-11-08T05:09:36.467 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3600 IOs, 1709090816 bytes 2023-11-08T05:09:36.589 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3700 IOs, 1709705216 bytes 2023-11-08T05:09:36.783 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3800 IOs, 1753519104 bytes 2023-11-08T05:09:36.896 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3900 IOs, 1754147840 bytes 2023-11-08T05:09:37.010 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4000 IOs, 1754788864 bytes 2023-11-08T05:09:37.211 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4100 IOs, 1806394368 bytes 2023-11-08T05:09:37.338 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4200 IOs, 1811274752 bytes 2023-11-08T05:09:37.550 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4300 IOs, 1867576320 bytes 2023-11-08T05:09:37.854 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4400 IOs, 1962068992 bytes 2023-11-08T05:09:38.193 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4500 IOs, 2055506944 bytes 2023-11-08T05:09:38.328 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4600 IOs, 2072407040 bytes 2023-11-08T05:09:38.541 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4700 IOs, 2119481344 bytes 2023-11-08T05:09:38.723 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4800 IOs, 2160262144 bytes 2023-11-08T05:09:38.913 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4900 IOs, 2210220032 bytes 2023-11-08T05:09:39.193 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5000 IOs, 2277304320 bytes 2023-11-08T05:09:39.416 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5100 IOs, 2345502720 bytes 2023-11-08T05:09:39.640 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5200 IOs, 2409714688 bytes 2023-11-08T05:09:39.954 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5300 IOs, 2523835392 bytes 2023-11-08T05:09:40.130 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5400 IOs, 2559250432 bytes 2023-11-08T05:09:40.375 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5500 IOs, 2626232320 bytes 2023-11-08T05:09:40.675 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5600 IOs, 2727654400 bytes 2023-11-08T05:09:40.833 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5700 IOs, 2750653440 bytes 2023-11-08T05:09:41.069 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5800 IOs, 2810571776 bytes 2023-11-08T05:09:41.243 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5900 IOs, 2846063616 bytes 2023-11-08T05:09:41.329 INFO:tasks.ceph.cluster2.osd.0.smithi087.stderr:2023-11-08T05:09:41.327+0000 7f1ac5683700 -1 reset not still connected to 0x55f9f36d9790 2023-11-08T05:09:41.374 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:6000 IOs, 2862430208 bytes 2023-11-08T05:09:41.616 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:6100 IOs, 2933153792 bytes 2023-11-08T05:09:41.780 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:6200 IOs, 2985989120 bytes 2023-11-08T05:09:41.931 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:6300 IOs, 2993642496 bytes 2023-11-08T05:09:42.102 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:6400 IOs, 3023705088 bytes 2023-11-08T05:09:42.221 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:6500 IOs, 3028764672 bytes 2023-11-08T05:09:42.406 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:6600 IOs, 3069815808 bytes 2023-11-08T05:09:42.549 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:6700 IOs, 3090471936 bytes 2023-11-08T05:09:42.670 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:6800 IOs, 3119211520 bytes 2023-11-08T05:09:42.880 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:6900 IOs, 3148067840 bytes 2023-11-08T05:09:43.107 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:7000 IOs, 3209044992 bytes 2023-11-08T05:09:43.147 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ error_code=124 2023-11-08T05:09:43.147 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ set -e 2023-11-08T05:09:43.147 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' 124 -eq 124 ']' 2023-11-08T05:09:43.148 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:09:43.148 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-11-08T05:09:43.148 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image=image_30 2023-11-08T05:09:43.148 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ create_image_and_enable_mirror cluster2 mirror image_30 snapshot 128M 2023-11-08T05:09:43.148 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:09:43.149 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:09:43.149 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:09:43.149 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:09:43.149 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_30 2023-11-08T05:09:43.149 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:09:43.150 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local mode=snapshot 2023-11-08T05:09:43.150 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' -n snapshot ']' 2023-11-08T05:09:43.150 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:09:43.150 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ create_image cluster2 mirror image_30 128M 2023-11-08T05:09:43.150 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:09:43.151 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:09:43.151 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:09:43.151 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:09:43.151 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_30 2023-11-08T05:09:43.151 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:09:43.152 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local size=128 2023-11-08T05:09:43.152 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' -n 128M ']' 2023-11-08T05:09:43.152 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ size=128M 2023-11-08T05:09:43.152 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:09:43.152 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_30 2023-11-08T05:09:43.315 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' image = image ']' 2023-11-08T05:09:43.316 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ enable_mirror cluster2 mirror image_30 snapshot 2023-11-08T05:09:43.316 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:09:43.316 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:09:43.317 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_30 2023-11-08T05:09:43.317 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local mode=snapshot 2023-11-08T05:09:43.317 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_30 snapshot 2023-11-08T05:09:44.279 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:Mirroring enabled 2023-11-08T05:09:44.288 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster=cluster2 info mirror/image_30 2023-11-08T05:09:44.321 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:rbd image 'image_30': 2023-11-08T05:09:44.321 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: size 128 MiB in 32 objects 2023-11-08T05:09:44.321 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: order 22 (4 MiB objects) 2023-11-08T05:09:44.321 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: snapshot_count: 1 2023-11-08T05:09:44.322 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: id: 13bab3f1f151 2023-11-08T05:09:44.322 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: block_name_prefix: rbd_data.13bab3f1f151 2023-11-08T05:09:44.322 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: format: 2 2023-11-08T05:09:44.322 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: features: layering, exclusive-lock 2023-11-08T05:09:44.322 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: op_features: 2023-11-08T05:09:44.323 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: flags: 2023-11-08T05:09:44.323 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: create_timestamp: Wed Nov 8 05:09:43 2023 2023-11-08T05:09:44.323 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: access_timestamp: Wed Nov 8 05:09:43 2023 2023-11-08T05:09:44.323 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: modify_timestamp: Wed Nov 8 05:09:43 2023 2023-11-08T05:09:44.323 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring state: enabled 2023-11-08T05:09:44.324 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring mode: snapshot 2023-11-08T05:09:44.324 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring global id: c78f7ab4-c752-49b4-8cd9-bc194f1db211 2023-11-08T05:09:44.324 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring primary: true 2023-11-08T05:09:44.325 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' -n '' ']' 2023-11-08T05:09:44.325 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ stress_write_image cluster2 mirror image_30 2023-11-08T05:09:44.325 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:09:44.326 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:09:44.326 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_30 2023-11-08T05:09:44.326 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-11-08T05:09:44.327 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local duration=10 2023-11-08T05:09:44.327 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ set +e 2023-11-08T05:09:44.327 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_30 --debug-rbd=20 --debug-journaler=20 2023-11-08T05:09:44.550 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:100 IOs, 41185280 bytes 2023-11-08T05:09:44.729 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:200 IOs, 77733888 bytes 2023-11-08T05:09:44.914 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:300 IOs, 106819584 bytes 2023-11-08T05:09:45.121 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:400 IOs, 163975168 bytes 2023-11-08T05:09:45.272 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:500 IOs, 195379200 bytes 2023-11-08T05:09:45.457 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:600 IOs, 242765824 bytes 2023-11-08T05:09:45.582 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:700 IOs, 251462656 bytes 2023-11-08T05:09:45.757 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:800 IOs, 289741824 bytes 2023-11-08T05:09:45.941 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:900 IOs, 329159680 bytes 2023-11-08T05:09:46.068 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1000 IOs, 341022720 bytes 2023-11-08T05:09:46.202 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1100 IOs, 356597760 bytes 2023-11-08T05:09:46.458 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1200 IOs, 428196864 bytes 2023-11-08T05:09:46.564 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1300 IOs, 428816384 bytes 2023-11-08T05:09:46.805 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1400 IOs, 494256128 bytes 2023-11-08T05:09:46.982 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1500 IOs, 523873280 bytes 2023-11-08T05:09:47.134 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1600 IOs, 549954560 bytes 2023-11-08T05:09:47.296 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1700 IOs, 592031744 bytes 2023-11-08T05:09:47.597 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1800 IOs, 676831232 bytes 2023-11-08T05:09:47.793 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1900 IOs, 746273792 bytes 2023-11-08T05:09:48.021 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2000 IOs, 787681280 bytes 2023-11-08T05:09:48.207 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2100 IOs, 832596992 bytes 2023-11-08T05:09:48.410 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2200 IOs, 879266816 bytes 2023-11-08T05:09:48.575 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2300 IOs, 910510080 bytes 2023-11-08T05:09:48.715 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2400 IOs, 927717376 bytes 2023-11-08T05:09:48.942 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2500 IOs, 984845312 bytes 2023-11-08T05:09:49.196 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2600 IOs, 1088850944 bytes 2023-11-08T05:09:49.275 INFO:tasks.ceph.cluster2.osd.2.smithi087.stderr:2023-11-08T05:09:49.273+0000 7fa243317700 -1 reset not still connected to 0x5613a5dfc000 2023-11-08T05:09:49.413 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2700 IOs, 1111056384 bytes 2023-11-08T05:09:49.637 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2800 IOs, 1164342272 bytes 2023-11-08T05:09:49.851 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2900 IOs, 1216478208 bytes 2023-11-08T05:09:50.018 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3000 IOs, 1250896896 bytes 2023-11-08T05:09:50.178 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3100 IOs, 1277004800 bytes 2023-11-08T05:09:50.402 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3200 IOs, 1330824192 bytes 2023-11-08T05:09:50.563 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3300 IOs, 1404601344 bytes 2023-11-08T05:09:50.908 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3400 IOs, 1489026048 bytes 2023-11-08T05:09:51.048 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3500 IOs, 1524235264 bytes 2023-11-08T05:09:51.215 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3600 IOs, 1558084608 bytes 2023-11-08T05:09:51.368 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3700 IOs, 1565037568 bytes 2023-11-08T05:09:51.614 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3800 IOs, 1643564032 bytes 2023-11-08T05:09:51.798 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3900 IOs, 1664518144 bytes 2023-11-08T05:09:51.909 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4000 IOs, 1666211840 bytes 2023-11-08T05:09:52.098 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4100 IOs, 1714190336 bytes 2023-11-08T05:09:52.247 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4200 IOs, 1740797952 bytes 2023-11-08T05:09:52.415 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4300 IOs, 1758439424 bytes 2023-11-08T05:09:52.560 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4400 IOs, 1778338816 bytes 2023-11-08T05:09:52.694 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4500 IOs, 1793064960 bytes 2023-11-08T05:09:52.854 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4600 IOs, 1821924352 bytes 2023-11-08T05:09:53.005 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4700 IOs, 1843867648 bytes 2023-11-08T05:09:53.234 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4800 IOs, 1911603200 bytes 2023-11-08T05:09:53.439 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4900 IOs, 1965481984 bytes 2023-11-08T05:09:53.625 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5000 IOs, 2017625088 bytes 2023-11-08T05:09:53.791 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5100 IOs, 2054504448 bytes 2023-11-08T05:09:54.000 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5200 IOs, 2103409664 bytes 2023-11-08T05:09:54.333 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ error_code=124 2023-11-08T05:09:54.334 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ set -e 2023-11-08T05:09:54.334 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' 124 -eq 124 ']' 2023-11-08T05:09:54.334 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:09:54.334 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-11-08T05:09:54.335 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image=image_31 2023-11-08T05:09:54.335 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ create_image_and_enable_mirror cluster2 mirror image_31 snapshot 128M 2023-11-08T05:09:54.335 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:09:54.335 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:09:54.335 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:09:54.335 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:09:54.336 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_31 2023-11-08T05:09:54.336 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:09:54.336 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local mode=snapshot 2023-11-08T05:09:54.336 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' -n snapshot ']' 2023-11-08T05:09:54.336 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:09:54.337 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ create_image cluster2 mirror image_31 128M 2023-11-08T05:09:54.337 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:09:54.337 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:09:54.337 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:09:54.338 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:09:54.338 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_31 2023-11-08T05:09:54.338 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:09:54.338 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local size=128 2023-11-08T05:09:54.339 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' -n 128M ']' 2023-11-08T05:09:54.339 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ size=128M 2023-11-08T05:09:54.339 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:09:54.339 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_31 2023-11-08T05:09:54.374 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' image = image ']' 2023-11-08T05:09:54.374 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ enable_mirror cluster2 mirror image_31 snapshot 2023-11-08T05:09:54.375 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:09:54.375 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:09:54.375 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_31 2023-11-08T05:09:54.375 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local mode=snapshot 2023-11-08T05:09:54.376 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_31 snapshot 2023-11-08T05:09:55.328 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:Mirroring enabled 2023-11-08T05:09:55.335 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster=cluster2 info mirror/image_31 2023-11-08T05:09:55.366 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:rbd image 'image_31': 2023-11-08T05:09:55.367 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: size 128 MiB in 32 objects 2023-11-08T05:09:55.367 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: order 22 (4 MiB objects) 2023-11-08T05:09:55.367 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: snapshot_count: 1 2023-11-08T05:09:55.367 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: id: 13c38aaa06ed 2023-11-08T05:09:55.368 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: block_name_prefix: rbd_data.13c38aaa06ed 2023-11-08T05:09:55.368 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: format: 2 2023-11-08T05:09:55.368 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: features: layering, exclusive-lock 2023-11-08T05:09:55.368 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: op_features: 2023-11-08T05:09:55.368 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: flags: 2023-11-08T05:09:55.369 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: create_timestamp: Wed Nov 8 05:09:54 2023 2023-11-08T05:09:55.369 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: access_timestamp: Wed Nov 8 05:09:54 2023 2023-11-08T05:09:55.369 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: modify_timestamp: Wed Nov 8 05:09:54 2023 2023-11-08T05:09:55.369 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring state: enabled 2023-11-08T05:09:55.369 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring mode: snapshot 2023-11-08T05:09:55.370 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring global id: 50f6deef-3c6f-490a-bab8-f906a3136c2a 2023-11-08T05:09:55.370 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring primary: true 2023-11-08T05:09:55.371 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' -n '' ']' 2023-11-08T05:09:55.371 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ stress_write_image cluster2 mirror image_31 2023-11-08T05:09:55.371 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:09:55.372 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:09:55.372 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_31 2023-11-08T05:09:55.372 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-11-08T05:09:55.372 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local duration=5 2023-11-08T05:09:55.372 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ set +e 2023-11-08T05:09:55.373 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_31 --debug-rbd=20 --debug-journaler=20 2023-11-08T05:09:55.597 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:100 IOs, 45312000 bytes 2023-11-08T05:09:55.763 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:200 IOs, 79107072 bytes 2023-11-08T05:09:55.902 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:300 IOs, 93363200 bytes 2023-11-08T05:09:56.116 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:400 IOs, 152079360 bytes 2023-11-08T05:09:56.326 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:500 IOs, 209502208 bytes 2023-11-08T05:09:56.483 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:600 IOs, 232958976 bytes 2023-11-08T05:09:56.752 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:700 IOs, 309926912 bytes 2023-11-08T05:09:56.907 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:800 IOs, 329393152 bytes 2023-11-08T05:09:57.133 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:900 IOs, 390748160 bytes 2023-11-08T05:09:57.274 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1000 IOs, 426964992 bytes 2023-11-08T05:09:57.537 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1100 IOs, 480535552 bytes 2023-11-08T05:09:57.689 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1200 IOs, 505312256 bytes 2023-11-08T05:09:57.822 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1300 IOs, 551235584 bytes 2023-11-08T05:09:58.070 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1400 IOs, 582686720 bytes 2023-11-08T05:09:58.204 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1500 IOs, 599144448 bytes 2023-11-08T05:09:58.483 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1600 IOs, 701212672 bytes 2023-11-08T05:09:58.670 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1700 IOs, 730768384 bytes 2023-11-08T05:09:58.821 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1800 IOs, 754408448 bytes 2023-11-08T05:09:58.956 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1900 IOs, 773566464 bytes 2023-11-08T05:09:59.217 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2000 IOs, 846681088 bytes 2023-11-08T05:09:59.385 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2100 IOs, 879553536 bytes 2023-11-08T05:09:59.561 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2200 IOs, 918873088 bytes 2023-11-08T05:09:59.753 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2300 IOs, 958276608 bytes 2023-11-08T05:09:59.865 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2400 IOs, 958888960 bytes 2023-11-08T05:10:00.077 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2500 IOs, 1015117824 bytes 2023-11-08T05:10:00.277 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2600 IOs, 1078776832 bytes 2023-11-08T05:10:00.380 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ error_code=124 2023-11-08T05:10:00.381 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ set -e 2023-11-08T05:10:00.381 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' 124 -eq 124 ']' 2023-11-08T05:10:00.381 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:10:00.381 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-11-08T05:10:00.382 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image=image_32 2023-11-08T05:10:00.382 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ create_image_and_enable_mirror cluster2 mirror image_32 snapshot 128M 2023-11-08T05:10:00.382 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:10:00.382 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:10:00.382 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:10:00.383 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:10:00.383 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_32 2023-11-08T05:10:00.383 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:10:00.383 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local mode=snapshot 2023-11-08T05:10:00.384 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' -n snapshot ']' 2023-11-08T05:10:00.384 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:10:00.384 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ create_image cluster2 mirror image_32 128M 2023-11-08T05:10:00.384 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:10:00.384 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:10:00.385 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:10:00.385 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:10:00.385 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_32 2023-11-08T05:10:00.385 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:10:00.385 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local size=128 2023-11-08T05:10:00.386 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' -n 128M ']' 2023-11-08T05:10:00.386 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ size=128M 2023-11-08T05:10:00.386 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:10:00.386 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_32 2023-11-08T05:10:00.418 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' image = image ']' 2023-11-08T05:10:00.419 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ enable_mirror cluster2 mirror image_32 snapshot 2023-11-08T05:10:00.419 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:10:00.419 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:10:00.419 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_32 2023-11-08T05:10:00.420 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local mode=snapshot 2023-11-08T05:10:00.420 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_32 snapshot 2023-11-08T05:10:01.383 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:Mirroring enabled 2023-11-08T05:10:01.391 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster=cluster2 info mirror/image_32 2023-11-08T05:10:01.425 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:rbd image 'image_32': 2023-11-08T05:10:01.425 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: size 128 MiB in 32 objects 2023-11-08T05:10:01.425 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: order 22 (4 MiB objects) 2023-11-08T05:10:01.426 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: snapshot_count: 1 2023-11-08T05:10:01.426 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: id: 13cba19aba9c 2023-11-08T05:10:01.426 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: block_name_prefix: rbd_data.13cba19aba9c 2023-11-08T05:10:01.426 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: format: 2 2023-11-08T05:10:01.426 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: features: layering, exclusive-lock 2023-11-08T05:10:01.427 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: op_features: 2023-11-08T05:10:01.427 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: flags: 2023-11-08T05:10:01.427 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: create_timestamp: Wed Nov 8 05:10:00 2023 2023-11-08T05:10:01.427 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: access_timestamp: Wed Nov 8 05:10:00 2023 2023-11-08T05:10:01.427 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: modify_timestamp: Wed Nov 8 05:10:00 2023 2023-11-08T05:10:01.428 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring state: enabled 2023-11-08T05:10:01.428 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring mode: snapshot 2023-11-08T05:10:01.428 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring global id: e6ec2241-9334-4ae3-ae69-78e6eb1a509b 2023-11-08T05:10:01.428 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring primary: true 2023-11-08T05:10:01.429 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' -n '' ']' 2023-11-08T05:10:01.429 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ stress_write_image cluster2 mirror image_32 2023-11-08T05:10:01.429 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:10:01.430 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:10:01.430 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_32 2023-11-08T05:10:01.430 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-11-08T05:10:01.431 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local duration=7 2023-11-08T05:10:01.431 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ set +e 2023-11-08T05:10:01.431 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_32 --debug-rbd=20 --debug-journaler=20 2023-11-08T05:10:01.642 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:100 IOs, 35346432 bytes 2023-11-08T05:10:01.868 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:200 IOs, 99387392 bytes 2023-11-08T05:10:02.039 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:300 IOs, 123151360 bytes 2023-11-08T05:10:02.211 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:400 IOs, 161489920 bytes 2023-11-08T05:10:02.381 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:500 IOs, 191936512 bytes 2023-11-08T05:10:02.498 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:600 IOs, 192558080 bytes 2023-11-08T05:10:02.676 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:700 IOs, 228238336 bytes 2023-11-08T05:10:02.866 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:800 IOs, 270690304 bytes 2023-11-08T05:10:03.013 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:900 IOs, 304640000 bytes 2023-11-08T05:10:03.274 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1000 IOs, 372642816 bytes 2023-11-08T05:10:03.482 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1100 IOs, 415958016 bytes 2023-11-08T05:10:03.651 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1200 IOs, 448102400 bytes 2023-11-08T05:10:03.815 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1300 IOs, 475932672 bytes 2023-11-08T05:10:03.979 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1400 IOs, 507319296 bytes 2023-11-08T05:10:04.146 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1500 IOs, 538261504 bytes 2023-11-08T05:10:04.269 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1600 IOs, 549023744 bytes 2023-11-08T05:10:04.441 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1700 IOs, 581283840 bytes 2023-11-08T05:10:04.581 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1800 IOs, 601061376 bytes 2023-11-08T05:10:04.847 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1900 IOs, 682123264 bytes 2023-11-08T05:10:05.163 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2000 IOs, 809054208 bytes 2023-11-08T05:10:05.452 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2100 IOs, 878575616 bytes 2023-11-08T05:10:05.648 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2200 IOs, 913052672 bytes 2023-11-08T05:10:05.918 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2300 IOs, 996654080 bytes 2023-11-08T05:10:06.110 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2400 IOs, 1036125184 bytes 2023-11-08T05:10:06.323 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2500 IOs, 1089627136 bytes 2023-11-08T05:10:06.530 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2600 IOs, 1144445952 bytes 2023-11-08T05:10:06.743 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2700 IOs, 1197753344 bytes 2023-11-08T05:10:06.905 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2800 IOs, 1219814400 bytes 2023-11-08T05:10:07.040 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2900 IOs, 1249886208 bytes 2023-11-08T05:10:07.218 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3000 IOs, 1288250368 bytes 2023-11-08T05:10:07.377 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3100 IOs, 1301080064 bytes 2023-11-08T05:10:07.562 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3200 IOs, 1338196992 bytes 2023-11-08T05:10:07.729 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3300 IOs, 1366874112 bytes 2023-11-08T05:10:07.938 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3400 IOs, 1433953280 bytes 2023-11-08T05:10:08.162 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3500 IOs, 1475588096 bytes 2023-11-08T05:10:08.345 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3600 IOs, 1517081600 bytes 2023-11-08T05:10:08.438 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ error_code=124 2023-11-08T05:10:08.438 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ set -e 2023-11-08T05:10:08.438 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' 124 -eq 124 ']' 2023-11-08T05:10:08.438 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:10:08.439 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-11-08T05:10:08.439 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image=image_33 2023-11-08T05:10:08.439 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ create_image_and_enable_mirror cluster2 mirror image_33 snapshot 128M 2023-11-08T05:10:08.439 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:10:08.440 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:10:08.440 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:10:08.440 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:10:08.440 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_33 2023-11-08T05:10:08.440 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:10:08.441 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local mode=snapshot 2023-11-08T05:10:08.441 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' -n snapshot ']' 2023-11-08T05:10:08.441 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:10:08.441 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ create_image cluster2 mirror image_33 128M 2023-11-08T05:10:08.441 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:10:08.441 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:10:08.442 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:10:08.442 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:10:08.442 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_33 2023-11-08T05:10:08.442 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:10:08.442 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local size=128 2023-11-08T05:10:08.443 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' -n 128M ']' 2023-11-08T05:10:08.443 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ size=128M 2023-11-08T05:10:08.443 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:10:08.443 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_33 2023-11-08T05:10:08.476 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' image = image ']' 2023-11-08T05:10:08.477 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ enable_mirror cluster2 mirror image_33 snapshot 2023-11-08T05:10:08.477 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:10:08.477 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:10:08.477 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_33 2023-11-08T05:10:08.477 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local mode=snapshot 2023-11-08T05:10:08.478 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_33 snapshot 2023-11-08T05:10:09.451 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:Mirroring enabled 2023-11-08T05:10:09.459 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster=cluster2 info mirror/image_33 2023-11-08T05:10:09.493 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:rbd image 'image_33': 2023-11-08T05:10:09.494 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: size 128 MiB in 32 objects 2023-11-08T05:10:09.494 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: order 22 (4 MiB objects) 2023-11-08T05:10:09.495 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: snapshot_count: 1 2023-11-08T05:10:09.495 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: id: 13d54a22fcdb 2023-11-08T05:10:09.495 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: block_name_prefix: rbd_data.13d54a22fcdb 2023-11-08T05:10:09.496 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: format: 2 2023-11-08T05:10:09.496 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: features: layering, exclusive-lock 2023-11-08T05:10:09.496 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: op_features: 2023-11-08T05:10:09.497 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: flags: 2023-11-08T05:10:09.497 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: create_timestamp: Wed Nov 8 05:10:08 2023 2023-11-08T05:10:09.498 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: access_timestamp: Wed Nov 8 05:10:08 2023 2023-11-08T05:10:09.498 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: modify_timestamp: Wed Nov 8 05:10:08 2023 2023-11-08T05:10:09.498 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring state: enabled 2023-11-08T05:10:09.499 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring mode: snapshot 2023-11-08T05:10:09.499 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring global id: 6f088a8f-d677-4bda-b50c-7fba63dd233e 2023-11-08T05:10:09.500 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring primary: true 2023-11-08T05:10:09.502 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' -n '' ']' 2023-11-08T05:10:09.502 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ stress_write_image cluster2 mirror image_33 2023-11-08T05:10:09.503 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:10:09.503 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:10:09.503 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_33 2023-11-08T05:10:09.503 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-11-08T05:10:09.504 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local duration=13 2023-11-08T05:10:09.504 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ set +e 2023-11-08T05:10:09.504 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_33 --debug-rbd=20 --debug-journaler=20 2023-11-08T05:10:09.675 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:100 IOs, 19776512 bytes 2023-11-08T05:10:09.816 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:200 IOs, 39009280 bytes 2023-11-08T05:10:09.961 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:300 IOs, 65342464 bytes 2023-11-08T05:10:10.075 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:400 IOs, 81866752 bytes 2023-11-08T05:10:10.243 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:500 IOs, 104915968 bytes 2023-11-08T05:10:10.457 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:600 IOs, 160491520 bytes 2023-11-08T05:10:10.613 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:700 IOs, 192004096 bytes 2023-11-08T05:10:10.770 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:800 IOs, 210849792 bytes 2023-11-08T05:10:10.927 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:900 IOs, 240993280 bytes 2023-11-08T05:10:11.088 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1000 IOs, 273204224 bytes 2023-11-08T05:10:11.251 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1100 IOs, 306960384 bytes 2023-11-08T05:10:11.377 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1200 IOs, 321552384 bytes 2023-11-08T05:10:11.494 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1300 IOs, 322164736 bytes 2023-11-08T05:10:11.609 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1400 IOs, 330614784 bytes 2023-11-08T05:10:11.778 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1500 IOs, 368256000 bytes 2023-11-08T05:10:12.022 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1600 IOs, 432308224 bytes 2023-11-08T05:10:12.139 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1700 IOs, 432949248 bytes 2023-11-08T05:10:12.361 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1800 IOs, 494823424 bytes 2023-11-08T05:10:12.498 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1900 IOs, 504991744 bytes 2023-11-08T05:10:12.744 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2000 IOs, 583569408 bytes 2023-11-08T05:10:12.989 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2100 IOs, 638296064 bytes 2023-11-08T05:10:13.198 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2200 IOs, 686234624 bytes 2023-11-08T05:10:13.431 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2300 IOs, 756688896 bytes 2023-11-08T05:10:13.593 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2400 IOs, 777285632 bytes 2023-11-08T05:10:13.735 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2500 IOs, 792922112 bytes 2023-11-08T05:10:13.872 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2600 IOs, 807693312 bytes 2023-11-08T05:10:14.043 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2700 IOs, 842432512 bytes 2023-11-08T05:10:14.245 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2800 IOs, 887258112 bytes 2023-11-08T05:10:14.563 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2900 IOs, 993779712 bytes 2023-11-08T05:10:14.780 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3000 IOs, 1050618880 bytes 2023-11-08T05:10:14.994 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3100 IOs, 1111348224 bytes 2023-11-08T05:10:15.215 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3200 IOs, 1175929856 bytes 2023-11-08T05:10:15.330 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3300 IOs, 1176776704 bytes 2023-11-08T05:10:15.494 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3400 IOs, 1210678272 bytes 2023-11-08T05:10:15.670 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3500 IOs, 1238565888 bytes 2023-11-08T05:10:15.946 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3600 IOs, 1329333248 bytes 2023-11-08T05:10:16.094 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3700 IOs, 1346522112 bytes 2023-11-08T05:10:16.284 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3800 IOs, 1396072448 bytes 2023-11-08T05:10:16.507 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3900 IOs, 1453216768 bytes 2023-11-08T05:10:16.707 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4000 IOs, 1500896256 bytes 2023-11-08T05:10:16.964 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4100 IOs, 1577073664 bytes 2023-11-08T05:10:17.097 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4200 IOs, 1584817152 bytes 2023-11-08T05:10:17.225 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4300 IOs, 1595517952 bytes 2023-11-08T05:10:17.474 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4400 IOs, 1672044544 bytes 2023-11-08T05:10:17.687 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4500 IOs, 1734703104 bytes 2023-11-08T05:10:17.839 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4600 IOs, 1755847680 bytes 2023-11-08T05:10:18.001 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4700 IOs, 1788058624 bytes 2023-11-08T05:10:18.188 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4800 IOs, 1828606976 bytes 2023-11-08T05:10:18.474 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4900 IOs, 1921341440 bytes 2023-11-08T05:10:18.699 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5000 IOs, 1985255424 bytes 2023-11-08T05:10:18.836 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5100 IOs, 1996002304 bytes 2023-11-08T05:10:19.034 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5200 IOs, 2047984640 bytes 2023-11-08T05:10:19.213 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5300 IOs, 2084486144 bytes 2023-11-08T05:10:19.410 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5400 IOs, 2132545536 bytes 2023-11-08T05:10:19.579 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5500 IOs, 2166359040 bytes 2023-11-08T05:10:19.756 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5600 IOs, 2222979072 bytes 2023-11-08T05:10:19.935 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5700 IOs, 2238676992 bytes 2023-11-08T05:10:20.082 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5800 IOs, 2254389248 bytes 2023-11-08T05:10:20.338 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5900 IOs, 2339241984 bytes 2023-11-08T05:10:20.506 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:6000 IOs, 2367694848 bytes 2023-11-08T05:10:20.658 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:6100 IOs, 2390380544 bytes 2023-11-08T05:10:20.798 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:6200 IOs, 2408863744 bytes 2023-11-08T05:10:20.950 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:6300 IOs, 2426851328 bytes 2023-11-08T05:10:21.091 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:6400 IOs, 2445375488 bytes 2023-11-08T05:10:21.242 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:6500 IOs, 2476284928 bytes 2023-11-08T05:10:21.459 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:6600 IOs, 2514889728 bytes 2023-11-08T05:10:21.622 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:6700 IOs, 2548689920 bytes 2023-11-08T05:10:21.823 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:6800 IOs, 2594110464 bytes 2023-11-08T05:10:22.021 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:6900 IOs, 2642885632 bytes 2023-11-08T05:10:22.295 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:7000 IOs, 2703316992 bytes 2023-11-08T05:10:22.504 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ error_code=124 2023-11-08T05:10:22.505 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ set -e 2023-11-08T05:10:22.505 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' 124 -eq 124 ']' 2023-11-08T05:10:22.505 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:10:22.505 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-11-08T05:10:22.505 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image=image_34 2023-11-08T05:10:22.506 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ create_image_and_enable_mirror cluster2 mirror image_34 snapshot 128M 2023-11-08T05:10:22.506 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:10:22.506 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:10:22.506 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:10:22.506 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:10:22.507 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_34 2023-11-08T05:10:22.507 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:10:22.507 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local mode=snapshot 2023-11-08T05:10:22.507 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' -n snapshot ']' 2023-11-08T05:10:22.507 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:10:22.508 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ create_image cluster2 mirror image_34 128M 2023-11-08T05:10:22.508 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:10:22.508 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:10:22.508 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:10:22.508 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:10:22.509 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_34 2023-11-08T05:10:22.509 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:10:22.509 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local size=128 2023-11-08T05:10:22.509 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' -n 128M ']' 2023-11-08T05:10:22.509 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ size=128M 2023-11-08T05:10:22.510 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:10:22.510 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_34 2023-11-08T05:10:22.545 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' image = image ']' 2023-11-08T05:10:22.545 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ enable_mirror cluster2 mirror image_34 snapshot 2023-11-08T05:10:22.545 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:10:22.545 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:10:22.546 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_34 2023-11-08T05:10:22.546 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local mode=snapshot 2023-11-08T05:10:22.546 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_34 snapshot 2023-11-08T05:10:23.558 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:Mirroring enabled 2023-11-08T05:10:23.567 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster=cluster2 info mirror/image_34 2023-11-08T05:10:23.600 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:rbd image 'image_34': 2023-11-08T05:10:23.601 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: size 128 MiB in 32 objects 2023-11-08T05:10:23.601 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: order 22 (4 MiB objects) 2023-11-08T05:10:23.601 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: snapshot_count: 1 2023-11-08T05:10:23.601 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: id: 13de80f3047e 2023-11-08T05:10:23.601 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: block_name_prefix: rbd_data.13de80f3047e 2023-11-08T05:10:23.602 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: format: 2 2023-11-08T05:10:23.602 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: features: layering, exclusive-lock 2023-11-08T05:10:23.602 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: op_features: 2023-11-08T05:10:23.602 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: flags: 2023-11-08T05:10:23.603 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: create_timestamp: Wed Nov 8 05:10:22 2023 2023-11-08T05:10:23.603 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: access_timestamp: Wed Nov 8 05:10:22 2023 2023-11-08T05:10:23.603 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: modify_timestamp: Wed Nov 8 05:10:22 2023 2023-11-08T05:10:23.603 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring state: enabled 2023-11-08T05:10:23.603 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring mode: snapshot 2023-11-08T05:10:23.604 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring global id: 9fa92707-a7d6-4c46-967b-87ad9015703b 2023-11-08T05:10:23.604 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring primary: true 2023-11-08T05:10:23.605 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' -n '' ']' 2023-11-08T05:10:23.605 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ stress_write_image cluster2 mirror image_34 2023-11-08T05:10:23.605 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:10:23.605 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:10:23.606 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_34 2023-11-08T05:10:23.606 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-11-08T05:10:23.606 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local duration=8 2023-11-08T05:10:23.607 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ set +e 2023-11-08T05:10:23.607 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_34 --debug-rbd=20 --debug-journaler=20 2023-11-08T05:10:23.942 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:100 IOs, 95115264 bytes 2023-11-08T05:10:24.104 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:200 IOs, 125959168 bytes 2023-11-08T05:10:24.321 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:300 IOs, 182797312 bytes 2023-11-08T05:10:24.434 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:400 IOs, 183431168 bytes 2023-11-08T05:10:24.555 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:500 IOs, 193094656 bytes 2023-11-08T05:10:24.682 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:600 IOs, 203845632 bytes 2023-11-08T05:10:24.859 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:700 IOs, 239604736 bytes 2023-11-08T05:10:25.021 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:800 IOs, 267863040 bytes 2023-11-08T05:10:25.202 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:900 IOs, 303172608 bytes 2023-11-08T05:10:25.368 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1000 IOs, 344487936 bytes 2023-11-08T05:10:25.634 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1100 IOs, 423354368 bytes 2023-11-08T05:10:25.834 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1200 IOs, 490572800 bytes 2023-11-08T05:10:25.986 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1300 IOs, 503235584 bytes 2023-11-08T05:10:26.111 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1400 IOs, 518543360 bytes 2023-11-08T05:10:26.374 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1500 IOs, 588534784 bytes 2023-11-08T05:10:26.605 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1600 IOs, 651324416 bytes 2023-11-08T05:10:26.798 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1700 IOs, 718171136 bytes 2023-11-08T05:10:27.014 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1800 IOs, 752401408 bytes 2023-11-08T05:10:27.321 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1900 IOs, 854917120 bytes 2023-11-08T05:10:27.464 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2000 IOs, 876506112 bytes 2023-11-08T05:10:27.660 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2100 IOs, 921736192 bytes 2023-11-08T05:10:27.801 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2200 IOs, 939762688 bytes 2023-11-08T05:10:27.948 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2300 IOs, 960930816 bytes 2023-11-08T05:10:28.129 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2400 IOs, 998270976 bytes 2023-11-08T05:10:28.166 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.6472939819048606) > weight (0.25) 2023-11-08T05:10:28.167 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.26834482838229357) > weight (0.25) 2023-11-08T05:10:28.167 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.2 2023-11-08T05:10:28.167 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Sent signal 15 2023-11-08T05:10:28.168 INFO:tasks.rbd_mirror_thrash:waiting for 1 secs before reviving daemons 2023-11-08T05:10:28.168 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi087.stderr:2023-11-08T05:10:28.166+0000 7fcbfbb22700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 (PID: 123132) UID: 1000 2023-11-08T05:10:28.373 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2500 IOs, 1049746432 bytes 2023-11-08T05:10:28.554 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2600 IOs, 1107002368 bytes 2023-11-08T05:10:28.572 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi087.stderr:daemon-helper: command failed with exit status 1 2023-11-08T05:10:28.732 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2700 IOs, 1122750464 bytes 2023-11-08T05:10:28.934 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2800 IOs, 1178866688 bytes 2023-11-08T05:10:29.169 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.2 2023-11-08T05:10:29.170 INFO:teuthology.orchestra.run:waiting for 600 2023-11-08T05:10:29.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-08T05:10:29.171 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.2 2023-11-08T05:10:29.171 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Restarting daemon 2023-11-08T05:10:29.172 DEBUG:teuthology.orchestra.run.smithi087:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 2023-11-08T05:10:29.174 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Started 2023-11-08T05:10:29.175 INFO:tasks.rbd_mirror_thrash:waiting for 98 secs before thrashing 2023-11-08T05:10:29.295 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2900 IOs, 1299570688 bytes 2023-11-08T05:10:29.526 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3000 IOs, 1351801856 bytes 2023-11-08T05:10:29.751 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3100 IOs, 1427496960 bytes 2023-11-08T05:10:29.969 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3200 IOs, 1465271296 bytes 2023-11-08T05:10:30.179 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3300 IOs, 1540765696 bytes 2023-11-08T05:10:30.374 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3400 IOs, 1574863872 bytes 2023-11-08T05:10:30.555 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3500 IOs, 1594573824 bytes 2023-11-08T05:10:30.714 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3600 IOs, 1616658432 bytes 2023-11-08T05:10:30.852 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3700 IOs, 1627498496 bytes 2023-11-08T05:10:31.086 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3800 IOs, 1689579520 bytes 2023-11-08T05:10:31.207 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3900 IOs, 1711217664 bytes 2023-11-08T05:10:31.410 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4000 IOs, 1745405952 bytes 2023-11-08T05:10:31.611 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ error_code=124 2023-11-08T05:10:31.611 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ set -e 2023-11-08T05:10:31.611 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' 124 -eq 124 ']' 2023-11-08T05:10:31.612 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:10:31.612 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-11-08T05:10:31.612 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image=image_35 2023-11-08T05:10:31.612 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ create_image_and_enable_mirror cluster2 mirror image_35 snapshot 128M 2023-11-08T05:10:31.613 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:10:31.613 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:10:31.613 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:10:31.613 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:10:31.614 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_35 2023-11-08T05:10:31.614 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:10:31.614 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local mode=snapshot 2023-11-08T05:10:31.614 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' -n snapshot ']' 2023-11-08T05:10:31.614 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:10:31.615 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ create_image cluster2 mirror image_35 128M 2023-11-08T05:10:31.615 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:10:31.615 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:10:31.615 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:10:31.615 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:10:31.616 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_35 2023-11-08T05:10:31.616 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:10:31.616 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local size=128 2023-11-08T05:10:31.616 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' -n 128M ']' 2023-11-08T05:10:31.616 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ size=128M 2023-11-08T05:10:31.616 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:10:31.617 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_35 2023-11-08T05:10:31.651 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' image = image ']' 2023-11-08T05:10:31.651 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ enable_mirror cluster2 mirror image_35 snapshot 2023-11-08T05:10:31.652 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:10:31.652 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:10:31.652 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_35 2023-11-08T05:10:31.652 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local mode=snapshot 2023-11-08T05:10:31.652 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_35 snapshot 2023-11-08T05:10:32.070 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:Mirroring enabled 2023-11-08T05:10:32.077 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster=cluster2 info mirror/image_35 2023-11-08T05:10:32.108 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:rbd image 'image_35': 2023-11-08T05:10:32.108 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: size 128 MiB in 32 objects 2023-11-08T05:10:32.108 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: order 22 (4 MiB objects) 2023-11-08T05:10:32.108 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: snapshot_count: 1 2023-11-08T05:10:32.109 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: id: 13eb10cd99f9 2023-11-08T05:10:32.109 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: block_name_prefix: rbd_data.13eb10cd99f9 2023-11-08T05:10:32.109 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: format: 2 2023-11-08T05:10:32.109 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: features: layering, exclusive-lock 2023-11-08T05:10:32.109 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: op_features: 2023-11-08T05:10:32.110 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: flags: 2023-11-08T05:10:32.110 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: create_timestamp: Wed Nov 8 05:10:31 2023 2023-11-08T05:10:32.110 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: access_timestamp: Wed Nov 8 05:10:31 2023 2023-11-08T05:10:32.110 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: modify_timestamp: Wed Nov 8 05:10:31 2023 2023-11-08T05:10:32.110 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring state: enabled 2023-11-08T05:10:32.111 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring mode: snapshot 2023-11-08T05:10:32.111 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring global id: 4f580749-5b1c-4916-870b-0406e147094b 2023-11-08T05:10:32.111 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring primary: true 2023-11-08T05:10:32.112 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' -n '' ']' 2023-11-08T05:10:32.112 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ stress_write_image cluster2 mirror image_35 2023-11-08T05:10:32.112 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:10:32.113 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:10:32.113 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_35 2023-11-08T05:10:32.113 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-11-08T05:10:32.113 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local duration=7 2023-11-08T05:10:32.113 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ set +e 2023-11-08T05:10:32.114 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_35 --debug-rbd=20 --debug-journaler=20 2023-11-08T05:10:32.308 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:100 IOs, 51039232 bytes 2023-11-08T05:10:32.450 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:200 IOs, 52094976 bytes 2023-11-08T05:10:32.615 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:300 IOs, 84474880 bytes 2023-11-08T05:10:32.736 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:400 IOs, 94222336 bytes 2023-11-08T05:10:32.923 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:500 IOs, 140178432 bytes 2023-11-08T05:10:33.041 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:600 IOs, 152089600 bytes 2023-11-08T05:10:33.190 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:700 IOs, 171024384 bytes 2023-11-08T05:10:33.393 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:800 IOs, 223010816 bytes 2023-11-08T05:10:33.580 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:900 IOs, 268206080 bytes 2023-11-08T05:10:33.712 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1000 IOs, 279940096 bytes 2023-11-08T05:10:33.852 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1100 IOs, 299657216 bytes 2023-11-08T05:10:33.985 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1200 IOs, 313550848 bytes 2023-11-08T05:10:34.169 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1300 IOs, 352267264 bytes 2023-11-08T05:10:34.416 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1400 IOs, 412567552 bytes 2023-11-08T05:10:34.651 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1500 IOs, 490898432 bytes 2023-11-08T05:10:34.826 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1600 IOs, 513103872 bytes 2023-11-08T05:10:35.008 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1700 IOs, 555839488 bytes 2023-11-08T05:10:35.071 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.576133471778233) > weight (0.25) 2023-11-08T05:10:35.072 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.947757118407947) > weight (0.25) 2023-11-08T05:10:35.072 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.2 2023-11-08T05:10:35.072 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Sent signal 15 2023-11-08T05:10:35.072 INFO:tasks.rbd_mirror_thrash:waiting for 2 secs before reviving daemons 2023-11-08T05:10:35.075 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi087.stderr:2023-11-08T05:10:35.073+0000 7f42693d5700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 (PID: 125224) UID: 1000 2023-11-08T05:10:35.224 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1800 IOs, 603347968 bytes 2023-11-08T05:10:35.275 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi087.stderr:daemon-helper: command failed with exit status 1 2023-11-08T05:10:35.363 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1900 IOs, 615232512 bytes 2023-11-08T05:10:35.544 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2000 IOs, 651864064 bytes 2023-11-08T05:10:35.745 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2100 IOs, 695413760 bytes 2023-11-08T05:10:35.910 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2200 IOs, 732125184 bytes 2023-11-08T05:10:36.115 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2300 IOs, 777915392 bytes 2023-11-08T05:10:36.259 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2400 IOs, 801197056 bytes 2023-11-08T05:10:36.387 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2500 IOs, 812180480 bytes 2023-11-08T05:10:36.496 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2600 IOs, 826467328 bytes 2023-11-08T05:10:36.669 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2700 IOs, 846365696 bytes 2023-11-08T05:10:36.847 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2800 IOs, 884130816 bytes 2023-11-08T05:10:37.057 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2900 IOs, 944539648 bytes 2023-11-08T05:10:37.073 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.2 2023-11-08T05:10:37.074 INFO:teuthology.orchestra.run:waiting for 600 2023-11-08T05:10:37.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-08T05:10:37.075 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.2 2023-11-08T05:10:37.075 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Restarting daemon 2023-11-08T05:10:37.076 DEBUG:teuthology.orchestra.run.smithi087:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 2023-11-08T05:10:37.079 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Started 2023-11-08T05:10:37.079 INFO:tasks.rbd_mirror_thrash:waiting for 114 secs before thrashing 2023-11-08T05:10:37.260 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3000 IOs, 975475712 bytes 2023-11-08T05:10:37.421 INFO:tasks.ceph.cluster2.osd.2.smithi087.stderr:2023-11-08T05:10:37.410+0000 7fa243317700 -1 reset not still connected to 0x5613b907dba0 2023-11-08T05:10:37.421 INFO:tasks.ceph.cluster2.osd.2.smithi087.stderr:2023-11-08T05:10:37.412+0000 7fa243317700 -1 reset not still connected to 0x5613d3c16680 2023-11-08T05:10:37.526 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3100 IOs, 1030389760 bytes 2023-11-08T05:10:37.751 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3200 IOs, 1089303552 bytes 2023-11-08T05:10:37.991 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3300 IOs, 1158899712 bytes 2023-11-08T05:10:38.148 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3400 IOs, 1188791296 bytes 2023-11-08T05:10:38.296 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3500 IOs, 1206736896 bytes 2023-11-08T05:10:38.426 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3600 IOs, 1231884288 bytes 2023-11-08T05:10:38.639 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3700 IOs, 1269041152 bytes 2023-11-08T05:10:38.826 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3800 IOs, 1306760192 bytes 2023-11-08T05:10:39.120 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ error_code=124 2023-11-08T05:10:39.120 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ set -e 2023-11-08T05:10:39.120 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' 124 -eq 124 ']' 2023-11-08T05:10:39.121 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:10:39.121 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-11-08T05:10:39.121 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image=image_36 2023-11-08T05:10:39.121 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ create_image_and_enable_mirror cluster2 mirror image_36 snapshot 128M 2023-11-08T05:10:39.121 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:10:39.122 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:10:39.122 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:10:39.122 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:10:39.122 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_36 2023-11-08T05:10:39.123 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:10:39.123 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local mode=snapshot 2023-11-08T05:10:39.123 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' -n snapshot ']' 2023-11-08T05:10:39.123 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:10:39.123 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ create_image cluster2 mirror image_36 128M 2023-11-08T05:10:39.123 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:10:39.124 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:10:39.124 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:10:39.124 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:10:39.124 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_36 2023-11-08T05:10:39.124 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:10:39.125 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local size=128 2023-11-08T05:10:39.125 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' -n 128M ']' 2023-11-08T05:10:39.125 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ size=128M 2023-11-08T05:10:39.125 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:10:39.125 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_36 2023-11-08T05:10:39.159 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' image = image ']' 2023-11-08T05:10:39.160 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ enable_mirror cluster2 mirror image_36 snapshot 2023-11-08T05:10:39.160 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:10:39.160 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:10:39.160 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_36 2023-11-08T05:10:39.160 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local mode=snapshot 2023-11-08T05:10:39.161 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_36 snapshot 2023-11-08T05:10:40.144 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:Mirroring enabled 2023-11-08T05:10:40.152 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster=cluster2 info mirror/image_36 2023-11-08T05:10:40.186 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:rbd image 'image_36': 2023-11-08T05:10:40.186 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: size 128 MiB in 32 objects 2023-11-08T05:10:40.186 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: order 22 (4 MiB objects) 2023-11-08T05:10:40.186 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: snapshot_count: 1 2023-11-08T05:10:40.187 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: id: 13fd3a94a48b 2023-11-08T05:10:40.187 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: block_name_prefix: rbd_data.13fd3a94a48b 2023-11-08T05:10:40.187 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: format: 2 2023-11-08T05:10:40.187 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: features: layering, exclusive-lock 2023-11-08T05:10:40.187 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: op_features: 2023-11-08T05:10:40.188 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: flags: 2023-11-08T05:10:40.188 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: create_timestamp: Wed Nov 8 05:10:39 2023 2023-11-08T05:10:40.188 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: access_timestamp: Wed Nov 8 05:10:39 2023 2023-11-08T05:10:40.188 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: modify_timestamp: Wed Nov 8 05:10:39 2023 2023-11-08T05:10:40.188 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring state: enabled 2023-11-08T05:10:40.189 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring mode: snapshot 2023-11-08T05:10:40.189 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring global id: a868743f-f258-4032-ae79-7cb04d60e193 2023-11-08T05:10:40.189 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring primary: true 2023-11-08T05:10:40.190 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' -n '' ']' 2023-11-08T05:10:40.190 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ stress_write_image cluster2 mirror image_36 2023-11-08T05:10:40.191 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:10:40.191 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:10:40.191 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_36 2023-11-08T05:10:40.191 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-11-08T05:10:40.191 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local duration=12 2023-11-08T05:10:40.192 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ set +e 2023-11-08T05:10:40.192 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_36 --debug-rbd=20 --debug-journaler=20 2023-11-08T05:10:40.365 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:100 IOs, 19388416 bytes 2023-11-08T05:10:40.518 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:200 IOs, 45841408 bytes 2023-11-08T05:10:40.740 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:300 IOs, 105024512 bytes 2023-11-08T05:10:40.916 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:400 IOs, 134444032 bytes 2023-11-08T05:10:41.036 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:500 IOs, 160819200 bytes 2023-11-08T05:10:41.307 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:600 IOs, 222456832 bytes 2023-11-08T05:10:41.502 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:700 IOs, 269241344 bytes 2023-11-08T05:10:41.700 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:800 IOs, 318284800 bytes 2023-11-08T05:10:41.831 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:900 IOs, 331720704 bytes 2023-11-08T05:10:41.977 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1000 IOs, 347767808 bytes 2023-11-08T05:10:42.109 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1100 IOs, 361885696 bytes 2023-11-08T05:10:42.266 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1200 IOs, 387139584 bytes 2023-11-08T05:10:42.470 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1300 IOs, 427609088 bytes 2023-11-08T05:10:42.607 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1400 IOs, 442651648 bytes 2023-11-08T05:10:42.827 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1500 IOs, 504819712 bytes 2023-11-08T05:10:43.070 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1600 IOs, 574382080 bytes 2023-11-08T05:10:43.193 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1700 IOs, 585377792 bytes 2023-11-08T05:10:43.310 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1800 IOs, 590099456 bytes 2023-11-08T05:10:43.524 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1900 IOs, 641041408 bytes 2023-11-08T05:10:43.628 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2000 IOs, 641645568 bytes 2023-11-08T05:10:43.792 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2100 IOs, 675746816 bytes 2023-11-08T05:10:44.071 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2200 IOs, 766370816 bytes 2023-11-08T05:10:44.226 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2300 IOs, 789691392 bytes 2023-11-08T05:10:44.419 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2400 IOs, 845737984 bytes 2023-11-08T05:10:44.579 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2500 IOs, 859436032 bytes 2023-11-08T05:10:44.794 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2600 IOs, 917096448 bytes 2023-11-08T05:10:44.998 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2700 IOs, 960999424 bytes 2023-11-08T05:10:45.213 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2800 IOs, 1013695488 bytes 2023-11-08T05:10:45.403 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2900 IOs, 1054780416 bytes 2023-11-08T05:10:45.601 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3000 IOs, 1101521920 bytes 2023-11-08T05:10:45.741 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3100 IOs, 1121025024 bytes 2023-11-08T05:10:45.896 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3200 IOs, 1143016448 bytes 2023-11-08T05:10:46.170 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3300 IOs, 1228951552 bytes 2023-11-08T05:10:46.293 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3400 IOs, 1234598912 bytes 2023-11-08T05:10:46.534 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3500 IOs, 1300810752 bytes 2023-11-08T05:10:46.690 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3600 IOs, 1319949312 bytes 2023-11-08T05:10:46.861 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3700 IOs, 1350373376 bytes 2023-11-08T05:10:46.989 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3800 IOs, 1363878912 bytes 2023-11-08T05:10:47.222 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3900 IOs, 1433026560 bytes 2023-11-08T05:10:47.402 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4000 IOs, 1462589440 bytes 2023-11-08T05:10:47.637 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4100 IOs, 1526303744 bytes 2023-11-08T05:10:47.877 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4200 IOs, 1600549888 bytes 2023-11-08T05:10:48.143 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4300 IOs, 1667735552 bytes 2023-11-08T05:10:48.309 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4400 IOs, 1698402304 bytes 2023-11-08T05:10:48.488 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4500 IOs, 1735756800 bytes 2023-11-08T05:10:48.629 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4600 IOs, 1752842240 bytes 2023-11-08T05:10:48.742 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4700 IOs, 1753444352 bytes 2023-11-08T05:10:48.949 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4800 IOs, 1811458048 bytes 2023-11-08T05:10:49.082 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4900 IOs, 1822824448 bytes 2023-11-08T05:10:49.186 INFO:tasks.ceph.cluster2.osd.1.smithi087.stderr:2023-11-08T05:10:49.183+0000 7fbb7a4fc700 -1 reset not still connected to 0x55770f770270 2023-11-08T05:10:49.201 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5000 IOs, 1829131264 bytes 2023-11-08T05:10:49.364 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5100 IOs, 1856902144 bytes 2023-11-08T05:10:49.465 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5200 IOs, 1857509376 bytes 2023-11-08T05:10:49.611 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5300 IOs, 1870008320 bytes 2023-11-08T05:10:49.776 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5400 IOs, 1900791808 bytes 2023-11-08T05:10:49.954 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5500 IOs, 1939170304 bytes 2023-11-08T05:10:50.082 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5600 IOs, 1954374656 bytes 2023-11-08T05:10:50.328 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5700 IOs, 2010648576 bytes 2023-11-08T05:10:50.495 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5800 IOs, 2043488256 bytes 2023-11-08T05:10:50.644 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5900 IOs, 2059617280 bytes 2023-11-08T05:10:50.822 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:6000 IOs, 2094683136 bytes 2023-11-08T05:10:50.932 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:6100 IOs, 2097944576 bytes 2023-11-08T05:10:51.117 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:6200 IOs, 2138468352 bytes 2023-11-08T05:10:51.263 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:6300 IOs, 2153864192 bytes 2023-11-08T05:10:51.408 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:6400 IOs, 2169427968 bytes 2023-11-08T05:10:51.707 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:6500 IOs, 2281345024 bytes 2023-11-08T05:10:51.857 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:6600 IOs, 2307496960 bytes 2023-11-08T05:10:51.963 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:6700 IOs, 2308117504 bytes 2023-11-08T05:10:52.152 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:6800 IOs, 2348755968 bytes 2023-11-08T05:10:52.198 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ error_code=124 2023-11-08T05:10:52.198 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ set -e 2023-11-08T05:10:52.198 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' 124 -eq 124 ']' 2023-11-08T05:10:52.198 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:10:52.198 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-11-08T05:10:52.199 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image=image_37 2023-11-08T05:10:52.199 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ create_image_and_enable_mirror cluster2 mirror image_37 snapshot 128M 2023-11-08T05:10:52.199 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:10:52.199 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:10:52.199 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:10:52.200 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:10:52.200 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_37 2023-11-08T05:10:52.200 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:10:52.200 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local mode=snapshot 2023-11-08T05:10:52.200 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' -n snapshot ']' 2023-11-08T05:10:52.200 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:10:52.201 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ create_image cluster2 mirror image_37 128M 2023-11-08T05:10:52.201 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:10:52.201 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:10:52.201 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:10:52.201 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:10:52.201 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_37 2023-11-08T05:10:52.202 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:10:52.202 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local size=128 2023-11-08T05:10:52.202 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' -n 128M ']' 2023-11-08T05:10:52.202 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ size=128M 2023-11-08T05:10:52.202 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:10:52.203 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_37 2023-11-08T05:10:52.208 INFO:tasks.ceph.cluster2.osd.2.smithi087.stderr:2023-11-08T05:10:52.206+0000 7fa243317700 -1 reset not still connected to 0x5613b889e4e0 2023-11-08T05:10:52.208 INFO:tasks.ceph.cluster2.osd.2.smithi087.stderr:2023-11-08T05:10:52.206+0000 7fa243317700 -1 reset not still connected to 0x5613bbe344e0 2023-11-08T05:10:52.442 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' image = image ']' 2023-11-08T05:10:52.443 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ enable_mirror cluster2 mirror image_37 snapshot 2023-11-08T05:10:52.443 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:10:52.443 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:10:52.443 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_37 2023-11-08T05:10:52.443 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local mode=snapshot 2023-11-08T05:10:52.444 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_37 snapshot 2023-11-08T05:10:53.197 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:Mirroring enabled 2023-11-08T05:10:53.205 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster=cluster2 info mirror/image_37 2023-11-08T05:10:53.235 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:rbd image 'image_37': 2023-11-08T05:10:53.236 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: size 128 MiB in 32 objects 2023-11-08T05:10:53.236 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: order 22 (4 MiB objects) 2023-11-08T05:10:53.236 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: snapshot_count: 1 2023-11-08T05:10:53.236 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: id: 140832268671 2023-11-08T05:10:53.236 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: block_name_prefix: rbd_data.140832268671 2023-11-08T05:10:53.237 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: format: 2 2023-11-08T05:10:53.237 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: features: layering, exclusive-lock 2023-11-08T05:10:53.237 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: op_features: 2023-11-08T05:10:53.237 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: flags: 2023-11-08T05:10:53.237 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: create_timestamp: Wed Nov 8 05:10:52 2023 2023-11-08T05:10:53.237 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: access_timestamp: Wed Nov 8 05:10:52 2023 2023-11-08T05:10:53.238 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: modify_timestamp: Wed Nov 8 05:10:52 2023 2023-11-08T05:10:53.238 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring state: enabled 2023-11-08T05:10:53.238 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring mode: snapshot 2023-11-08T05:10:53.238 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring global id: 7c36d62d-ec90-4bfc-b81d-9054fbd92513 2023-11-08T05:10:53.239 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring primary: true 2023-11-08T05:10:53.239 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' -n '' ']' 2023-11-08T05:10:53.240 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ stress_write_image cluster2 mirror image_37 2023-11-08T05:10:53.240 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:10:53.240 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:10:53.240 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_37 2023-11-08T05:10:53.240 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-11-08T05:10:53.241 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local duration=7 2023-11-08T05:10:53.241 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ set +e 2023-11-08T05:10:53.242 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_37 --debug-rbd=20 --debug-journaler=20 2023-11-08T05:10:53.484 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:100 IOs, 49039360 bytes 2023-11-08T05:10:53.640 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:200 IOs, 76091392 bytes 2023-11-08T05:10:53.893 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:300 IOs, 147837952 bytes 2023-11-08T05:10:54.071 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:400 IOs, 186366976 bytes 2023-11-08T05:10:54.250 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:500 IOs, 219792384 bytes 2023-11-08T05:10:54.395 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:600 IOs, 258026496 bytes 2023-11-08T05:10:54.555 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:700 IOs, 271827968 bytes 2023-11-08T05:10:54.699 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:800 IOs, 306145280 bytes 2023-11-08T05:10:54.887 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:900 IOs, 337705984 bytes 2023-11-08T05:10:55.037 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1000 IOs, 354948096 bytes 2023-11-08T05:10:55.176 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1100 IOs, 369806336 bytes 2023-11-08T05:10:55.402 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1200 IOs, 443423744 bytes 2023-11-08T05:10:55.589 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1300 IOs, 472343552 bytes 2023-11-08T05:10:55.805 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1400 IOs, 531184640 bytes 2023-11-08T05:10:55.934 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1500 IOs, 554806272 bytes 2023-11-08T05:10:56.107 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1600 IOs, 579926016 bytes 2023-11-08T05:10:56.261 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1700 IOs, 595392512 bytes 2023-11-08T05:10:56.548 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1800 IOs, 684468224 bytes 2023-11-08T05:10:56.739 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1900 IOs, 724757504 bytes 2023-11-08T05:10:56.890 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2000 IOs, 764042240 bytes 2023-11-08T05:10:57.114 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2100 IOs, 807675904 bytes 2023-11-08T05:10:57.303 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2200 IOs, 859265024 bytes 2023-11-08T05:10:57.493 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2300 IOs, 885242880 bytes 2023-11-08T05:10:57.725 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2400 IOs, 963654656 bytes 2023-11-08T05:10:57.900 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2500 IOs, 980233216 bytes 2023-11-08T05:10:58.092 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2600 IOs, 1021699072 bytes 2023-11-08T05:10:58.262 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2700 IOs, 1054850048 bytes 2023-11-08T05:10:58.516 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2800 IOs, 1129050112 bytes 2023-11-08T05:10:58.743 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2900 IOs, 1191537664 bytes 2023-11-08T05:10:58.923 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3000 IOs, 1230834688 bytes 2023-11-08T05:10:59.114 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3100 IOs, 1297057792 bytes 2023-11-08T05:10:59.406 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3200 IOs, 1345807360 bytes 2023-11-08T05:10:59.552 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3300 IOs, 1360768000 bytes 2023-11-08T05:10:59.729 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3400 IOs, 1396567040 bytes 2023-11-08T05:10:59.938 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3500 IOs, 1450833920 bytes 2023-11-08T05:11:00.192 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3600 IOs, 1526345728 bytes 2023-11-08T05:11:00.250 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ error_code=124 2023-11-08T05:11:00.250 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ set -e 2023-11-08T05:11:00.250 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' 124 -eq 124 ']' 2023-11-08T05:11:00.250 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:11:00.250 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-11-08T05:11:00.251 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image=image_38 2023-11-08T05:11:00.251 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ create_image_and_enable_mirror cluster2 mirror image_38 snapshot 128M 2023-11-08T05:11:00.251 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:11:00.251 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:11:00.251 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:11:00.251 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:11:00.252 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_38 2023-11-08T05:11:00.252 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:11:00.252 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local mode=snapshot 2023-11-08T05:11:00.252 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' -n snapshot ']' 2023-11-08T05:11:00.252 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:11:00.253 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ create_image cluster2 mirror image_38 128M 2023-11-08T05:11:00.253 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:11:00.253 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:11:00.253 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:11:00.253 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:11:00.253 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_38 2023-11-08T05:11:00.254 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:11:00.254 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local size=128 2023-11-08T05:11:00.254 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' -n 128M ']' 2023-11-08T05:11:00.254 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ size=128M 2023-11-08T05:11:00.254 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:11:00.254 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_38 2023-11-08T05:11:00.291 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' image = image ']' 2023-11-08T05:11:00.291 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ enable_mirror cluster2 mirror image_38 snapshot 2023-11-08T05:11:00.291 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:11:00.291 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:11:00.291 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_38 2023-11-08T05:11:00.292 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local mode=snapshot 2023-11-08T05:11:00.292 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_38 snapshot 2023-11-08T05:11:01.293 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:Mirroring enabled 2023-11-08T05:11:01.302 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster=cluster2 info mirror/image_38 2023-11-08T05:11:01.335 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:rbd image 'image_38': 2023-11-08T05:11:01.335 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: size 128 MiB in 32 objects 2023-11-08T05:11:01.336 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: order 22 (4 MiB objects) 2023-11-08T05:11:01.336 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: snapshot_count: 1 2023-11-08T05:11:01.336 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: id: 14148a1fe53e 2023-11-08T05:11:01.336 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: block_name_prefix: rbd_data.14148a1fe53e 2023-11-08T05:11:01.336 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: format: 2 2023-11-08T05:11:01.337 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: features: layering, exclusive-lock 2023-11-08T05:11:01.337 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: op_features: 2023-11-08T05:11:01.337 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: flags: 2023-11-08T05:11:01.337 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: create_timestamp: Wed Nov 8 05:11:00 2023 2023-11-08T05:11:01.337 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: access_timestamp: Wed Nov 8 05:11:00 2023 2023-11-08T05:11:01.337 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: modify_timestamp: Wed Nov 8 05:11:00 2023 2023-11-08T05:11:01.338 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring state: enabled 2023-11-08T05:11:01.338 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring mode: snapshot 2023-11-08T05:11:01.338 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring global id: c3a4ad2f-2430-4db4-a78e-1b1f4255422f 2023-11-08T05:11:01.338 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring primary: true 2023-11-08T05:11:01.339 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' -n '' ']' 2023-11-08T05:11:01.339 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ stress_write_image cluster2 mirror image_38 2023-11-08T05:11:01.340 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:11:01.340 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:11:01.340 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_38 2023-11-08T05:11:01.340 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-11-08T05:11:01.341 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local duration=9 2023-11-08T05:11:01.341 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ set +e 2023-11-08T05:11:01.341 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_38 --debug-rbd=20 --debug-journaler=20 2023-11-08T05:11:01.667 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:100 IOs, 88420352 bytes 2023-11-08T05:11:01.873 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:200 IOs, 151690240 bytes 2023-11-08T05:11:02.035 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:300 IOs, 179041280 bytes 2023-11-08T05:11:02.334 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:400 IOs, 272595968 bytes 2023-11-08T05:11:02.646 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:500 IOs, 368612352 bytes 2023-11-08T05:11:02.788 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:600 IOs, 390329344 bytes 2023-11-08T05:11:03.011 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:700 IOs, 449757184 bytes 2023-11-08T05:11:03.134 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:800 IOs, 457062400 bytes 2023-11-08T05:11:03.297 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:900 IOs, 498414592 bytes 2023-11-08T05:11:03.690 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1000 IOs, 618305536 bytes 2023-11-08T05:11:03.841 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1100 IOs, 642021376 bytes 2023-11-08T05:11:04.094 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1200 IOs, 715159552 bytes 2023-11-08T05:11:04.310 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1300 IOs, 771133440 bytes 2023-11-08T05:11:04.485 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1400 IOs, 802361344 bytes 2023-11-08T05:11:04.711 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1500 IOs, 855170048 bytes 2023-11-08T05:11:04.857 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1600 IOs, 879736832 bytes 2023-11-08T05:11:05.038 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1700 IOs, 912542720 bytes 2023-11-08T05:11:05.185 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1800 IOs, 937741312 bytes 2023-11-08T05:11:05.535 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1900 IOs, 1065040896 bytes 2023-11-08T05:11:05.757 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2000 IOs, 1112403968 bytes 2023-11-08T05:11:05.868 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2100 IOs, 1113026560 bytes 2023-11-08T05:11:06.060 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2200 IOs, 1156909056 bytes 2023-11-08T05:11:06.195 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2300 IOs, 1173655552 bytes 2023-11-08T05:11:06.476 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2400 IOs, 1229207552 bytes 2023-11-08T05:11:06.616 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2500 IOs, 1240782848 bytes 2023-11-08T05:11:06.853 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2600 IOs, 1304599552 bytes 2023-11-08T05:11:07.004 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2700 IOs, 1326402560 bytes 2023-11-08T05:11:07.120 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2800 IOs, 1327017984 bytes 2023-11-08T05:11:07.314 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2900 IOs, 1370075136 bytes 2023-11-08T05:11:07.512 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3000 IOs, 1414766592 bytes 2023-11-08T05:11:07.721 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3100 IOs, 1466727424 bytes 2023-11-08T05:11:07.841 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3200 IOs, 1467375616 bytes 2023-11-08T05:11:08.138 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3300 IOs, 1567261696 bytes 2023-11-08T05:11:08.341 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3400 IOs, 1613879296 bytes 2023-11-08T05:11:08.479 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3500 IOs, 1636268032 bytes 2023-11-08T05:11:08.631 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3600 IOs, 1651000320 bytes 2023-11-08T05:11:08.866 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3700 IOs, 1714383872 bytes 2023-11-08T05:11:09.155 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3800 IOs, 1826422784 bytes 2023-11-08T05:11:09.354 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3900 IOs, 1851674624 bytes 2023-11-08T05:11:09.517 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4000 IOs, 1884213248 bytes 2023-11-08T05:11:09.777 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4100 IOs, 1953096704 bytes 2023-11-08T05:11:09.883 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4200 IOs, 1953682432 bytes 2023-11-08T05:11:10.073 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4300 IOs, 1999496192 bytes 2023-11-08T05:11:10.181 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4400 IOs, 2000117760 bytes 2023-11-08T05:11:10.340 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4500 IOs, 2022214656 bytes 2023-11-08T05:11:10.348 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ error_code=124 2023-11-08T05:11:10.349 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ set -e 2023-11-08T05:11:10.349 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' 124 -eq 124 ']' 2023-11-08T05:11:10.349 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:11:10.349 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-11-08T05:11:10.350 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image=image_39 2023-11-08T05:11:10.350 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ create_image_and_enable_mirror cluster2 mirror image_39 snapshot 128M 2023-11-08T05:11:10.350 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:11:10.350 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:11:10.351 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:11:10.351 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:11:10.351 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_39 2023-11-08T05:11:10.351 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:11:10.351 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local mode=snapshot 2023-11-08T05:11:10.351 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' -n snapshot ']' 2023-11-08T05:11:10.352 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:11:10.352 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ create_image cluster2 mirror image_39 128M 2023-11-08T05:11:10.352 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:11:10.352 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:11:10.352 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:11:10.353 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:11:10.353 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_39 2023-11-08T05:11:10.353 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:11:10.353 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local size=128 2023-11-08T05:11:10.353 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' -n 128M ']' 2023-11-08T05:11:10.353 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ size=128M 2023-11-08T05:11:10.354 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:11:10.354 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_39 2023-11-08T05:11:10.388 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' image = image ']' 2023-11-08T05:11:10.389 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ enable_mirror cluster2 mirror image_39 snapshot 2023-11-08T05:11:10.389 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:11:10.389 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:11:10.389 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_39 2023-11-08T05:11:10.389 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local mode=snapshot 2023-11-08T05:11:10.390 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_39 snapshot 2023-11-08T05:11:11.385 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:Mirroring enabled 2023-11-08T05:11:11.393 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster=cluster2 info mirror/image_39 2023-11-08T05:11:11.426 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:rbd image 'image_39': 2023-11-08T05:11:11.427 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: size 128 MiB in 32 objects 2023-11-08T05:11:11.427 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: order 22 (4 MiB objects) 2023-11-08T05:11:11.427 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: snapshot_count: 1 2023-11-08T05:11:11.427 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: id: 141f1e7f8633 2023-11-08T05:11:11.428 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: block_name_prefix: rbd_data.141f1e7f8633 2023-11-08T05:11:11.428 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: format: 2 2023-11-08T05:11:11.428 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: features: layering, exclusive-lock 2023-11-08T05:11:11.428 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: op_features: 2023-11-08T05:11:11.428 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: flags: 2023-11-08T05:11:11.429 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: create_timestamp: Wed Nov 8 05:11:10 2023 2023-11-08T05:11:11.429 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: access_timestamp: Wed Nov 8 05:11:10 2023 2023-11-08T05:11:11.429 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: modify_timestamp: Wed Nov 8 05:11:10 2023 2023-11-08T05:11:11.429 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring state: enabled 2023-11-08T05:11:11.429 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring mode: snapshot 2023-11-08T05:11:11.430 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring global id: a4b03d0b-f737-4c8f-8a6c-6971fcf77e8a 2023-11-08T05:11:11.430 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring primary: true 2023-11-08T05:11:11.431 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' -n '' ']' 2023-11-08T05:11:11.431 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ stress_write_image cluster2 mirror image_39 2023-11-08T05:11:11.431 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:11:11.431 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:11:11.431 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_39 2023-11-08T05:11:11.432 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-11-08T05:11:11.432 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local duration=12 2023-11-08T05:11:11.433 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ set +e 2023-11-08T05:11:11.433 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_39 --debug-rbd=20 --debug-journaler=20 2023-11-08T05:11:11.571 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:100 IOs, 620544 bytes 2023-11-08T05:11:11.711 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:200 IOs, 25280512 bytes 2023-11-08T05:11:11.848 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:300 IOs, 40843264 bytes 2023-11-08T05:11:12.036 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:400 IOs, 84910080 bytes 2023-11-08T05:11:12.157 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:500 IOs, 93232128 bytes 2023-11-08T05:11:12.287 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:600 IOs, 106166272 bytes 2023-11-08T05:11:12.522 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:700 IOs, 171773952 bytes 2023-11-08T05:11:12.679 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:800 IOs, 198993920 bytes 2023-11-08T05:11:12.792 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:900 IOs, 217675776 bytes 2023-11-08T05:11:12.960 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1000 IOs, 241083392 bytes 2023-11-08T05:11:13.141 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1100 IOs, 286191616 bytes 2023-11-08T05:11:13.364 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1200 IOs, 356519936 bytes 2023-11-08T05:11:13.733 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1300 IOs, 471718912 bytes 2023-11-08T05:11:13.900 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1400 IOs, 502424576 bytes 2023-11-08T05:11:14.126 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1500 IOs, 566278144 bytes 2023-11-08T05:11:14.341 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1600 IOs, 622695424 bytes 2023-11-08T05:11:14.566 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1700 IOs, 680124416 bytes 2023-11-08T05:11:14.734 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1800 IOs, 715586560 bytes 2023-11-08T05:11:14.895 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1900 IOs, 741706752 bytes 2023-11-08T05:11:15.140 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2000 IOs, 812273664 bytes 2023-11-08T05:11:15.289 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2100 IOs, 834379776 bytes 2023-11-08T05:11:15.404 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2200 IOs, 835214336 bytes 2023-11-08T05:11:15.577 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2300 IOs, 866145280 bytes 2023-11-08T05:11:15.879 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2400 IOs, 966477824 bytes 2023-11-08T05:11:16.031 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2500 IOs, 986791936 bytes 2023-11-08T05:11:16.144 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2600 IOs, 992721920 bytes 2023-11-08T05:11:16.360 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2700 IOs, 1053607936 bytes 2023-11-08T05:11:16.485 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2800 IOs, 1055424512 bytes 2023-11-08T05:11:16.665 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2900 IOs, 1087518720 bytes 2023-11-08T05:11:16.845 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3000 IOs, 1128008704 bytes 2023-11-08T05:11:17.022 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3100 IOs, 1166121984 bytes 2023-11-08T05:11:17.243 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3200 IOs, 1223489536 bytes 2023-11-08T05:11:17.368 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3300 IOs, 1232904192 bytes 2023-11-08T05:11:17.564 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3400 IOs, 1278008320 bytes 2023-11-08T05:11:17.772 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3500 IOs, 1323811840 bytes 2023-11-08T05:11:17.895 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3600 IOs, 1330833408 bytes 2023-11-08T05:11:18.003 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3700 IOs, 1333005312 bytes 2023-11-08T05:11:18.189 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3800 IOs, 1376737280 bytes 2023-11-08T05:11:18.360 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3900 IOs, 1408588800 bytes 2023-11-08T05:11:18.531 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4000 IOs, 1444834304 bytes 2023-11-08T05:11:18.760 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4100 IOs, 1505225728 bytes 2023-11-08T05:11:18.963 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4200 IOs, 1554511872 bytes 2023-11-08T05:11:19.208 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4300 IOs, 1628276736 bytes 2023-11-08T05:11:19.404 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4400 IOs, 1676557312 bytes 2023-11-08T05:11:19.624 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4500 IOs, 1729765376 bytes 2023-11-08T05:11:19.767 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4600 IOs, 1748241408 bytes 2023-11-08T05:11:19.937 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4700 IOs, 1767282688 bytes 2023-11-08T05:11:20.080 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4800 IOs, 1790617600 bytes 2023-11-08T05:11:20.283 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4900 IOs, 1827205120 bytes 2023-11-08T05:11:20.462 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5000 IOs, 1851817984 bytes 2023-11-08T05:11:20.682 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5100 IOs, 1886852096 bytes 2023-11-08T05:11:20.832 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5200 IOs, 1926433792 bytes 2023-11-08T05:11:21.031 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5300 IOs, 1951919104 bytes 2023-11-08T05:11:21.176 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5400 IOs, 1973152768 bytes 2023-11-08T05:11:21.316 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5500 IOs, 1989967872 bytes 2023-11-08T05:11:21.566 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5600 IOs, 2067008512 bytes 2023-11-08T05:11:21.757 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5700 IOs, 2108069888 bytes 2023-11-08T05:11:21.959 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5800 IOs, 2162040832 bytes 2023-11-08T05:11:22.204 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5900 IOs, 2234049536 bytes 2023-11-08T05:11:22.335 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:6000 IOs, 2246672384 bytes 2023-11-08T05:11:22.533 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:6100 IOs, 2294011904 bytes 2023-11-08T05:11:22.656 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:6200 IOs, 2315964416 bytes 2023-11-08T05:11:22.857 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:6300 IOs, 2359023616 bytes 2023-11-08T05:11:23.013 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:6400 IOs, 2387832832 bytes 2023-11-08T05:11:23.220 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:6500 IOs, 2432224256 bytes 2023-11-08T05:11:23.371 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:6600 IOs, 2453940224 bytes 2023-11-08T05:11:23.438 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ error_code=124 2023-11-08T05:11:23.438 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ set -e 2023-11-08T05:11:23.439 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' 124 -eq 124 ']' 2023-11-08T05:11:23.439 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:11:23.439 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-11-08T05:11:23.439 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image=image_40 2023-11-08T05:11:23.439 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ create_image_and_enable_mirror cluster2 mirror image_40 snapshot 128M 2023-11-08T05:11:23.440 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:11:23.440 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:11:23.440 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:11:23.440 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:11:23.440 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_40 2023-11-08T05:11:23.441 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:11:23.441 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local mode=snapshot 2023-11-08T05:11:23.441 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' -n snapshot ']' 2023-11-08T05:11:23.441 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:11:23.441 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ create_image cluster2 mirror image_40 128M 2023-11-08T05:11:23.442 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:11:23.442 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:11:23.442 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:11:23.442 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:11:23.442 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_40 2023-11-08T05:11:23.443 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:11:23.443 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local size=128 2023-11-08T05:11:23.443 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' -n 128M ']' 2023-11-08T05:11:23.443 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ size=128M 2023-11-08T05:11:23.443 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:11:23.444 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_40 2023-11-08T05:11:23.477 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' image = image ']' 2023-11-08T05:11:23.477 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ enable_mirror cluster2 mirror image_40 snapshot 2023-11-08T05:11:23.478 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:11:23.478 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:11:23.478 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_40 2023-11-08T05:11:23.478 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local mode=snapshot 2023-11-08T05:11:23.478 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_40 snapshot 2023-11-08T05:11:24.463 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:Mirroring enabled 2023-11-08T05:11:24.471 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster=cluster2 info mirror/image_40 2023-11-08T05:11:24.505 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:rbd image 'image_40': 2023-11-08T05:11:24.506 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: size 128 MiB in 32 objects 2023-11-08T05:11:24.506 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: order 22 (4 MiB objects) 2023-11-08T05:11:24.506 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: snapshot_count: 1 2023-11-08T05:11:24.506 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: id: 14288bfc45dc 2023-11-08T05:11:24.507 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: block_name_prefix: rbd_data.14288bfc45dc 2023-11-08T05:11:24.507 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: format: 2 2023-11-08T05:11:24.507 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: features: layering, exclusive-lock 2023-11-08T05:11:24.507 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: op_features: 2023-11-08T05:11:24.507 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: flags: 2023-11-08T05:11:24.507 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: create_timestamp: Wed Nov 8 05:11:23 2023 2023-11-08T05:11:24.508 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: access_timestamp: Wed Nov 8 05:11:23 2023 2023-11-08T05:11:24.508 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: modify_timestamp: Wed Nov 8 05:11:23 2023 2023-11-08T05:11:24.508 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring state: enabled 2023-11-08T05:11:24.508 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring mode: snapshot 2023-11-08T05:11:24.508 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring global id: 17ed47a8-b4d3-4db4-8592-15e686f59163 2023-11-08T05:11:24.509 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring primary: true 2023-11-08T05:11:24.509 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' -n '' ']' 2023-11-08T05:11:24.509 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ stress_write_image cluster2 mirror image_40 2023-11-08T05:11:24.510 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:11:24.510 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:11:24.510 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_40 2023-11-08T05:11:24.510 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-11-08T05:11:24.511 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local duration=13 2023-11-08T05:11:24.511 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ set +e 2023-11-08T05:11:24.511 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_40 --debug-rbd=20 --debug-journaler=20 2023-11-08T05:11:24.728 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:100 IOs, 38635520 bytes 2023-11-08T05:11:24.889 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:200 IOs, 66269184 bytes 2023-11-08T05:11:25.046 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:300 IOs, 96738304 bytes 2023-11-08T05:11:25.256 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:400 IOs, 147744768 bytes 2023-11-08T05:11:25.530 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:500 IOs, 243319808 bytes 2023-11-08T05:11:25.700 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:600 IOs, 257961984 bytes 2023-11-08T05:11:25.848 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:700 IOs, 276803584 bytes 2023-11-08T05:11:26.005 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:800 IOs, 308919296 bytes 2023-11-08T05:11:26.144 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:900 IOs, 333669376 bytes 2023-11-08T05:11:26.296 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1000 IOs, 355215360 bytes 2023-11-08T05:11:26.414 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1100 IOs, 358747136 bytes 2023-11-08T05:11:26.692 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1200 IOs, 449366016 bytes 2023-11-08T05:11:26.875 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1300 IOs, 490549248 bytes 2023-11-08T05:11:27.068 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1400 IOs, 550422528 bytes 2023-11-08T05:11:27.300 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1500 IOs, 596528128 bytes 2023-11-08T05:11:27.547 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1600 IOs, 664129536 bytes 2023-11-08T05:11:27.764 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1700 IOs, 724857856 bytes 2023-11-08T05:11:28.210 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1800 IOs, 805758976 bytes 2023-11-08T05:11:28.418 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1900 IOs, 851584000 bytes 2023-11-08T05:11:28.639 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2000 IOs, 917585920 bytes 2023-11-08T05:11:28.805 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2100 IOs, 944301056 bytes 2023-11-08T05:11:29.003 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2200 IOs, 979975168 bytes 2023-11-08T05:11:29.215 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2300 IOs, 1045632000 bytes 2023-11-08T05:11:29.416 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2400 IOs, 1082268672 bytes 2023-11-08T05:11:29.668 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2500 IOs, 1163122688 bytes 2023-11-08T05:11:29.907 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2600 IOs, 1229137920 bytes 2023-11-08T05:11:30.150 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2700 IOs, 1300835328 bytes 2023-11-08T05:11:30.301 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2800 IOs, 1323885568 bytes 2023-11-08T05:11:30.552 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2900 IOs, 1369818112 bytes 2023-11-08T05:11:30.758 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3000 IOs, 1418249216 bytes 2023-11-08T05:11:30.958 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3100 IOs, 1464159232 bytes 2023-11-08T05:11:31.163 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3200 IOs, 1514455040 bytes 2023-11-08T05:11:31.413 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3300 IOs, 1584832512 bytes 2023-11-08T05:11:31.575 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3400 IOs, 1620931584 bytes 2023-11-08T05:11:31.776 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3500 IOs, 1666219008 bytes 2023-11-08T05:11:31.895 INFO:tasks.ceph.cluster2.osd.2.smithi087.stderr:2023-11-08T05:11:31.893+0000 7fa243317700 -1 reset not still connected to 0x5613d4310270 2023-11-08T05:11:32.000 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3600 IOs, 1727350784 bytes 2023-11-08T05:11:32.166 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3700 IOs, 1760026624 bytes 2023-11-08T05:11:32.396 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3800 IOs, 1821371392 bytes 2023-11-08T05:11:32.535 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3900 IOs, 1844465664 bytes 2023-11-08T05:11:32.775 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4000 IOs, 1931949056 bytes 2023-11-08T05:11:32.947 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4100 IOs, 1952458752 bytes 2023-11-08T05:11:33.130 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4200 IOs, 1996743680 bytes 2023-11-08T05:11:33.476 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4300 IOs, 2113905664 bytes 2023-11-08T05:11:33.696 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4400 IOs, 2163082240 bytes 2023-11-08T05:11:33.883 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4500 IOs, 2200117248 bytes 2023-11-08T05:11:34.061 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4600 IOs, 2258120704 bytes 2023-11-08T05:11:34.255 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4700 IOs, 2283717632 bytes 2023-11-08T05:11:34.461 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4800 IOs, 2333629440 bytes 2023-11-08T05:11:34.685 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4900 IOs, 2391413760 bytes 2023-11-08T05:11:34.827 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5000 IOs, 2405466112 bytes 2023-11-08T05:11:35.049 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5100 IOs, 2463989760 bytes 2023-11-08T05:11:35.282 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5200 IOs, 2528358400 bytes 2023-11-08T05:11:35.469 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5300 IOs, 2565526528 bytes 2023-11-08T05:11:35.658 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5400 IOs, 2602011648 bytes 2023-11-08T05:11:35.822 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5500 IOs, 2629918720 bytes 2023-11-08T05:11:36.033 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5600 IOs, 2706774016 bytes 2023-11-08T05:11:36.232 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5700 IOs, 2726382592 bytes 2023-11-08T05:11:36.363 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5800 IOs, 2738298880 bytes 2023-11-08T05:11:36.566 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5900 IOs, 2793211904 bytes 2023-11-08T05:11:36.798 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:6000 IOs, 2865573888 bytes 2023-11-08T05:11:36.986 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:6100 IOs, 2896344064 bytes 2023-11-08T05:11:37.200 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:6200 IOs, 2947634176 bytes 2023-11-08T05:11:37.326 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:6300 IOs, 2953172992 bytes 2023-11-08T05:11:37.475 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:6400 IOs, 2968322048 bytes 2023-11-08T05:11:37.518 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ error_code=124 2023-11-08T05:11:37.518 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ set -e 2023-11-08T05:11:37.518 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' 124 -eq 124 ']' 2023-11-08T05:11:37.518 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:11:37.519 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-11-08T05:11:37.519 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image=image_41 2023-11-08T05:11:37.519 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ create_image_and_enable_mirror cluster2 mirror image_41 snapshot 128M 2023-11-08T05:11:37.519 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:11:37.519 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:11:37.520 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:11:37.520 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:11:37.520 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_41 2023-11-08T05:11:37.520 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:11:37.520 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local mode=snapshot 2023-11-08T05:11:37.521 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' -n snapshot ']' 2023-11-08T05:11:37.521 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:11:37.521 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ create_image cluster2 mirror image_41 128M 2023-11-08T05:11:37.521 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:11:37.521 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:11:37.522 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:11:37.522 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:11:37.522 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_41 2023-11-08T05:11:37.522 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:11:37.522 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local size=128 2023-11-08T05:11:37.523 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' -n 128M ']' 2023-11-08T05:11:37.523 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ size=128M 2023-11-08T05:11:37.523 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:11:37.523 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_41 2023-11-08T05:11:37.558 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' image = image ']' 2023-11-08T05:11:37.558 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ enable_mirror cluster2 mirror image_41 snapshot 2023-11-08T05:11:37.559 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:11:37.559 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:11:37.559 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_41 2023-11-08T05:11:37.559 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local mode=snapshot 2023-11-08T05:11:37.560 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_41 snapshot 2023-11-08T05:11:38.539 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:Mirroring enabled 2023-11-08T05:11:38.547 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster=cluster2 info mirror/image_41 2023-11-08T05:11:38.579 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:rbd image 'image_41': 2023-11-08T05:11:38.580 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: size 128 MiB in 32 objects 2023-11-08T05:11:38.580 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: order 22 (4 MiB objects) 2023-11-08T05:11:38.580 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: snapshot_count: 1 2023-11-08T05:11:38.580 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: id: 14314384d312 2023-11-08T05:11:38.580 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: block_name_prefix: rbd_data.14314384d312 2023-11-08T05:11:38.581 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: format: 2 2023-11-08T05:11:38.581 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: features: layering, exclusive-lock 2023-11-08T05:11:38.581 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: op_features: 2023-11-08T05:11:38.581 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: flags: 2023-11-08T05:11:38.581 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: create_timestamp: Wed Nov 8 05:11:37 2023 2023-11-08T05:11:38.582 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: access_timestamp: Wed Nov 8 05:11:37 2023 2023-11-08T05:11:38.582 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: modify_timestamp: Wed Nov 8 05:11:37 2023 2023-11-08T05:11:38.582 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring state: enabled 2023-11-08T05:11:38.582 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring mode: snapshot 2023-11-08T05:11:38.582 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring global id: de215ca4-d87f-4583-9648-dab2d031119f 2023-11-08T05:11:38.583 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring primary: true 2023-11-08T05:11:38.583 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' -n '' ']' 2023-11-08T05:11:38.584 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ stress_write_image cluster2 mirror image_41 2023-11-08T05:11:38.584 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:11:38.584 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:11:38.584 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_41 2023-11-08T05:11:38.584 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-11-08T05:11:38.585 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local duration=5 2023-11-08T05:11:38.585 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ set +e 2023-11-08T05:11:38.586 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_41 --debug-rbd=20 --debug-journaler=20 2023-11-08T05:11:38.770 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:100 IOs, 20220928 bytes 2023-11-08T05:11:38.947 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:200 IOs, 59555840 bytes 2023-11-08T05:11:39.062 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:300 IOs, 66754560 bytes 2023-11-08T05:11:39.219 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:400 IOs, 84122624 bytes 2023-11-08T05:11:39.414 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:500 IOs, 130162688 bytes 2023-11-08T05:11:39.599 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:600 IOs, 170384384 bytes 2023-11-08T05:11:39.808 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:700 IOs, 220157952 bytes 2023-11-08T05:11:40.068 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:800 IOs, 300986368 bytes 2023-11-08T05:11:40.217 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:900 IOs, 320172032 bytes 2023-11-08T05:11:40.379 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1000 IOs, 361013248 bytes 2023-11-08T05:11:40.570 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1100 IOs, 399278080 bytes 2023-11-08T05:11:40.816 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1200 IOs, 459395072 bytes 2023-11-08T05:11:41.036 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1300 IOs, 507122688 bytes 2023-11-08T05:11:41.222 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1400 IOs, 544105472 bytes 2023-11-08T05:11:41.373 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1500 IOs, 568261632 bytes 2023-11-08T05:11:41.669 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1600 IOs, 641837056 bytes 2023-11-08T05:11:41.830 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1700 IOs, 664373248 bytes 2023-11-08T05:11:41.994 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1800 IOs, 705684480 bytes 2023-11-08T05:11:42.157 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1900 IOs, 719577088 bytes 2023-11-08T05:11:42.329 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2000 IOs, 744861696 bytes 2023-11-08T05:11:42.554 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2100 IOs, 811419648 bytes 2023-11-08T05:11:42.776 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2200 IOs, 859507712 bytes 2023-11-08T05:11:42.900 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2300 IOs, 867861504 bytes 2023-11-08T05:11:43.013 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2400 IOs, 889533440 bytes 2023-11-08T05:11:43.170 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2500 IOs, 890171392 bytes 2023-11-08T05:11:43.415 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2600 IOs, 958908416 bytes 2023-11-08T05:11:43.590 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ error_code=124 2023-11-08T05:11:43.591 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ set -e 2023-11-08T05:11:43.591 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' 124 -eq 124 ']' 2023-11-08T05:11:43.591 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:11:43.591 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-11-08T05:11:43.591 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image=image_42 2023-11-08T05:11:43.592 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ create_image_and_enable_mirror cluster2 mirror image_42 snapshot 128M 2023-11-08T05:11:43.592 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:11:43.592 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:11:43.592 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:11:43.592 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:11:43.593 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_42 2023-11-08T05:11:43.593 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:11:43.593 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local mode=snapshot 2023-11-08T05:11:43.593 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' -n snapshot ']' 2023-11-08T05:11:43.593 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:11:43.594 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ create_image cluster2 mirror image_42 128M 2023-11-08T05:11:43.594 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:11:43.594 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:11:43.594 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:11:43.594 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:11:43.595 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_42 2023-11-08T05:11:43.595 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:11:43.595 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local size=128 2023-11-08T05:11:43.595 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' -n 128M ']' 2023-11-08T05:11:43.595 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ size=128M 2023-11-08T05:11:43.595 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:11:43.596 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_42 2023-11-08T05:11:43.631 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' image = image ']' 2023-11-08T05:11:43.631 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ enable_mirror cluster2 mirror image_42 snapshot 2023-11-08T05:11:43.631 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:11:43.631 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:11:43.632 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_42 2023-11-08T05:11:43.632 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local mode=snapshot 2023-11-08T05:11:43.632 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_42 snapshot 2023-11-08T05:11:44.582 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:Mirroring enabled 2023-11-08T05:11:44.590 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster=cluster2 info mirror/image_42 2023-11-08T05:11:44.622 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:rbd image 'image_42': 2023-11-08T05:11:44.622 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: size 128 MiB in 32 objects 2023-11-08T05:11:44.622 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: order 22 (4 MiB objects) 2023-11-08T05:11:44.622 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: snapshot_count: 1 2023-11-08T05:11:44.623 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: id: 143c6b179029 2023-11-08T05:11:44.623 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: block_name_prefix: rbd_data.143c6b179029 2023-11-08T05:11:44.623 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: format: 2 2023-11-08T05:11:44.623 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: features: layering, exclusive-lock 2023-11-08T05:11:44.624 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: op_features: 2023-11-08T05:11:44.624 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: flags: 2023-11-08T05:11:44.624 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: create_timestamp: Wed Nov 8 05:11:43 2023 2023-11-08T05:11:44.624 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: access_timestamp: Wed Nov 8 05:11:43 2023 2023-11-08T05:11:44.625 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: modify_timestamp: Wed Nov 8 05:11:43 2023 2023-11-08T05:11:44.625 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring state: enabled 2023-11-08T05:11:44.625 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring mode: snapshot 2023-11-08T05:11:44.625 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring global id: 2409fb55-b412-43a5-9d65-43c4ee962379 2023-11-08T05:11:44.625 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring primary: true 2023-11-08T05:11:44.626 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' -n '' ']' 2023-11-08T05:11:44.627 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ stress_write_image cluster2 mirror image_42 2023-11-08T05:11:44.627 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:11:44.627 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:11:44.627 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_42 2023-11-08T05:11:44.628 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-11-08T05:11:44.628 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local duration=6 2023-11-08T05:11:44.628 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ set +e 2023-11-08T05:11:44.628 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_42 --debug-rbd=20 --debug-journaler=20 2023-11-08T05:11:45.034 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:100 IOs, 34913280 bytes 2023-11-08T05:11:45.198 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:200 IOs, 70628352 bytes 2023-11-08T05:11:45.352 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:300 IOs, 88117248 bytes 2023-11-08T05:11:45.524 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:400 IOs, 114513920 bytes 2023-11-08T05:11:45.712 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:500 IOs, 157009920 bytes 2023-11-08T05:11:45.967 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:600 IOs, 233091072 bytes 2023-11-08T05:11:46.250 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:700 IOs, 325125120 bytes 2023-11-08T05:11:46.535 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:800 IOs, 408126464 bytes 2023-11-08T05:11:46.661 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:900 IOs, 416714752 bytes 2023-11-08T05:11:46.884 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1000 IOs, 480637952 bytes 2023-11-08T05:11:47.094 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1100 IOs, 536609792 bytes 2023-11-08T05:11:47.260 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1200 IOs, 561803264 bytes 2023-11-08T05:11:47.419 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1300 IOs, 591693824 bytes 2023-11-08T05:11:47.565 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1400 IOs, 611791872 bytes 2023-11-08T05:11:47.725 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1500 IOs, 640609280 bytes 2023-11-08T05:11:47.969 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1600 IOs, 707676160 bytes 2023-11-08T05:11:48.178 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1700 IOs, 782602240 bytes 2023-11-08T05:11:48.445 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1800 IOs, 851623936 bytes 2023-11-08T05:11:48.613 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1900 IOs, 883215360 bytes 2023-11-08T05:11:48.796 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2000 IOs, 910721024 bytes 2023-11-08T05:11:49.082 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2100 IOs, 1000562688 bytes 2023-11-08T05:11:49.218 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2200 IOs, 1013093376 bytes 2023-11-08T05:11:49.440 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2300 IOs, 1069717504 bytes 2023-11-08T05:11:49.666 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2400 IOs, 1129058304 bytes 2023-11-08T05:11:49.879 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2500 IOs, 1175725056 bytes 2023-11-08T05:11:50.001 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2600 IOs, 1197410304 bytes 2023-11-08T05:11:50.234 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2700 IOs, 1249966080 bytes 2023-11-08T05:11:50.347 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2800 IOs, 1250577408 bytes 2023-11-08T05:11:50.567 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2900 IOs, 1305937920 bytes 2023-11-08T05:11:50.634 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ error_code=124 2023-11-08T05:11:50.634 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ set -e 2023-11-08T05:11:50.635 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' 124 -eq 124 ']' 2023-11-08T05:11:50.635 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:11:50.635 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-11-08T05:11:50.635 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image=image_43 2023-11-08T05:11:50.635 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ create_image_and_enable_mirror cluster2 mirror image_43 snapshot 128M 2023-11-08T05:11:50.636 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:11:50.636 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:11:50.636 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:11:50.636 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:11:50.636 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_43 2023-11-08T05:11:50.637 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:11:50.637 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local mode=snapshot 2023-11-08T05:11:50.637 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' -n snapshot ']' 2023-11-08T05:11:50.637 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:11:50.637 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ create_image cluster2 mirror image_43 128M 2023-11-08T05:11:50.638 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:11:50.638 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:11:50.638 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:11:50.638 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:11:50.638 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_43 2023-11-08T05:11:50.639 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:11:50.639 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local size=128 2023-11-08T05:11:50.639 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' -n 128M ']' 2023-11-08T05:11:50.639 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ size=128M 2023-11-08T05:11:50.639 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:11:50.639 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_43 2023-11-08T05:11:50.676 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' image = image ']' 2023-11-08T05:11:50.677 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ enable_mirror cluster2 mirror image_43 snapshot 2023-11-08T05:11:50.677 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:11:50.677 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:11:50.677 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_43 2023-11-08T05:11:50.677 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local mode=snapshot 2023-11-08T05:11:50.678 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_43 snapshot 2023-11-08T05:11:51.648 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:Mirroring enabled 2023-11-08T05:11:51.656 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster=cluster2 info mirror/image_43 2023-11-08T05:11:51.688 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:rbd image 'image_43': 2023-11-08T05:11:51.689 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: size 128 MiB in 32 objects 2023-11-08T05:11:51.689 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: order 22 (4 MiB objects) 2023-11-08T05:11:51.689 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: snapshot_count: 1 2023-11-08T05:11:51.689 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: id: 144711c5f4bc 2023-11-08T05:11:51.690 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: block_name_prefix: rbd_data.144711c5f4bc 2023-11-08T05:11:51.690 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: format: 2 2023-11-08T05:11:51.690 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: features: layering, exclusive-lock 2023-11-08T05:11:51.690 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: op_features: 2023-11-08T05:11:51.690 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: flags: 2023-11-08T05:11:51.691 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: create_timestamp: Wed Nov 8 05:11:50 2023 2023-11-08T05:11:51.691 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: access_timestamp: Wed Nov 8 05:11:50 2023 2023-11-08T05:11:51.691 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: modify_timestamp: Wed Nov 8 05:11:50 2023 2023-11-08T05:11:51.691 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring state: enabled 2023-11-08T05:11:51.691 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring mode: snapshot 2023-11-08T05:11:51.692 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring global id: 0cc6b9f9-7eeb-414c-bd5f-685fa6d23f33 2023-11-08T05:11:51.692 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring primary: true 2023-11-08T05:11:51.693 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' -n '' ']' 2023-11-08T05:11:51.693 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ stress_write_image cluster2 mirror image_43 2023-11-08T05:11:51.693 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:11:51.694 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:11:51.694 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_43 2023-11-08T05:11:51.694 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-11-08T05:11:51.694 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local duration=12 2023-11-08T05:11:51.694 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ set +e 2023-11-08T05:11:51.695 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_43 --debug-rbd=20 --debug-journaler=20 2023-11-08T05:11:52.133 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:100 IOs, 55002112 bytes 2023-11-08T05:11:52.356 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:200 IOs, 115183616 bytes 2023-11-08T05:11:52.655 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:300 IOs, 236005376 bytes 2023-11-08T05:11:52.803 INFO:tasks.ceph.cluster2.osd.1.smithi087.stderr:2023-11-08T05:11:52.801+0000 7fbb7a4fc700 -1 reset not still connected to 0x557710953790 2023-11-08T05:11:52.908 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:400 IOs, 290716672 bytes 2023-11-08T05:11:53.097 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:500 IOs, 327798784 bytes 2023-11-08T05:11:53.246 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:600 IOs, 345458688 bytes 2023-11-08T05:11:53.444 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:700 IOs, 379674624 bytes 2023-11-08T05:11:53.700 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:800 IOs, 462739456 bytes 2023-11-08T05:11:53.838 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:900 IOs, 493527040 bytes 2023-11-08T05:11:53.990 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1000 IOs, 516374528 bytes 2023-11-08T05:11:54.173 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1100 IOs, 540023808 bytes 2023-11-08T05:11:54.288 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1200 IOs, 540644352 bytes 2023-11-08T05:11:54.416 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1300 IOs, 547504128 bytes 2023-11-08T05:11:54.728 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1400 IOs, 644193280 bytes 2023-11-08T05:11:54.964 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1500 IOs, 730489856 bytes 2023-11-08T05:11:55.295 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1600 IOs, 822340608 bytes 2023-11-08T05:11:55.509 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1700 IOs, 862321664 bytes 2023-11-08T05:11:55.724 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1800 IOs, 916284416 bytes 2023-11-08T05:11:55.858 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1900 IOs, 932444160 bytes 2023-11-08T05:11:56.093 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2000 IOs, 996797440 bytes 2023-11-08T05:11:56.324 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2100 IOs, 1090187264 bytes 2023-11-08T05:11:56.468 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2200 IOs, 1099689984 bytes 2023-11-08T05:11:56.671 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2300 IOs, 1142661120 bytes 2023-11-08T05:11:56.833 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2400 IOs, 1162407936 bytes 2023-11-08T05:11:57.064 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2500 IOs, 1220300800 bytes 2023-11-08T05:11:57.242 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2600 IOs, 1255360512 bytes 2023-11-08T05:11:57.392 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2700 IOs, 1299650560 bytes 2023-11-08T05:11:57.558 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2800 IOs, 1300272128 bytes 2023-11-08T05:11:57.693 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2900 IOs, 1313343488 bytes 2023-11-08T05:11:57.890 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3000 IOs, 1369416704 bytes 2023-11-08T05:11:58.171 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3100 IOs, 1444997120 bytes 2023-11-08T05:11:58.314 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3200 IOs, 1473053696 bytes 2023-11-08T05:11:58.534 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3300 IOs, 1526090752 bytes 2023-11-08T05:11:58.698 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3400 IOs, 1548325888 bytes 2023-11-08T05:11:58.899 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3500 IOs, 1568598016 bytes 2023-11-08T05:11:59.096 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3600 IOs, 1611478016 bytes 2023-11-08T05:11:59.514 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3700 IOs, 1716699136 bytes 2023-11-08T05:11:59.760 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3800 IOs, 1766514688 bytes 2023-11-08T05:11:59.979 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3900 IOs, 1818847232 bytes 2023-11-08T05:12:00.176 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4000 IOs, 1859454976 bytes 2023-11-08T05:12:00.365 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4100 IOs, 1905221632 bytes 2023-11-08T05:12:00.639 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4200 IOs, 1970923520 bytes 2023-11-08T05:12:00.792 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4300 IOs, 1995209728 bytes 2023-11-08T05:12:01.051 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4400 IOs, 2073936896 bytes 2023-11-08T05:12:01.219 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4500 IOs, 2093919232 bytes 2023-11-08T05:12:01.402 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4600 IOs, 2131826688 bytes 2023-11-08T05:12:01.594 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4700 IOs, 2166489088 bytes 2023-11-08T05:12:01.770 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4800 IOs, 2194298880 bytes 2023-11-08T05:12:01.928 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4900 IOs, 2216339456 bytes 2023-11-08T05:12:02.165 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5000 IOs, 2276457472 bytes 2023-11-08T05:12:02.405 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5100 IOs, 2336655360 bytes 2023-11-08T05:12:02.568 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5200 IOs, 2358793216 bytes 2023-11-08T05:12:02.702 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5300 IOs, 2370403328 bytes 2023-11-08T05:12:02.957 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5400 IOs, 2442327040 bytes 2023-11-08T05:12:03.150 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5500 IOs, 2487910400 bytes 2023-11-08T05:12:03.395 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5600 IOs, 2558873600 bytes 2023-11-08T05:12:03.577 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5700 IOs, 2599177216 bytes 2023-11-08T05:12:03.699 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ error_code=124 2023-11-08T05:12:03.699 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ set -e 2023-11-08T05:12:03.699 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' 124 -eq 124 ']' 2023-11-08T05:12:03.699 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:12:03.700 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-11-08T05:12:03.700 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image=image_44 2023-11-08T05:12:03.700 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ create_image_and_enable_mirror cluster2 mirror image_44 snapshot 128M 2023-11-08T05:12:03.700 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:12:03.700 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:12:03.701 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:12:03.701 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:12:03.701 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_44 2023-11-08T05:12:03.701 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:12:03.701 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local mode=snapshot 2023-11-08T05:12:03.702 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' -n snapshot ']' 2023-11-08T05:12:03.702 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:12:03.702 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ create_image cluster2 mirror image_44 128M 2023-11-08T05:12:03.702 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:12:03.702 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:12:03.703 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:12:03.703 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:12:03.703 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_44 2023-11-08T05:12:03.703 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:12:03.703 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local size=128 2023-11-08T05:12:03.703 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' -n 128M ']' 2023-11-08T05:12:03.704 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ size=128M 2023-11-08T05:12:03.704 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:12:03.704 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_44 2023-11-08T05:12:03.741 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' image = image ']' 2023-11-08T05:12:03.742 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ enable_mirror cluster2 mirror image_44 snapshot 2023-11-08T05:12:03.742 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:12:03.742 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:12:03.742 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_44 2023-11-08T05:12:03.743 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local mode=snapshot 2023-11-08T05:12:03.743 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_44 snapshot 2023-11-08T05:12:04.193 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:Mirroring enabled 2023-11-08T05:12:04.200 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster=cluster2 info mirror/image_44 2023-11-08T05:12:04.230 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:rbd image 'image_44': 2023-11-08T05:12:04.231 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: size 128 MiB in 32 objects 2023-11-08T05:12:04.231 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: order 22 (4 MiB objects) 2023-11-08T05:12:04.231 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: snapshot_count: 1 2023-11-08T05:12:04.231 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: id: 14509941062c 2023-11-08T05:12:04.231 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: block_name_prefix: rbd_data.14509941062c 2023-11-08T05:12:04.231 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: format: 2 2023-11-08T05:12:04.232 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: features: layering, exclusive-lock 2023-11-08T05:12:04.232 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: op_features: 2023-11-08T05:12:04.232 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: flags: 2023-11-08T05:12:04.232 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: create_timestamp: Wed Nov 8 05:12:03 2023 2023-11-08T05:12:04.232 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: access_timestamp: Wed Nov 8 05:12:03 2023 2023-11-08T05:12:04.232 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: modify_timestamp: Wed Nov 8 05:12:03 2023 2023-11-08T05:12:04.233 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring state: enabled 2023-11-08T05:12:04.233 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring mode: snapshot 2023-11-08T05:12:04.233 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring global id: 211c5392-1879-4787-844d-06dee2447ed6 2023-11-08T05:12:04.233 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring primary: true 2023-11-08T05:12:04.234 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' -n '' ']' 2023-11-08T05:12:04.234 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ stress_write_image cluster2 mirror image_44 2023-11-08T05:12:04.235 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:12:04.235 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:12:04.235 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_44 2023-11-08T05:12:04.235 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-11-08T05:12:04.236 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local duration=13 2023-11-08T05:12:04.236 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ set +e 2023-11-08T05:12:04.236 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_44 --debug-rbd=20 --debug-journaler=20 2023-11-08T05:12:04.491 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:100 IOs, 53620736 bytes 2023-11-08T05:12:04.620 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:200 IOs, 61910016 bytes 2023-11-08T05:12:04.878 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:300 IOs, 148796416 bytes 2023-11-08T05:12:05.011 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:400 IOs, 165798912 bytes 2023-11-08T05:12:05.245 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:500 IOs, 236302336 bytes 2023-11-08T05:12:05.428 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:600 IOs, 272036864 bytes 2023-11-08T05:12:05.533 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:700 IOs, 272666624 bytes 2023-11-08T05:12:05.739 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:800 IOs, 316359680 bytes 2023-11-08T05:12:05.947 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:900 IOs, 366557184 bytes 2023-11-08T05:12:06.102 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1000 IOs, 389786624 bytes 2023-11-08T05:12:06.323 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1100 IOs, 444168192 bytes 2023-11-08T05:12:06.460 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1200 IOs, 465186816 bytes 2023-11-08T05:12:06.629 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1300 IOs, 494336000 bytes 2023-11-08T05:12:06.853 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1400 IOs, 567772160 bytes 2023-11-08T05:12:07.070 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1500 IOs, 617926656 bytes 2023-11-08T05:12:07.176 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.0 2023-11-08T05:12:07.176 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Sent signal 15 2023-11-08T05:12:07.177 INFO:tasks.rbd_mirror_thrash:waiting for 7 secs before reviving daemons 2023-11-08T05:12:07.178 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi087.stderr:2023-11-08T05:12:07.175+0000 7f1520ecf700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 130123) UID: 1000 2023-11-08T05:12:07.201 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1600 IOs, 618552320 bytes 2023-11-08T05:12:07.385 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi087.stderr:daemon-helper: command failed with exit status 1 2023-11-08T05:12:07.386 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1700 IOs, 643216384 bytes 2023-11-08T05:12:07.513 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1800 IOs, 651003904 bytes 2023-11-08T05:12:07.682 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1900 IOs, 680682496 bytes 2023-11-08T05:12:07.797 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2000 IOs, 684735488 bytes 2023-11-08T05:12:07.915 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2100 IOs, 696135680 bytes 2023-11-08T05:12:08.171 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2200 IOs, 765402112 bytes 2023-11-08T05:12:08.346 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2300 IOs, 799036416 bytes 2023-11-08T05:12:08.560 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2400 IOs, 851769344 bytes 2023-11-08T05:12:08.701 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2500 IOs, 870257664 bytes 2023-11-08T05:12:08.896 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2600 IOs, 911857664 bytes 2023-11-08T05:12:09.083 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2700 IOs, 964200448 bytes 2023-11-08T05:12:09.356 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2800 IOs, 1031480320 bytes 2023-11-08T05:12:09.496 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2900 IOs, 1043153920 bytes 2023-11-08T05:12:09.644 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3000 IOs, 1064476672 bytes 2023-11-08T05:12:09.837 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3100 IOs, 1105683456 bytes 2023-11-08T05:12:09.995 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3200 IOs, 1143532544 bytes 2023-11-08T05:12:10.242 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3300 IOs, 1189487616 bytes 2023-11-08T05:12:10.374 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3400 IOs, 1199964160 bytes 2023-11-08T05:12:10.613 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3500 IOs, 1265593344 bytes 2023-11-08T05:12:10.795 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3600 IOs, 1293067264 bytes 2023-11-08T05:12:11.012 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3700 IOs, 1366019072 bytes 2023-11-08T05:12:11.227 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3800 IOs, 1398923264 bytes 2023-11-08T05:12:11.383 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3900 IOs, 1435329536 bytes 2023-11-08T05:12:11.683 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4000 IOs, 1548193792 bytes 2023-11-08T05:12:11.878 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4100 IOs, 1573633024 bytes 2023-11-08T05:12:12.045 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4200 IOs, 1602482176 bytes 2023-11-08T05:12:12.194 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4300 IOs, 1627822080 bytes 2023-11-08T05:12:12.441 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4400 IOs, 1686597632 bytes 2023-11-08T05:12:12.637 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4500 IOs, 1737311232 bytes 2023-11-08T05:12:12.887 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4600 IOs, 1818603520 bytes 2023-11-08T05:12:13.062 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4700 IOs, 1844326400 bytes 2023-11-08T05:12:13.242 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4800 IOs, 1877147648 bytes 2023-11-08T05:12:13.421 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4900 IOs, 1910427648 bytes 2023-11-08T05:12:13.618 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5000 IOs, 1954317312 bytes 2023-11-08T05:12:13.802 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5100 IOs, 1992111104 bytes 2023-11-08T05:12:13.966 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5200 IOs, 2024364032 bytes 2023-11-08T05:12:14.177 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.0 2023-11-08T05:12:14.178 INFO:teuthology.orchestra.run:waiting for 600 2023-11-08T05:12:14.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-08T05:12:14.178 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.0 2023-11-08T05:12:14.179 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2023-11-08T05:12:14.179 DEBUG:teuthology.orchestra.run.smithi087:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2023-11-08T05:12:14.182 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2023-11-08T05:12:14.182 INFO:tasks.rbd_mirror_thrash:waiting for 103 secs before thrashing 2023-11-08T05:12:14.189 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5300 IOs, 2074439680 bytes 2023-11-08T05:12:14.327 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5400 IOs, 2079504384 bytes 2023-11-08T05:12:14.530 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5500 IOs, 2133848064 bytes 2023-11-08T05:12:14.719 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5600 IOs, 2168501248 bytes 2023-11-08T05:12:14.930 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5700 IOs, 2212012032 bytes 2023-11-08T05:12:15.148 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5800 IOs, 2263896064 bytes 2023-11-08T05:12:15.278 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5900 IOs, 2275934208 bytes 2023-11-08T05:12:15.483 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:6000 IOs, 2325184512 bytes 2023-11-08T05:12:15.749 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:6100 IOs, 2404110336 bytes 2023-11-08T05:12:15.983 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:6200 IOs, 2463331328 bytes 2023-11-08T05:12:16.146 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:6300 IOs, 2491255808 bytes 2023-11-08T05:12:16.418 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:6400 IOs, 2601314304 bytes 2023-11-08T05:12:16.757 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:6500 IOs, 2697932800 bytes 2023-11-08T05:12:17.016 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:6600 IOs, 2767515648 bytes 2023-11-08T05:12:17.185 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:6700 IOs, 2795843584 bytes 2023-11-08T05:12:17.241 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ error_code=124 2023-11-08T05:12:17.241 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ set -e 2023-11-08T05:12:17.242 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' 124 -eq 124 ']' 2023-11-08T05:12:17.242 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:12:17.242 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-11-08T05:12:17.242 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image=image_45 2023-11-08T05:12:17.242 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ create_image_and_enable_mirror cluster2 mirror image_45 snapshot 128M 2023-11-08T05:12:17.243 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:12:17.243 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:12:17.243 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:12:17.243 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:12:17.243 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_45 2023-11-08T05:12:17.244 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:12:17.244 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local mode=snapshot 2023-11-08T05:12:17.244 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' -n snapshot ']' 2023-11-08T05:12:17.244 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:12:17.244 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ create_image cluster2 mirror image_45 128M 2023-11-08T05:12:17.244 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:12:17.245 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:12:17.245 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:12:17.245 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:12:17.245 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_45 2023-11-08T05:12:17.245 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:12:17.246 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local size=128 2023-11-08T05:12:17.246 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' -n 128M ']' 2023-11-08T05:12:17.246 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ size=128M 2023-11-08T05:12:17.246 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:12:17.246 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_45 2023-11-08T05:12:17.282 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' image = image ']' 2023-11-08T05:12:17.282 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ enable_mirror cluster2 mirror image_45 snapshot 2023-11-08T05:12:17.282 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:12:17.283 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:12:17.283 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_45 2023-11-08T05:12:17.283 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local mode=snapshot 2023-11-08T05:12:17.283 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_45 snapshot 2023-11-08T05:12:18.282 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:Mirroring enabled 2023-11-08T05:12:18.289 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster=cluster2 info mirror/image_45 2023-11-08T05:12:18.322 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:rbd image 'image_45': 2023-11-08T05:12:18.322 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: size 128 MiB in 32 objects 2023-11-08T05:12:18.322 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: order 22 (4 MiB objects) 2023-11-08T05:12:18.322 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: snapshot_count: 1 2023-11-08T05:12:18.323 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: id: 145d4d7d9316 2023-11-08T05:12:18.323 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: block_name_prefix: rbd_data.145d4d7d9316 2023-11-08T05:12:18.323 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: format: 2 2023-11-08T05:12:18.323 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: features: layering, exclusive-lock 2023-11-08T05:12:18.323 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: op_features: 2023-11-08T05:12:18.323 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: flags: 2023-11-08T05:12:18.324 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: create_timestamp: Wed Nov 8 05:12:17 2023 2023-11-08T05:12:18.324 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: access_timestamp: Wed Nov 8 05:12:17 2023 2023-11-08T05:12:18.324 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: modify_timestamp: Wed Nov 8 05:12:17 2023 2023-11-08T05:12:18.324 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring state: enabled 2023-11-08T05:12:18.324 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring mode: snapshot 2023-11-08T05:12:18.325 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring global id: 20b7d4e5-6730-4919-b471-a15668fab78b 2023-11-08T05:12:18.325 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring primary: true 2023-11-08T05:12:18.326 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' -n '' ']' 2023-11-08T05:12:18.326 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ stress_write_image cluster2 mirror image_45 2023-11-08T05:12:18.326 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:12:18.326 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:12:18.327 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_45 2023-11-08T05:12:18.327 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-11-08T05:12:18.327 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local duration=10 2023-11-08T05:12:18.328 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ set +e 2023-11-08T05:12:18.328 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_45 --debug-rbd=20 --debug-journaler=20 2023-11-08T05:12:18.480 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:100 IOs, 6844416 bytes 2023-11-08T05:12:18.588 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:200 IOs, 20436992 bytes 2023-11-08T05:12:18.781 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:300 IOs, 53219328 bytes 2023-11-08T05:12:19.028 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:400 IOs, 121068544 bytes 2023-11-08T05:12:19.177 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:500 IOs, 138762240 bytes 2023-11-08T05:12:19.397 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:600 IOs, 191899648 bytes 2023-11-08T05:12:19.607 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:700 IOs, 247108608 bytes 2023-11-08T05:12:19.742 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:800 IOs, 265298944 bytes 2023-11-08T05:12:19.885 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:900 IOs, 282980352 bytes 2023-11-08T05:12:20.129 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1000 IOs, 330683392 bytes 2023-11-08T05:12:20.363 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1100 IOs, 387707904 bytes 2023-11-08T05:12:20.489 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1200 IOs, 395693056 bytes 2023-11-08T05:12:20.655 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1300 IOs, 414701568 bytes 2023-11-08T05:12:20.952 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1400 IOs, 498560000 bytes 2023-11-08T05:12:21.191 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1500 IOs, 539267072 bytes 2023-11-08T05:12:21.513 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1600 IOs, 648123392 bytes 2023-11-08T05:12:21.669 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1700 IOs, 668975104 bytes 2023-11-08T05:12:21.832 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1800 IOs, 693708800 bytes 2023-11-08T05:12:22.053 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1900 IOs, 755004416 bytes 2023-11-08T05:12:22.371 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2000 IOs, 851307520 bytes 2023-11-08T05:12:22.564 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2100 IOs, 890893312 bytes 2023-11-08T05:12:22.806 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2200 IOs, 949738496 bytes 2023-11-08T05:12:22.943 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2300 IOs, 966983680 bytes 2023-11-08T05:12:23.184 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2400 IOs, 1030405120 bytes 2023-11-08T05:12:23.342 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2500 IOs, 1055047680 bytes 2023-11-08T05:12:23.476 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2600 IOs, 1073038336 bytes 2023-11-08T05:12:23.715 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2700 IOs, 1136358400 bytes 2023-11-08T05:12:23.924 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2800 IOs, 1192505344 bytes 2023-11-08T05:12:24.095 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2900 IOs, 1214652416 bytes 2023-11-08T05:12:24.320 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3000 IOs, 1276514304 bytes 2023-11-08T05:12:24.493 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3100 IOs, 1310892032 bytes 2023-11-08T05:12:24.730 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3200 IOs, 1374967808 bytes 2023-11-08T05:12:24.891 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3300 IOs, 1402756096 bytes 2023-11-08T05:12:25.095 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3400 IOs, 1448487936 bytes 2023-11-08T05:12:25.252 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3500 IOs, 1484909568 bytes 2023-11-08T05:12:25.413 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3600 IOs, 1496976384 bytes 2023-11-08T05:12:25.596 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3700 IOs, 1535375360 bytes 2023-11-08T05:12:25.769 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3800 IOs, 1571614720 bytes 2023-11-08T05:12:26.041 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3900 IOs, 1667727360 bytes 2023-11-08T05:12:26.239 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4000 IOs, 1691578368 bytes 2023-11-08T05:12:26.386 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4100 IOs, 1720482816 bytes 2023-11-08T05:12:26.576 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4200 IOs, 1754019840 bytes 2023-11-08T05:12:26.805 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4300 IOs, 1817519104 bytes 2023-11-08T05:12:26.965 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4400 IOs, 1842376704 bytes 2023-11-08T05:12:27.130 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4500 IOs, 1881504768 bytes 2023-11-08T05:12:27.308 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4600 IOs, 1912008704 bytes 2023-11-08T05:12:27.510 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4700 IOs, 1933374464 bytes 2023-11-08T05:12:27.628 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4800 IOs, 1937908736 bytes 2023-11-08T05:12:27.784 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4900 IOs, 1956168704 bytes 2023-11-08T05:12:27.948 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5000 IOs, 1986709504 bytes 2023-11-08T05:12:28.158 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5100 IOs, 2040689664 bytes 2023-11-08T05:12:28.267 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5200 IOs, 2041324544 bytes 2023-11-08T05:12:28.339 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ error_code=124 2023-11-08T05:12:28.339 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ set -e 2023-11-08T05:12:28.340 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' 124 -eq 124 ']' 2023-11-08T05:12:28.340 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:12:28.340 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-11-08T05:12:28.340 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image=image_46 2023-11-08T05:12:28.340 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ create_image_and_enable_mirror cluster2 mirror image_46 snapshot 128M 2023-11-08T05:12:28.341 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:12:28.341 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:12:28.341 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:12:28.341 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:12:28.341 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_46 2023-11-08T05:12:28.342 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:12:28.342 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local mode=snapshot 2023-11-08T05:12:28.342 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' -n snapshot ']' 2023-11-08T05:12:28.342 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:12:28.342 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ create_image cluster2 mirror image_46 128M 2023-11-08T05:12:28.343 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:12:28.343 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:12:28.343 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:12:28.343 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:12:28.343 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_46 2023-11-08T05:12:28.343 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:12:28.344 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local size=128 2023-11-08T05:12:28.344 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' -n 128M ']' 2023-11-08T05:12:28.344 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ size=128M 2023-11-08T05:12:28.344 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:12:28.344 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_46 2023-11-08T05:12:28.380 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' image = image ']' 2023-11-08T05:12:28.380 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ enable_mirror cluster2 mirror image_46 snapshot 2023-11-08T05:12:28.380 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:12:28.381 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:12:28.381 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_46 2023-11-08T05:12:28.381 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local mode=snapshot 2023-11-08T05:12:28.381 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_46 snapshot 2023-11-08T05:12:29.359 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:Mirroring enabled 2023-11-08T05:12:29.369 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster=cluster2 info mirror/image_46 2023-11-08T05:12:29.402 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:rbd image 'image_46': 2023-11-08T05:12:29.403 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: size 128 MiB in 32 objects 2023-11-08T05:12:29.403 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: order 22 (4 MiB objects) 2023-11-08T05:12:29.403 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: snapshot_count: 1 2023-11-08T05:12:29.403 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: id: 1465264e4997 2023-11-08T05:12:29.403 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: block_name_prefix: rbd_data.1465264e4997 2023-11-08T05:12:29.404 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: format: 2 2023-11-08T05:12:29.404 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: features: layering, exclusive-lock 2023-11-08T05:12:29.404 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: op_features: 2023-11-08T05:12:29.404 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: flags: 2023-11-08T05:12:29.405 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: create_timestamp: Wed Nov 8 05:12:28 2023 2023-11-08T05:12:29.405 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: access_timestamp: Wed Nov 8 05:12:28 2023 2023-11-08T05:12:29.405 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: modify_timestamp: Wed Nov 8 05:12:28 2023 2023-11-08T05:12:29.406 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring state: enabled 2023-11-08T05:12:29.406 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring mode: snapshot 2023-11-08T05:12:29.406 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring global id: ab12d762-6014-4867-a6ea-6c7d605eb202 2023-11-08T05:12:29.406 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring primary: true 2023-11-08T05:12:29.408 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' -n '' ']' 2023-11-08T05:12:29.408 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ stress_write_image cluster2 mirror image_46 2023-11-08T05:12:29.409 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:12:29.409 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:12:29.409 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_46 2023-11-08T05:12:29.410 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-11-08T05:12:29.410 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local duration=12 2023-11-08T05:12:29.410 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ set +e 2023-11-08T05:12:29.411 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_46 --debug-rbd=20 --debug-journaler=20 2023-11-08T05:12:29.602 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:100 IOs, 27021312 bytes 2023-11-08T05:12:29.839 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:200 IOs, 109661184 bytes 2023-11-08T05:12:30.016 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:300 IOs, 134973440 bytes 2023-11-08T05:12:30.183 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:400 IOs, 167877632 bytes 2023-11-08T05:12:30.291 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:500 IOs, 168495104 bytes 2023-11-08T05:12:30.462 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:600 IOs, 206765056 bytes 2023-11-08T05:12:30.633 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:700 IOs, 235695104 bytes 2023-11-08T05:12:30.864 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:800 IOs, 296844288 bytes 2023-11-08T05:12:31.080 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.0 2023-11-08T05:12:31.080 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Sent signal 15 2023-11-08T05:12:31.081 INFO:tasks.rbd_mirror_thrash:waiting for 2 secs before reviving daemons 2023-11-08T05:12:31.081 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi087.stderr:2023-11-08T05:12:31.079+0000 7f1918286700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 (PID: 131126) UID: 1000 2023-11-08T05:12:31.094 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:900 IOs, 356687872 bytes 2023-11-08T05:12:31.296 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1000 IOs, 393998336 bytes 2023-11-08T05:12:31.482 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi087.stderr:daemon-helper: command failed with exit status 1 2023-11-08T05:12:31.628 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1100 IOs, 497395712 bytes 2023-11-08T05:12:31.778 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1200 IOs, 519291904 bytes 2023-11-08T05:12:31.917 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1300 IOs, 534052864 bytes 2023-11-08T05:12:32.076 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1400 IOs, 568710144 bytes 2023-11-08T05:12:32.323 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1500 IOs, 638418944 bytes 2023-11-08T05:12:32.490 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1600 IOs, 661008384 bytes 2023-11-08T05:12:32.614 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1700 IOs, 668193792 bytes 2023-11-08T05:12:32.744 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1800 IOs, 678266880 bytes 2023-11-08T05:12:32.853 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1900 IOs, 680700928 bytes 2023-11-08T05:12:33.033 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2000 IOs, 716445696 bytes 2023-11-08T05:12:33.082 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.0 2023-11-08T05:12:33.082 INFO:teuthology.orchestra.run:waiting for 600 2023-11-08T05:12:33.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-08T05:12:33.083 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.0 2023-11-08T05:12:33.083 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Restarting daemon 2023-11-08T05:12:33.083 DEBUG:teuthology.orchestra.run.smithi087:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 2023-11-08T05:12:33.089 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Started 2023-11-08T05:12:33.090 INFO:tasks.rbd_mirror_thrash:waiting for 88 secs before thrashing 2023-11-08T05:12:33.296 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2100 IOs, 790078464 bytes 2023-11-08T05:12:33.513 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2200 IOs, 839947264 bytes 2023-11-08T05:12:33.705 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2300 IOs, 888690688 bytes 2023-11-08T05:12:33.918 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2400 IOs, 939508736 bytes 2023-11-08T05:12:34.184 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2500 IOs, 1016249344 bytes 2023-11-08T05:12:34.364 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2600 IOs, 1051756544 bytes 2023-11-08T05:12:34.540 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2700 IOs, 1086885888 bytes 2023-11-08T05:12:34.895 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2800 IOs, 1195644928 bytes 2023-11-08T05:12:35.061 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2900 IOs, 1222343680 bytes 2023-11-08T05:12:35.252 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3000 IOs, 1263671296 bytes 2023-11-08T05:12:35.389 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3100 IOs, 1276964864 bytes 2023-11-08T05:12:35.658 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3200 IOs, 1349633024 bytes 2023-11-08T05:12:35.832 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3300 IOs, 1395984384 bytes 2023-11-08T05:12:35.996 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3400 IOs, 1405946880 bytes 2023-11-08T05:12:36.184 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3500 IOs, 1451865088 bytes 2023-11-08T05:12:36.349 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3600 IOs, 1480960000 bytes 2023-11-08T05:12:36.530 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3700 IOs, 1533604864 bytes 2023-11-08T05:12:36.743 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3800 IOs, 1573727232 bytes 2023-11-08T05:12:36.920 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3900 IOs, 1628360704 bytes 2023-11-08T05:12:37.146 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4000 IOs, 1661181952 bytes 2023-11-08T05:12:37.291 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4100 IOs, 1672578048 bytes 2023-11-08T05:12:37.502 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4200 IOs, 1702906880 bytes 2023-11-08T05:12:37.652 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4300 IOs, 1712145408 bytes 2023-11-08T05:12:37.836 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4400 IOs, 1746692096 bytes 2023-11-08T05:12:38.022 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4500 IOs, 1780408320 bytes 2023-11-08T05:12:38.237 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4600 IOs, 1818588160 bytes 2023-11-08T05:12:38.378 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4700 IOs, 1819163648 bytes 2023-11-08T05:12:38.599 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4800 IOs, 1878593536 bytes 2023-11-08T05:12:38.745 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4900 IOs, 1895916544 bytes 2023-11-08T05:12:38.907 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5000 IOs, 1920141312 bytes 2023-11-08T05:12:39.261 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5100 IOs, 2038452224 bytes 2023-11-08T05:12:39.455 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5200 IOs, 2091768832 bytes 2023-11-08T05:12:39.657 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5300 IOs, 2128669696 bytes 2023-11-08T05:12:39.846 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5400 IOs, 2171500544 bytes 2023-11-08T05:12:40.038 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5500 IOs, 2214575104 bytes 2023-11-08T05:12:40.183 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5600 IOs, 2233059328 bytes 2023-11-08T05:12:40.341 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5700 IOs, 2252206080 bytes 2023-11-08T05:12:40.530 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5800 IOs, 2302709760 bytes 2023-11-08T05:12:40.722 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5900 IOs, 2331675648 bytes 2023-11-08T05:12:40.959 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:6000 IOs, 2406483968 bytes 2023-11-08T05:12:41.250 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:6100 IOs, 2492938240 bytes 2023-11-08T05:12:41.413 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ error_code=124 2023-11-08T05:12:41.414 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ set -e 2023-11-08T05:12:41.414 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' 124 -eq 124 ']' 2023-11-08T05:12:41.414 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:12:41.414 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-11-08T05:12:41.415 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image=image_47 2023-11-08T05:12:41.415 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ create_image_and_enable_mirror cluster2 mirror image_47 snapshot 128M 2023-11-08T05:12:41.415 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:12:41.415 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:12:41.415 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:12:41.416 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:12:41.416 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_47 2023-11-08T05:12:41.416 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:12:41.416 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local mode=snapshot 2023-11-08T05:12:41.416 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' -n snapshot ']' 2023-11-08T05:12:41.416 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:12:41.417 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ create_image cluster2 mirror image_47 128M 2023-11-08T05:12:41.417 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:12:41.417 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:12:41.417 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:12:41.417 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:12:41.418 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_47 2023-11-08T05:12:41.418 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:12:41.418 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local size=128 2023-11-08T05:12:41.418 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' -n 128M ']' 2023-11-08T05:12:41.418 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ size=128M 2023-11-08T05:12:41.419 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:12:41.419 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_47 2023-11-08T05:12:41.452 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' image = image ']' 2023-11-08T05:12:41.453 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ enable_mirror cluster2 mirror image_47 snapshot 2023-11-08T05:12:41.453 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:12:41.453 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:12:41.453 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_47 2023-11-08T05:12:41.453 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local mode=snapshot 2023-11-08T05:12:41.454 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_47 snapshot 2023-11-08T05:12:42.457 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:Mirroring enabled 2023-11-08T05:12:42.466 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster=cluster2 info mirror/image_47 2023-11-08T05:12:42.500 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:rbd image 'image_47': 2023-11-08T05:12:42.501 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: size 128 MiB in 32 objects 2023-11-08T05:12:42.501 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: order 22 (4 MiB objects) 2023-11-08T05:12:42.501 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: snapshot_count: 1 2023-11-08T05:12:42.501 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: id: 1475ee9a2b1e 2023-11-08T05:12:42.502 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: block_name_prefix: rbd_data.1475ee9a2b1e 2023-11-08T05:12:42.502 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: format: 2 2023-11-08T05:12:42.502 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: features: layering, exclusive-lock 2023-11-08T05:12:42.502 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: op_features: 2023-11-08T05:12:42.502 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: flags: 2023-11-08T05:12:42.503 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: create_timestamp: Wed Nov 8 05:12:41 2023 2023-11-08T05:12:42.503 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: access_timestamp: Wed Nov 8 05:12:41 2023 2023-11-08T05:12:42.503 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: modify_timestamp: Wed Nov 8 05:12:41 2023 2023-11-08T05:12:42.503 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring state: enabled 2023-11-08T05:12:42.504 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring mode: snapshot 2023-11-08T05:12:42.504 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring global id: 8f6b2565-b6aa-4361-bc36-f1a120eb271e 2023-11-08T05:12:42.504 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring primary: true 2023-11-08T05:12:42.505 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' -n '' ']' 2023-11-08T05:12:42.506 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ stress_write_image cluster2 mirror image_47 2023-11-08T05:12:42.506 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:12:42.506 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:12:42.506 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_47 2023-11-08T05:12:42.507 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-11-08T05:12:42.507 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local duration=13 2023-11-08T05:12:42.507 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ set +e 2023-11-08T05:12:42.507 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_47 --debug-rbd=20 --debug-journaler=20 2023-11-08T05:12:42.656 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:100 IOs, 583680 bytes 2023-11-08T05:12:42.808 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:200 IOs, 46073856 bytes 2023-11-08T05:12:43.068 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:300 IOs, 106854400 bytes 2023-11-08T05:12:43.223 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:400 IOs, 131289088 bytes 2023-11-08T05:12:43.375 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:500 IOs, 150873088 bytes 2023-11-08T05:12:43.539 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:600 IOs, 182156288 bytes 2023-11-08T05:12:43.698 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:700 IOs, 207224832 bytes 2023-11-08T05:12:43.879 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:800 IOs, 244269056 bytes 2023-11-08T05:12:44.080 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:900 IOs, 294673408 bytes 2023-11-08T05:12:44.298 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1000 IOs, 348536832 bytes 2023-11-08T05:12:44.455 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1100 IOs, 369369088 bytes 2023-11-08T05:12:44.638 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1200 IOs, 408148992 bytes 2023-11-08T05:12:44.794 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1300 IOs, 432117760 bytes 2023-11-08T05:12:44.930 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1400 IOs, 443307008 bytes 2023-11-08T05:12:45.150 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1500 IOs, 502443008 bytes 2023-11-08T05:12:45.329 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1600 IOs, 538635264 bytes 2023-11-08T05:12:45.554 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1700 IOs, 596785152 bytes 2023-11-08T05:12:45.672 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1800 IOs, 604619776 bytes 2023-11-08T05:12:45.794 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1900 IOs, 608272384 bytes 2023-11-08T05:12:46.133 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2000 IOs, 721282048 bytes 2023-11-08T05:12:46.274 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2100 IOs, 736898048 bytes 2023-11-08T05:12:46.404 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2200 IOs, 749236224 bytes 2023-11-08T05:12:46.550 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2300 IOs, 770677760 bytes 2023-11-08T05:12:46.677 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2400 IOs, 780095488 bytes 2023-11-08T05:12:46.856 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2500 IOs, 836662272 bytes 2023-11-08T05:12:47.098 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2600 IOs, 882543616 bytes 2023-11-08T05:12:47.373 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2700 IOs, 971751424 bytes 2023-11-08T05:12:47.523 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2800 IOs, 998008832 bytes 2023-11-08T05:12:47.742 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2900 IOs, 1057145856 bytes 2023-11-08T05:12:47.976 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3000 IOs, 1125559296 bytes 2023-11-08T05:12:48.082 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3100 IOs, 1131334656 bytes 2023-11-08T05:12:48.328 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3200 IOs, 1193705472 bytes 2023-11-08T05:12:48.522 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3300 IOs, 1233482752 bytes 2023-11-08T05:12:48.777 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3400 IOs, 1309407232 bytes 2023-11-08T05:12:49.003 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3500 IOs, 1367751680 bytes 2023-11-08T05:12:49.436 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3600 IOs, 1453656064 bytes 2023-11-08T05:12:49.690 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3700 IOs, 1530052608 bytes 2023-11-08T05:12:49.906 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3800 IOs, 1577162752 bytes 2023-11-08T05:12:50.126 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3900 IOs, 1626976256 bytes 2023-11-08T05:12:50.285 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4000 IOs, 1651878912 bytes 2023-11-08T05:12:50.528 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4100 IOs, 1719103488 bytes 2023-11-08T05:12:50.722 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4200 IOs, 1761886208 bytes 2023-11-08T05:12:50.939 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4300 IOs, 1818790912 bytes 2023-11-08T05:12:51.050 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4400 IOs, 1825118208 bytes 2023-11-08T05:12:51.264 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4500 IOs, 1868289024 bytes 2023-11-08T05:12:51.393 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4600 IOs, 1884028928 bytes 2023-11-08T05:12:51.550 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4700 IOs, 1928295424 bytes 2023-11-08T05:12:51.851 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4800 IOs, 2008017920 bytes 2023-11-08T05:12:52.012 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4900 IOs, 2036321280 bytes 2023-11-08T05:12:52.152 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5000 IOs, 2054155264 bytes 2023-11-08T05:12:52.353 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5100 IOs, 2099538944 bytes 2023-11-08T05:12:52.569 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5200 IOs, 2154981376 bytes 2023-11-08T05:12:52.701 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5300 IOs, 2168247296 bytes 2023-11-08T05:12:52.902 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5400 IOs, 2219340800 bytes 2023-11-08T05:12:53.166 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5500 IOs, 2296208384 bytes 2023-11-08T05:12:53.443 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5600 IOs, 2376542208 bytes 2023-11-08T05:12:53.548 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5700 IOs, 2379772928 bytes 2023-11-08T05:12:53.665 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5800 IOs, 2380358656 bytes 2023-11-08T05:12:53.928 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5900 IOs, 2437531648 bytes 2023-11-08T05:12:54.182 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:6000 IOs, 2548888576 bytes 2023-11-08T05:12:54.375 INFO:tasks.ceph.cluster2.osd.1.smithi087.stderr:2023-11-08T05:12:54.372+0000 7fbb7a4fc700 -1 reset not still connected to 0x5576fc5df1e0 2023-11-08T05:12:54.376 INFO:tasks.ceph.cluster2.osd.1.smithi087.stderr:2023-11-08T05:12:54.372+0000 7fbb7a4fc700 -1 reset not still connected to 0x5576fd0d4f70 2023-11-08T05:12:54.376 INFO:tasks.ceph.cluster2.osd.1.smithi087.stderr:2023-11-08T05:12:54.374+0000 7fbb7a4fc700 -1 reset not still connected to 0x5576fdcac750 2023-11-08T05:12:54.377 INFO:tasks.ceph.cluster2.osd.1.smithi087.stderr:2023-11-08T05:12:54.374+0000 7fbb7a4fc700 -1 reset not still connected to 0x5576ffb3bad0 2023-11-08T05:12:54.378 INFO:tasks.ceph.cluster2.osd.1.smithi087.stderr:2023-11-08T05:12:54.375+0000 7fbb7a4fc700 -1 reset not still connected to 0x557716cb0270 2023-11-08T05:12:54.378 INFO:tasks.ceph.cluster2.osd.1.smithi087.stderr:2023-11-08T05:12:54.375+0000 7fbb7a4fc700 -1 reset not still connected to 0x557728a596c0 2023-11-08T05:12:54.522 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:6100 IOs, 2628149248 bytes 2023-11-08T05:12:54.680 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:6200 IOs, 2655611904 bytes 2023-11-08T05:12:54.893 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:6300 IOs, 2702502912 bytes 2023-11-08T05:12:55.073 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:6400 IOs, 2742719488 bytes 2023-11-08T05:12:55.308 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:6500 IOs, 2800637952 bytes 2023-11-08T05:12:55.513 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ error_code=124 2023-11-08T05:12:55.514 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ set -e 2023-11-08T05:12:55.514 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' 124 -eq 124 ']' 2023-11-08T05:12:55.514 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:12:55.515 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-11-08T05:12:55.515 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image=image_48 2023-11-08T05:12:55.515 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ create_image_and_enable_mirror cluster2 mirror image_48 snapshot 128M 2023-11-08T05:12:55.516 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:12:55.516 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:12:55.516 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:12:55.516 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:12:55.516 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_48 2023-11-08T05:12:55.517 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:12:55.517 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local mode=snapshot 2023-11-08T05:12:55.517 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' -n snapshot ']' 2023-11-08T05:12:55.517 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:12:55.517 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ create_image cluster2 mirror image_48 128M 2023-11-08T05:12:55.517 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:12:55.518 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:12:55.518 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:12:55.518 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:12:55.518 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_48 2023-11-08T05:12:55.519 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:12:55.519 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local size=128 2023-11-08T05:12:55.520 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' -n 128M ']' 2023-11-08T05:12:55.520 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ size=128M 2023-11-08T05:12:55.520 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:12:55.520 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_48 2023-11-08T05:12:55.552 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' image = image ']' 2023-11-08T05:12:55.553 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ enable_mirror cluster2 mirror image_48 snapshot 2023-11-08T05:12:55.553 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:12:55.553 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:12:55.553 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_48 2023-11-08T05:12:55.554 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local mode=snapshot 2023-11-08T05:12:55.554 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_48 snapshot 2023-11-08T05:12:56.549 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:Mirroring enabled 2023-11-08T05:12:56.557 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster=cluster2 info mirror/image_48 2023-11-08T05:12:56.590 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:rbd image 'image_48': 2023-11-08T05:12:56.591 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: size 128 MiB in 32 objects 2023-11-08T05:12:56.591 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: order 22 (4 MiB objects) 2023-11-08T05:12:56.591 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: snapshot_count: 1 2023-11-08T05:12:56.591 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: id: 147e8b5c4877 2023-11-08T05:12:56.591 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: block_name_prefix: rbd_data.147e8b5c4877 2023-11-08T05:12:56.592 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: format: 2 2023-11-08T05:12:56.592 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: features: layering, exclusive-lock 2023-11-08T05:12:56.592 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: op_features: 2023-11-08T05:12:56.592 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: flags: 2023-11-08T05:12:56.592 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: create_timestamp: Wed Nov 8 05:12:55 2023 2023-11-08T05:12:56.593 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: access_timestamp: Wed Nov 8 05:12:55 2023 2023-11-08T05:12:56.593 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: modify_timestamp: Wed Nov 8 05:12:55 2023 2023-11-08T05:12:56.593 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring state: enabled 2023-11-08T05:12:56.593 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring mode: snapshot 2023-11-08T05:12:56.593 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring global id: c3d37bcb-5eef-4c9f-a31c-472ae9f8d4c1 2023-11-08T05:12:56.594 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring primary: true 2023-11-08T05:12:56.594 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' -n '' ']' 2023-11-08T05:12:56.595 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ stress_write_image cluster2 mirror image_48 2023-11-08T05:12:56.595 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:12:56.595 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:12:56.595 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_48 2023-11-08T05:12:56.595 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-11-08T05:12:56.596 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local duration=9 2023-11-08T05:12:56.596 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ set +e 2023-11-08T05:12:56.597 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_48 --debug-rbd=20 --debug-journaler=20 2023-11-08T05:12:56.825 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:100 IOs, 37734400 bytes 2023-11-08T05:12:56.937 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:200 IOs, 43180032 bytes 2023-11-08T05:12:57.114 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:300 IOs, 85825536 bytes 2023-11-08T05:12:57.323 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:400 IOs, 140940288 bytes 2023-11-08T05:12:57.577 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:500 IOs, 219198464 bytes 2023-11-08T05:12:57.697 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:600 IOs, 223741952 bytes 2023-11-08T05:12:57.907 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:700 IOs, 275086336 bytes 2023-11-08T05:12:58.183 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:800 IOs, 361594880 bytes 2023-11-08T05:12:58.372 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:900 IOs, 398244864 bytes 2023-11-08T05:12:58.643 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1000 IOs, 476296192 bytes 2023-11-08T05:12:58.815 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1100 IOs, 529319936 bytes 2023-11-08T05:12:59.049 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1200 IOs, 570077184 bytes 2023-11-08T05:12:59.264 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1300 IOs, 623923200 bytes 2023-11-08T05:12:59.440 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1400 IOs, 654518272 bytes 2023-11-08T05:12:59.624 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1500 IOs, 697720832 bytes 2023-11-08T05:12:59.758 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1600 IOs, 707160064 bytes 2023-11-08T05:12:59.867 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1700 IOs, 708987904 bytes 2023-11-08T05:13:00.041 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1800 IOs, 743267328 bytes 2023-11-08T05:13:00.290 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1900 IOs, 818160640 bytes 2023-11-08T05:13:00.482 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2000 IOs, 866008064 bytes 2023-11-08T05:13:00.693 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2100 IOs, 910017536 bytes 2023-11-08T05:13:00.840 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2200 IOs, 931900416 bytes 2023-11-08T05:13:01.047 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2300 IOs, 983385088 bytes 2023-11-08T05:13:01.283 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2400 IOs, 1046426624 bytes 2023-11-08T05:13:01.388 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2500 IOs, 1047023616 bytes 2023-11-08T05:13:01.522 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2600 IOs, 1050610688 bytes 2023-11-08T05:13:01.632 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2700 IOs, 1051237376 bytes 2023-11-08T05:13:01.873 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2800 IOs, 1123665920 bytes 2023-11-08T05:13:02.063 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2900 IOs, 1168879616 bytes 2023-11-08T05:13:02.268 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3000 IOs, 1205292032 bytes 2023-11-08T05:13:02.372 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3100 IOs, 1205915648 bytes 2023-11-08T05:13:02.538 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3200 IOs, 1234114560 bytes 2023-11-08T05:13:02.705 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3300 IOs, 1261117440 bytes 2023-11-08T05:13:02.987 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3400 IOs, 1343978496 bytes 2023-11-08T05:13:03.162 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3500 IOs, 1376764928 bytes 2023-11-08T05:13:03.380 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3600 IOs, 1417363456 bytes 2023-11-08T05:13:03.559 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3700 IOs, 1454240768 bytes 2023-11-08T05:13:03.785 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3800 IOs, 1509401600 bytes 2023-11-08T05:13:03.973 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3900 IOs, 1552465920 bytes 2023-11-08T05:13:04.186 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4000 IOs, 1606032384 bytes 2023-11-08T05:13:04.304 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4100 IOs, 1606629376 bytes 2023-11-08T05:13:04.494 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4200 IOs, 1646043136 bytes 2023-11-08T05:13:04.691 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4300 IOs, 1688219648 bytes 2023-11-08T05:13:04.852 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4400 IOs, 1713661952 bytes 2023-11-08T05:13:05.016 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4500 IOs, 1740981248 bytes 2023-11-08T05:13:05.140 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4600 IOs, 1753496576 bytes 2023-11-08T05:13:05.290 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4700 IOs, 1774703616 bytes 2023-11-08T05:13:05.469 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4800 IOs, 1812090880 bytes 2023-11-08T05:13:05.601 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ error_code=124 2023-11-08T05:13:05.602 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ set -e 2023-11-08T05:13:05.602 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' 124 -eq 124 ']' 2023-11-08T05:13:05.602 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:13:05.602 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-11-08T05:13:05.602 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image=image_49 2023-11-08T05:13:05.603 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ create_image_and_enable_mirror cluster2 mirror image_49 snapshot 128M 2023-11-08T05:13:05.603 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:13:05.603 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:13:05.603 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:13:05.603 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:13:05.604 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_49 2023-11-08T05:13:05.604 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:13:05.604 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local mode=snapshot 2023-11-08T05:13:05.604 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' -n snapshot ']' 2023-11-08T05:13:05.604 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:13:05.604 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ create_image cluster2 mirror image_49 128M 2023-11-08T05:13:05.605 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:13:05.605 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:13:05.605 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:13:05.605 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:13:05.605 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_49 2023-11-08T05:13:05.606 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:13:05.606 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local size=128 2023-11-08T05:13:05.606 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' -n 128M ']' 2023-11-08T05:13:05.606 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ size=128M 2023-11-08T05:13:05.606 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:13:05.607 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_49 2023-11-08T05:13:05.641 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' image = image ']' 2023-11-08T05:13:05.641 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ enable_mirror cluster2 mirror image_49 snapshot 2023-11-08T05:13:05.641 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:13:05.642 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:13:05.642 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_49 2023-11-08T05:13:05.642 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local mode=snapshot 2023-11-08T05:13:05.642 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_49 snapshot 2023-11-08T05:13:06.594 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:Mirroring enabled 2023-11-08T05:13:06.602 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster=cluster2 info mirror/image_49 2023-11-08T05:13:06.634 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:rbd image 'image_49': 2023-11-08T05:13:06.634 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: size 128 MiB in 32 objects 2023-11-08T05:13:06.634 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: order 22 (4 MiB objects) 2023-11-08T05:13:06.635 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: snapshot_count: 1 2023-11-08T05:13:06.635 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: id: 14897d04906f 2023-11-08T05:13:06.635 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: block_name_prefix: rbd_data.14897d04906f 2023-11-08T05:13:06.635 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: format: 2 2023-11-08T05:13:06.635 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: features: layering, exclusive-lock 2023-11-08T05:13:06.636 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: op_features: 2023-11-08T05:13:06.636 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: flags: 2023-11-08T05:13:06.636 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: create_timestamp: Wed Nov 8 05:13:05 2023 2023-11-08T05:13:06.636 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: access_timestamp: Wed Nov 8 05:13:05 2023 2023-11-08T05:13:06.636 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: modify_timestamp: Wed Nov 8 05:13:05 2023 2023-11-08T05:13:06.637 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring state: enabled 2023-11-08T05:13:06.637 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring mode: snapshot 2023-11-08T05:13:06.637 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring global id: e7efac79-0655-48fa-85f6-d32c1c0bf780 2023-11-08T05:13:06.637 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring primary: true 2023-11-08T05:13:06.638 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' -n '' ']' 2023-11-08T05:13:06.638 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ stress_write_image cluster2 mirror image_49 2023-11-08T05:13:06.639 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:13:06.639 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:13:06.639 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_49 2023-11-08T05:13:06.639 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-11-08T05:13:06.639 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local duration=7 2023-11-08T05:13:06.640 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ set +e 2023-11-08T05:13:06.640 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_49 --debug-rbd=20 --debug-journaler=20 2023-11-08T05:13:06.856 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:100 IOs, 53921792 bytes 2023-11-08T05:13:07.008 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:200 IOs, 70896640 bytes 2023-11-08T05:13:07.183 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:300 IOs, 97170432 bytes 2023-11-08T05:13:07.412 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:400 IOs, 156581888 bytes 2023-11-08T05:13:07.576 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:500 IOs, 188989440 bytes 2023-11-08T05:13:07.813 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:600 IOs, 253389824 bytes 2023-11-08T05:13:08.018 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:700 IOs, 313884672 bytes 2023-11-08T05:13:08.158 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:800 IOs, 320026624 bytes 2023-11-08T05:13:08.413 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:900 IOs, 392111104 bytes 2023-11-08T05:13:08.560 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1000 IOs, 413960192 bytes 2023-11-08T05:13:08.761 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1100 IOs, 463641600 bytes 2023-11-08T05:13:08.960 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1200 IOs, 529316864 bytes 2023-11-08T05:13:09.165 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1300 IOs, 559774720 bytes 2023-11-08T05:13:09.303 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1400 IOs, 571470848 bytes 2023-11-08T05:13:09.475 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1500 IOs, 603689984 bytes 2023-11-08T05:13:09.652 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1600 IOs, 641085440 bytes 2023-11-08T05:13:09.819 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1700 IOs, 671563776 bytes 2023-11-08T05:13:10.052 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1800 IOs, 738191360 bytes 2023-11-08T05:13:10.193 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1900 IOs, 756407296 bytes 2023-11-08T05:13:10.337 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2000 IOs, 773314560 bytes 2023-11-08T05:13:10.742 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2100 IOs, 927400960 bytes 2023-11-08T05:13:10.884 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2200 IOs, 936085504 bytes 2023-11-08T05:13:11.066 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2300 IOs, 970804224 bytes 2023-11-08T05:13:11.185 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2400 IOs, 985217024 bytes 2023-11-08T05:13:11.450 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2500 IOs, 1059818496 bytes 2023-11-08T05:13:11.641 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2600 IOs, 1103753216 bytes 2023-11-08T05:13:11.823 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2700 IOs, 1141561344 bytes 2023-11-08T05:13:11.993 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2800 IOs, 1172454400 bytes 2023-11-08T05:13:12.271 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2900 IOs, 1261109248 bytes 2023-11-08T05:13:12.451 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3000 IOs, 1295374336 bytes 2023-11-08T05:13:12.765 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3100 IOs, 1391788032 bytes 2023-11-08T05:13:12.957 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3200 IOs, 1434684416 bytes 2023-11-08T05:13:13.117 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3300 IOs, 1459395584 bytes 2023-11-08T05:13:13.496 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3400 IOs, 1576873984 bytes 2023-11-08T05:13:13.644 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ error_code=124 2023-11-08T05:13:13.645 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ set -e 2023-11-08T05:13:13.645 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' 124 -eq 124 ']' 2023-11-08T05:13:13.645 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:13:13.646 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-11-08T05:13:13.646 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image=image_50 2023-11-08T05:13:13.646 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ create_image_and_enable_mirror cluster2 mirror image_50 snapshot 128M 2023-11-08T05:13:13.646 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:13:13.646 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:13:13.647 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:13:13.647 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:13:13.647 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_50 2023-11-08T05:13:13.647 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:13:13.647 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local mode=snapshot 2023-11-08T05:13:13.648 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' -n snapshot ']' 2023-11-08T05:13:13.648 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:13:13.648 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ create_image cluster2 mirror image_50 128M 2023-11-08T05:13:13.648 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:13:13.648 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:13:13.649 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:13:13.649 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:13:13.649 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_50 2023-11-08T05:13:13.649 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:13:13.649 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local size=128 2023-11-08T05:13:13.650 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' -n 128M ']' 2023-11-08T05:13:13.650 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ size=128M 2023-11-08T05:13:13.650 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:13:13.650 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_50 2023-11-08T05:13:13.684 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' image = image ']' 2023-11-08T05:13:13.685 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ enable_mirror cluster2 mirror image_50 snapshot 2023-11-08T05:13:13.685 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:13:13.685 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:13:13.685 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_50 2023-11-08T05:13:13.686 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local mode=snapshot 2023-11-08T05:13:13.686 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_50 snapshot 2023-11-08T05:13:14.631 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:Mirroring enabled 2023-11-08T05:13:14.639 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster=cluster2 info mirror/image_50 2023-11-08T05:13:14.671 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:rbd image 'image_50': 2023-11-08T05:13:14.672 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: size 128 MiB in 32 objects 2023-11-08T05:13:14.672 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: order 22 (4 MiB objects) 2023-11-08T05:13:14.672 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: snapshot_count: 1 2023-11-08T05:13:14.672 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: id: 149430554caf 2023-11-08T05:13:14.673 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: block_name_prefix: rbd_data.149430554caf 2023-11-08T05:13:14.673 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: format: 2 2023-11-08T05:13:14.673 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: features: layering, exclusive-lock 2023-11-08T05:13:14.673 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: op_features: 2023-11-08T05:13:14.673 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: flags: 2023-11-08T05:13:14.674 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: create_timestamp: Wed Nov 8 05:13:13 2023 2023-11-08T05:13:14.674 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: access_timestamp: Wed Nov 8 05:13:13 2023 2023-11-08T05:13:14.674 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: modify_timestamp: Wed Nov 8 05:13:13 2023 2023-11-08T05:13:14.674 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring state: enabled 2023-11-08T05:13:14.674 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring mode: snapshot 2023-11-08T05:13:14.675 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring global id: 39de3a57-ef7b-4ccc-a818-26ba71efeed5 2023-11-08T05:13:14.675 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring primary: true 2023-11-08T05:13:14.676 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' -n '' ']' 2023-11-08T05:13:14.676 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ stress_write_image cluster2 mirror image_50 2023-11-08T05:13:14.676 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:13:14.677 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:13:14.677 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_50 2023-11-08T05:13:14.677 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-11-08T05:13:14.677 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local duration=14 2023-11-08T05:13:14.677 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ set +e 2023-11-08T05:13:14.677 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_50 --debug-rbd=20 --debug-journaler=20 2023-11-08T05:13:15.171 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:100 IOs, 86121472 bytes 2023-11-08T05:13:15.301 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:200 IOs, 92102656 bytes 2023-11-08T05:13:15.499 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:300 IOs, 147932160 bytes 2023-11-08T05:13:15.685 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:400 IOs, 185824256 bytes 2023-11-08T05:13:15.848 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:500 IOs, 214388736 bytes 2023-11-08T05:13:15.994 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:600 IOs, 232442880 bytes 2023-11-08T05:13:16.217 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:700 IOs, 292203520 bytes 2023-11-08T05:13:16.495 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:800 IOs, 375032832 bytes 2023-11-08T05:13:16.636 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:900 IOs, 387562496 bytes 2023-11-08T05:13:16.782 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1000 IOs, 403978240 bytes 2023-11-08T05:13:16.944 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1100 IOs, 427499520 bytes 2023-11-08T05:13:17.110 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1200 IOs, 471179264 bytes 2023-11-08T05:13:17.345 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1300 IOs, 522191872 bytes 2023-11-08T05:13:17.583 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1400 IOs, 596775936 bytes 2023-11-08T05:13:17.788 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1500 IOs, 640785408 bytes 2023-11-08T05:13:17.924 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1600 IOs, 649317376 bytes 2023-11-08T05:13:18.131 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1700 IOs, 695554048 bytes 2023-11-08T05:13:18.279 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1800 IOs, 708189184 bytes 2023-11-08T05:13:18.407 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1900 IOs, 716562432 bytes 2023-11-08T05:13:18.517 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2000 IOs, 717186048 bytes 2023-11-08T05:13:18.729 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2100 IOs, 774084608 bytes 2023-11-08T05:13:18.916 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2200 IOs, 809362432 bytes 2023-11-08T05:13:19.070 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2300 IOs, 832818176 bytes 2023-11-08T05:13:19.288 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2400 IOs, 887389184 bytes 2023-11-08T05:13:19.497 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2500 IOs, 935147520 bytes 2023-11-08T05:13:19.758 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2600 IOs, 1016946688 bytes 2023-11-08T05:13:19.918 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2700 IOs, 1040640000 bytes 2023-11-08T05:13:20.077 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2800 IOs, 1065455616 bytes 2023-11-08T05:13:20.272 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2900 IOs, 1105224704 bytes 2023-11-08T05:13:20.449 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3000 IOs, 1137171456 bytes 2023-11-08T05:13:20.556 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3100 IOs, 1138470912 bytes 2023-11-08T05:13:20.697 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3200 IOs, 1151234048 bytes 2023-11-08T05:13:20.894 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3300 IOs, 1197853696 bytes 2023-11-08T05:13:21.121 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3400 IOs, 1252343808 bytes 2023-11-08T05:13:21.309 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3500 IOs, 1291857920 bytes 2023-11-08T05:13:21.598 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3600 IOs, 1379655680 bytes 2023-11-08T05:13:21.905 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3700 IOs, 1505784832 bytes 2023-11-08T05:13:22.133 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3800 IOs, 1550051328 bytes 2023-11-08T05:13:22.317 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3900 IOs, 1590368256 bytes 2023-11-08T05:13:22.478 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4000 IOs, 1639409664 bytes 2023-11-08T05:13:22.620 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4100 IOs, 1644982272 bytes 2023-11-08T05:13:22.877 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4200 IOs, 1709803520 bytes 2023-11-08T05:13:23.188 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4300 IOs, 1800871936 bytes 2023-11-08T05:13:23.395 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4400 IOs, 1841541120 bytes 2023-11-08T05:13:23.601 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4500 IOs, 1884821504 bytes 2023-11-08T05:13:23.769 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4600 IOs, 1906500608 bytes 2023-11-08T05:13:23.997 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4700 IOs, 1955151872 bytes 2023-11-08T05:13:24.140 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4800 IOs, 1977915392 bytes 2023-11-08T05:13:24.369 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4900 IOs, 2027569152 bytes 2023-11-08T05:13:24.489 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5000 IOs, 2028154880 bytes 2023-11-08T05:13:24.728 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5100 IOs, 2092107776 bytes 2023-11-08T05:13:24.932 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5200 IOs, 2134902784 bytes 2023-11-08T05:13:25.169 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5300 IOs, 2210255872 bytes 2023-11-08T05:13:25.404 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5400 IOs, 2258142208 bytes 2023-11-08T05:13:25.732 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5500 IOs, 2394770432 bytes 2023-11-08T05:13:25.920 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5600 IOs, 2399086592 bytes 2023-11-08T05:13:26.118 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5700 IOs, 2440196096 bytes 2023-11-08T05:13:26.259 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5800 IOs, 2456201216 bytes 2023-11-08T05:13:26.449 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5900 IOs, 2489682944 bytes 2023-11-08T05:13:26.590 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:6000 IOs, 2503211008 bytes 2023-11-08T05:13:26.829 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:6100 IOs, 2585001984 bytes 2023-11-08T05:13:27.236 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:6200 IOs, 2654512128 bytes 2023-11-08T05:13:27.376 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:6300 IOs, 2669927424 bytes 2023-11-08T05:13:27.641 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:6400 IOs, 2751653888 bytes 2023-11-08T05:13:27.828 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:6500 IOs, 2799847424 bytes 2023-11-08T05:13:28.057 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:6600 IOs, 2849148928 bytes 2023-11-08T05:13:28.259 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:6700 IOs, 2904711168 bytes 2023-11-08T05:13:28.457 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:6800 IOs, 2938647552 bytes 2023-11-08T05:13:28.681 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ error_code=124 2023-11-08T05:13:28.682 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ set -e 2023-11-08T05:13:28.682 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' 124 -eq 124 ']' 2023-11-08T05:13:28.682 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:13:28.682 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_pool_images cluster2 mirror 50 2023-11-08T05:13:28.683 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:13:28.683 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:13:28.683 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image_count=50 2023-11-08T05:13:28.683 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local s 2023-11-08T05:13:28.684 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local count 2023-11-08T05:13:28.684 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local last_count=0 2023-11-08T05:13:28.684 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ true 2023-11-08T05:13:28.684 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ seq 1 40 2023-11-08T05:13:28.685 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in `seq 1 40` 2023-11-08T05:13:28.685 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test 1 -ne 1 2023-11-08T05:13:28.685 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ rbd --cluster cluster2 -p mirror mirror pool status 2023-11-08T05:13:28.686 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ grep 'images: ' 2023-11-08T05:13:28.686 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cut '-d ' -f 2 2023-11-08T05:13:28.715 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd: failed to query services: (13) Permission denied 2023-11-08T05:13:28.718 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ count=50 2023-11-08T05:13:28.718 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test 50 = 50 2023-11-08T05:13:28.718 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:13:28.719 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_pool_healthy cluster2 mirror 2023-11-08T05:13:28.719 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:13:28.719 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:13:28.719 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local s 2023-11-08T05:13:28.719 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local state 2023-11-08T05:13:28.720 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ seq 1 40 2023-11-08T05:13:28.720 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in `seq 1 40` 2023-11-08T05:13:28.721 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test 1 -ne 1 2023-11-08T05:13:28.721 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ rbd --cluster cluster2 -p mirror mirror pool status 2023-11-08T05:13:28.721 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ grep 'image health:' 2023-11-08T05:13:28.721 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cut '-d ' -f 3 2023-11-08T05:13:28.755 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd: failed to query services: (13) Permission denied 2023-11-08T05:13:28.755 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ state=OK 2023-11-08T05:13:28.755 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test OK = ERROR 2023-11-08T05:13:28.756 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test OK = OK 2023-11-08T05:13:28.756 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:13:28.756 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_pool_images cluster1 mirror 50 2023-11-08T05:13:28.756 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:13:28.756 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:13:28.757 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image_count=50 2023-11-08T05:13:28.757 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local s 2023-11-08T05:13:28.757 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local count 2023-11-08T05:13:28.757 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local last_count=0 2023-11-08T05:13:28.757 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ true 2023-11-08T05:13:28.757 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ seq 1 40 2023-11-08T05:13:28.758 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in `seq 1 40` 2023-11-08T05:13:28.758 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test 1 -ne 1 2023-11-08T05:13:28.758 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ rbd --cluster cluster1 -p mirror mirror pool status 2023-11-08T05:13:28.759 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ grep 'images: ' 2023-11-08T05:13:28.759 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cut '-d ' -f 2 2023-11-08T05:13:28.786 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd: failed to query services: (13) Permission denied 2023-11-08T05:13:28.788 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ count=50 2023-11-08T05:13:28.789 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test 50 = 50 2023-11-08T05:13:28.789 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:13:28.789 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_pool_healthy cluster1 mirror 2023-11-08T05:13:28.789 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:13:28.789 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:13:28.790 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local s 2023-11-08T05:13:28.790 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local state 2023-11-08T05:13:28.790 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ seq 1 40 2023-11-08T05:13:28.791 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in `seq 1 40` 2023-11-08T05:13:28.791 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test 1 -ne 1 2023-11-08T05:13:28.791 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ rbd --cluster cluster1 -p mirror mirror pool status 2023-11-08T05:13:28.792 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ grep 'image health:' 2023-11-08T05:13:28.792 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cut '-d ' -f 3 2023-11-08T05:13:28.819 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd: failed to query services: (13) Permission denied 2023-11-08T05:13:28.822 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ state=WARNING 2023-11-08T05:13:28.822 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test WARNING = ERROR 2023-11-08T05:13:28.822 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test WARNING = OK 2023-11-08T05:13:28.822 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in `seq 1 40` 2023-11-08T05:13:28.823 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test 2 -ne 1 2023-11-08T05:13:28.823 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 30 2023-11-08T05:13:57.184 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.4390717826312459) > weight (0.25) 2023-11-08T05:13:57.184 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.8981390689725133) > weight (0.25) 2023-11-08T05:13:57.185 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.8949018329276506) > weight (0.25) 2023-11-08T05:13:57.185 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.3 2023-11-08T05:13:57.185 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Sent signal 15 2023-11-08T05:13:57.186 INFO:tasks.rbd_mirror_thrash:waiting for 3 secs before reviving daemons 2023-11-08T05:13:57.186 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi087.stderr:2023-11-08T05:13:57.184+0000 7f99286a5700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.3 (PID: 127184) UID: 1000 2023-11-08T05:13:57.387 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi087.stderr:daemon-helper: command failed with exit status 1 2023-11-08T05:13:58.521 INFO:tasks.ceph.cluster2.osd.2.smithi087.stderr:2023-11-08T05:13:58.519+0000 7fa243317700 -1 reset not still connected to 0x5613b884d6c0 2023-11-08T05:13:58.521 INFO:tasks.ceph.cluster2.osd.2.smithi087.stderr:2023-11-08T05:13:58.519+0000 7fa243317700 -1 reset not still connected to 0x5613bcfb7450 2023-11-08T05:13:58.522 INFO:tasks.ceph.cluster2.osd.2.smithi087.stderr:2023-11-08T05:13:58.519+0000 7fa243317700 -1 reset not still connected to 0x5613d4311e10 2023-11-08T05:13:58.533 INFO:tasks.ceph.cluster1.osd.2.smithi077.stderr:2023-11-08T05:13:58.532+0000 7f9900abf700 -1 reset not still connected to 0x555e828aa000 2023-11-08T05:13:58.534 INFO:tasks.ceph.cluster1.osd.2.smithi077.stderr:2023-11-08T05:13:58.532+0000 7f9900abf700 -1 reset not still connected to 0x555e828ab040 2023-11-08T05:13:58.534 INFO:tasks.ceph.cluster1.osd.2.smithi077.stderr:2023-11-08T05:13:58.532+0000 7f9900abf700 -1 reset not still connected to 0x555e828ab860 2023-11-08T05:13:58.534 INFO:tasks.ceph.cluster1.osd.2.smithi077.stderr:2023-11-08T05:13:58.532+0000 7f9900abf700 -1 reset not still connected to 0x555e82ab92b0 2023-11-08T05:13:58.534 INFO:tasks.ceph.cluster1.osd.2.smithi077.stderr:2023-11-08T05:13:58.532+0000 7f9900abf700 -1 reset not still connected to 0x555e8343f5f0 2023-11-08T05:13:58.534 INFO:tasks.ceph.cluster1.osd.2.smithi077.stderr:2023-11-08T05:13:58.532+0000 7f9900abf700 -1 reset not still connected to 0x555e83dabd40 2023-11-08T05:13:58.535 INFO:tasks.ceph.cluster1.osd.2.smithi077.stderr:2023-11-08T05:13:58.532+0000 7f9900abf700 -1 reset not still connected to 0x555e9809ec30 2023-11-08T05:13:58.823 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ rbd --cluster cluster1 -p mirror mirror pool status 2023-11-08T05:13:58.824 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ grep 'image health:' 2023-11-08T05:13:58.824 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cut '-d ' -f 3 2023-11-08T05:13:58.854 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd: failed to query services: (13) Permission denied 2023-11-08T05:13:58.856 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ state=OK 2023-11-08T05:13:58.857 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test OK = ERROR 2023-11-08T05:13:58.857 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test OK = OK 2023-11-08T05:13:58.857 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:13:58.857 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ testlog 'TEST: compare many images' 2023-11-08T05:13:58.858 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ tee -a /tmp/tmp.Ludk8ikCAB/rbd-mirror.test.log 2023-11-08T05:13:58.858 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ date '+%F %T' 2023-11-08T05:13:58.859 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ echo 2023-11-08 05:13:58 TEST: compare many images 2023-11-08T05:13:58.859 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:2023-11-08 05:13:58 TEST: compare many images 2023-11-08T05:13:58.860 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ seq 1 50 2023-11-08T05:13:58.860 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-11-08T05:13:58.860 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image=image_1 2023-11-08T05:13:58.860 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ create_snap cluster2 mirror image_1 snap 2023-11-08T05:13:58.860 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:13:58.861 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:13:58.861 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_1 2023-11-08T05:13:58.861 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap_name=snap 2023-11-08T05:13:58.861 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 -p mirror snap create image_1@snap --debug-rbd=20 --debug-journaler=20 2023-11-08T05:14:00.187 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.3 2023-11-08T05:14:00.187 INFO:teuthology.orchestra.run:waiting for 600 2023-11-08T05:14:00.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-08T05:14:00.188 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.3 2023-11-08T05:14:00.188 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Restarting daemon 2023-11-08T05:14:00.188 DEBUG:teuthology.orchestra.run.smithi087:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.3 2023-11-08T05:14:00.191 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Started 2023-11-08T05:14:00.191 INFO:tasks.rbd_mirror_thrash:waiting for 71 secs before thrashing 2023-11-08T05:14:00.255 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_image_replay_started cluster1 mirror image_1 2023-11-08T05:14:00.256 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:14:00.256 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:14:00.256 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_1 2023-11-08T05:14:00.256 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_image_replay_state cluster1 mirror image_1 started 2023-11-08T05:14:00.257 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:14:00.257 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:14:00.257 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_1 2023-11-08T05:14:00.257 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local state=started 2023-11-08T05:14:00.257 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local s 2023-11-08T05:14:00.258 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-11-08T05:14:00.258 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 1 2023-11-08T05:14:01.090 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.3675433229523758) > weight (0.25) 2023-11-08T05:14:01.091 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.1 2023-11-08T05:14:01.091 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Sent signal 15 2023-11-08T05:14:01.091 INFO:tasks.rbd_mirror_thrash:waiting for 6 secs before reviving daemons 2023-11-08T05:14:01.092 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi087.stderr:2023-11-08T05:14:01.090+0000 7f91418ab700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 (PID: 132113) UID: 1000 2023-11-08T05:14:01.257 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test_image_replay_state cluster1 mirror image_1 started 2023-11-08T05:14:01.257 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:14:01.258 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:14:01.258 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_1 2023-11-08T05:14:01.258 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local test_state=started 2023-11-08T05:14:01.258 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local status_result 2023-11-08T05:14:01.258 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local current_state=stopped 2023-11-08T05:14:01.259 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_1 2023-11-08T05:14:01.259 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_instance=cluster1 2023-11-08T05:14:01.259 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T05:14:01.259 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1 2023-11-08T05:14:01.259 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ grep -i state 2023-11-08T05:14:01.260 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance=cluster1 2023-11-08T05:14:01.260 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local loop_instance 2023-11-08T05:14:01.260 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-11-08T05:14:01.260 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sleep 0 2023-11-08T05:14:01.263 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-11-08T05:14:01.264 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ seq 0 3 2023-11-08T05:14:01.264 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T05:14:01.264 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_1 2023-11-08T05:14:01.264 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1:0 2023-11-08T05:14:01.264 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T05:14:01.265 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance 2023-11-08T05:14:01.265 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-11-08T05:14:01.265 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local val=cluster1:0 2023-11-08T05:14:01.265 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_var_name=cluster 2023-11-08T05:14:01.265 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance_var_name=instance 2023-11-08T05:14:01.266 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cluster=cluster1 2023-11-08T05:14:01.266 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ instance=0 2023-11-08T05:14:01.266 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' 0 = cluster1:0 ']' 2023-11-08T05:14:01.266 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval cluster=cluster1 2023-11-08T05:14:01.266 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:14:01.267 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval instance=0 2023-11-08T05:14:01.267 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=0 2023-11-08T05:14:01.267 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-11-08T05:14:01.267 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local local_cluster=cluster1:0 2023-11-08T05:14:01.267 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster=cluster1 2023-11-08T05:14:01.267 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance 2023-11-08T05:14:01.268 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-11-08T05:14:01.268 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local val=cluster1:0 2023-11-08T05:14:01.268 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster_var_name=local_cluster 2023-11-08T05:14:01.268 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance_var_name=instance 2023-11-08T05:14:01.268 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:14:01.268 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=0 2023-11-08T05:14:01.269 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ '[' 0 = cluster1:0 ']' 2023-11-08T05:14:01.269 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval local_cluster=cluster1 2023-11-08T05:14:01.269 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ local_cluster=cluster1 2023-11-08T05:14:01.269 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval instance=0 2023-11-08T05:14:01.269 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ instance=0 2023-11-08T05:14:01.269 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-11-08T05:14:01.276 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-08T05:14:01.276 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-08T05:14:01.276 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-08T05:14:01.276 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_1 2023-11-08T05:14:01.292 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi087.stderr:daemon-helper: command failed with exit status 1 2023-11-08T05:14:01.386 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ return 0 2023-11-08T05:14:01.388 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ status_result=' "state": "Replaying"' 2023-11-08T05:14:01.388 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ echo ' "state": "Replaying"' 2023-11-08T05:14:01.388 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ grep -i Replaying 2023-11-08T05:14:01.389 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: "state": "Replaying" 2023-11-08T05:14:01.391 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ current_state=started 2023-11-08T05:14:01.391 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test started = started 2023-11-08T05:14:01.391 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:14:01.391 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_1 2023-11-08T05:14:01.392 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local local_cluster=cluster1 2023-11-08T05:14:01.392 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:14:01.392 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:14:01.392 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_1 2023-11-08T05:14:01.392 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' snapshot = journal ']' 2023-11-08T05:14:01.393 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' snapshot = snapshot ']' 2023-11-08T05:14:01.393 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_1 2023-11-08T05:14:01.393 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local local_cluster=cluster1 2023-11-08T05:14:01.393 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:14:01.393 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:14:01.393 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_1 2023-11-08T05:14:01.394 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ mkfname cluster2-mirror-image_1.status 2023-11-08T05:14:01.394 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ echo cluster2-mirror-image_1.status 2023-11-08T05:14:01.394 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -e 's|[/ ]|_|g' 2023-11-08T05:14:01.394 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local status_log=/tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_1.status 2023-11-08T05:14:01.394 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ mkfname cluster1-mirror-image_1.status 2023-11-08T05:14:01.395 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ echo cluster1-mirror-image_1.status 2023-11-08T05:14:01.395 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -e 's|[/ ]|_|g' 2023-11-08T05:14:01.396 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local local_status_log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_1.status 2023-11-08T05:14:01.396 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ mirror_image_snapshot cluster2 mirror image_1 2023-11-08T05:14:01.396 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:14:01.396 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:14:01.397 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_1 2023-11-08T05:14:01.397 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_1 2023-11-08T05:14:02.257 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:Snapshot ID: 119 2023-11-08T05:14:02.267 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_1 /tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_1.status 2023-11-08T05:14:02.267 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:14:02.268 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:14:02.268 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_1 2023-11-08T05:14:02.268 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_1.status 2023-11-08T05:14:02.268 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_1 --format xml 2023-11-08T05:14:02.269 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T05:14:02.308 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-11-08T05:14:02.310 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snapshot_id=119 2023-11-08T05:14:02.311 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ true 2023-11-08T05:14:02.311 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-11-08T05:14:02.311 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0.2 2023-11-08T05:14:02.512 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_1 /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_1.status 2023-11-08T05:14:02.512 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:14:02.512 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:14:02.513 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_1 2023-11-08T05:14:02.513 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_1.status 2023-11-08T05:14:02.513 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_1 --format xml 2023-11-08T05:14:02.513 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T05:14:02.551 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ true 2023-11-08T05:14:02.552 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-08T05:14:02.553 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:-:1.1: Document is empty 2023-11-08T05:14:02.553 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: 2023-11-08T05:14:02.553 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:^ 2023-11-08T05:14:02.554 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local primary_snapshot_id= 2023-11-08T05:14:02.554 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test 119 = '' 2023-11-08T05:14:02.554 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-11-08T05:14:02.555 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0.4 2023-11-08T05:14:02.955 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_1 /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_1.status 2023-11-08T05:14:02.956 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:14:02.956 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:14:02.956 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_1 2023-11-08T05:14:02.957 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_1.status 2023-11-08T05:14:02.957 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_1 --format xml 2023-11-08T05:14:02.957 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T05:14:02.997 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ true 2023-11-08T05:14:02.998 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-08T05:14:02.998 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:-:1.1: Document is empty 2023-11-08T05:14:02.999 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: 2023-11-08T05:14:02.999 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:^ 2023-11-08T05:14:03.000 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local primary_snapshot_id= 2023-11-08T05:14:03.001 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test 119 = '' 2023-11-08T05:14:03.001 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-11-08T05:14:03.001 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0.8 2023-11-08T05:14:03.800 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_1 /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_1.status 2023-11-08T05:14:03.801 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:14:03.801 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:14:03.801 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_1 2023-11-08T05:14:03.801 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_1.status 2023-11-08T05:14:03.801 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_1 --format xml 2023-11-08T05:14:03.802 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T05:14:03.838 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-08T05:14:03.840 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local primary_snapshot_id=119 2023-11-08T05:14:03.841 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test 119 = 119 2023-11-08T05:14:03.841 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:14:03.841 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_snap_present cluster1 mirror image_1 snap 2023-11-08T05:14:03.841 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:14:03.842 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:14:03.842 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_1 2023-11-08T05:14:03.842 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap_name=snap 2023-11-08T05:14:03.842 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local s 2023-11-08T05:14:03.843 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-11-08T05:14:03.843 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 1 2023-11-08T05:14:04.841 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 info mirror/image_1@snap 2023-11-08T05:14:04.880 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:rbd image 'image_1': 2023-11-08T05:14:04.881 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: size 128 MiB in 32 objects 2023-11-08T05:14:04.881 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: order 22 (4 MiB objects) 2023-11-08T05:14:04.881 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: snapshot_count: 2 2023-11-08T05:14:04.881 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: id: 10e75c34e398 2023-11-08T05:14:04.882 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: block_name_prefix: rbd_data.10e75c34e398 2023-11-08T05:14:04.882 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: format: 2 2023-11-08T05:14:04.882 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: features: layering, exclusive-lock, non-primary 2023-11-08T05:14:04.882 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: op_features: 2023-11-08T05:14:04.882 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: flags: 2023-11-08T05:14:04.883 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: create_timestamp: Wed Nov 8 05:04:23 2023 2023-11-08T05:14:04.883 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: access_timestamp: Wed Nov 8 05:04:23 2023 2023-11-08T05:14:04.883 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: modify_timestamp: Wed Nov 8 05:04:23 2023 2023-11-08T05:14:04.883 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: protected: False 2023-11-08T05:14:04.883 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring state: enabled 2023-11-08T05:14:04.884 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring mode: snapshot 2023-11-08T05:14:04.884 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring global id: 8f09f37c-3f8d-4d10-ad86-50b1dda83090 2023-11-08T05:14:04.884 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring primary: false 2023-11-08T05:14:04.885 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:14:04.885 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ compare_image_snaps mirror image_1 snap 2023-11-08T05:14:04.886 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:14:04.886 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_1 2023-11-08T05:14:04.886 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap_name=snap 2023-11-08T05:14:04.886 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local ret=0 2023-11-08T05:14:04.886 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local rmt_export=/tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_1.export 2023-11-08T05:14:04.887 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local loc_export=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_1.export 2023-11-08T05:14:04.887 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rm -f /tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_1.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_1.export 2023-11-08T05:14:04.887 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 -p mirror export image_1@snap /tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_1.export 2023-11-08T05:14:05.070 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-11-08T05:14:05.073 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 -p mirror export image_1@snap /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_1.export 2023-11-08T05:14:05.289 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-11-08T05:14:05.292 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ cmp /tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_1.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_1.export 2023-11-08T05:14:05.354 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rm -f /tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_1.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_1.export 2023-11-08T05:14:05.395 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:14:05.395 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-11-08T05:14:05.395 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image=image_2 2023-11-08T05:14:05.395 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ create_snap cluster2 mirror image_2 snap 2023-11-08T05:14:05.396 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:14:05.396 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:14:05.396 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_2 2023-11-08T05:14:05.396 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap_name=snap 2023-11-08T05:14:05.396 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 -p mirror snap create image_2@snap --debug-rbd=20 --debug-journaler=20 2023-11-08T05:14:07.092 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.1 2023-11-08T05:14:07.093 INFO:teuthology.orchestra.run:waiting for 600 2023-11-08T05:14:07.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-08T05:14:07.093 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.1 2023-11-08T05:14:07.094 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Restarting daemon 2023-11-08T05:14:07.094 DEBUG:teuthology.orchestra.run.smithi087:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 2023-11-08T05:14:07.097 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Started 2023-11-08T05:14:07.097 INFO:tasks.rbd_mirror_thrash:waiting for 75 secs before thrashing 2023-11-08T05:14:07.272 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_image_replay_started cluster1 mirror image_2 2023-11-08T05:14:07.273 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:14:07.273 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:14:07.273 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_2 2023-11-08T05:14:07.273 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_image_replay_state cluster1 mirror image_2 started 2023-11-08T05:14:07.273 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:14:07.274 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:14:07.274 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_2 2023-11-08T05:14:07.274 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local state=started 2023-11-08T05:14:07.274 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local s 2023-11-08T05:14:07.274 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-11-08T05:14:07.275 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 1 2023-11-08T05:14:08.274 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test_image_replay_state cluster1 mirror image_2 started 2023-11-08T05:14:08.274 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:14:08.275 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:14:08.275 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_2 2023-11-08T05:14:08.275 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local test_state=started 2023-11-08T05:14:08.275 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local status_result 2023-11-08T05:14:08.276 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local current_state=stopped 2023-11-08T05:14:08.276 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_2 2023-11-08T05:14:08.276 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_instance=cluster1 2023-11-08T05:14:08.276 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ grep -i state 2023-11-08T05:14:08.276 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T05:14:08.276 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1 2023-11-08T05:14:08.277 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance=cluster1 2023-11-08T05:14:08.277 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local loop_instance 2023-11-08T05:14:08.277 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-11-08T05:14:08.277 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sleep 0 2023-11-08T05:14:08.280 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-11-08T05:14:08.280 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ seq 0 3 2023-11-08T05:14:08.281 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T05:14:08.281 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_2 2023-11-08T05:14:08.281 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1:0 2023-11-08T05:14:08.281 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T05:14:08.281 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance 2023-11-08T05:14:08.282 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-11-08T05:14:08.282 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local val=cluster1:0 2023-11-08T05:14:08.282 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_var_name=cluster 2023-11-08T05:14:08.282 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance_var_name=instance 2023-11-08T05:14:08.282 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cluster=cluster1 2023-11-08T05:14:08.283 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ instance=0 2023-11-08T05:14:08.283 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' 0 = cluster1:0 ']' 2023-11-08T05:14:08.283 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval cluster=cluster1 2023-11-08T05:14:08.283 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:14:08.283 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval instance=0 2023-11-08T05:14:08.284 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=0 2023-11-08T05:14:08.284 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-11-08T05:14:08.284 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local local_cluster=cluster1:0 2023-11-08T05:14:08.284 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster=cluster1 2023-11-08T05:14:08.284 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance 2023-11-08T05:14:08.285 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-11-08T05:14:08.285 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local val=cluster1:0 2023-11-08T05:14:08.285 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster_var_name=local_cluster 2023-11-08T05:14:08.285 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance_var_name=instance 2023-11-08T05:14:08.285 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:14:08.286 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=0 2023-11-08T05:14:08.286 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ '[' 0 = cluster1:0 ']' 2023-11-08T05:14:08.286 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval local_cluster=cluster1 2023-11-08T05:14:08.286 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ local_cluster=cluster1 2023-11-08T05:14:08.286 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval instance=0 2023-11-08T05:14:08.287 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ instance=0 2023-11-08T05:14:08.287 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-11-08T05:14:08.296 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-08T05:14:08.296 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-08T05:14:08.296 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-08T05:14:08.296 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_2 2023-11-08T05:14:08.410 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:no valid command found; 1 closest matches: 2023-11-08T05:14:08.410 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd mirror status mirror/image_24 2023-11-08T05:14:08.411 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:admin_socket: invalid command 2023-11-08T05:14:08.413 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T05:14:08.414 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_2 2023-11-08T05:14:08.414 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1:1 2023-11-08T05:14:08.414 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T05:14:08.414 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance 2023-11-08T05:14:08.415 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-11-08T05:14:08.415 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local val=cluster1:1 2023-11-08T05:14:08.415 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_var_name=cluster 2023-11-08T05:14:08.415 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance_var_name=instance 2023-11-08T05:14:08.415 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cluster=cluster1 2023-11-08T05:14:08.416 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ instance=1 2023-11-08T05:14:08.416 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' 1 = cluster1:1 ']' 2023-11-08T05:14:08.416 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval cluster=cluster1 2023-11-08T05:14:08.416 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:14:08.416 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval instance=1 2023-11-08T05:14:08.417 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=1 2023-11-08T05:14:08.417 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-11-08T05:14:08.417 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local local_cluster=cluster1:1 2023-11-08T05:14:08.417 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster=cluster1 2023-11-08T05:14:08.417 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance 2023-11-08T05:14:08.417 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-11-08T05:14:08.418 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local val=cluster1:1 2023-11-08T05:14:08.418 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster_var_name=local_cluster 2023-11-08T05:14:08.418 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance_var_name=instance 2023-11-08T05:14:08.418 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:14:08.419 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=1 2023-11-08T05:14:08.419 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ '[' 1 = cluster1:1 ']' 2023-11-08T05:14:08.419 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval local_cluster=cluster1 2023-11-08T05:14:08.419 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ local_cluster=cluster1 2023-11-08T05:14:08.419 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval instance=1 2023-11-08T05:14:08.419 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ instance=1 2023-11-08T05:14:08.420 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-11-08T05:14:08.427 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-08T05:14:08.428 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-08T05:14:08.428 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-08T05:14:08.428 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_2 2023-11-08T05:14:08.528 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:no valid command found; 3 closest matches: 2023-11-08T05:14:08.528 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd mirror status mirror/image_20 2023-11-08T05:14:08.529 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd mirror status mirror/image_26 2023-11-08T05:14:08.529 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd mirror status mirror/image_28 2023-11-08T05:14:08.529 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:admin_socket: invalid command 2023-11-08T05:14:08.533 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T05:14:08.533 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_2 2023-11-08T05:14:08.533 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1:2 2023-11-08T05:14:08.533 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T05:14:08.533 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance 2023-11-08T05:14:08.534 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-11-08T05:14:08.534 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local val=cluster1:2 2023-11-08T05:14:08.534 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_var_name=cluster 2023-11-08T05:14:08.534 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance_var_name=instance 2023-11-08T05:14:08.534 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cluster=cluster1 2023-11-08T05:14:08.535 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ instance=2 2023-11-08T05:14:08.537 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' 2 = cluster1:2 ']' 2023-11-08T05:14:08.537 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval cluster=cluster1 2023-11-08T05:14:08.537 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:14:08.538 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval instance=2 2023-11-08T05:14:08.538 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=2 2023-11-08T05:14:08.538 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-11-08T05:14:08.538 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local local_cluster=cluster1:2 2023-11-08T05:14:08.538 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster=cluster1 2023-11-08T05:14:08.539 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance 2023-11-08T05:14:08.539 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-11-08T05:14:08.539 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local val=cluster1:2 2023-11-08T05:14:08.539 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster_var_name=local_cluster 2023-11-08T05:14:08.539 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance_var_name=instance 2023-11-08T05:14:08.540 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:14:08.540 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=2 2023-11-08T05:14:08.540 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ '[' 2 = cluster1:2 ']' 2023-11-08T05:14:08.540 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval local_cluster=cluster1 2023-11-08T05:14:08.540 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ local_cluster=cluster1 2023-11-08T05:14:08.541 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval instance=2 2023-11-08T05:14:08.541 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ instance=2 2023-11-08T05:14:08.541 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-11-08T05:14:08.546 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-08T05:14:08.546 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-08T05:14:08.547 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-08T05:14:08.547 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_2 2023-11-08T05:14:08.645 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:no valid command found; 2 closest matches: 2023-11-08T05:14:08.646 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd mirror status mirror/image_21 2023-11-08T05:14:08.646 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd mirror status mirror/image_29 2023-11-08T05:14:08.646 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:admin_socket: invalid command 2023-11-08T05:14:08.649 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T05:14:08.649 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_2 2023-11-08T05:14:08.649 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1:3 2023-11-08T05:14:08.650 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T05:14:08.650 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance 2023-11-08T05:14:08.650 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ set_cluster_instance cluster1:3 cluster instance 2023-11-08T05:14:08.650 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local val=cluster1:3 2023-11-08T05:14:08.650 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_var_name=cluster 2023-11-08T05:14:08.651 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance_var_name=instance 2023-11-08T05:14:08.651 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cluster=cluster1 2023-11-08T05:14:08.651 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ instance=3 2023-11-08T05:14:08.651 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' 3 = cluster1:3 ']' 2023-11-08T05:14:08.652 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval cluster=cluster1 2023-11-08T05:14:08.652 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:14:08.652 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval instance=3 2023-11-08T05:14:08.652 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=3 2023-11-08T05:14:08.652 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ daemon_asok_file cluster1:3 cluster1 2023-11-08T05:14:08.653 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local local_cluster=cluster1:3 2023-11-08T05:14:08.653 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster=cluster1 2023-11-08T05:14:08.653 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance 2023-11-08T05:14:08.653 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2023-11-08T05:14:08.653 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local val=cluster1:3 2023-11-08T05:14:08.654 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster_var_name=local_cluster 2023-11-08T05:14:08.654 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance_var_name=instance 2023-11-08T05:14:08.654 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:14:08.654 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=3 2023-11-08T05:14:08.654 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ '[' 3 = cluster1:3 ']' 2023-11-08T05:14:08.655 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval local_cluster=cluster1 2023-11-08T05:14:08.655 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ local_cluster=cluster1 2023-11-08T05:14:08.655 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval instance=3 2023-11-08T05:14:08.655 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ instance=3 2023-11-08T05:14:08.655 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2023-11-08T05:14:08.663 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-11-08T05:14:08.663 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-11-08T05:14:08.663 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-11-08T05:14:08.664 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_2 2023-11-08T05:14:08.752 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:no valid command found; 1 closest matches: 2023-11-08T05:14:08.753 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd mirror status mirror/image_27 2023-11-08T05:14:08.753 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:admin_socket: invalid command 2023-11-08T05:14:08.754 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-11-08T05:14:08.754 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sleep 1 2023-11-08T05:14:09.755 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-11-08T05:14:09.755 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ seq 0 3 2023-11-08T05:14:09.758 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T05:14:09.758 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_2 2023-11-08T05:14:09.759 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1:0 2023-11-08T05:14:09.759 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T05:14:09.759 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance 2023-11-08T05:14:09.759 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-11-08T05:14:09.759 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local val=cluster1:0 2023-11-08T05:14:09.760 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_var_name=cluster 2023-11-08T05:14:09.760 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance_var_name=instance 2023-11-08T05:14:09.760 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cluster=cluster1 2023-11-08T05:14:09.760 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ instance=0 2023-11-08T05:14:09.760 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' 0 = cluster1:0 ']' 2023-11-08T05:14:09.761 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval cluster=cluster1 2023-11-08T05:14:09.761 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:14:09.761 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval instance=0 2023-11-08T05:14:09.761 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=0 2023-11-08T05:14:09.761 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-11-08T05:14:09.762 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local local_cluster=cluster1:0 2023-11-08T05:14:09.762 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster=cluster1 2023-11-08T05:14:09.762 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance 2023-11-08T05:14:09.762 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-11-08T05:14:09.762 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local val=cluster1:0 2023-11-08T05:14:09.763 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster_var_name=local_cluster 2023-11-08T05:14:09.763 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance_var_name=instance 2023-11-08T05:14:09.763 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:14:09.763 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=0 2023-11-08T05:14:09.763 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ '[' 0 = cluster1:0 ']' 2023-11-08T05:14:09.764 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval local_cluster=cluster1 2023-11-08T05:14:09.764 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ local_cluster=cluster1 2023-11-08T05:14:09.764 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval instance=0 2023-11-08T05:14:09.764 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ instance=0 2023-11-08T05:14:09.764 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-11-08T05:14:09.773 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-08T05:14:09.773 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-08T05:14:09.773 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-08T05:14:09.774 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_2 2023-11-08T05:14:09.873 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:no valid command found; 1 closest matches: 2023-11-08T05:14:09.874 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd mirror status mirror/image_24 2023-11-08T05:14:09.874 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:admin_socket: invalid command 2023-11-08T05:14:09.876 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T05:14:09.876 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_2 2023-11-08T05:14:09.877 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1:1 2023-11-08T05:14:09.877 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T05:14:09.877 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance 2023-11-08T05:14:09.877 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-11-08T05:14:09.878 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local val=cluster1:1 2023-11-08T05:14:09.878 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_var_name=cluster 2023-11-08T05:14:09.878 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance_var_name=instance 2023-11-08T05:14:09.878 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cluster=cluster1 2023-11-08T05:14:09.879 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ instance=1 2023-11-08T05:14:09.879 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' 1 = cluster1:1 ']' 2023-11-08T05:14:09.879 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval cluster=cluster1 2023-11-08T05:14:09.879 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:14:09.879 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval instance=1 2023-11-08T05:14:09.879 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=1 2023-11-08T05:14:09.880 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-11-08T05:14:09.880 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local local_cluster=cluster1:1 2023-11-08T05:14:09.880 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster=cluster1 2023-11-08T05:14:09.880 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance 2023-11-08T05:14:09.880 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-11-08T05:14:09.881 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local val=cluster1:1 2023-11-08T05:14:09.881 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster_var_name=local_cluster 2023-11-08T05:14:09.881 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance_var_name=instance 2023-11-08T05:14:09.881 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:14:09.881 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=1 2023-11-08T05:14:09.882 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ '[' 1 = cluster1:1 ']' 2023-11-08T05:14:09.882 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval local_cluster=cluster1 2023-11-08T05:14:09.882 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ local_cluster=cluster1 2023-11-08T05:14:09.882 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval instance=1 2023-11-08T05:14:09.882 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ instance=1 2023-11-08T05:14:09.883 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-11-08T05:14:09.891 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-08T05:14:09.891 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-08T05:14:09.891 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-08T05:14:09.891 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_2 2023-11-08T05:14:09.989 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:no valid command found; 3 closest matches: 2023-11-08T05:14:09.990 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd mirror status mirror/image_20 2023-11-08T05:14:09.990 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd mirror status mirror/image_26 2023-11-08T05:14:09.990 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd mirror status mirror/image_28 2023-11-08T05:14:09.990 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:admin_socket: invalid command 2023-11-08T05:14:09.993 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T05:14:09.993 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_2 2023-11-08T05:14:09.994 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1:2 2023-11-08T05:14:09.994 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T05:14:09.994 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance 2023-11-08T05:14:09.994 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-11-08T05:14:09.994 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local val=cluster1:2 2023-11-08T05:14:09.995 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_var_name=cluster 2023-11-08T05:14:09.995 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance_var_name=instance 2023-11-08T05:14:09.995 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cluster=cluster1 2023-11-08T05:14:09.995 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ instance=2 2023-11-08T05:14:09.995 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' 2 = cluster1:2 ']' 2023-11-08T05:14:09.996 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval cluster=cluster1 2023-11-08T05:14:09.996 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:14:09.996 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval instance=2 2023-11-08T05:14:09.996 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=2 2023-11-08T05:14:09.996 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-11-08T05:14:09.997 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local local_cluster=cluster1:2 2023-11-08T05:14:09.997 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster=cluster1 2023-11-08T05:14:09.997 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance 2023-11-08T05:14:09.997 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-11-08T05:14:09.997 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local val=cluster1:2 2023-11-08T05:14:09.998 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster_var_name=local_cluster 2023-11-08T05:14:09.998 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance_var_name=instance 2023-11-08T05:14:09.998 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:14:09.998 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=2 2023-11-08T05:14:09.998 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ '[' 2 = cluster1:2 ']' 2023-11-08T05:14:09.999 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval local_cluster=cluster1 2023-11-08T05:14:09.999 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ local_cluster=cluster1 2023-11-08T05:14:09.999 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval instance=2 2023-11-08T05:14:09.999 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ instance=2 2023-11-08T05:14:09.999 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-11-08T05:14:10.006 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-08T05:14:10.007 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-08T05:14:10.007 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-08T05:14:10.007 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_2 2023-11-08T05:14:10.105 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:no valid command found; 2 closest matches: 2023-11-08T05:14:10.105 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd mirror status mirror/image_21 2023-11-08T05:14:10.106 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd mirror status mirror/image_29 2023-11-08T05:14:10.106 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:admin_socket: invalid command 2023-11-08T05:14:10.108 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T05:14:10.108 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_2 2023-11-08T05:14:10.109 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1:3 2023-11-08T05:14:10.109 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T05:14:10.109 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance 2023-11-08T05:14:10.109 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ set_cluster_instance cluster1:3 cluster instance 2023-11-08T05:14:10.109 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local val=cluster1:3 2023-11-08T05:14:10.110 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_var_name=cluster 2023-11-08T05:14:10.110 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance_var_name=instance 2023-11-08T05:14:10.110 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cluster=cluster1 2023-11-08T05:14:10.110 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ instance=3 2023-11-08T05:14:10.110 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' 3 = cluster1:3 ']' 2023-11-08T05:14:10.111 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval cluster=cluster1 2023-11-08T05:14:10.111 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:14:10.111 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval instance=3 2023-11-08T05:14:10.111 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=3 2023-11-08T05:14:10.111 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ daemon_asok_file cluster1:3 cluster1 2023-11-08T05:14:10.112 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local local_cluster=cluster1:3 2023-11-08T05:14:10.112 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster=cluster1 2023-11-08T05:14:10.112 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance 2023-11-08T05:14:10.112 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2023-11-08T05:14:10.112 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local val=cluster1:3 2023-11-08T05:14:10.113 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster_var_name=local_cluster 2023-11-08T05:14:10.113 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance_var_name=instance 2023-11-08T05:14:10.113 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:14:10.113 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=3 2023-11-08T05:14:10.113 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ '[' 3 = cluster1:3 ']' 2023-11-08T05:14:10.113 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval local_cluster=cluster1 2023-11-08T05:14:10.114 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ local_cluster=cluster1 2023-11-08T05:14:10.114 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval instance=3 2023-11-08T05:14:10.114 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ instance=3 2023-11-08T05:14:10.114 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2023-11-08T05:14:10.123 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-11-08T05:14:10.123 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-11-08T05:14:10.123 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-11-08T05:14:10.124 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_2 2023-11-08T05:14:10.212 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:no valid command found; 1 closest matches: 2023-11-08T05:14:10.212 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd mirror status mirror/image_27 2023-11-08T05:14:10.213 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:admin_socket: invalid command 2023-11-08T05:14:10.213 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-11-08T05:14:10.213 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sleep 2 2023-11-08T05:14:12.214 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-11-08T05:14:12.215 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ seq 0 3 2023-11-08T05:14:12.218 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T05:14:12.218 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_2 2023-11-08T05:14:12.218 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1:0 2023-11-08T05:14:12.219 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T05:14:12.219 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance 2023-11-08T05:14:12.219 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-11-08T05:14:12.219 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local val=cluster1:0 2023-11-08T05:14:12.220 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_var_name=cluster 2023-11-08T05:14:12.220 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance_var_name=instance 2023-11-08T05:14:12.220 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cluster=cluster1 2023-11-08T05:14:12.220 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ instance=0 2023-11-08T05:14:12.220 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' 0 = cluster1:0 ']' 2023-11-08T05:14:12.221 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval cluster=cluster1 2023-11-08T05:14:12.221 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:14:12.221 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval instance=0 2023-11-08T05:14:12.221 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=0 2023-11-08T05:14:12.221 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-11-08T05:14:12.222 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local local_cluster=cluster1:0 2023-11-08T05:14:12.222 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster=cluster1 2023-11-08T05:14:12.222 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance 2023-11-08T05:14:12.222 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-11-08T05:14:12.222 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local val=cluster1:0 2023-11-08T05:14:12.223 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster_var_name=local_cluster 2023-11-08T05:14:12.223 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance_var_name=instance 2023-11-08T05:14:12.223 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:14:12.223 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=0 2023-11-08T05:14:12.224 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ '[' 0 = cluster1:0 ']' 2023-11-08T05:14:12.224 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval local_cluster=cluster1 2023-11-08T05:14:12.224 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ local_cluster=cluster1 2023-11-08T05:14:12.224 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval instance=0 2023-11-08T05:14:12.224 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ instance=0 2023-11-08T05:14:12.224 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-11-08T05:14:12.235 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-08T05:14:12.235 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-08T05:14:12.235 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-08T05:14:12.236 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_2 2023-11-08T05:14:12.343 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:no valid command found; 1 closest matches: 2023-11-08T05:14:12.344 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd mirror status mirror/image_24 2023-11-08T05:14:12.344 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:admin_socket: invalid command 2023-11-08T05:14:12.348 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T05:14:12.348 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_2 2023-11-08T05:14:12.348 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1:1 2023-11-08T05:14:12.348 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T05:14:12.348 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance 2023-11-08T05:14:12.349 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-11-08T05:14:12.349 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local val=cluster1:1 2023-11-08T05:14:12.349 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_var_name=cluster 2023-11-08T05:14:12.349 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance_var_name=instance 2023-11-08T05:14:12.349 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cluster=cluster1 2023-11-08T05:14:12.350 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ instance=1 2023-11-08T05:14:12.350 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' 1 = cluster1:1 ']' 2023-11-08T05:14:12.350 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval cluster=cluster1 2023-11-08T05:14:12.351 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:14:12.351 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval instance=1 2023-11-08T05:14:12.351 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=1 2023-11-08T05:14:12.351 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-11-08T05:14:12.352 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local local_cluster=cluster1:1 2023-11-08T05:14:12.352 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster=cluster1 2023-11-08T05:14:12.352 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance 2023-11-08T05:14:12.352 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-11-08T05:14:12.353 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local val=cluster1:1 2023-11-08T05:14:12.353 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster_var_name=local_cluster 2023-11-08T05:14:12.353 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance_var_name=instance 2023-11-08T05:14:12.354 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:14:12.354 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=1 2023-11-08T05:14:12.354 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ '[' 1 = cluster1:1 ']' 2023-11-08T05:14:12.354 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval local_cluster=cluster1 2023-11-08T05:14:12.355 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ local_cluster=cluster1 2023-11-08T05:14:12.355 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval instance=1 2023-11-08T05:14:12.355 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ instance=1 2023-11-08T05:14:12.355 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-11-08T05:14:12.361 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-08T05:14:12.361 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-08T05:14:12.361 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-08T05:14:12.361 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_2 2023-11-08T05:14:12.462 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:no valid command found; 3 closest matches: 2023-11-08T05:14:12.463 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd mirror status mirror/image_20 2023-11-08T05:14:12.463 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd mirror status mirror/image_26 2023-11-08T05:14:12.463 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd mirror status mirror/image_28 2023-11-08T05:14:12.464 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:admin_socket: invalid command 2023-11-08T05:14:12.468 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T05:14:12.468 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_2 2023-11-08T05:14:12.468 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1:2 2023-11-08T05:14:12.469 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T05:14:12.469 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance 2023-11-08T05:14:12.469 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-11-08T05:14:12.470 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local val=cluster1:2 2023-11-08T05:14:12.470 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_var_name=cluster 2023-11-08T05:14:12.470 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance_var_name=instance 2023-11-08T05:14:12.470 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cluster=cluster1 2023-11-08T05:14:12.471 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ instance=2 2023-11-08T05:14:12.471 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' 2 = cluster1:2 ']' 2023-11-08T05:14:12.471 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval cluster=cluster1 2023-11-08T05:14:12.472 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:14:12.472 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval instance=2 2023-11-08T05:14:12.472 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=2 2023-11-08T05:14:12.472 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-11-08T05:14:12.473 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local local_cluster=cluster1:2 2023-11-08T05:14:12.473 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster=cluster1 2023-11-08T05:14:12.473 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance 2023-11-08T05:14:12.473 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-11-08T05:14:12.474 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local val=cluster1:2 2023-11-08T05:14:12.474 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster_var_name=local_cluster 2023-11-08T05:14:12.474 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance_var_name=instance 2023-11-08T05:14:12.475 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:14:12.475 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=2 2023-11-08T05:14:12.475 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ '[' 2 = cluster1:2 ']' 2023-11-08T05:14:12.475 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval local_cluster=cluster1 2023-11-08T05:14:12.476 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ local_cluster=cluster1 2023-11-08T05:14:12.476 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval instance=2 2023-11-08T05:14:12.476 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ instance=2 2023-11-08T05:14:12.477 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-11-08T05:14:12.480 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-08T05:14:12.480 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-08T05:14:12.480 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-08T05:14:12.481 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_2 2023-11-08T05:14:12.579 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:no valid command found; 2 closest matches: 2023-11-08T05:14:12.579 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd mirror status mirror/image_21 2023-11-08T05:14:12.579 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd mirror status mirror/image_29 2023-11-08T05:14:12.580 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:admin_socket: invalid command 2023-11-08T05:14:12.582 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T05:14:12.583 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_2 2023-11-08T05:14:12.583 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1:3 2023-11-08T05:14:12.583 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T05:14:12.583 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance 2023-11-08T05:14:12.583 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ set_cluster_instance cluster1:3 cluster instance 2023-11-08T05:14:12.584 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local val=cluster1:3 2023-11-08T05:14:12.584 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_var_name=cluster 2023-11-08T05:14:12.584 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance_var_name=instance 2023-11-08T05:14:12.584 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cluster=cluster1 2023-11-08T05:14:12.584 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ instance=3 2023-11-08T05:14:12.585 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' 3 = cluster1:3 ']' 2023-11-08T05:14:12.585 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval cluster=cluster1 2023-11-08T05:14:12.585 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:14:12.585 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval instance=3 2023-11-08T05:14:12.585 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=3 2023-11-08T05:14:12.586 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ daemon_asok_file cluster1:3 cluster1 2023-11-08T05:14:12.586 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local local_cluster=cluster1:3 2023-11-08T05:14:12.586 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster=cluster1 2023-11-08T05:14:12.586 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance 2023-11-08T05:14:12.586 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2023-11-08T05:14:12.587 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local val=cluster1:3 2023-11-08T05:14:12.587 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster_var_name=local_cluster 2023-11-08T05:14:12.587 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance_var_name=instance 2023-11-08T05:14:12.587 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:14:12.587 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=3 2023-11-08T05:14:12.588 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ '[' 3 = cluster1:3 ']' 2023-11-08T05:14:12.588 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval local_cluster=cluster1 2023-11-08T05:14:12.588 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ local_cluster=cluster1 2023-11-08T05:14:12.588 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval instance=3 2023-11-08T05:14:12.588 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ instance=3 2023-11-08T05:14:12.588 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2023-11-08T05:14:12.596 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-11-08T05:14:12.596 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-11-08T05:14:12.596 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-11-08T05:14:12.597 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_2 2023-11-08T05:14:12.686 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:no valid command found; 1 closest matches: 2023-11-08T05:14:12.686 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd mirror status mirror/image_27 2023-11-08T05:14:12.686 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:admin_socket: invalid command 2023-11-08T05:14:12.687 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-11-08T05:14:12.687 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sleep 4 2023-11-08T05:14:16.688 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-11-08T05:14:16.689 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ seq 0 3 2023-11-08T05:14:16.692 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T05:14:16.692 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_2 2023-11-08T05:14:16.692 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1:0 2023-11-08T05:14:16.692 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T05:14:16.693 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance 2023-11-08T05:14:16.693 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-11-08T05:14:16.693 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local val=cluster1:0 2023-11-08T05:14:16.693 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_var_name=cluster 2023-11-08T05:14:16.693 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance_var_name=instance 2023-11-08T05:14:16.694 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cluster=cluster1 2023-11-08T05:14:16.694 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ instance=0 2023-11-08T05:14:16.694 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' 0 = cluster1:0 ']' 2023-11-08T05:14:16.694 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval cluster=cluster1 2023-11-08T05:14:16.694 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:14:16.695 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval instance=0 2023-11-08T05:14:16.695 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=0 2023-11-08T05:14:16.695 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-11-08T05:14:16.695 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local local_cluster=cluster1:0 2023-11-08T05:14:16.695 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster=cluster1 2023-11-08T05:14:16.696 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance 2023-11-08T05:14:16.696 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-11-08T05:14:16.696 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local val=cluster1:0 2023-11-08T05:14:16.696 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster_var_name=local_cluster 2023-11-08T05:14:16.696 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance_var_name=instance 2023-11-08T05:14:16.697 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:14:16.697 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=0 2023-11-08T05:14:16.697 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ '[' 0 = cluster1:0 ']' 2023-11-08T05:14:16.697 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval local_cluster=cluster1 2023-11-08T05:14:16.697 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ local_cluster=cluster1 2023-11-08T05:14:16.698 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval instance=0 2023-11-08T05:14:16.698 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ instance=0 2023-11-08T05:14:16.698 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-11-08T05:14:16.709 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-08T05:14:16.709 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-08T05:14:16.709 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-08T05:14:16.710 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_2 2023-11-08T05:14:16.814 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:no valid command found; 1 closest matches: 2023-11-08T05:14:16.814 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd mirror status mirror/image_24 2023-11-08T05:14:16.815 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:admin_socket: invalid command 2023-11-08T05:14:16.818 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T05:14:16.818 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_2 2023-11-08T05:14:16.818 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1:1 2023-11-08T05:14:16.818 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T05:14:16.818 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance 2023-11-08T05:14:16.819 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-11-08T05:14:16.819 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local val=cluster1:1 2023-11-08T05:14:16.819 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_var_name=cluster 2023-11-08T05:14:16.819 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance_var_name=instance 2023-11-08T05:14:16.819 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cluster=cluster1 2023-11-08T05:14:16.820 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ instance=1 2023-11-08T05:14:16.820 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' 1 = cluster1:1 ']' 2023-11-08T05:14:16.820 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval cluster=cluster1 2023-11-08T05:14:16.820 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:14:16.820 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval instance=1 2023-11-08T05:14:16.821 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=1 2023-11-08T05:14:16.821 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-11-08T05:14:16.821 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local local_cluster=cluster1:1 2023-11-08T05:14:16.821 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster=cluster1 2023-11-08T05:14:16.821 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance 2023-11-08T05:14:16.822 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-11-08T05:14:16.822 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local val=cluster1:1 2023-11-08T05:14:16.822 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster_var_name=local_cluster 2023-11-08T05:14:16.822 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance_var_name=instance 2023-11-08T05:14:16.822 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:14:16.823 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=1 2023-11-08T05:14:16.823 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ '[' 1 = cluster1:1 ']' 2023-11-08T05:14:16.823 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval local_cluster=cluster1 2023-11-08T05:14:16.823 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ local_cluster=cluster1 2023-11-08T05:14:16.823 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval instance=1 2023-11-08T05:14:16.824 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ instance=1 2023-11-08T05:14:16.824 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-11-08T05:14:16.831 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-08T05:14:16.832 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-08T05:14:16.832 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-08T05:14:16.832 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_2 2023-11-08T05:14:16.930 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:no valid command found; 3 closest matches: 2023-11-08T05:14:16.931 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd mirror status mirror/image_20 2023-11-08T05:14:16.931 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd mirror status mirror/image_26 2023-11-08T05:14:16.931 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd mirror status mirror/image_28 2023-11-08T05:14:16.931 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:admin_socket: invalid command 2023-11-08T05:14:16.934 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T05:14:16.934 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_2 2023-11-08T05:14:16.934 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1:2 2023-11-08T05:14:16.934 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T05:14:16.935 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance 2023-11-08T05:14:16.935 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-11-08T05:14:16.935 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local val=cluster1:2 2023-11-08T05:14:16.935 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_var_name=cluster 2023-11-08T05:14:16.935 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance_var_name=instance 2023-11-08T05:14:16.936 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cluster=cluster1 2023-11-08T05:14:16.936 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ instance=2 2023-11-08T05:14:16.936 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' 2 = cluster1:2 ']' 2023-11-08T05:14:16.936 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval cluster=cluster1 2023-11-08T05:14:16.936 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:14:16.937 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval instance=2 2023-11-08T05:14:16.937 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=2 2023-11-08T05:14:16.937 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-11-08T05:14:16.937 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local local_cluster=cluster1:2 2023-11-08T05:14:16.937 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster=cluster1 2023-11-08T05:14:16.938 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance 2023-11-08T05:14:16.938 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-11-08T05:14:16.938 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local val=cluster1:2 2023-11-08T05:14:16.938 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster_var_name=local_cluster 2023-11-08T05:14:16.938 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance_var_name=instance 2023-11-08T05:14:16.938 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:14:16.939 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=2 2023-11-08T05:14:16.939 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ '[' 2 = cluster1:2 ']' 2023-11-08T05:14:16.939 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval local_cluster=cluster1 2023-11-08T05:14:16.939 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ local_cluster=cluster1 2023-11-08T05:14:16.939 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval instance=2 2023-11-08T05:14:16.940 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ instance=2 2023-11-08T05:14:16.940 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-11-08T05:14:16.947 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-08T05:14:16.949 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-08T05:14:16.949 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-08T05:14:16.949 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_2 2023-11-08T05:14:17.045 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:no valid command found; 2 closest matches: 2023-11-08T05:14:17.045 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd mirror status mirror/image_21 2023-11-08T05:14:17.046 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd mirror status mirror/image_29 2023-11-08T05:14:17.046 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:admin_socket: invalid command 2023-11-08T05:14:17.049 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T05:14:17.049 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_2 2023-11-08T05:14:17.049 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1:3 2023-11-08T05:14:17.049 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T05:14:17.049 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance 2023-11-08T05:14:17.050 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ set_cluster_instance cluster1:3 cluster instance 2023-11-08T05:14:17.050 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local val=cluster1:3 2023-11-08T05:14:17.050 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_var_name=cluster 2023-11-08T05:14:17.050 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance_var_name=instance 2023-11-08T05:14:17.050 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cluster=cluster1 2023-11-08T05:14:17.051 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ instance=3 2023-11-08T05:14:17.051 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' 3 = cluster1:3 ']' 2023-11-08T05:14:17.051 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval cluster=cluster1 2023-11-08T05:14:17.051 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:14:17.051 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval instance=3 2023-11-08T05:14:17.052 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=3 2023-11-08T05:14:17.052 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ daemon_asok_file cluster1:3 cluster1 2023-11-08T05:14:17.052 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local local_cluster=cluster1:3 2023-11-08T05:14:17.052 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster=cluster1 2023-11-08T05:14:17.052 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance 2023-11-08T05:14:17.053 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2023-11-08T05:14:17.053 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local val=cluster1:3 2023-11-08T05:14:17.053 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster_var_name=local_cluster 2023-11-08T05:14:17.053 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance_var_name=instance 2023-11-08T05:14:17.054 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:14:17.054 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=3 2023-11-08T05:14:17.054 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ '[' 3 = cluster1:3 ']' 2023-11-08T05:14:17.054 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval local_cluster=cluster1 2023-11-08T05:14:17.055 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ local_cluster=cluster1 2023-11-08T05:14:17.055 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval instance=3 2023-11-08T05:14:17.055 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ instance=3 2023-11-08T05:14:17.055 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2023-11-08T05:14:17.063 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-11-08T05:14:17.063 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-11-08T05:14:17.063 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-11-08T05:14:17.064 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_2 2023-11-08T05:14:17.153 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:no valid command found; 1 closest matches: 2023-11-08T05:14:17.153 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd mirror status mirror/image_27 2023-11-08T05:14:17.154 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:admin_socket: invalid command 2023-11-08T05:14:17.154 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-11-08T05:14:17.154 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sleep 8 2023-11-08T05:14:25.155 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-11-08T05:14:25.155 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ seq 0 3 2023-11-08T05:14:25.158 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T05:14:25.158 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_2 2023-11-08T05:14:25.158 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1:0 2023-11-08T05:14:25.159 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T05:14:25.159 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance 2023-11-08T05:14:25.159 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-11-08T05:14:25.159 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local val=cluster1:0 2023-11-08T05:14:25.159 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_var_name=cluster 2023-11-08T05:14:25.160 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance_var_name=instance 2023-11-08T05:14:25.160 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cluster=cluster1 2023-11-08T05:14:25.160 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ instance=0 2023-11-08T05:14:25.160 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' 0 = cluster1:0 ']' 2023-11-08T05:14:25.160 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval cluster=cluster1 2023-11-08T05:14:25.161 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:14:25.161 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval instance=0 2023-11-08T05:14:25.161 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=0 2023-11-08T05:14:25.161 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-11-08T05:14:25.161 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local local_cluster=cluster1:0 2023-11-08T05:14:25.162 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster=cluster1 2023-11-08T05:14:25.162 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance 2023-11-08T05:14:25.162 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-11-08T05:14:25.162 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local val=cluster1:0 2023-11-08T05:14:25.162 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster_var_name=local_cluster 2023-11-08T05:14:25.163 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance_var_name=instance 2023-11-08T05:14:25.163 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:14:25.163 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=0 2023-11-08T05:14:25.163 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ '[' 0 = cluster1:0 ']' 2023-11-08T05:14:25.163 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval local_cluster=cluster1 2023-11-08T05:14:25.164 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ local_cluster=cluster1 2023-11-08T05:14:25.164 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval instance=0 2023-11-08T05:14:25.164 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ instance=0 2023-11-08T05:14:25.164 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-11-08T05:14:25.174 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-08T05:14:25.175 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-08T05:14:25.175 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-08T05:14:25.175 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_2 2023-11-08T05:14:25.281 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:no valid command found; 1 closest matches: 2023-11-08T05:14:25.282 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd mirror status mirror/image_24 2023-11-08T05:14:25.282 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:admin_socket: invalid command 2023-11-08T05:14:25.286 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T05:14:25.286 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_2 2023-11-08T05:14:25.287 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1:1 2023-11-08T05:14:25.287 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T05:14:25.287 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance 2023-11-08T05:14:25.287 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-11-08T05:14:25.287 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local val=cluster1:1 2023-11-08T05:14:25.288 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_var_name=cluster 2023-11-08T05:14:25.288 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance_var_name=instance 2023-11-08T05:14:25.288 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cluster=cluster1 2023-11-08T05:14:25.288 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ instance=1 2023-11-08T05:14:25.288 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' 1 = cluster1:1 ']' 2023-11-08T05:14:25.288 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval cluster=cluster1 2023-11-08T05:14:25.289 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:14:25.289 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval instance=1 2023-11-08T05:14:25.289 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=1 2023-11-08T05:14:25.289 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-11-08T05:14:25.289 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local local_cluster=cluster1:1 2023-11-08T05:14:25.290 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster=cluster1 2023-11-08T05:14:25.290 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance 2023-11-08T05:14:25.290 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-11-08T05:14:25.290 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local val=cluster1:1 2023-11-08T05:14:25.291 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster_var_name=local_cluster 2023-11-08T05:14:25.291 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance_var_name=instance 2023-11-08T05:14:25.291 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:14:25.291 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=1 2023-11-08T05:14:25.291 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ '[' 1 = cluster1:1 ']' 2023-11-08T05:14:25.291 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval local_cluster=cluster1 2023-11-08T05:14:25.292 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ local_cluster=cluster1 2023-11-08T05:14:25.292 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval instance=1 2023-11-08T05:14:25.292 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ instance=1 2023-11-08T05:14:25.292 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-11-08T05:14:25.299 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-08T05:14:25.299 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-08T05:14:25.300 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-08T05:14:25.300 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_2 2023-11-08T05:14:25.401 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:no valid command found; 3 closest matches: 2023-11-08T05:14:25.401 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd mirror status mirror/image_20 2023-11-08T05:14:25.401 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd mirror status mirror/image_26 2023-11-08T05:14:25.402 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd mirror status mirror/image_28 2023-11-08T05:14:25.402 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:admin_socket: invalid command 2023-11-08T05:14:25.405 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T05:14:25.405 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_2 2023-11-08T05:14:25.405 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1:2 2023-11-08T05:14:25.405 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T05:14:25.405 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance 2023-11-08T05:14:25.406 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-11-08T05:14:25.406 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local val=cluster1:2 2023-11-08T05:14:25.406 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_var_name=cluster 2023-11-08T05:14:25.406 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance_var_name=instance 2023-11-08T05:14:25.407 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cluster=cluster1 2023-11-08T05:14:25.407 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ instance=2 2023-11-08T05:14:25.407 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' 2 = cluster1:2 ']' 2023-11-08T05:14:25.407 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval cluster=cluster1 2023-11-08T05:14:25.407 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:14:25.407 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval instance=2 2023-11-08T05:14:25.408 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=2 2023-11-08T05:14:25.408 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-11-08T05:14:25.408 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local local_cluster=cluster1:2 2023-11-08T05:14:25.408 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster=cluster1 2023-11-08T05:14:25.409 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance 2023-11-08T05:14:25.409 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-11-08T05:14:25.409 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local val=cluster1:2 2023-11-08T05:14:25.409 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster_var_name=local_cluster 2023-11-08T05:14:25.409 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance_var_name=instance 2023-11-08T05:14:25.409 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:14:25.410 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=2 2023-11-08T05:14:25.410 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ '[' 2 = cluster1:2 ']' 2023-11-08T05:14:25.410 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval local_cluster=cluster1 2023-11-08T05:14:25.410 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ local_cluster=cluster1 2023-11-08T05:14:25.411 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval instance=2 2023-11-08T05:14:25.411 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ instance=2 2023-11-08T05:14:25.411 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-11-08T05:14:25.418 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-08T05:14:25.419 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-08T05:14:25.419 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-08T05:14:25.419 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_2 2023-11-08T05:14:25.521 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:no valid command found; 2 closest matches: 2023-11-08T05:14:25.522 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd mirror status mirror/image_21 2023-11-08T05:14:25.522 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd mirror status mirror/image_29 2023-11-08T05:14:25.522 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:admin_socket: invalid command 2023-11-08T05:14:25.525 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T05:14:25.525 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_2 2023-11-08T05:14:25.525 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1:3 2023-11-08T05:14:25.526 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T05:14:25.526 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance 2023-11-08T05:14:25.526 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ set_cluster_instance cluster1:3 cluster instance 2023-11-08T05:14:25.527 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local val=cluster1:3 2023-11-08T05:14:25.527 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_var_name=cluster 2023-11-08T05:14:25.527 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance_var_name=instance 2023-11-08T05:14:25.527 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cluster=cluster1 2023-11-08T05:14:25.527 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ instance=3 2023-11-08T05:14:25.527 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' 3 = cluster1:3 ']' 2023-11-08T05:14:25.528 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval cluster=cluster1 2023-11-08T05:14:25.528 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:14:25.528 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval instance=3 2023-11-08T05:14:25.528 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=3 2023-11-08T05:14:25.528 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ daemon_asok_file cluster1:3 cluster1 2023-11-08T05:14:25.529 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local local_cluster=cluster1:3 2023-11-08T05:14:25.529 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster=cluster1 2023-11-08T05:14:25.529 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance 2023-11-08T05:14:25.529 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2023-11-08T05:14:25.529 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local val=cluster1:3 2023-11-08T05:14:25.530 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster_var_name=local_cluster 2023-11-08T05:14:25.530 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance_var_name=instance 2023-11-08T05:14:25.530 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:14:25.530 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=3 2023-11-08T05:14:25.531 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ '[' 3 = cluster1:3 ']' 2023-11-08T05:14:25.531 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval local_cluster=cluster1 2023-11-08T05:14:25.531 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ local_cluster=cluster1 2023-11-08T05:14:25.531 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval instance=3 2023-11-08T05:14:25.531 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ instance=3 2023-11-08T05:14:25.532 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2023-11-08T05:14:25.539 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-11-08T05:14:25.540 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-11-08T05:14:25.540 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-11-08T05:14:25.540 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_2 2023-11-08T05:14:25.629 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:no valid command found; 1 closest matches: 2023-11-08T05:14:25.629 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd mirror status mirror/image_27 2023-11-08T05:14:25.630 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:admin_socket: invalid command 2023-11-08T05:14:25.630 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-11-08T05:14:25.631 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sleep 8 2023-11-08T05:14:33.631 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-11-08T05:14:33.632 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ seq 0 3 2023-11-08T05:14:33.635 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T05:14:33.636 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_2 2023-11-08T05:14:33.636 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1:0 2023-11-08T05:14:33.636 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T05:14:33.636 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance 2023-11-08T05:14:33.636 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-11-08T05:14:33.637 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local val=cluster1:0 2023-11-08T05:14:33.637 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_var_name=cluster 2023-11-08T05:14:33.637 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance_var_name=instance 2023-11-08T05:14:33.637 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cluster=cluster1 2023-11-08T05:14:33.637 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ instance=0 2023-11-08T05:14:33.637 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' 0 = cluster1:0 ']' 2023-11-08T05:14:33.638 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval cluster=cluster1 2023-11-08T05:14:33.638 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:14:33.638 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval instance=0 2023-11-08T05:14:33.638 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=0 2023-11-08T05:14:33.639 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-11-08T05:14:33.639 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local local_cluster=cluster1:0 2023-11-08T05:14:33.639 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster=cluster1 2023-11-08T05:14:33.639 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance 2023-11-08T05:14:33.639 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-11-08T05:14:33.640 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local val=cluster1:0 2023-11-08T05:14:33.640 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster_var_name=local_cluster 2023-11-08T05:14:33.640 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance_var_name=instance 2023-11-08T05:14:33.640 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:14:33.640 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=0 2023-11-08T05:14:33.640 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ '[' 0 = cluster1:0 ']' 2023-11-08T05:14:33.641 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval local_cluster=cluster1 2023-11-08T05:14:33.641 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ local_cluster=cluster1 2023-11-08T05:14:33.641 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval instance=0 2023-11-08T05:14:33.641 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ instance=0 2023-11-08T05:14:33.641 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-11-08T05:14:33.652 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-08T05:14:33.652 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-08T05:14:33.652 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-08T05:14:33.652 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_2 2023-11-08T05:14:33.758 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:no valid command found; 1 closest matches: 2023-11-08T05:14:33.759 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd mirror status mirror/image_24 2023-11-08T05:14:33.759 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:admin_socket: invalid command 2023-11-08T05:14:33.761 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T05:14:33.762 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_2 2023-11-08T05:14:33.762 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1:1 2023-11-08T05:14:33.762 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T05:14:33.762 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance 2023-11-08T05:14:33.763 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-11-08T05:14:33.765 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local val=cluster1:1 2023-11-08T05:14:33.765 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_var_name=cluster 2023-11-08T05:14:33.765 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance_var_name=instance 2023-11-08T05:14:33.766 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cluster=cluster1 2023-11-08T05:14:33.766 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ instance=1 2023-11-08T05:14:33.766 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' 1 = cluster1:1 ']' 2023-11-08T05:14:33.766 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval cluster=cluster1 2023-11-08T05:14:33.766 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:14:33.767 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval instance=1 2023-11-08T05:14:33.767 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=1 2023-11-08T05:14:33.767 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-11-08T05:14:33.767 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local local_cluster=cluster1:1 2023-11-08T05:14:33.767 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster=cluster1 2023-11-08T05:14:33.767 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance 2023-11-08T05:14:33.768 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-11-08T05:14:33.768 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local val=cluster1:1 2023-11-08T05:14:33.768 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster_var_name=local_cluster 2023-11-08T05:14:33.768 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance_var_name=instance 2023-11-08T05:14:33.768 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:14:33.769 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=1 2023-11-08T05:14:33.769 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ '[' 1 = cluster1:1 ']' 2023-11-08T05:14:33.769 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval local_cluster=cluster1 2023-11-08T05:14:33.769 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ local_cluster=cluster1 2023-11-08T05:14:33.769 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval instance=1 2023-11-08T05:14:33.769 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ instance=1 2023-11-08T05:14:33.770 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-11-08T05:14:33.776 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-08T05:14:33.776 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-08T05:14:33.776 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-08T05:14:33.776 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_2 2023-11-08T05:14:33.875 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:no valid command found; 3 closest matches: 2023-11-08T05:14:33.876 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd mirror status mirror/image_20 2023-11-08T05:14:33.876 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd mirror status mirror/image_26 2023-11-08T05:14:33.877 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd mirror status mirror/image_28 2023-11-08T05:14:33.877 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:admin_socket: invalid command 2023-11-08T05:14:33.882 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T05:14:33.882 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_2 2023-11-08T05:14:33.883 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1:2 2023-11-08T05:14:33.883 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T05:14:33.883 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance 2023-11-08T05:14:33.884 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-11-08T05:14:33.884 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local val=cluster1:2 2023-11-08T05:14:33.884 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_var_name=cluster 2023-11-08T05:14:33.885 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance_var_name=instance 2023-11-08T05:14:33.885 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cluster=cluster1 2023-11-08T05:14:33.886 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ instance=2 2023-11-08T05:14:33.886 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' 2 = cluster1:2 ']' 2023-11-08T05:14:33.886 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval cluster=cluster1 2023-11-08T05:14:33.886 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:14:33.887 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval instance=2 2023-11-08T05:14:33.887 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=2 2023-11-08T05:14:33.887 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-11-08T05:14:33.887 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local local_cluster=cluster1:2 2023-11-08T05:14:33.887 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster=cluster1 2023-11-08T05:14:33.888 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance 2023-11-08T05:14:33.888 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-11-08T05:14:33.888 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local val=cluster1:2 2023-11-08T05:14:33.888 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster_var_name=local_cluster 2023-11-08T05:14:33.888 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance_var_name=instance 2023-11-08T05:14:33.889 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:14:33.889 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=2 2023-11-08T05:14:33.889 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ '[' 2 = cluster1:2 ']' 2023-11-08T05:14:33.889 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval local_cluster=cluster1 2023-11-08T05:14:33.889 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ local_cluster=cluster1 2023-11-08T05:14:33.889 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval instance=2 2023-11-08T05:14:33.890 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ instance=2 2023-11-08T05:14:33.890 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-11-08T05:14:33.893 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-08T05:14:33.894 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-08T05:14:33.894 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-08T05:14:33.894 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_2 2023-11-08T05:14:33.994 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:no valid command found; 2 closest matches: 2023-11-08T05:14:33.994 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd mirror status mirror/image_21 2023-11-08T05:14:33.994 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd mirror status mirror/image_29 2023-11-08T05:14:33.995 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:admin_socket: invalid command 2023-11-08T05:14:33.997 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T05:14:33.998 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_2 2023-11-08T05:14:33.998 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1:3 2023-11-08T05:14:33.998 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T05:14:33.998 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance 2023-11-08T05:14:33.998 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ set_cluster_instance cluster1:3 cluster instance 2023-11-08T05:14:33.999 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local val=cluster1:3 2023-11-08T05:14:33.999 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_var_name=cluster 2023-11-08T05:14:33.999 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance_var_name=instance 2023-11-08T05:14:33.999 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cluster=cluster1 2023-11-08T05:14:33.999 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ instance=3 2023-11-08T05:14:34.000 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' 3 = cluster1:3 ']' 2023-11-08T05:14:34.000 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval cluster=cluster1 2023-11-08T05:14:34.000 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:14:34.000 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval instance=3 2023-11-08T05:14:34.000 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=3 2023-11-08T05:14:34.001 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ daemon_asok_file cluster1:3 cluster1 2023-11-08T05:14:34.001 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local local_cluster=cluster1:3 2023-11-08T05:14:34.001 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster=cluster1 2023-11-08T05:14:34.001 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance 2023-11-08T05:14:34.001 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2023-11-08T05:14:34.002 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local val=cluster1:3 2023-11-08T05:14:34.002 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster_var_name=local_cluster 2023-11-08T05:14:34.002 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance_var_name=instance 2023-11-08T05:14:34.002 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:14:34.002 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=3 2023-11-08T05:14:34.003 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ '[' 3 = cluster1:3 ']' 2023-11-08T05:14:34.003 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval local_cluster=cluster1 2023-11-08T05:14:34.003 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ local_cluster=cluster1 2023-11-08T05:14:34.003 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval instance=3 2023-11-08T05:14:34.003 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ instance=3 2023-11-08T05:14:34.004 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2023-11-08T05:14:34.011 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-11-08T05:14:34.011 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-11-08T05:14:34.011 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-11-08T05:14:34.012 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_2 2023-11-08T05:14:34.118 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ return 0 2023-11-08T05:14:34.119 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ status_result=' "state": "Replaying"' 2023-11-08T05:14:34.119 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ echo ' "state": "Replaying"' 2023-11-08T05:14:34.119 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ grep -i Replaying 2023-11-08T05:14:34.120 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: "state": "Replaying" 2023-11-08T05:14:34.122 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ current_state=started 2023-11-08T05:14:34.123 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test started = started 2023-11-08T05:14:34.123 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:14:34.123 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_2 2023-11-08T05:14:34.124 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local local_cluster=cluster1 2023-11-08T05:14:34.124 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:14:34.124 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:14:34.124 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_2 2023-11-08T05:14:34.125 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' snapshot = journal ']' 2023-11-08T05:14:34.125 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' snapshot = snapshot ']' 2023-11-08T05:14:34.125 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_2 2023-11-08T05:14:34.125 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local local_cluster=cluster1 2023-11-08T05:14:34.125 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:14:34.125 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:14:34.126 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_2 2023-11-08T05:14:34.126 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ mkfname cluster2-mirror-image_2.status 2023-11-08T05:14:34.126 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ echo cluster2-mirror-image_2.status 2023-11-08T05:14:34.126 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -e 's|[/ ]|_|g' 2023-11-08T05:14:34.126 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local status_log=/tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_2.status 2023-11-08T05:14:34.127 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ mkfname cluster1-mirror-image_2.status 2023-11-08T05:14:34.127 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ echo cluster1-mirror-image_2.status 2023-11-08T05:14:34.127 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -e 's|[/ ]|_|g' 2023-11-08T05:14:34.128 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local local_status_log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_2.status 2023-11-08T05:14:34.128 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ mirror_image_snapshot cluster2 mirror image_2 2023-11-08T05:14:34.128 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:14:34.129 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:14:34.129 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_2 2023-11-08T05:14:34.129 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_2 2023-11-08T05:14:34.298 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:Snapshot ID: 121 2023-11-08T05:14:34.307 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_2 /tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_2.status 2023-11-08T05:14:34.307 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:14:34.307 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:14:34.308 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_2 2023-11-08T05:14:34.308 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_2.status 2023-11-08T05:14:34.308 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_2 --format xml 2023-11-08T05:14:34.308 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T05:14:34.343 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-11-08T05:14:34.345 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snapshot_id=121 2023-11-08T05:14:34.345 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ true 2023-11-08T05:14:34.345 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-11-08T05:14:34.346 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0.2 2023-11-08T05:14:34.547 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_2 /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_2.status 2023-11-08T05:14:34.548 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:14:34.548 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:14:34.548 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_2 2023-11-08T05:14:34.549 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_2.status 2023-11-08T05:14:34.549 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_2 --format xml 2023-11-08T05:14:34.549 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T05:14:34.586 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ true 2023-11-08T05:14:34.586 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-08T05:14:34.587 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:-:1.1: Document is empty 2023-11-08T05:14:34.587 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: 2023-11-08T05:14:34.588 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:^ 2023-11-08T05:14:34.588 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local primary_snapshot_id= 2023-11-08T05:14:34.588 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test 121 = '' 2023-11-08T05:14:34.589 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-11-08T05:14:34.589 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0.4 2023-11-08T05:14:34.989 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_2 /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_2.status 2023-11-08T05:14:34.990 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:14:34.990 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:14:34.990 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_2 2023-11-08T05:14:34.991 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_2.status 2023-11-08T05:14:34.991 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_2 --format xml 2023-11-08T05:14:34.991 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T05:14:35.030 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ true 2023-11-08T05:14:35.031 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-08T05:14:35.032 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:-:1.1: Document is empty 2023-11-08T05:14:35.032 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: 2023-11-08T05:14:35.032 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:^ 2023-11-08T05:14:35.033 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local primary_snapshot_id= 2023-11-08T05:14:35.033 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test 121 = '' 2023-11-08T05:14:35.033 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-11-08T05:14:35.033 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0.8 2023-11-08T05:14:35.834 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_2 /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_2.status 2023-11-08T05:14:35.834 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:14:35.834 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:14:35.835 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_2 2023-11-08T05:14:35.835 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_2.status 2023-11-08T05:14:35.835 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_2 --format xml 2023-11-08T05:14:35.835 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T05:14:35.871 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-08T05:14:35.874 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local primary_snapshot_id=121 2023-11-08T05:14:35.874 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test 121 = 121 2023-11-08T05:14:35.874 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:14:35.874 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_snap_present cluster1 mirror image_2 snap 2023-11-08T05:14:35.875 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:14:35.875 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:14:35.875 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_2 2023-11-08T05:14:35.875 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap_name=snap 2023-11-08T05:14:35.875 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local s 2023-11-08T05:14:35.876 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-11-08T05:14:35.876 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 1 2023-11-08T05:14:36.874 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 info mirror/image_2@snap 2023-11-08T05:14:37.117 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:rbd image 'image_2': 2023-11-08T05:14:37.117 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: size 128 MiB in 32 objects 2023-11-08T05:14:37.117 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: order 22 (4 MiB objects) 2023-11-08T05:14:37.118 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: snapshot_count: 2 2023-11-08T05:14:37.118 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: id: 111239e974bf 2023-11-08T05:14:37.118 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: block_name_prefix: rbd_data.111239e974bf 2023-11-08T05:14:37.118 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: format: 2 2023-11-08T05:14:37.119 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: features: layering, exclusive-lock, non-primary 2023-11-08T05:14:37.119 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: op_features: 2023-11-08T05:14:37.119 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: flags: 2023-11-08T05:14:37.119 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: create_timestamp: Wed Nov 8 05:04:35 2023 2023-11-08T05:14:37.120 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: access_timestamp: Wed Nov 8 05:04:35 2023 2023-11-08T05:14:37.120 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: modify_timestamp: Wed Nov 8 05:04:35 2023 2023-11-08T05:14:37.120 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: protected: False 2023-11-08T05:14:37.120 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring state: enabled 2023-11-08T05:14:37.120 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring mode: snapshot 2023-11-08T05:14:37.121 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring global id: c9cc5896-9129-4803-9473-1d795c399c55 2023-11-08T05:14:37.121 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring primary: false 2023-11-08T05:14:37.122 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:14:37.122 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ compare_image_snaps mirror image_2 snap 2023-11-08T05:14:37.122 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:14:37.123 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_2 2023-11-08T05:14:37.123 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap_name=snap 2023-11-08T05:14:37.123 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local ret=0 2023-11-08T05:14:37.123 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local rmt_export=/tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_2.export 2023-11-08T05:14:37.124 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local loc_export=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_2.export 2023-11-08T05:14:37.124 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rm -f /tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_2.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_2.export 2023-11-08T05:14:37.124 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 -p mirror export image_2@snap /tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_2.export 2023-11-08T05:14:37.324 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-11-08T05:14:37.327 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 -p mirror export image_2@snap /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_2.export 2023-11-08T05:14:37.555 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-11-08T05:14:37.558 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ cmp /tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_2.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_2.export 2023-11-08T05:14:37.615 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rm -f /tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_2.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_2.export 2023-11-08T05:14:37.655 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:14:37.655 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-11-08T05:14:37.655 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image=image_3 2023-11-08T05:14:37.655 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ create_snap cluster2 mirror image_3 snap 2023-11-08T05:14:37.656 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:14:37.656 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:14:37.656 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_3 2023-11-08T05:14:37.656 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap_name=snap 2023-11-08T05:14:37.656 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 -p mirror snap create image_3@snap --debug-rbd=20 --debug-journaler=20 2023-11-08T05:14:39.314 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_image_replay_started cluster1 mirror image_3 2023-11-08T05:14:39.314 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:14:39.315 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:14:39.315 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_3 2023-11-08T05:14:39.315 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_image_replay_state cluster1 mirror image_3 started 2023-11-08T05:14:39.315 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:14:39.315 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:14:39.316 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_3 2023-11-08T05:14:39.316 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local state=started 2023-11-08T05:14:39.316 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local s 2023-11-08T05:14:39.316 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-11-08T05:14:39.316 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 1 2023-11-08T05:14:40.315 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test_image_replay_state cluster1 mirror image_3 started 2023-11-08T05:14:40.315 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:14:40.315 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:14:40.316 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_3 2023-11-08T05:14:40.316 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local test_state=started 2023-11-08T05:14:40.316 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local status_result 2023-11-08T05:14:40.316 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local current_state=stopped 2023-11-08T05:14:40.316 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_3 2023-11-08T05:14:40.317 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_instance=cluster1 2023-11-08T05:14:40.317 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T05:14:40.317 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ grep -i state 2023-11-08T05:14:40.317 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1 2023-11-08T05:14:40.317 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance=cluster1 2023-11-08T05:14:40.317 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local loop_instance 2023-11-08T05:14:40.318 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-11-08T05:14:40.318 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sleep 0 2023-11-08T05:14:40.320 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-11-08T05:14:40.320 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ seq 0 3 2023-11-08T05:14:40.320 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T05:14:40.321 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_3 2023-11-08T05:14:40.321 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1:0 2023-11-08T05:14:40.321 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T05:14:40.321 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance 2023-11-08T05:14:40.321 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-11-08T05:14:40.322 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local val=cluster1:0 2023-11-08T05:14:40.322 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_var_name=cluster 2023-11-08T05:14:40.322 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance_var_name=instance 2023-11-08T05:14:40.323 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cluster=cluster1 2023-11-08T05:14:40.323 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ instance=0 2023-11-08T05:14:40.323 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' 0 = cluster1:0 ']' 2023-11-08T05:14:40.323 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval cluster=cluster1 2023-11-08T05:14:40.324 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:14:40.324 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval instance=0 2023-11-08T05:14:40.324 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=0 2023-11-08T05:14:40.324 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-11-08T05:14:40.325 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local local_cluster=cluster1:0 2023-11-08T05:14:40.327 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster=cluster1 2023-11-08T05:14:40.327 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance 2023-11-08T05:14:40.327 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-11-08T05:14:40.327 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local val=cluster1:0 2023-11-08T05:14:40.328 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster_var_name=local_cluster 2023-11-08T05:14:40.328 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance_var_name=instance 2023-11-08T05:14:40.328 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:14:40.328 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=0 2023-11-08T05:14:40.328 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ '[' 0 = cluster1:0 ']' 2023-11-08T05:14:40.329 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval local_cluster=cluster1 2023-11-08T05:14:40.329 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ local_cluster=cluster1 2023-11-08T05:14:40.329 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval instance=0 2023-11-08T05:14:40.329 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ instance=0 2023-11-08T05:14:40.329 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-11-08T05:14:40.336 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-08T05:14:40.336 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-08T05:14:40.337 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-08T05:14:40.337 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_3 2023-11-08T05:14:40.444 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:no valid command found; 2 closest matches: 2023-11-08T05:14:40.445 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd mirror status mirror/image_31 2023-11-08T05:14:40.445 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd mirror status mirror/image_35 2023-11-08T05:14:40.445 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:admin_socket: invalid command 2023-11-08T05:14:40.448 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T05:14:40.448 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_3 2023-11-08T05:14:40.448 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1:1 2023-11-08T05:14:40.449 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T05:14:40.449 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance 2023-11-08T05:14:40.449 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-11-08T05:14:40.449 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local val=cluster1:1 2023-11-08T05:14:40.449 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_var_name=cluster 2023-11-08T05:14:40.450 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance_var_name=instance 2023-11-08T05:14:40.450 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cluster=cluster1 2023-11-08T05:14:40.450 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ instance=1 2023-11-08T05:14:40.451 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' 1 = cluster1:1 ']' 2023-11-08T05:14:40.451 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval cluster=cluster1 2023-11-08T05:14:40.451 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:14:40.451 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval instance=1 2023-11-08T05:14:40.451 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=1 2023-11-08T05:14:40.452 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-11-08T05:14:40.452 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local local_cluster=cluster1:1 2023-11-08T05:14:40.452 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster=cluster1 2023-11-08T05:14:40.452 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance 2023-11-08T05:14:40.452 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-11-08T05:14:40.453 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local val=cluster1:1 2023-11-08T05:14:40.453 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster_var_name=local_cluster 2023-11-08T05:14:40.453 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance_var_name=instance 2023-11-08T05:14:40.453 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:14:40.453 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=1 2023-11-08T05:14:40.453 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ '[' 1 = cluster1:1 ']' 2023-11-08T05:14:40.454 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval local_cluster=cluster1 2023-11-08T05:14:40.454 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ local_cluster=cluster1 2023-11-08T05:14:40.454 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval instance=1 2023-11-08T05:14:40.454 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ instance=1 2023-11-08T05:14:40.454 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-11-08T05:14:40.463 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-08T05:14:40.463 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-08T05:14:40.463 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-08T05:14:40.463 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_3 2023-11-08T05:14:40.562 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:no valid command found; 2 closest matches: 2023-11-08T05:14:40.562 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd mirror status mirror/image_32 2023-11-08T05:14:40.563 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd mirror status mirror/image_39 2023-11-08T05:14:40.563 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:admin_socket: invalid command 2023-11-08T05:14:40.565 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T05:14:40.566 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_3 2023-11-08T05:14:40.566 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1:2 2023-11-08T05:14:40.566 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T05:14:40.566 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance 2023-11-08T05:14:40.567 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-11-08T05:14:40.567 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local val=cluster1:2 2023-11-08T05:14:40.567 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_var_name=cluster 2023-11-08T05:14:40.567 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance_var_name=instance 2023-11-08T05:14:40.567 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cluster=cluster1 2023-11-08T05:14:40.567 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ instance=2 2023-11-08T05:14:40.568 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' 2 = cluster1:2 ']' 2023-11-08T05:14:40.568 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval cluster=cluster1 2023-11-08T05:14:40.568 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:14:40.568 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval instance=2 2023-11-08T05:14:40.568 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=2 2023-11-08T05:14:40.569 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-11-08T05:14:40.569 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local local_cluster=cluster1:2 2023-11-08T05:14:40.569 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster=cluster1 2023-11-08T05:14:40.569 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance 2023-11-08T05:14:40.569 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-11-08T05:14:40.569 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local val=cluster1:2 2023-11-08T05:14:40.570 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster_var_name=local_cluster 2023-11-08T05:14:40.570 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance_var_name=instance 2023-11-08T05:14:40.570 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:14:40.570 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=2 2023-11-08T05:14:40.570 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ '[' 2 = cluster1:2 ']' 2023-11-08T05:14:40.571 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval local_cluster=cluster1 2023-11-08T05:14:40.571 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ local_cluster=cluster1 2023-11-08T05:14:40.571 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval instance=2 2023-11-08T05:14:40.571 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ instance=2 2023-11-08T05:14:40.571 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-11-08T05:14:40.579 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-08T05:14:40.579 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-08T05:14:40.580 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-08T05:14:40.580 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_3 2023-11-08T05:14:40.688 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ return 0 2023-11-08T05:14:40.688 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ status_result=' "state": "Replaying"' 2023-11-08T05:14:40.688 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ echo ' "state": "Replaying"' 2023-11-08T05:14:40.689 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ grep -i Replaying 2023-11-08T05:14:40.690 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: "state": "Replaying" 2023-11-08T05:14:40.693 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ current_state=started 2023-11-08T05:14:40.693 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test started = started 2023-11-08T05:14:40.693 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:14:40.693 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_3 2023-11-08T05:14:40.693 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local local_cluster=cluster1 2023-11-08T05:14:40.694 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:14:40.694 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:14:40.694 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_3 2023-11-08T05:14:40.694 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' snapshot = journal ']' 2023-11-08T05:14:40.694 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' snapshot = snapshot ']' 2023-11-08T05:14:40.695 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_3 2023-11-08T05:14:40.695 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local local_cluster=cluster1 2023-11-08T05:14:40.695 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:14:40.695 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:14:40.695 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_3 2023-11-08T05:14:40.695 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ mkfname cluster2-mirror-image_3.status 2023-11-08T05:14:40.696 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ echo cluster2-mirror-image_3.status 2023-11-08T05:14:40.696 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -e 's|[/ ]|_|g' 2023-11-08T05:14:40.696 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local status_log=/tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_3.status 2023-11-08T05:14:40.696 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ mkfname cluster1-mirror-image_3.status 2023-11-08T05:14:40.696 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ echo cluster1-mirror-image_3.status 2023-11-08T05:14:40.696 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -e 's|[/ ]|_|g' 2023-11-08T05:14:40.697 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local local_status_log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_3.status 2023-11-08T05:14:40.697 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ mirror_image_snapshot cluster2 mirror image_3 2023-11-08T05:14:40.697 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:14:40.697 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:14:40.697 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_3 2023-11-08T05:14:40.698 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_3 2023-11-08T05:14:41.316 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:Snapshot ID: 123 2023-11-08T05:14:41.325 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_3 /tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_3.status 2023-11-08T05:14:41.325 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:14:41.325 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:14:41.326 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_3 2023-11-08T05:14:41.326 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_3.status 2023-11-08T05:14:41.326 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_3 --format xml 2023-11-08T05:14:41.326 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T05:14:41.362 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-11-08T05:14:41.363 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snapshot_id=123 2023-11-08T05:14:41.363 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ true 2023-11-08T05:14:41.364 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-11-08T05:14:41.364 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0.2 2023-11-08T05:14:41.565 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_3 /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_3.status 2023-11-08T05:14:41.565 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:14:41.565 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:14:41.565 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_3 2023-11-08T05:14:41.566 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_3.status 2023-11-08T05:14:41.566 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_3 --format xml 2023-11-08T05:14:41.566 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T05:14:41.605 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ true 2023-11-08T05:14:41.606 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-08T05:14:41.607 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:-:1.1: Document is empty 2023-11-08T05:14:41.607 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: 2023-11-08T05:14:41.607 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:^ 2023-11-08T05:14:41.608 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local primary_snapshot_id= 2023-11-08T05:14:41.608 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test 123 = '' 2023-11-08T05:14:41.608 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-11-08T05:14:41.608 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0.4 2023-11-08T05:14:42.009 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_3 /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_3.status 2023-11-08T05:14:42.009 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:14:42.009 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:14:42.009 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_3 2023-11-08T05:14:42.010 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_3.status 2023-11-08T05:14:42.010 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_3 --format xml 2023-11-08T05:14:42.010 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T05:14:42.049 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ true 2023-11-08T05:14:42.050 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-08T05:14:42.051 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:-:1.1: Document is empty 2023-11-08T05:14:42.051 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: 2023-11-08T05:14:42.051 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:^ 2023-11-08T05:14:42.052 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local primary_snapshot_id= 2023-11-08T05:14:42.053 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test 123 = '' 2023-11-08T05:14:42.053 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-11-08T05:14:42.053 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0.8 2023-11-08T05:14:42.853 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_3 /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_3.status 2023-11-08T05:14:42.854 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:14:42.854 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:14:42.854 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_3 2023-11-08T05:14:42.854 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_3.status 2023-11-08T05:14:42.854 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_3 --format xml 2023-11-08T05:14:42.855 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T05:14:42.891 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-08T05:14:42.893 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local primary_snapshot_id=123 2023-11-08T05:14:42.893 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test 123 = 123 2023-11-08T05:14:42.894 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:14:42.894 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_snap_present cluster1 mirror image_3 snap 2023-11-08T05:14:42.894 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:14:42.894 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:14:42.895 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_3 2023-11-08T05:14:42.895 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap_name=snap 2023-11-08T05:14:42.895 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local s 2023-11-08T05:14:42.895 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-11-08T05:14:42.895 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 1 2023-11-08T05:14:43.894 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 info mirror/image_3@snap 2023-11-08T05:14:43.933 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:rbd image 'image_3': 2023-11-08T05:14:43.933 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: size 128 MiB in 32 objects 2023-11-08T05:14:43.933 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: order 22 (4 MiB objects) 2023-11-08T05:14:43.934 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: snapshot_count: 2 2023-11-08T05:14:43.934 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: id: 11d8e069cedc 2023-11-08T05:14:43.934 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: block_name_prefix: rbd_data.11d8e069cedc 2023-11-08T05:14:43.934 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: format: 2 2023-11-08T05:14:43.934 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: features: layering, exclusive-lock, non-primary 2023-11-08T05:14:43.935 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: op_features: 2023-11-08T05:14:43.935 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: flags: 2023-11-08T05:14:43.935 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: create_timestamp: Wed Nov 8 05:04:48 2023 2023-11-08T05:14:43.935 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: access_timestamp: Wed Nov 8 05:04:48 2023 2023-11-08T05:14:43.935 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: modify_timestamp: Wed Nov 8 05:04:48 2023 2023-11-08T05:14:43.936 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: protected: False 2023-11-08T05:14:43.936 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring state: enabled 2023-11-08T05:14:43.936 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring mode: snapshot 2023-11-08T05:14:43.936 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring global id: 64454ccd-8f11-4a7c-ba33-492d579d2c3b 2023-11-08T05:14:43.936 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring primary: false 2023-11-08T05:14:43.937 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:14:43.938 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ compare_image_snaps mirror image_3 snap 2023-11-08T05:14:43.938 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:14:43.938 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_3 2023-11-08T05:14:43.938 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap_name=snap 2023-11-08T05:14:43.938 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local ret=0 2023-11-08T05:14:43.939 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local rmt_export=/tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_3.export 2023-11-08T05:14:43.939 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local loc_export=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_3.export 2023-11-08T05:14:43.939 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rm -f /tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_3.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_3.export 2023-11-08T05:14:43.939 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 -p mirror export image_3@snap /tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_3.export 2023-11-08T05:14:44.130 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-11-08T05:14:44.132 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 -p mirror export image_3@snap /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_3.export 2023-11-08T05:14:44.361 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-11-08T05:14:44.365 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ cmp /tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_3.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_3.export 2023-11-08T05:14:44.423 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rm -f /tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_3.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_3.export 2023-11-08T05:14:44.463 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:14:44.464 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-11-08T05:14:44.464 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image=image_4 2023-11-08T05:14:44.464 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ create_snap cluster2 mirror image_4 snap 2023-11-08T05:14:44.464 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:14:44.465 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:14:44.465 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_4 2023-11-08T05:14:44.465 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap_name=snap 2023-11-08T05:14:44.465 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 -p mirror snap create image_4@snap --debug-rbd=20 --debug-journaler=20 2023-11-08T05:14:46.338 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_image_replay_started cluster1 mirror image_4 2023-11-08T05:14:46.338 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:14:46.339 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:14:46.339 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_4 2023-11-08T05:14:46.339 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_image_replay_state cluster1 mirror image_4 started 2023-11-08T05:14:46.339 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:14:46.339 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:14:46.340 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_4 2023-11-08T05:14:46.340 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local state=started 2023-11-08T05:14:46.340 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local s 2023-11-08T05:14:46.340 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-11-08T05:14:46.340 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 1 2023-11-08T05:14:47.339 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test_image_replay_state cluster1 mirror image_4 started 2023-11-08T05:14:47.339 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:14:47.340 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:14:47.340 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_4 2023-11-08T05:14:47.340 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local test_state=started 2023-11-08T05:14:47.340 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local status_result 2023-11-08T05:14:47.340 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local current_state=stopped 2023-11-08T05:14:47.341 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_4 2023-11-08T05:14:47.342 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ grep -i state 2023-11-08T05:14:47.342 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_instance=cluster1 2023-11-08T05:14:47.342 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T05:14:47.342 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1 2023-11-08T05:14:47.343 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance=cluster1 2023-11-08T05:14:47.343 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local loop_instance 2023-11-08T05:14:47.343 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-11-08T05:14:47.343 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sleep 0 2023-11-08T05:14:47.343 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-11-08T05:14:47.343 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ seq 0 3 2023-11-08T05:14:47.345 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T05:14:47.346 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_4 2023-11-08T05:14:47.346 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1:0 2023-11-08T05:14:47.346 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T05:14:47.346 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance 2023-11-08T05:14:47.347 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-11-08T05:14:47.347 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local val=cluster1:0 2023-11-08T05:14:47.347 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_var_name=cluster 2023-11-08T05:14:47.347 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance_var_name=instance 2023-11-08T05:14:47.347 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cluster=cluster1 2023-11-08T05:14:47.348 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ instance=0 2023-11-08T05:14:47.348 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' 0 = cluster1:0 ']' 2023-11-08T05:14:47.348 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval cluster=cluster1 2023-11-08T05:14:47.348 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:14:47.348 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval instance=0 2023-11-08T05:14:47.349 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=0 2023-11-08T05:14:47.349 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-11-08T05:14:47.349 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local local_cluster=cluster1:0 2023-11-08T05:14:47.350 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster=cluster1 2023-11-08T05:14:47.351 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance 2023-11-08T05:14:47.351 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-11-08T05:14:47.351 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local val=cluster1:0 2023-11-08T05:14:47.351 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster_var_name=local_cluster 2023-11-08T05:14:47.352 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance_var_name=instance 2023-11-08T05:14:47.352 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:14:47.352 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=0 2023-11-08T05:14:47.352 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ '[' 0 = cluster1:0 ']' 2023-11-08T05:14:47.352 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval local_cluster=cluster1 2023-11-08T05:14:47.353 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ local_cluster=cluster1 2023-11-08T05:14:47.353 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval instance=0 2023-11-08T05:14:47.353 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ instance=0 2023-11-08T05:14:47.353 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-11-08T05:14:47.361 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-08T05:14:47.361 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-08T05:14:47.361 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-08T05:14:47.362 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_4 2023-11-08T05:14:47.468 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:no valid command found; 3 closest matches: 2023-11-08T05:14:47.469 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd mirror status mirror/image_42 2023-11-08T05:14:47.469 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd mirror status mirror/image_46 2023-11-08T05:14:47.469 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd mirror status mirror/image_48 2023-11-08T05:14:47.469 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:admin_socket: invalid command 2023-11-08T05:14:47.472 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T05:14:47.472 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_4 2023-11-08T05:14:47.473 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1:1 2023-11-08T05:14:47.473 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T05:14:47.473 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance 2023-11-08T05:14:47.473 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-11-08T05:14:47.473 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local val=cluster1:1 2023-11-08T05:14:47.474 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_var_name=cluster 2023-11-08T05:14:47.474 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance_var_name=instance 2023-11-08T05:14:47.474 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cluster=cluster1 2023-11-08T05:14:47.474 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ instance=1 2023-11-08T05:14:47.474 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' 1 = cluster1:1 ']' 2023-11-08T05:14:47.475 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval cluster=cluster1 2023-11-08T05:14:47.475 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:14:47.475 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval instance=1 2023-11-08T05:14:47.475 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=1 2023-11-08T05:14:47.475 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-11-08T05:14:47.476 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local local_cluster=cluster1:1 2023-11-08T05:14:47.476 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster=cluster1 2023-11-08T05:14:47.476 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance 2023-11-08T05:14:47.476 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-11-08T05:14:47.476 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local val=cluster1:1 2023-11-08T05:14:47.477 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster_var_name=local_cluster 2023-11-08T05:14:47.477 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance_var_name=instance 2023-11-08T05:14:47.477 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:14:47.477 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=1 2023-11-08T05:14:47.477 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ '[' 1 = cluster1:1 ']' 2023-11-08T05:14:47.478 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval local_cluster=cluster1 2023-11-08T05:14:47.478 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ local_cluster=cluster1 2023-11-08T05:14:47.478 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval instance=1 2023-11-08T05:14:47.478 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ instance=1 2023-11-08T05:14:47.478 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-11-08T05:14:47.486 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-08T05:14:47.486 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-08T05:14:47.487 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-08T05:14:47.487 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_4 2023-11-08T05:14:47.586 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:no valid command found; 2 closest matches: 2023-11-08T05:14:47.586 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd mirror status mirror/image_41 2023-11-08T05:14:47.586 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd mirror status mirror/image_43 2023-11-08T05:14:47.587 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:admin_socket: invalid command 2023-11-08T05:14:47.589 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T05:14:47.590 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_4 2023-11-08T05:14:47.590 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1:2 2023-11-08T05:14:47.590 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T05:14:47.590 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance 2023-11-08T05:14:47.591 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-11-08T05:14:47.591 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local val=cluster1:2 2023-11-08T05:14:47.591 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_var_name=cluster 2023-11-08T05:14:47.591 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance_var_name=instance 2023-11-08T05:14:47.591 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cluster=cluster1 2023-11-08T05:14:47.592 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ instance=2 2023-11-08T05:14:47.592 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' 2 = cluster1:2 ']' 2023-11-08T05:14:47.592 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval cluster=cluster1 2023-11-08T05:14:47.592 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:14:47.592 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval instance=2 2023-11-08T05:14:47.593 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=2 2023-11-08T05:14:47.593 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-11-08T05:14:47.593 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local local_cluster=cluster1:2 2023-11-08T05:14:47.593 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster=cluster1 2023-11-08T05:14:47.593 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance 2023-11-08T05:14:47.593 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-11-08T05:14:47.594 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local val=cluster1:2 2023-11-08T05:14:47.594 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster_var_name=local_cluster 2023-11-08T05:14:47.594 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance_var_name=instance 2023-11-08T05:14:47.594 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:14:47.594 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=2 2023-11-08T05:14:47.595 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ '[' 2 = cluster1:2 ']' 2023-11-08T05:14:47.595 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval local_cluster=cluster1 2023-11-08T05:14:47.595 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ local_cluster=cluster1 2023-11-08T05:14:47.595 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval instance=2 2023-11-08T05:14:47.595 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ instance=2 2023-11-08T05:14:47.596 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-11-08T05:14:47.603 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-08T05:14:47.604 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-08T05:14:47.604 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-08T05:14:47.604 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_4 2023-11-08T05:14:47.713 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ return 0 2023-11-08T05:14:47.713 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ status_result=' "state": "Replaying"' 2023-11-08T05:14:47.714 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ echo ' "state": "Replaying"' 2023-11-08T05:14:47.714 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ grep -i Replaying 2023-11-08T05:14:47.715 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: "state": "Replaying" 2023-11-08T05:14:47.717 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ current_state=started 2023-11-08T05:14:47.717 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test started = started 2023-11-08T05:14:47.717 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:14:47.717 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_4 2023-11-08T05:14:47.718 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local local_cluster=cluster1 2023-11-08T05:14:47.718 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:14:47.718 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:14:47.718 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_4 2023-11-08T05:14:47.719 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' snapshot = journal ']' 2023-11-08T05:14:47.719 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' snapshot = snapshot ']' 2023-11-08T05:14:47.719 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_4 2023-11-08T05:14:47.720 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local local_cluster=cluster1 2023-11-08T05:14:47.720 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:14:47.720 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:14:47.720 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_4 2023-11-08T05:14:47.720 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ mkfname cluster2-mirror-image_4.status 2023-11-08T05:14:47.721 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ echo cluster2-mirror-image_4.status 2023-11-08T05:14:47.721 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -e 's|[/ ]|_|g' 2023-11-08T05:14:47.721 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local status_log=/tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_4.status 2023-11-08T05:14:47.721 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ mkfname cluster1-mirror-image_4.status 2023-11-08T05:14:47.721 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ echo cluster1-mirror-image_4.status 2023-11-08T05:14:47.722 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -e 's|[/ ]|_|g' 2023-11-08T05:14:47.722 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local local_status_log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_4.status 2023-11-08T05:14:47.723 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ mirror_image_snapshot cluster2 mirror image_4 2023-11-08T05:14:47.723 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:14:47.723 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:14:47.723 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_4 2023-11-08T05:14:47.723 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_4 2023-11-08T05:14:48.334 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:Snapshot ID: 125 2023-11-08T05:14:48.343 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_4 /tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_4.status 2023-11-08T05:14:48.343 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:14:48.344 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:14:48.344 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_4 2023-11-08T05:14:48.344 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_4.status 2023-11-08T05:14:48.344 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_4 --format xml 2023-11-08T05:14:48.344 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T05:14:48.381 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-11-08T05:14:48.383 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snapshot_id=125 2023-11-08T05:14:48.383 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ true 2023-11-08T05:14:48.383 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-11-08T05:14:48.383 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0.2 2023-11-08T05:14:48.584 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_4 /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_4.status 2023-11-08T05:14:48.585 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:14:48.585 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:14:48.585 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_4 2023-11-08T05:14:48.585 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_4.status 2023-11-08T05:14:48.585 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_4 --format xml 2023-11-08T05:14:48.586 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T05:14:48.623 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ true 2023-11-08T05:14:48.623 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-08T05:14:48.624 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:-:1.1: Document is empty 2023-11-08T05:14:48.625 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: 2023-11-08T05:14:48.625 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:^ 2023-11-08T05:14:48.626 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local primary_snapshot_id= 2023-11-08T05:14:48.626 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test 125 = '' 2023-11-08T05:14:48.626 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-11-08T05:14:48.626 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0.4 2023-11-08T05:14:49.026 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_4 /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_4.status 2023-11-08T05:14:49.027 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:14:49.027 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:14:49.027 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_4 2023-11-08T05:14:49.028 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_4.status 2023-11-08T05:14:49.028 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_4 --format xml 2023-11-08T05:14:49.028 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T05:14:49.068 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ true 2023-11-08T05:14:49.068 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-08T05:14:49.069 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:-:1.1: Document is empty 2023-11-08T05:14:49.070 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: 2023-11-08T05:14:49.070 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:^ 2023-11-08T05:14:49.071 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local primary_snapshot_id= 2023-11-08T05:14:49.071 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test 125 = '' 2023-11-08T05:14:49.071 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-11-08T05:14:49.072 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0.8 2023-11-08T05:14:49.871 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_4 /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_4.status 2023-11-08T05:14:49.872 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:14:49.872 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:14:49.872 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_4 2023-11-08T05:14:49.872 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_4.status 2023-11-08T05:14:49.872 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_4 --format xml 2023-11-08T05:14:49.873 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T05:14:49.909 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-08T05:14:49.912 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local primary_snapshot_id=125 2023-11-08T05:14:49.912 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test 125 = 125 2023-11-08T05:14:49.912 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:14:49.912 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_snap_present cluster1 mirror image_4 snap 2023-11-08T05:14:49.913 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:14:49.913 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:14:49.913 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_4 2023-11-08T05:14:49.913 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap_name=snap 2023-11-08T05:14:49.913 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local s 2023-11-08T05:14:49.914 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-11-08T05:14:49.914 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 1 2023-11-08T05:14:50.912 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 info mirror/image_4@snap 2023-11-08T05:14:50.948 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:rbd image 'image_4': 2023-11-08T05:14:50.949 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: size 128 MiB in 32 objects 2023-11-08T05:14:50.949 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: order 22 (4 MiB objects) 2023-11-08T05:14:50.949 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: snapshot_count: 2 2023-11-08T05:14:50.949 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: id: 128749b89ad2 2023-11-08T05:14:50.949 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: block_name_prefix: rbd_data.128749b89ad2 2023-11-08T05:14:50.950 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: format: 2 2023-11-08T05:14:50.950 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: features: layering, exclusive-lock, non-primary 2023-11-08T05:14:50.950 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: op_features: 2023-11-08T05:14:50.950 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: flags: 2023-11-08T05:14:50.950 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: create_timestamp: Wed Nov 8 05:04:55 2023 2023-11-08T05:14:50.951 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: access_timestamp: Wed Nov 8 05:04:55 2023 2023-11-08T05:14:50.951 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: modify_timestamp: Wed Nov 8 05:04:55 2023 2023-11-08T05:14:50.951 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: protected: False 2023-11-08T05:14:50.951 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring state: enabled 2023-11-08T05:14:50.951 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring mode: snapshot 2023-11-08T05:14:50.952 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring global id: 93d1c29c-aa4a-4563-b1cd-30608ea2b43f 2023-11-08T05:14:50.952 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring primary: false 2023-11-08T05:14:50.953 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:14:50.953 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ compare_image_snaps mirror image_4 snap 2023-11-08T05:14:50.954 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:14:50.954 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_4 2023-11-08T05:14:50.954 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap_name=snap 2023-11-08T05:14:50.954 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local ret=0 2023-11-08T05:14:50.955 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local rmt_export=/tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_4.export 2023-11-08T05:14:50.955 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local loc_export=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_4.export 2023-11-08T05:14:50.955 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rm -f /tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_4.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_4.export 2023-11-08T05:14:50.955 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 -p mirror export image_4@snap /tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_4.export 2023-11-08T05:14:51.138 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-11-08T05:14:51.141 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 -p mirror export image_4@snap /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_4.export 2023-11-08T05:14:51.376 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-11-08T05:14:51.379 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ cmp /tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_4.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_4.export 2023-11-08T05:14:51.436 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rm -f /tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_4.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_4.export 2023-11-08T05:14:51.476 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:14:51.476 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-11-08T05:14:51.476 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image=image_5 2023-11-08T05:14:51.476 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ create_snap cluster2 mirror image_5 snap 2023-11-08T05:14:51.477 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:14:51.477 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:14:51.477 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_5 2023-11-08T05:14:51.477 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap_name=snap 2023-11-08T05:14:51.477 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 -p mirror snap create image_5@snap --debug-rbd=20 --debug-journaler=20 2023-11-08T05:14:53.349 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_image_replay_started cluster1 mirror image_5 2023-11-08T05:14:53.349 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:14:53.350 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:14:53.350 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_5 2023-11-08T05:14:53.350 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_image_replay_state cluster1 mirror image_5 started 2023-11-08T05:14:53.350 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:14:53.350 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:14:53.351 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_5 2023-11-08T05:14:53.351 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local state=started 2023-11-08T05:14:53.351 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local s 2023-11-08T05:14:53.351 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-11-08T05:14:53.351 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 1 2023-11-08T05:14:54.351 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test_image_replay_state cluster1 mirror image_5 started 2023-11-08T05:14:54.351 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:14:54.351 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:14:54.352 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_5 2023-11-08T05:14:54.352 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local test_state=started 2023-11-08T05:14:54.352 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local status_result 2023-11-08T05:14:54.352 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local current_state=stopped 2023-11-08T05:14:54.352 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_5 2023-11-08T05:14:54.353 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_instance=cluster1 2023-11-08T05:14:54.353 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T05:14:54.353 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ grep -i state 2023-11-08T05:14:54.353 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1 2023-11-08T05:14:54.353 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance=cluster1 2023-11-08T05:14:54.354 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local loop_instance 2023-11-08T05:14:54.354 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-11-08T05:14:54.354 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sleep 0 2023-11-08T05:14:54.356 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-11-08T05:14:54.356 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ seq 0 3 2023-11-08T05:14:54.357 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T05:14:54.357 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_5 2023-11-08T05:14:54.357 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1:0 2023-11-08T05:14:54.357 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T05:14:54.357 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance 2023-11-08T05:14:54.358 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-11-08T05:14:54.358 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local val=cluster1:0 2023-11-08T05:14:54.358 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_var_name=cluster 2023-11-08T05:14:54.358 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance_var_name=instance 2023-11-08T05:14:54.358 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cluster=cluster1 2023-11-08T05:14:54.359 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ instance=0 2023-11-08T05:14:54.359 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' 0 = cluster1:0 ']' 2023-11-08T05:14:54.359 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval cluster=cluster1 2023-11-08T05:14:54.359 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:14:54.359 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval instance=0 2023-11-08T05:14:54.360 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=0 2023-11-08T05:14:54.360 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-11-08T05:14:54.360 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local local_cluster=cluster1:0 2023-11-08T05:14:54.360 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster=cluster1 2023-11-08T05:14:54.360 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance 2023-11-08T05:14:54.361 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-11-08T05:14:54.362 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local val=cluster1:0 2023-11-08T05:14:54.362 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster_var_name=local_cluster 2023-11-08T05:14:54.363 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance_var_name=instance 2023-11-08T05:14:54.363 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:14:54.363 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=0 2023-11-08T05:14:54.363 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ '[' 0 = cluster1:0 ']' 2023-11-08T05:14:54.363 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval local_cluster=cluster1 2023-11-08T05:14:54.364 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ local_cluster=cluster1 2023-11-08T05:14:54.364 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval instance=0 2023-11-08T05:14:54.364 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ instance=0 2023-11-08T05:14:54.364 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-11-08T05:14:54.373 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-08T05:14:54.373 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-08T05:14:54.373 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-08T05:14:54.373 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_5 2023-11-08T05:14:54.481 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:Invalid command: unused arguments: ['mirror/image_5'] 2023-11-08T05:14:54.481 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd mirror status : get status for rbd mirror 2023-11-08T05:14:54.482 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:admin_socket: invalid command 2023-11-08T05:14:54.485 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T05:14:54.485 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_5 2023-11-08T05:14:54.485 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1:1 2023-11-08T05:14:54.485 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T05:14:54.485 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance 2023-11-08T05:14:54.486 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-11-08T05:14:54.486 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local val=cluster1:1 2023-11-08T05:14:54.486 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_var_name=cluster 2023-11-08T05:14:54.486 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance_var_name=instance 2023-11-08T05:14:54.486 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cluster=cluster1 2023-11-08T05:14:54.487 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ instance=1 2023-11-08T05:14:54.487 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' 1 = cluster1:1 ']' 2023-11-08T05:14:54.487 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval cluster=cluster1 2023-11-08T05:14:54.487 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:14:54.487 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval instance=1 2023-11-08T05:14:54.488 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=1 2023-11-08T05:14:54.488 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-11-08T05:14:54.488 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local local_cluster=cluster1:1 2023-11-08T05:14:54.488 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster=cluster1 2023-11-08T05:14:54.488 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance 2023-11-08T05:14:54.489 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-11-08T05:14:54.489 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local val=cluster1:1 2023-11-08T05:14:54.489 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster_var_name=local_cluster 2023-11-08T05:14:54.489 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance_var_name=instance 2023-11-08T05:14:54.489 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:14:54.490 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=1 2023-11-08T05:14:54.490 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ '[' 1 = cluster1:1 ']' 2023-11-08T05:14:54.490 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval local_cluster=cluster1 2023-11-08T05:14:54.490 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ local_cluster=cluster1 2023-11-08T05:14:54.490 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval instance=1 2023-11-08T05:14:54.491 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ instance=1 2023-11-08T05:14:54.491 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-11-08T05:14:54.499 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-08T05:14:54.499 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-08T05:14:54.499 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-08T05:14:54.499 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_5 2023-11-08T05:14:54.599 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:no valid command found; 1 closest matches: 2023-11-08T05:14:54.599 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd mirror status mirror/image_50 2023-11-08T05:14:54.600 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:admin_socket: invalid command 2023-11-08T05:14:54.603 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T05:14:54.603 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_5 2023-11-08T05:14:54.603 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1:2 2023-11-08T05:14:54.604 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T05:14:54.604 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance 2023-11-08T05:14:54.604 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-11-08T05:14:54.604 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local val=cluster1:2 2023-11-08T05:14:54.604 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_var_name=cluster 2023-11-08T05:14:54.605 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance_var_name=instance 2023-11-08T05:14:54.605 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cluster=cluster1 2023-11-08T05:14:54.605 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ instance=2 2023-11-08T05:14:54.605 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' 2 = cluster1:2 ']' 2023-11-08T05:14:54.605 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval cluster=cluster1 2023-11-08T05:14:54.605 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:14:54.606 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval instance=2 2023-11-08T05:14:54.606 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=2 2023-11-08T05:14:54.606 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-11-08T05:14:54.606 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local local_cluster=cluster1:2 2023-11-08T05:14:54.606 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster=cluster1 2023-11-08T05:14:54.607 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance 2023-11-08T05:14:54.607 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-11-08T05:14:54.607 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local val=cluster1:2 2023-11-08T05:14:54.607 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster_var_name=local_cluster 2023-11-08T05:14:54.607 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance_var_name=instance 2023-11-08T05:14:54.608 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:14:54.608 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=2 2023-11-08T05:14:54.608 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ '[' 2 = cluster1:2 ']' 2023-11-08T05:14:54.608 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval local_cluster=cluster1 2023-11-08T05:14:54.608 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ local_cluster=cluster1 2023-11-08T05:14:54.608 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval instance=2 2023-11-08T05:14:54.609 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ instance=2 2023-11-08T05:14:54.609 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-11-08T05:14:54.616 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-08T05:14:54.617 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-08T05:14:54.617 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-08T05:14:54.617 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_5 2023-11-08T05:14:54.726 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ return 0 2023-11-08T05:14:54.727 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ status_result=' "state": "Replaying"' 2023-11-08T05:14:54.727 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ echo ' "state": "Replaying"' 2023-11-08T05:14:54.727 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ grep -i Replaying 2023-11-08T05:14:54.728 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: "state": "Replaying" 2023-11-08T05:14:54.730 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ current_state=started 2023-11-08T05:14:54.730 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test started = started 2023-11-08T05:14:54.730 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:14:54.730 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_5 2023-11-08T05:14:54.731 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local local_cluster=cluster1 2023-11-08T05:14:54.731 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:14:54.731 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:14:54.731 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_5 2023-11-08T05:14:54.731 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' snapshot = journal ']' 2023-11-08T05:14:54.732 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' snapshot = snapshot ']' 2023-11-08T05:14:54.732 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_5 2023-11-08T05:14:54.732 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local local_cluster=cluster1 2023-11-08T05:14:54.732 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:14:54.732 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:14:54.733 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_5 2023-11-08T05:14:54.733 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ mkfname cluster2-mirror-image_5.status 2023-11-08T05:14:54.733 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ echo cluster2-mirror-image_5.status 2023-11-08T05:14:54.733 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -e 's|[/ ]|_|g' 2023-11-08T05:14:54.733 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local status_log=/tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_5.status 2023-11-08T05:14:54.734 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ mkfname cluster1-mirror-image_5.status 2023-11-08T05:14:54.734 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ echo cluster1-mirror-image_5.status 2023-11-08T05:14:54.734 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -e 's|[/ ]|_|g' 2023-11-08T05:14:54.735 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local local_status_log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_5.status 2023-11-08T05:14:54.735 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ mirror_image_snapshot cluster2 mirror image_5 2023-11-08T05:14:54.735 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:14:54.736 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:14:54.736 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_5 2023-11-08T05:14:54.736 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_5 2023-11-08T05:14:55.363 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:Snapshot ID: 127 2023-11-08T05:14:55.372 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_5 /tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_5.status 2023-11-08T05:14:55.372 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:14:55.373 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:14:55.373 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_5 2023-11-08T05:14:55.373 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_5.status 2023-11-08T05:14:55.373 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_5 --format xml 2023-11-08T05:14:55.374 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T05:14:55.409 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-11-08T05:14:55.411 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snapshot_id=127 2023-11-08T05:14:55.412 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ true 2023-11-08T05:14:55.412 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-11-08T05:14:55.412 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0.2 2023-11-08T05:14:55.613 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_5 /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_5.status 2023-11-08T05:14:55.613 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:14:55.613 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:14:55.614 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_5 2023-11-08T05:14:55.614 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_5.status 2023-11-08T05:14:55.614 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_5 --format xml 2023-11-08T05:14:55.614 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T05:14:55.652 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ true 2023-11-08T05:14:55.652 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-08T05:14:55.653 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:-:1.1: Document is empty 2023-11-08T05:14:55.653 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: 2023-11-08T05:14:55.654 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:^ 2023-11-08T05:14:55.654 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local primary_snapshot_id= 2023-11-08T05:14:55.655 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test 127 = '' 2023-11-08T05:14:55.655 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-11-08T05:14:55.655 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0.4 2023-11-08T05:14:56.055 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_5 /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_5.status 2023-11-08T05:14:56.055 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:14:56.056 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:14:56.056 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_5 2023-11-08T05:14:56.056 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_5.status 2023-11-08T05:14:56.056 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_5 --format xml 2023-11-08T05:14:56.056 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T05:14:56.096 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ true 2023-11-08T05:14:56.097 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-08T05:14:56.101 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:-:1.1: Document is empty 2023-11-08T05:14:56.101 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: 2023-11-08T05:14:56.102 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:^ 2023-11-08T05:14:56.102 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local primary_snapshot_id= 2023-11-08T05:14:56.103 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test 127 = '' 2023-11-08T05:14:56.103 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-11-08T05:14:56.103 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0.8 2023-11-08T05:14:56.900 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_5 /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_5.status 2023-11-08T05:14:56.900 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:14:56.900 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:14:56.901 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_5 2023-11-08T05:14:56.901 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_5.status 2023-11-08T05:14:56.901 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_5 --format xml 2023-11-08T05:14:56.901 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T05:14:56.940 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-08T05:14:56.942 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local primary_snapshot_id=127 2023-11-08T05:14:56.942 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test 127 = 127 2023-11-08T05:14:56.942 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:14:56.943 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_snap_present cluster1 mirror image_5 snap 2023-11-08T05:14:56.943 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:14:56.943 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:14:56.943 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_5 2023-11-08T05:14:56.943 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap_name=snap 2023-11-08T05:14:56.944 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local s 2023-11-08T05:14:56.944 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-11-08T05:14:56.944 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 1 2023-11-08T05:14:57.942 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 info mirror/image_5@snap 2023-11-08T05:14:57.977 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:rbd image 'image_5': 2023-11-08T05:14:57.978 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: size 128 MiB in 32 objects 2023-11-08T05:14:57.978 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: order 22 (4 MiB objects) 2023-11-08T05:14:57.978 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: snapshot_count: 2 2023-11-08T05:14:57.978 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: id: 10e74e4f5c74 2023-11-08T05:14:57.978 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: block_name_prefix: rbd_data.10e74e4f5c74 2023-11-08T05:14:57.979 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: format: 2 2023-11-08T05:14:57.979 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: features: layering, exclusive-lock, non-primary 2023-11-08T05:14:57.979 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: op_features: 2023-11-08T05:14:57.979 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: flags: 2023-11-08T05:14:57.979 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: create_timestamp: Wed Nov 8 05:05:06 2023 2023-11-08T05:14:57.980 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: access_timestamp: Wed Nov 8 05:05:06 2023 2023-11-08T05:14:57.980 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: modify_timestamp: Wed Nov 8 05:05:06 2023 2023-11-08T05:14:57.980 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: protected: False 2023-11-08T05:14:57.980 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring state: enabled 2023-11-08T05:14:57.980 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring mode: snapshot 2023-11-08T05:14:57.981 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring global id: c80b0fb2-a1fa-458e-baf2-277b0fc2657c 2023-11-08T05:14:57.981 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring primary: false 2023-11-08T05:14:57.982 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:14:57.982 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ compare_image_snaps mirror image_5 snap 2023-11-08T05:14:57.983 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:14:57.983 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_5 2023-11-08T05:14:57.983 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap_name=snap 2023-11-08T05:14:57.983 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local ret=0 2023-11-08T05:14:57.983 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local rmt_export=/tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_5.export 2023-11-08T05:14:57.984 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local loc_export=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_5.export 2023-11-08T05:14:57.984 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rm -f /tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_5.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_5.export 2023-11-08T05:14:57.984 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 -p mirror export image_5@snap /tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_5.export 2023-11-08T05:14:58.154 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-11-08T05:14:58.154 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 -p mirror export image_5@snap /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_5.export 2023-11-08T05:14:58.402 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-11-08T05:14:58.405 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ cmp /tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_5.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_5.export 2023-11-08T05:14:58.463 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rm -f /tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_5.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_5.export 2023-11-08T05:14:58.505 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:14:58.505 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-11-08T05:14:58.505 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image=image_6 2023-11-08T05:14:58.505 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ create_snap cluster2 mirror image_6 snap 2023-11-08T05:14:58.506 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:14:58.506 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:14:58.506 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_6 2023-11-08T05:14:58.506 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap_name=snap 2023-11-08T05:14:58.507 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 -p mirror snap create image_6@snap --debug-rbd=20 --debug-journaler=20 2023-11-08T05:15:00.378 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_image_replay_started cluster1 mirror image_6 2023-11-08T05:15:00.379 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:15:00.379 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:15:00.379 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_6 2023-11-08T05:15:00.379 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_image_replay_state cluster1 mirror image_6 started 2023-11-08T05:15:00.380 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:15:00.380 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:15:00.380 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_6 2023-11-08T05:15:00.380 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local state=started 2023-11-08T05:15:00.380 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local s 2023-11-08T05:15:00.381 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-11-08T05:15:00.381 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 1 2023-11-08T05:15:01.379 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test_image_replay_state cluster1 mirror image_6 started 2023-11-08T05:15:01.380 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:15:01.380 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:15:01.380 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_6 2023-11-08T05:15:01.380 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local test_state=started 2023-11-08T05:15:01.380 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local status_result 2023-11-08T05:15:01.381 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local current_state=stopped 2023-11-08T05:15:01.382 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_6 2023-11-08T05:15:01.382 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_instance=cluster1 2023-11-08T05:15:01.382 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T05:15:01.382 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ grep -i state 2023-11-08T05:15:01.383 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1 2023-11-08T05:15:01.383 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance=cluster1 2023-11-08T05:15:01.383 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local loop_instance 2023-11-08T05:15:01.383 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-11-08T05:15:01.383 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sleep 0 2023-11-08T05:15:01.384 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-11-08T05:15:01.384 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ seq 0 3 2023-11-08T05:15:01.385 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T05:15:01.386 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_6 2023-11-08T05:15:01.386 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1:0 2023-11-08T05:15:01.386 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T05:15:01.387 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance 2023-11-08T05:15:01.387 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-11-08T05:15:01.387 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local val=cluster1:0 2023-11-08T05:15:01.387 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_var_name=cluster 2023-11-08T05:15:01.387 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance_var_name=instance 2023-11-08T05:15:01.388 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cluster=cluster1 2023-11-08T05:15:01.388 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ instance=0 2023-11-08T05:15:01.388 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' 0 = cluster1:0 ']' 2023-11-08T05:15:01.388 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval cluster=cluster1 2023-11-08T05:15:01.388 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:15:01.389 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval instance=0 2023-11-08T05:15:01.390 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=0 2023-11-08T05:15:01.391 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-11-08T05:15:01.391 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local local_cluster=cluster1:0 2023-11-08T05:15:01.391 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster=cluster1 2023-11-08T05:15:01.391 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance 2023-11-08T05:15:01.391 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-11-08T05:15:01.392 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local val=cluster1:0 2023-11-08T05:15:01.392 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster_var_name=local_cluster 2023-11-08T05:15:01.392 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance_var_name=instance 2023-11-08T05:15:01.392 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:15:01.392 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=0 2023-11-08T05:15:01.392 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ '[' 0 = cluster1:0 ']' 2023-11-08T05:15:01.393 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval local_cluster=cluster1 2023-11-08T05:15:01.393 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ local_cluster=cluster1 2023-11-08T05:15:01.393 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval instance=0 2023-11-08T05:15:01.393 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ instance=0 2023-11-08T05:15:01.393 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-11-08T05:15:01.401 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-08T05:15:01.402 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-08T05:15:01.402 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-08T05:15:01.402 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_6 2023-11-08T05:15:01.509 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:Invalid command: unused arguments: ['mirror/image_6'] 2023-11-08T05:15:01.509 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd mirror status : get status for rbd mirror 2023-11-08T05:15:01.509 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:admin_socket: invalid command 2023-11-08T05:15:01.513 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T05:15:01.513 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_6 2023-11-08T05:15:01.513 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1:1 2023-11-08T05:15:01.513 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T05:15:01.513 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance 2023-11-08T05:15:01.514 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-11-08T05:15:01.514 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local val=cluster1:1 2023-11-08T05:15:01.514 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_var_name=cluster 2023-11-08T05:15:01.514 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance_var_name=instance 2023-11-08T05:15:01.514 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cluster=cluster1 2023-11-08T05:15:01.515 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ instance=1 2023-11-08T05:15:01.515 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' 1 = cluster1:1 ']' 2023-11-08T05:15:01.515 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval cluster=cluster1 2023-11-08T05:15:01.515 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:15:01.515 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval instance=1 2023-11-08T05:15:01.516 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=1 2023-11-08T05:15:01.516 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-11-08T05:15:01.516 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local local_cluster=cluster1:1 2023-11-08T05:15:01.516 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster=cluster1 2023-11-08T05:15:01.517 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance 2023-11-08T05:15:01.517 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-11-08T05:15:01.517 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local val=cluster1:1 2023-11-08T05:15:01.517 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster_var_name=local_cluster 2023-11-08T05:15:01.517 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance_var_name=instance 2023-11-08T05:15:01.517 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:15:01.518 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=1 2023-11-08T05:15:01.518 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ '[' 1 = cluster1:1 ']' 2023-11-08T05:15:01.518 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval local_cluster=cluster1 2023-11-08T05:15:01.518 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ local_cluster=cluster1 2023-11-08T05:15:01.518 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval instance=1 2023-11-08T05:15:01.519 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ instance=1 2023-11-08T05:15:01.519 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-11-08T05:15:01.526 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-08T05:15:01.527 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-08T05:15:01.527 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-08T05:15:01.527 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_6 2023-11-08T05:15:01.635 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ return 0 2023-11-08T05:15:01.636 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ status_result=' "state": "Replaying"' 2023-11-08T05:15:01.636 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ echo ' "state": "Replaying"' 2023-11-08T05:15:01.637 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ grep -i Replaying 2023-11-08T05:15:01.637 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: "state": "Replaying" 2023-11-08T05:15:01.639 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ current_state=started 2023-11-08T05:15:01.640 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test started = started 2023-11-08T05:15:01.640 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:15:01.640 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_6 2023-11-08T05:15:01.640 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local local_cluster=cluster1 2023-11-08T05:15:01.640 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:15:01.641 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:15:01.641 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_6 2023-11-08T05:15:01.641 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' snapshot = journal ']' 2023-11-08T05:15:01.641 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' snapshot = snapshot ']' 2023-11-08T05:15:01.641 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_6 2023-11-08T05:15:01.642 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local local_cluster=cluster1 2023-11-08T05:15:01.642 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:15:01.642 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:15:01.642 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_6 2023-11-08T05:15:01.643 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ mkfname cluster2-mirror-image_6.status 2023-11-08T05:15:01.643 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ echo cluster2-mirror-image_6.status 2023-11-08T05:15:01.643 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -e 's|[/ ]|_|g' 2023-11-08T05:15:01.643 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local status_log=/tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_6.status 2023-11-08T05:15:01.643 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ mkfname cluster1-mirror-image_6.status 2023-11-08T05:15:01.644 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ echo cluster1-mirror-image_6.status 2023-11-08T05:15:01.644 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -e 's|[/ ]|_|g' 2023-11-08T05:15:01.645 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local local_status_log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_6.status 2023-11-08T05:15:01.645 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ mirror_image_snapshot cluster2 mirror image_6 2023-11-08T05:15:01.645 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:15:01.645 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:15:01.646 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_6 2023-11-08T05:15:01.646 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_6 2023-11-08T05:15:02.382 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:Snapshot ID: 129 2023-11-08T05:15:02.392 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_6 /tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_6.status 2023-11-08T05:15:02.392 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:15:02.393 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:15:02.393 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_6 2023-11-08T05:15:02.393 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_6.status 2023-11-08T05:15:02.393 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_6 --format xml 2023-11-08T05:15:02.393 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T05:15:02.433 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-11-08T05:15:02.434 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snapshot_id=129 2023-11-08T05:15:02.435 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ true 2023-11-08T05:15:02.435 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-11-08T05:15:02.435 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0.2 2023-11-08T05:15:02.636 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_6 /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_6.status 2023-11-08T05:15:02.636 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:15:02.637 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:15:02.637 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_6 2023-11-08T05:15:02.637 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_6.status 2023-11-08T05:15:02.638 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_6 --format xml 2023-11-08T05:15:02.638 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T05:15:02.678 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ true 2023-11-08T05:15:02.678 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-08T05:15:02.679 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:-:1.1: Document is empty 2023-11-08T05:15:02.679 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: 2023-11-08T05:15:02.679 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:^ 2023-11-08T05:15:02.680 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local primary_snapshot_id= 2023-11-08T05:15:02.680 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test 129 = '' 2023-11-08T05:15:02.681 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-11-08T05:15:02.681 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0.4 2023-11-08T05:15:03.081 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_6 /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_6.status 2023-11-08T05:15:03.081 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:15:03.081 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:15:03.082 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_6 2023-11-08T05:15:03.082 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_6.status 2023-11-08T05:15:03.082 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_6 --format xml 2023-11-08T05:15:03.082 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T05:15:03.123 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ true 2023-11-08T05:15:03.123 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-08T05:15:03.124 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:-:1.1: Document is empty 2023-11-08T05:15:03.124 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: 2023-11-08T05:15:03.124 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:^ 2023-11-08T05:15:03.125 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local primary_snapshot_id= 2023-11-08T05:15:03.125 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test 129 = '' 2023-11-08T05:15:03.126 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-11-08T05:15:03.126 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0.8 2023-11-08T05:15:03.926 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_6 /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_6.status 2023-11-08T05:15:03.927 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:15:03.927 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:15:03.927 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_6 2023-11-08T05:15:03.927 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_6.status 2023-11-08T05:15:03.927 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_6 --format xml 2023-11-08T05:15:03.928 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T05:15:03.965 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-08T05:15:03.967 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local primary_snapshot_id=129 2023-11-08T05:15:03.967 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test 129 = 129 2023-11-08T05:15:03.968 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:15:03.968 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_snap_present cluster1 mirror image_6 snap 2023-11-08T05:15:03.968 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:15:03.968 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:15:03.968 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_6 2023-11-08T05:15:03.969 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap_name=snap 2023-11-08T05:15:03.969 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local s 2023-11-08T05:15:03.969 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-11-08T05:15:03.969 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 1 2023-11-08T05:15:04.968 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 info mirror/image_6@snap 2023-11-08T05:15:05.007 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:rbd image 'image_6': 2023-11-08T05:15:05.007 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: size 128 MiB in 32 objects 2023-11-08T05:15:05.007 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: order 22 (4 MiB objects) 2023-11-08T05:15:05.008 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: snapshot_count: 2 2023-11-08T05:15:05.008 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: id: 111232801467 2023-11-08T05:15:05.008 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: block_name_prefix: rbd_data.111232801467 2023-11-08T05:15:05.009 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: format: 2 2023-11-08T05:15:05.009 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: features: layering, exclusive-lock, non-primary 2023-11-08T05:15:05.009 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: op_features: 2023-11-08T05:15:05.009 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: flags: 2023-11-08T05:15:05.010 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: create_timestamp: Wed Nov 8 05:05:16 2023 2023-11-08T05:15:05.010 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: access_timestamp: Wed Nov 8 05:05:16 2023 2023-11-08T05:15:05.010 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: modify_timestamp: Wed Nov 8 05:05:16 2023 2023-11-08T05:15:05.011 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: protected: False 2023-11-08T05:15:05.011 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring state: enabled 2023-11-08T05:15:05.011 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring mode: snapshot 2023-11-08T05:15:05.011 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring global id: acb6ea51-6256-49db-9edb-d84ff9a099a2 2023-11-08T05:15:05.012 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring primary: false 2023-11-08T05:15:05.013 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:15:05.013 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ compare_image_snaps mirror image_6 snap 2023-11-08T05:15:05.014 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:15:05.014 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_6 2023-11-08T05:15:05.014 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap_name=snap 2023-11-08T05:15:05.014 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local ret=0 2023-11-08T05:15:05.014 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local rmt_export=/tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_6.export 2023-11-08T05:15:05.015 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local loc_export=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_6.export 2023-11-08T05:15:05.015 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rm -f /tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_6.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_6.export 2023-11-08T05:15:05.015 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 -p mirror export image_6@snap /tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_6.export 2023-11-08T05:15:05.188 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-11-08T05:15:05.191 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 -p mirror export image_6@snap /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_6.export 2023-11-08T05:15:05.419 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-11-08T05:15:05.422 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ cmp /tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_6.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_6.export 2023-11-08T05:15:05.479 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rm -f /tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_6.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_6.export 2023-11-08T05:15:05.520 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:15:05.520 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-11-08T05:15:05.520 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image=image_7 2023-11-08T05:15:05.521 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ create_snap cluster2 mirror image_7 snap 2023-11-08T05:15:05.521 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:15:05.521 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:15:05.521 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_7 2023-11-08T05:15:05.522 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap_name=snap 2023-11-08T05:15:05.522 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 -p mirror snap create image_7@snap --debug-rbd=20 --debug-journaler=20 2023-11-08T05:15:07.386 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_image_replay_started cluster1 mirror image_7 2023-11-08T05:15:07.387 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:15:07.387 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:15:07.387 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_7 2023-11-08T05:15:07.388 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_image_replay_state cluster1 mirror image_7 started 2023-11-08T05:15:07.388 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:15:07.388 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:15:07.388 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_7 2023-11-08T05:15:07.388 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local state=started 2023-11-08T05:15:07.388 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local s 2023-11-08T05:15:07.389 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-11-08T05:15:07.389 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 1 2023-11-08T05:15:08.387 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test_image_replay_state cluster1 mirror image_7 started 2023-11-08T05:15:08.388 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:15:08.388 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:15:08.388 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_7 2023-11-08T05:15:08.389 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local test_state=started 2023-11-08T05:15:08.389 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local status_result 2023-11-08T05:15:08.389 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local current_state=stopped 2023-11-08T05:15:08.393 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_7 2023-11-08T05:15:08.393 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_instance=cluster1 2023-11-08T05:15:08.394 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T05:15:08.394 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1 2023-11-08T05:15:08.394 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance=cluster1 2023-11-08T05:15:08.395 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local loop_instance 2023-11-08T05:15:08.395 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-11-08T05:15:08.395 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sleep 0 2023-11-08T05:15:08.395 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ grep -i state 2023-11-08T05:15:08.395 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-11-08T05:15:08.396 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ seq 0 3 2023-11-08T05:15:08.396 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T05:15:08.396 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_7 2023-11-08T05:15:08.396 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1:0 2023-11-08T05:15:08.396 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T05:15:08.397 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance 2023-11-08T05:15:08.397 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-11-08T05:15:08.397 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local val=cluster1:0 2023-11-08T05:15:08.397 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_var_name=cluster 2023-11-08T05:15:08.397 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance_var_name=instance 2023-11-08T05:15:08.398 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cluster=cluster1 2023-11-08T05:15:08.398 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ instance=0 2023-11-08T05:15:08.398 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' 0 = cluster1:0 ']' 2023-11-08T05:15:08.398 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval cluster=cluster1 2023-11-08T05:15:08.399 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:15:08.399 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval instance=0 2023-11-08T05:15:08.399 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=0 2023-11-08T05:15:08.399 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-11-08T05:15:08.399 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local local_cluster=cluster1:0 2023-11-08T05:15:08.399 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster=cluster1 2023-11-08T05:15:08.400 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance 2023-11-08T05:15:08.400 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-11-08T05:15:08.400 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local val=cluster1:0 2023-11-08T05:15:08.400 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster_var_name=local_cluster 2023-11-08T05:15:08.400 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance_var_name=instance 2023-11-08T05:15:08.401 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:15:08.401 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=0 2023-11-08T05:15:08.401 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ '[' 0 = cluster1:0 ']' 2023-11-08T05:15:08.401 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval local_cluster=cluster1 2023-11-08T05:15:08.401 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ local_cluster=cluster1 2023-11-08T05:15:08.402 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval instance=0 2023-11-08T05:15:08.402 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ instance=0 2023-11-08T05:15:08.402 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-11-08T05:15:08.408 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-08T05:15:08.409 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-08T05:15:08.409 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-08T05:15:08.409 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_7 2023-11-08T05:15:08.514 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:Invalid command: unused arguments: ['mirror/image_7'] 2023-11-08T05:15:08.515 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd mirror status : get status for rbd mirror 2023-11-08T05:15:08.515 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:admin_socket: invalid command 2023-11-08T05:15:08.518 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T05:15:08.518 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_7 2023-11-08T05:15:08.518 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1:1 2023-11-08T05:15:08.518 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T05:15:08.518 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance 2023-11-08T05:15:08.519 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-11-08T05:15:08.519 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local val=cluster1:1 2023-11-08T05:15:08.519 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_var_name=cluster 2023-11-08T05:15:08.519 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance_var_name=instance 2023-11-08T05:15:08.519 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cluster=cluster1 2023-11-08T05:15:08.520 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ instance=1 2023-11-08T05:15:08.520 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' 1 = cluster1:1 ']' 2023-11-08T05:15:08.520 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval cluster=cluster1 2023-11-08T05:15:08.520 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:15:08.520 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval instance=1 2023-11-08T05:15:08.521 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=1 2023-11-08T05:15:08.521 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-11-08T05:15:08.521 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local local_cluster=cluster1:1 2023-11-08T05:15:08.521 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster=cluster1 2023-11-08T05:15:08.521 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance 2023-11-08T05:15:08.521 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-11-08T05:15:08.522 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local val=cluster1:1 2023-11-08T05:15:08.522 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster_var_name=local_cluster 2023-11-08T05:15:08.522 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance_var_name=instance 2023-11-08T05:15:08.522 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:15:08.522 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=1 2023-11-08T05:15:08.523 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ '[' 1 = cluster1:1 ']' 2023-11-08T05:15:08.523 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval local_cluster=cluster1 2023-11-08T05:15:08.523 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ local_cluster=cluster1 2023-11-08T05:15:08.523 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval instance=1 2023-11-08T05:15:08.523 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ instance=1 2023-11-08T05:15:08.524 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-11-08T05:15:08.532 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-08T05:15:08.532 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-08T05:15:08.532 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-08T05:15:08.532 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_7 2023-11-08T05:15:08.630 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:Invalid command: unused arguments: ['mirror/image_7'] 2023-11-08T05:15:08.631 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd mirror status : get status for rbd mirror 2023-11-08T05:15:08.631 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:admin_socket: invalid command 2023-11-08T05:15:08.634 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T05:15:08.634 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_7 2023-11-08T05:15:08.634 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1:2 2023-11-08T05:15:08.634 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T05:15:08.634 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance 2023-11-08T05:15:08.635 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-11-08T05:15:08.635 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local val=cluster1:2 2023-11-08T05:15:08.635 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_var_name=cluster 2023-11-08T05:15:08.635 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance_var_name=instance 2023-11-08T05:15:08.635 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cluster=cluster1 2023-11-08T05:15:08.636 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ instance=2 2023-11-08T05:15:08.636 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' 2 = cluster1:2 ']' 2023-11-08T05:15:08.636 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval cluster=cluster1 2023-11-08T05:15:08.636 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:15:08.636 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval instance=2 2023-11-08T05:15:08.637 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=2 2023-11-08T05:15:08.637 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-11-08T05:15:08.637 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local local_cluster=cluster1:2 2023-11-08T05:15:08.638 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster=cluster1 2023-11-08T05:15:08.638 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance 2023-11-08T05:15:08.638 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-11-08T05:15:08.638 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local val=cluster1:2 2023-11-08T05:15:08.639 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster_var_name=local_cluster 2023-11-08T05:15:08.639 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance_var_name=instance 2023-11-08T05:15:08.639 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:15:08.639 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=2 2023-11-08T05:15:08.639 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ '[' 2 = cluster1:2 ']' 2023-11-08T05:15:08.639 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval local_cluster=cluster1 2023-11-08T05:15:08.640 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ local_cluster=cluster1 2023-11-08T05:15:08.640 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval instance=2 2023-11-08T05:15:08.640 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ instance=2 2023-11-08T05:15:08.640 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-11-08T05:15:08.648 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-08T05:15:08.648 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-08T05:15:08.648 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-08T05:15:08.649 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_7 2023-11-08T05:15:08.756 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ return 0 2023-11-08T05:15:08.757 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ status_result=' "state": "Replaying"' 2023-11-08T05:15:08.757 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ echo ' "state": "Replaying"' 2023-11-08T05:15:08.758 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ grep -i Replaying 2023-11-08T05:15:08.758 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: "state": "Replaying" 2023-11-08T05:15:08.760 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ current_state=started 2023-11-08T05:15:08.760 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test started = started 2023-11-08T05:15:08.761 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:15:08.761 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_7 2023-11-08T05:15:08.761 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local local_cluster=cluster1 2023-11-08T05:15:08.761 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:15:08.761 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:15:08.762 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_7 2023-11-08T05:15:08.762 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' snapshot = journal ']' 2023-11-08T05:15:08.762 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' snapshot = snapshot ']' 2023-11-08T05:15:08.762 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_7 2023-11-08T05:15:08.762 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local local_cluster=cluster1 2023-11-08T05:15:08.763 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:15:08.763 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:15:08.763 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_7 2023-11-08T05:15:08.763 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ mkfname cluster2-mirror-image_7.status 2023-11-08T05:15:08.763 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ echo cluster2-mirror-image_7.status 2023-11-08T05:15:08.764 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -e 's|[/ ]|_|g' 2023-11-08T05:15:08.764 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local status_log=/tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_7.status 2023-11-08T05:15:08.764 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ mkfname cluster1-mirror-image_7.status 2023-11-08T05:15:08.764 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ echo cluster1-mirror-image_7.status 2023-11-08T05:15:08.764 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -e 's|[/ ]|_|g' 2023-11-08T05:15:08.765 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local local_status_log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_7.status 2023-11-08T05:15:08.765 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ mirror_image_snapshot cluster2 mirror image_7 2023-11-08T05:15:08.766 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:15:08.766 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:15:08.766 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_7 2023-11-08T05:15:08.766 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_7 2023-11-08T05:15:09.399 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:Snapshot ID: 131 2023-11-08T05:15:09.408 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_7 /tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_7.status 2023-11-08T05:15:09.408 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:15:09.408 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:15:09.409 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_7 2023-11-08T05:15:09.409 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_7.status 2023-11-08T05:15:09.409 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_7 --format xml 2023-11-08T05:15:09.409 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T05:15:09.446 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-11-08T05:15:09.447 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snapshot_id=131 2023-11-08T05:15:09.448 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ true 2023-11-08T05:15:09.448 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-11-08T05:15:09.448 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0.2 2023-11-08T05:15:09.649 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_7 /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_7.status 2023-11-08T05:15:09.649 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:15:09.650 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:15:09.650 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_7 2023-11-08T05:15:09.650 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_7.status 2023-11-08T05:15:09.650 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_7 --format xml 2023-11-08T05:15:09.651 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T05:15:09.690 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ true 2023-11-08T05:15:09.690 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-08T05:15:09.691 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:-:1.1: Document is empty 2023-11-08T05:15:09.691 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: 2023-11-08T05:15:09.691 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:^ 2023-11-08T05:15:09.692 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local primary_snapshot_id= 2023-11-08T05:15:09.692 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test 131 = '' 2023-11-08T05:15:09.692 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-11-08T05:15:09.693 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0.4 2023-11-08T05:15:10.093 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_7 /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_7.status 2023-11-08T05:15:10.094 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:15:10.094 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:15:10.094 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_7 2023-11-08T05:15:10.094 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_7.status 2023-11-08T05:15:10.095 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_7 --format xml 2023-11-08T05:15:10.095 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T05:15:10.136 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ true 2023-11-08T05:15:10.137 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-08T05:15:10.138 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:-:1.1: Document is empty 2023-11-08T05:15:10.138 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: 2023-11-08T05:15:10.138 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:^ 2023-11-08T05:15:10.139 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local primary_snapshot_id= 2023-11-08T05:15:10.139 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test 131 = '' 2023-11-08T05:15:10.139 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-11-08T05:15:10.140 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0.8 2023-11-08T05:15:10.940 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_7 /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_7.status 2023-11-08T05:15:10.940 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:15:10.940 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:15:10.941 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_7 2023-11-08T05:15:10.941 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_7.status 2023-11-08T05:15:10.941 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_7 --format xml 2023-11-08T05:15:10.941 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T05:15:10.978 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-08T05:15:10.980 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local primary_snapshot_id=131 2023-11-08T05:15:10.980 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test 131 = 131 2023-11-08T05:15:10.980 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:15:10.981 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_snap_present cluster1 mirror image_7 snap 2023-11-08T05:15:10.981 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:15:10.981 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:15:10.981 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_7 2023-11-08T05:15:10.981 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap_name=snap 2023-11-08T05:15:10.982 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local s 2023-11-08T05:15:10.982 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-11-08T05:15:10.982 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 1 2023-11-08T05:15:11.193 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.5119944706866523) > weight (0.25) 2023-11-08T05:15:11.193 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.5059286874197627) > weight (0.25) 2023-11-08T05:15:11.193 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.2 2023-11-08T05:15:11.194 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Sent signal 15 2023-11-08T05:15:11.194 INFO:tasks.rbd_mirror_thrash:waiting for 1 secs before reviving daemons 2023-11-08T05:15:11.195 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi087.stderr:2023-11-08T05:15:11.193+0000 7fd436fa6700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 (PID: 133931) UID: 1000 2023-11-08T05:15:11.596 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi087.stderr:daemon-helper: command failed with exit status 1 2023-11-08T05:15:11.981 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 info mirror/image_7@snap 2023-11-08T05:15:12.015 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:rbd image 'image_7': 2023-11-08T05:15:12.015 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: size 128 MiB in 32 objects 2023-11-08T05:15:12.015 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: order 22 (4 MiB objects) 2023-11-08T05:15:12.015 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: snapshot_count: 2 2023-11-08T05:15:12.016 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: id: 11d892bbde20 2023-11-08T05:15:12.016 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: block_name_prefix: rbd_data.11d892bbde20 2023-11-08T05:15:12.016 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: format: 2 2023-11-08T05:15:12.016 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: features: layering, exclusive-lock, non-primary 2023-11-08T05:15:12.016 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: op_features: 2023-11-08T05:15:12.016 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: flags: 2023-11-08T05:15:12.017 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: create_timestamp: Wed Nov 8 05:05:24 2023 2023-11-08T05:15:12.017 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: access_timestamp: Wed Nov 8 05:05:24 2023 2023-11-08T05:15:12.017 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: modify_timestamp: Wed Nov 8 05:05:24 2023 2023-11-08T05:15:12.017 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: protected: False 2023-11-08T05:15:12.017 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring state: enabled 2023-11-08T05:15:12.018 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring mode: snapshot 2023-11-08T05:15:12.018 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring global id: baa8ce06-b6bc-450a-80c9-52c7757d4aa3 2023-11-08T05:15:12.018 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring primary: false 2023-11-08T05:15:12.019 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:15:12.020 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ compare_image_snaps mirror image_7 snap 2023-11-08T05:15:12.020 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:15:12.020 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_7 2023-11-08T05:15:12.020 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap_name=snap 2023-11-08T05:15:12.020 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local ret=0 2023-11-08T05:15:12.021 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local rmt_export=/tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_7.export 2023-11-08T05:15:12.021 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local loc_export=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_7.export 2023-11-08T05:15:12.021 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rm -f /tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_7.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_7.export 2023-11-08T05:15:12.021 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 -p mirror export image_7@snap /tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_7.export 2023-11-08T05:15:12.190 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-11-08T05:15:12.193 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 -p mirror export image_7@snap /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_7.export 2023-11-08T05:15:12.195 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.2 2023-11-08T05:15:12.195 INFO:teuthology.orchestra.run:waiting for 600 2023-11-08T05:15:12.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-08T05:15:12.196 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.2 2023-11-08T05:15:12.196 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Restarting daemon 2023-11-08T05:15:12.196 DEBUG:teuthology.orchestra.run.smithi087:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 2023-11-08T05:15:12.199 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Started 2023-11-08T05:15:12.199 INFO:tasks.rbd_mirror_thrash:waiting for 92 secs before thrashing 2023-11-08T05:15:12.434 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-11-08T05:15:12.437 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ cmp /tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_7.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_7.export 2023-11-08T05:15:12.495 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rm -f /tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_7.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_7.export 2023-11-08T05:15:12.535 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:15:12.535 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-11-08T05:15:12.535 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image=image_8 2023-11-08T05:15:12.535 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ create_snap cluster2 mirror image_8 snap 2023-11-08T05:15:12.536 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:15:12.536 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:15:12.536 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_8 2023-11-08T05:15:12.536 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap_name=snap 2023-11-08T05:15:12.536 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 -p mirror snap create image_8@snap --debug-rbd=20 --debug-journaler=20 2023-11-08T05:15:12.582 INFO:tasks.ceph.cluster2.osd.2.smithi087.stderr:2023-11-08T05:15:12.579+0000 7fa243317700 -1 reset not still connected to 0x5613a7399930 2023-11-08T05:15:12.583 INFO:tasks.ceph.cluster2.osd.2.smithi087.stderr:2023-11-08T05:15:12.579+0000 7fa243317700 -1 reset not still connected to 0x5613b889e4e0 2023-11-08T05:15:12.583 INFO:tasks.ceph.cluster2.osd.2.smithi087.stderr:2023-11-08T05:15:12.579+0000 7fa243317700 -1 reset not still connected to 0x5613bb67ed00 2023-11-08T05:15:12.583 INFO:tasks.ceph.cluster2.osd.2.smithi087.stderr:2023-11-08T05:15:12.579+0000 7fa243317700 -1 reset not still connected to 0x5613bc8465b0 2023-11-08T05:15:14.408 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_image_replay_started cluster1 mirror image_8 2023-11-08T05:15:14.409 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:15:14.409 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:15:14.409 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_8 2023-11-08T05:15:14.409 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_image_replay_state cluster1 mirror image_8 started 2023-11-08T05:15:14.410 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:15:14.410 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:15:14.410 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_8 2023-11-08T05:15:14.410 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local state=started 2023-11-08T05:15:14.411 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local s 2023-11-08T05:15:14.411 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-11-08T05:15:14.411 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 1 2023-11-08T05:15:15.409 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test_image_replay_state cluster1 mirror image_8 started 2023-11-08T05:15:15.410 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:15:15.410 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:15:15.410 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_8 2023-11-08T05:15:15.410 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local test_state=started 2023-11-08T05:15:15.411 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local status_result 2023-11-08T05:15:15.411 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local current_state=stopped 2023-11-08T05:15:15.412 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_8 2023-11-08T05:15:15.412 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_instance=cluster1 2023-11-08T05:15:15.413 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T05:15:15.413 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ grep -i state 2023-11-08T05:15:15.413 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1 2023-11-08T05:15:15.413 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance=cluster1 2023-11-08T05:15:15.413 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local loop_instance 2023-11-08T05:15:15.414 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-11-08T05:15:15.414 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sleep 0 2023-11-08T05:15:15.414 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-11-08T05:15:15.414 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ seq 0 3 2023-11-08T05:15:15.416 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T05:15:15.416 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_8 2023-11-08T05:15:15.416 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1:0 2023-11-08T05:15:15.416 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T05:15:15.417 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance 2023-11-08T05:15:15.417 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-11-08T05:15:15.417 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local val=cluster1:0 2023-11-08T05:15:15.417 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_var_name=cluster 2023-11-08T05:15:15.417 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance_var_name=instance 2023-11-08T05:15:15.418 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cluster=cluster1 2023-11-08T05:15:15.418 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ instance=0 2023-11-08T05:15:15.418 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' 0 = cluster1:0 ']' 2023-11-08T05:15:15.418 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval cluster=cluster1 2023-11-08T05:15:15.418 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:15:15.419 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval instance=0 2023-11-08T05:15:15.419 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=0 2023-11-08T05:15:15.419 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-11-08T05:15:15.421 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local local_cluster=cluster1:0 2023-11-08T05:15:15.421 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster=cluster1 2023-11-08T05:15:15.421 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance 2023-11-08T05:15:15.421 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-11-08T05:15:15.422 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local val=cluster1:0 2023-11-08T05:15:15.422 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster_var_name=local_cluster 2023-11-08T05:15:15.422 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance_var_name=instance 2023-11-08T05:15:15.422 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:15:15.422 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=0 2023-11-08T05:15:15.423 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ '[' 0 = cluster1:0 ']' 2023-11-08T05:15:15.423 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval local_cluster=cluster1 2023-11-08T05:15:15.423 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ local_cluster=cluster1 2023-11-08T05:15:15.423 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval instance=0 2023-11-08T05:15:15.423 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ instance=0 2023-11-08T05:15:15.423 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-11-08T05:15:15.432 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-08T05:15:15.433 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-08T05:15:15.433 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-08T05:15:15.433 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_8 2023-11-08T05:15:15.547 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ return 0 2023-11-08T05:15:15.548 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ status_result=' "state": "Replaying"' 2023-11-08T05:15:15.549 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ echo ' "state": "Replaying"' 2023-11-08T05:15:15.549 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ grep -i Replaying 2023-11-08T05:15:15.549 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: "state": "Replaying" 2023-11-08T05:15:15.551 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ current_state=started 2023-11-08T05:15:15.552 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test started = started 2023-11-08T05:15:15.552 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:15:15.552 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_8 2023-11-08T05:15:15.552 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local local_cluster=cluster1 2023-11-08T05:15:15.552 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:15:15.553 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:15:15.553 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_8 2023-11-08T05:15:15.553 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' snapshot = journal ']' 2023-11-08T05:15:15.553 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' snapshot = snapshot ']' 2023-11-08T05:15:15.554 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_8 2023-11-08T05:15:15.554 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local local_cluster=cluster1 2023-11-08T05:15:15.554 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:15:15.554 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:15:15.554 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_8 2023-11-08T05:15:15.555 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ mkfname cluster2-mirror-image_8.status 2023-11-08T05:15:15.555 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ echo cluster2-mirror-image_8.status 2023-11-08T05:15:15.555 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -e 's|[/ ]|_|g' 2023-11-08T05:15:15.555 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local status_log=/tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_8.status 2023-11-08T05:15:15.555 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ mkfname cluster1-mirror-image_8.status 2023-11-08T05:15:15.556 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ echo cluster1-mirror-image_8.status 2023-11-08T05:15:15.556 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -e 's|[/ ]|_|g' 2023-11-08T05:15:15.557 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local local_status_log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_8.status 2023-11-08T05:15:15.557 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ mirror_image_snapshot cluster2 mirror image_8 2023-11-08T05:15:15.557 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:15:15.557 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:15:15.558 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_8 2023-11-08T05:15:15.558 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_8 2023-11-08T05:15:16.417 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:Snapshot ID: 133 2023-11-08T05:15:16.428 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_8 /tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_8.status 2023-11-08T05:15:16.428 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:15:16.428 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:15:16.428 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_8 2023-11-08T05:15:16.429 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_8.status 2023-11-08T05:15:16.429 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_8 --format xml 2023-11-08T05:15:16.429 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T05:15:16.471 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-11-08T05:15:16.473 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snapshot_id=133 2023-11-08T05:15:16.474 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ true 2023-11-08T05:15:16.474 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-11-08T05:15:16.474 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0.2 2023-11-08T05:15:16.675 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_8 /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_8.status 2023-11-08T05:15:16.675 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:15:16.675 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:15:16.676 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_8 2023-11-08T05:15:16.676 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_8.status 2023-11-08T05:15:16.676 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_8 --format xml 2023-11-08T05:15:16.676 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T05:15:16.716 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ true 2023-11-08T05:15:16.717 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-08T05:15:16.718 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:-:1.1: Document is empty 2023-11-08T05:15:16.718 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: 2023-11-08T05:15:16.718 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:^ 2023-11-08T05:15:16.719 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local primary_snapshot_id= 2023-11-08T05:15:16.719 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test 133 = '' 2023-11-08T05:15:16.720 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-11-08T05:15:16.720 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0.4 2023-11-08T05:15:17.120 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_8 /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_8.status 2023-11-08T05:15:17.120 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:15:17.121 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:15:17.121 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_8 2023-11-08T05:15:17.121 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_8.status 2023-11-08T05:15:17.121 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_8 --format xml 2023-11-08T05:15:17.121 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T05:15:17.166 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ true 2023-11-08T05:15:17.166 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-08T05:15:17.167 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:-:1.1: Document is empty 2023-11-08T05:15:17.167 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: 2023-11-08T05:15:17.168 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:^ 2023-11-08T05:15:17.168 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local primary_snapshot_id= 2023-11-08T05:15:17.168 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test 133 = '' 2023-11-08T05:15:17.169 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-11-08T05:15:17.169 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0.8 2023-11-08T05:15:17.969 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_8 /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_8.status 2023-11-08T05:15:17.970 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:15:17.970 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:15:17.970 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_8 2023-11-08T05:15:17.970 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_8.status 2023-11-08T05:15:17.971 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_8 --format xml 2023-11-08T05:15:17.971 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T05:15:18.009 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-08T05:15:18.011 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local primary_snapshot_id=133 2023-11-08T05:15:18.011 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test 133 = 133 2023-11-08T05:15:18.011 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:15:18.012 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_snap_present cluster1 mirror image_8 snap 2023-11-08T05:15:18.012 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:15:18.012 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:15:18.012 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_8 2023-11-08T05:15:18.012 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap_name=snap 2023-11-08T05:15:18.013 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local s 2023-11-08T05:15:18.013 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-11-08T05:15:18.013 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 1 2023-11-08T05:15:19.012 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 info mirror/image_8@snap 2023-11-08T05:15:19.051 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:rbd image 'image_8': 2023-11-08T05:15:19.051 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: size 128 MiB in 32 objects 2023-11-08T05:15:19.052 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: order 22 (4 MiB objects) 2023-11-08T05:15:19.052 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: snapshot_count: 2 2023-11-08T05:15:19.052 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: id: 128729aa5b4d 2023-11-08T05:15:19.052 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: block_name_prefix: rbd_data.128729aa5b4d 2023-11-08T05:15:19.052 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: format: 2 2023-11-08T05:15:19.053 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: features: layering, exclusive-lock, non-primary 2023-11-08T05:15:19.053 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: op_features: 2023-11-08T05:15:19.053 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: flags: 2023-11-08T05:15:19.053 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: create_timestamp: Wed Nov 8 05:05:37 2023 2023-11-08T05:15:19.053 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: access_timestamp: Wed Nov 8 05:05:37 2023 2023-11-08T05:15:19.054 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: modify_timestamp: Wed Nov 8 05:05:37 2023 2023-11-08T05:15:19.054 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: protected: False 2023-11-08T05:15:19.054 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring state: enabled 2023-11-08T05:15:19.054 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring mode: snapshot 2023-11-08T05:15:19.054 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring global id: c78a25c8-e255-457e-8ce9-bb570e06b394 2023-11-08T05:15:19.055 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring primary: false 2023-11-08T05:15:19.058 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:15:19.058 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ compare_image_snaps mirror image_8 snap 2023-11-08T05:15:19.058 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:15:19.059 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_8 2023-11-08T05:15:19.059 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap_name=snap 2023-11-08T05:15:19.059 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local ret=0 2023-11-08T05:15:19.059 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local rmt_export=/tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_8.export 2023-11-08T05:15:19.060 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local loc_export=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_8.export 2023-11-08T05:15:19.060 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rm -f /tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_8.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_8.export 2023-11-08T05:15:19.060 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 -p mirror export image_8@snap /tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_8.export 2023-11-08T05:15:19.244 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-11-08T05:15:19.247 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 -p mirror export image_8@snap /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_8.export 2023-11-08T05:15:19.494 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-11-08T05:15:19.497 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ cmp /tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_8.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_8.export 2023-11-08T05:15:19.553 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rm -f /tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_8.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_8.export 2023-11-08T05:15:19.592 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:15:19.593 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-11-08T05:15:19.593 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image=image_9 2023-11-08T05:15:19.593 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ create_snap cluster2 mirror image_9 snap 2023-11-08T05:15:19.593 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:15:19.594 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:15:19.594 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_9 2023-11-08T05:15:19.594 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap_name=snap 2023-11-08T05:15:19.594 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 -p mirror snap create image_9@snap --debug-rbd=20 --debug-journaler=20 2023-11-08T05:15:21.424 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_image_replay_started cluster1 mirror image_9 2023-11-08T05:15:21.425 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:15:21.425 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:15:21.425 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_9 2023-11-08T05:15:21.426 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_image_replay_state cluster1 mirror image_9 started 2023-11-08T05:15:21.426 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:15:21.426 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:15:21.427 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_9 2023-11-08T05:15:21.427 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local state=started 2023-11-08T05:15:21.427 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local s 2023-11-08T05:15:21.427 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-11-08T05:15:21.427 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 1 2023-11-08T05:15:22.097 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.8110228041881675) > weight (0.25) 2023-11-08T05:15:22.098 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.6620351197691721) > weight (0.25) 2023-11-08T05:15:22.098 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.7837172146671899) > weight (0.25) 2023-11-08T05:15:22.099 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.3 with skip (0.6101864769867689) > weight (0.25) 2023-11-08T05:15:22.099 INFO:tasks.rbd_mirror_thrash:waiting for 113 secs before thrashing 2023-11-08T05:15:22.425 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test_image_replay_state cluster1 mirror image_9 started 2023-11-08T05:15:22.425 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:15:22.426 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:15:22.426 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_9 2023-11-08T05:15:22.426 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local test_state=started 2023-11-08T05:15:22.427 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local status_result 2023-11-08T05:15:22.427 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local current_state=stopped 2023-11-08T05:15:22.432 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_9 2023-11-08T05:15:22.433 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ grep -i state 2023-11-08T05:15:22.433 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_instance=cluster1 2023-11-08T05:15:22.433 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T05:15:22.433 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1 2023-11-08T05:15:22.433 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance=cluster1 2023-11-08T05:15:22.434 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local loop_instance 2023-11-08T05:15:22.434 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-11-08T05:15:22.434 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sleep 0 2023-11-08T05:15:22.434 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-11-08T05:15:22.434 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ seq 0 3 2023-11-08T05:15:22.435 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T05:15:22.435 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_9 2023-11-08T05:15:22.435 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1:0 2023-11-08T05:15:22.435 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T05:15:22.435 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance 2023-11-08T05:15:22.436 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-11-08T05:15:22.436 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local val=cluster1:0 2023-11-08T05:15:22.436 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_var_name=cluster 2023-11-08T05:15:22.436 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance_var_name=instance 2023-11-08T05:15:22.436 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cluster=cluster1 2023-11-08T05:15:22.437 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ instance=0 2023-11-08T05:15:22.437 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' 0 = cluster1:0 ']' 2023-11-08T05:15:22.437 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval cluster=cluster1 2023-11-08T05:15:22.437 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:15:22.437 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval instance=0 2023-11-08T05:15:22.438 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=0 2023-11-08T05:15:22.438 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-11-08T05:15:22.438 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local local_cluster=cluster1:0 2023-11-08T05:15:22.438 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster=cluster1 2023-11-08T05:15:22.439 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance 2023-11-08T05:15:22.439 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-11-08T05:15:22.439 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local val=cluster1:0 2023-11-08T05:15:22.439 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster_var_name=local_cluster 2023-11-08T05:15:22.439 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance_var_name=instance 2023-11-08T05:15:22.439 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:15:22.440 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=0 2023-11-08T05:15:22.440 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ '[' 0 = cluster1:0 ']' 2023-11-08T05:15:22.440 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval local_cluster=cluster1 2023-11-08T05:15:22.440 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ local_cluster=cluster1 2023-11-08T05:15:22.440 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval instance=0 2023-11-08T05:15:22.441 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ instance=0 2023-11-08T05:15:22.441 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-11-08T05:15:22.447 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-08T05:15:22.447 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-08T05:15:22.447 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-08T05:15:22.447 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_9 2023-11-08T05:15:22.558 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:Invalid command: unused arguments: ['mirror/image_9'] 2023-11-08T05:15:22.558 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd mirror status : get status for rbd mirror 2023-11-08T05:15:22.558 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:admin_socket: invalid command 2023-11-08T05:15:22.561 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T05:15:22.561 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_9 2023-11-08T05:15:22.562 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1:1 2023-11-08T05:15:22.562 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T05:15:22.562 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance 2023-11-08T05:15:22.562 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-11-08T05:15:22.563 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local val=cluster1:1 2023-11-08T05:15:22.563 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_var_name=cluster 2023-11-08T05:15:22.563 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance_var_name=instance 2023-11-08T05:15:22.563 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cluster=cluster1 2023-11-08T05:15:22.563 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ instance=1 2023-11-08T05:15:22.564 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' 1 = cluster1:1 ']' 2023-11-08T05:15:22.564 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval cluster=cluster1 2023-11-08T05:15:22.564 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:15:22.564 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval instance=1 2023-11-08T05:15:22.564 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=1 2023-11-08T05:15:22.565 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-11-08T05:15:22.565 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local local_cluster=cluster1:1 2023-11-08T05:15:22.565 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster=cluster1 2023-11-08T05:15:22.565 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance 2023-11-08T05:15:22.565 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-11-08T05:15:22.566 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local val=cluster1:1 2023-11-08T05:15:22.566 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster_var_name=local_cluster 2023-11-08T05:15:22.566 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance_var_name=instance 2023-11-08T05:15:22.566 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:15:22.566 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=1 2023-11-08T05:15:22.567 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ '[' 1 = cluster1:1 ']' 2023-11-08T05:15:22.567 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval local_cluster=cluster1 2023-11-08T05:15:22.567 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ local_cluster=cluster1 2023-11-08T05:15:22.567 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval instance=1 2023-11-08T05:15:22.567 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ instance=1 2023-11-08T05:15:22.568 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-11-08T05:15:22.575 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-08T05:15:22.576 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-08T05:15:22.576 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-08T05:15:22.576 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_9 2023-11-08T05:15:22.682 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ return 0 2023-11-08T05:15:22.683 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ status_result=' "state": "Replaying"' 2023-11-08T05:15:22.683 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ echo ' "state": "Replaying"' 2023-11-08T05:15:22.683 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ grep -i Replaying 2023-11-08T05:15:22.684 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: "state": "Replaying" 2023-11-08T05:15:22.686 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ current_state=started 2023-11-08T05:15:22.686 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test started = started 2023-11-08T05:15:22.686 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:15:22.687 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_9 2023-11-08T05:15:22.687 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local local_cluster=cluster1 2023-11-08T05:15:22.687 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:15:22.687 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:15:22.688 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_9 2023-11-08T05:15:22.688 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' snapshot = journal ']' 2023-11-08T05:15:22.688 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' snapshot = snapshot ']' 2023-11-08T05:15:22.689 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_9 2023-11-08T05:15:22.689 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local local_cluster=cluster1 2023-11-08T05:15:22.689 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:15:22.690 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:15:22.690 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_9 2023-11-08T05:15:22.691 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ mkfname cluster2-mirror-image_9.status 2023-11-08T05:15:22.691 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ echo cluster2-mirror-image_9.status 2023-11-08T05:15:22.691 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -e 's|[/ ]|_|g' 2023-11-08T05:15:22.692 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local status_log=/tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_9.status 2023-11-08T05:15:22.692 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ mkfname cluster1-mirror-image_9.status 2023-11-08T05:15:22.692 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ echo cluster1-mirror-image_9.status 2023-11-08T05:15:22.693 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -e 's|[/ ]|_|g' 2023-11-08T05:15:22.694 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local local_status_log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_9.status 2023-11-08T05:15:22.694 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ mirror_image_snapshot cluster2 mirror image_9 2023-11-08T05:15:22.695 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:15:22.695 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:15:22.695 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_9 2023-11-08T05:15:22.695 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_9 2023-11-08T05:15:23.430 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:Snapshot ID: 135 2023-11-08T05:15:23.439 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_9 /tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_9.status 2023-11-08T05:15:23.439 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:15:23.440 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:15:23.440 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_9 2023-11-08T05:15:23.440 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_9.status 2023-11-08T05:15:23.440 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_9 --format xml 2023-11-08T05:15:23.440 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T05:15:23.477 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-11-08T05:15:23.479 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snapshot_id=135 2023-11-08T05:15:23.479 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ true 2023-11-08T05:15:23.480 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-11-08T05:15:23.480 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0.2 2023-11-08T05:15:23.681 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_9 /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_9.status 2023-11-08T05:15:23.681 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:15:23.681 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:15:23.681 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_9 2023-11-08T05:15:23.682 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_9.status 2023-11-08T05:15:23.682 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_9 --format xml 2023-11-08T05:15:23.682 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T05:15:23.719 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ true 2023-11-08T05:15:23.719 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-08T05:15:23.720 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:-:1.1: Document is empty 2023-11-08T05:15:23.720 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: 2023-11-08T05:15:23.721 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:^ 2023-11-08T05:15:23.721 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local primary_snapshot_id= 2023-11-08T05:15:23.722 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test 135 = '' 2023-11-08T05:15:23.722 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-11-08T05:15:23.722 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0.4 2023-11-08T05:15:24.122 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_9 /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_9.status 2023-11-08T05:15:24.122 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:15:24.123 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:15:24.123 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_9 2023-11-08T05:15:24.123 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_9.status 2023-11-08T05:15:24.123 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_9 --format xml 2023-11-08T05:15:24.123 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T05:15:24.161 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ true 2023-11-08T05:15:24.161 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-08T05:15:24.163 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:-:1.1: Document is empty 2023-11-08T05:15:24.163 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: 2023-11-08T05:15:24.163 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:^ 2023-11-08T05:15:24.163 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local primary_snapshot_id= 2023-11-08T05:15:24.163 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test 135 = '' 2023-11-08T05:15:24.164 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-11-08T05:15:24.164 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0.8 2023-11-08T05:15:24.964 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_9 /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_9.status 2023-11-08T05:15:24.964 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:15:24.965 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:15:24.965 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_9 2023-11-08T05:15:24.965 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_9.status 2023-11-08T05:15:24.965 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_9 --format xml 2023-11-08T05:15:24.965 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T05:15:25.003 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-08T05:15:25.005 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local primary_snapshot_id=135 2023-11-08T05:15:25.005 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test 135 = 135 2023-11-08T05:15:25.005 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:15:25.006 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_snap_present cluster1 mirror image_9 snap 2023-11-08T05:15:25.006 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:15:25.006 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:15:25.006 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_9 2023-11-08T05:15:25.006 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap_name=snap 2023-11-08T05:15:25.007 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local s 2023-11-08T05:15:25.007 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-11-08T05:15:25.007 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 1 2023-11-08T05:15:26.006 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 info mirror/image_9@snap 2023-11-08T05:15:26.045 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:rbd image 'image_9': 2023-11-08T05:15:26.045 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: size 128 MiB in 32 objects 2023-11-08T05:15:26.045 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: order 22 (4 MiB objects) 2023-11-08T05:15:26.045 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: snapshot_count: 2 2023-11-08T05:15:26.046 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: id: 10e7bd729ca3 2023-11-08T05:15:26.046 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: block_name_prefix: rbd_data.10e7bd729ca3 2023-11-08T05:15:26.046 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: format: 2 2023-11-08T05:15:26.046 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: features: layering, exclusive-lock, non-primary 2023-11-08T05:15:26.046 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: op_features: 2023-11-08T05:15:26.047 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: flags: 2023-11-08T05:15:26.047 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: create_timestamp: Wed Nov 8 05:05:52 2023 2023-11-08T05:15:26.047 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: access_timestamp: Wed Nov 8 05:05:52 2023 2023-11-08T05:15:26.047 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: modify_timestamp: Wed Nov 8 05:05:52 2023 2023-11-08T05:15:26.047 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: protected: False 2023-11-08T05:15:26.048 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring state: enabled 2023-11-08T05:15:26.048 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring mode: snapshot 2023-11-08T05:15:26.048 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring global id: bd37ef4f-d302-424b-952f-0d71e891061b 2023-11-08T05:15:26.048 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring primary: false 2023-11-08T05:15:26.049 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:15:26.050 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ compare_image_snaps mirror image_9 snap 2023-11-08T05:15:26.050 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:15:26.050 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_9 2023-11-08T05:15:26.050 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap_name=snap 2023-11-08T05:15:26.050 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local ret=0 2023-11-08T05:15:26.051 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local rmt_export=/tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_9.export 2023-11-08T05:15:26.051 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local loc_export=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_9.export 2023-11-08T05:15:26.051 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rm -f /tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_9.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_9.export 2023-11-08T05:15:26.051 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 -p mirror export image_9@snap /tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_9.export 2023-11-08T05:15:26.228 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-11-08T05:15:26.231 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 -p mirror export image_9@snap /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_9.export 2023-11-08T05:15:26.473 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-11-08T05:15:26.476 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ cmp /tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_9.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_9.export 2023-11-08T05:15:26.532 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rm -f /tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_9.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_9.export 2023-11-08T05:15:26.571 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:15:26.572 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-11-08T05:15:26.572 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image=image_10 2023-11-08T05:15:26.572 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ create_snap cluster2 mirror image_10 snap 2023-11-08T05:15:26.573 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:15:26.573 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:15:26.573 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_10 2023-11-08T05:15:26.573 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap_name=snap 2023-11-08T05:15:26.573 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 -p mirror snap create image_10@snap --debug-rbd=20 --debug-journaler=20 2023-11-08T05:15:28.444 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_image_replay_started cluster1 mirror image_10 2023-11-08T05:15:28.444 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:15:28.445 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:15:28.445 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_10 2023-11-08T05:15:28.445 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_image_replay_state cluster1 mirror image_10 started 2023-11-08T05:15:28.445 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:15:28.445 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:15:28.446 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_10 2023-11-08T05:15:28.446 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local state=started 2023-11-08T05:15:28.446 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local s 2023-11-08T05:15:28.446 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-11-08T05:15:28.447 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 1 2023-11-08T05:15:29.446 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test_image_replay_state cluster1 mirror image_10 started 2023-11-08T05:15:29.447 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:15:29.447 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:15:29.447 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_10 2023-11-08T05:15:29.447 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local test_state=started 2023-11-08T05:15:29.447 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local status_result 2023-11-08T05:15:29.448 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local current_state=stopped 2023-11-08T05:15:29.448 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_10 2023-11-08T05:15:29.448 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_instance=cluster1 2023-11-08T05:15:29.448 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T05:15:29.449 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ grep -i state 2023-11-08T05:15:29.449 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1 2023-11-08T05:15:29.449 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance=cluster1 2023-11-08T05:15:29.449 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local loop_instance 2023-11-08T05:15:29.450 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-11-08T05:15:29.450 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sleep 0 2023-11-08T05:15:29.453 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-11-08T05:15:29.453 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ seq 0 3 2023-11-08T05:15:29.453 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T05:15:29.454 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_10 2023-11-08T05:15:29.454 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1:0 2023-11-08T05:15:29.454 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T05:15:29.454 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance 2023-11-08T05:15:29.455 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-11-08T05:15:29.455 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local val=cluster1:0 2023-11-08T05:15:29.455 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_var_name=cluster 2023-11-08T05:15:29.455 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance_var_name=instance 2023-11-08T05:15:29.455 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cluster=cluster1 2023-11-08T05:15:29.456 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ instance=0 2023-11-08T05:15:29.456 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' 0 = cluster1:0 ']' 2023-11-08T05:15:29.456 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval cluster=cluster1 2023-11-08T05:15:29.456 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:15:29.456 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval instance=0 2023-11-08T05:15:29.457 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=0 2023-11-08T05:15:29.457 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-11-08T05:15:29.457 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local local_cluster=cluster1:0 2023-11-08T05:15:29.457 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster=cluster1 2023-11-08T05:15:29.457 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance 2023-11-08T05:15:29.458 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-11-08T05:15:29.458 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local val=cluster1:0 2023-11-08T05:15:29.458 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster_var_name=local_cluster 2023-11-08T05:15:29.458 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance_var_name=instance 2023-11-08T05:15:29.458 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:15:29.459 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=0 2023-11-08T05:15:29.459 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ '[' 0 = cluster1:0 ']' 2023-11-08T05:15:29.459 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval local_cluster=cluster1 2023-11-08T05:15:29.459 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ local_cluster=cluster1 2023-11-08T05:15:29.460 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval instance=0 2023-11-08T05:15:29.460 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ instance=0 2023-11-08T05:15:29.460 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-11-08T05:15:29.467 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-08T05:15:29.467 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-08T05:15:29.468 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-08T05:15:29.468 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_10 2023-11-08T05:15:29.578 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:Invalid command: unused arguments: ['mirror/image_10'] 2023-11-08T05:15:29.579 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd mirror status : get status for rbd mirror 2023-11-08T05:15:29.579 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:admin_socket: invalid command 2023-11-08T05:15:29.581 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T05:15:29.582 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_10 2023-11-08T05:15:29.582 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1:1 2023-11-08T05:15:29.582 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T05:15:29.582 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance 2023-11-08T05:15:29.583 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-11-08T05:15:29.583 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local val=cluster1:1 2023-11-08T05:15:29.583 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_var_name=cluster 2023-11-08T05:15:29.583 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance_var_name=instance 2023-11-08T05:15:29.583 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cluster=cluster1 2023-11-08T05:15:29.584 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ instance=1 2023-11-08T05:15:29.584 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' 1 = cluster1:1 ']' 2023-11-08T05:15:29.584 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval cluster=cluster1 2023-11-08T05:15:29.584 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:15:29.584 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval instance=1 2023-11-08T05:15:29.585 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=1 2023-11-08T05:15:29.585 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-11-08T05:15:29.585 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local local_cluster=cluster1:1 2023-11-08T05:15:29.585 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster=cluster1 2023-11-08T05:15:29.585 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance 2023-11-08T05:15:29.586 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-11-08T05:15:29.586 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local val=cluster1:1 2023-11-08T05:15:29.586 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster_var_name=local_cluster 2023-11-08T05:15:29.586 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance_var_name=instance 2023-11-08T05:15:29.586 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:15:29.586 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=1 2023-11-08T05:15:29.587 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ '[' 1 = cluster1:1 ']' 2023-11-08T05:15:29.587 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval local_cluster=cluster1 2023-11-08T05:15:29.587 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ local_cluster=cluster1 2023-11-08T05:15:29.587 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval instance=1 2023-11-08T05:15:29.588 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ instance=1 2023-11-08T05:15:29.588 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-11-08T05:15:29.596 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-08T05:15:29.597 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-08T05:15:29.597 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-08T05:15:29.597 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_10 2023-11-08T05:15:29.702 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ return 0 2023-11-08T05:15:29.703 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ status_result=' "state": "Replaying"' 2023-11-08T05:15:29.703 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ echo ' "state": "Replaying"' 2023-11-08T05:15:29.704 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ grep -i Replaying 2023-11-08T05:15:29.704 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: "state": "Replaying" 2023-11-08T05:15:29.706 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ current_state=started 2023-11-08T05:15:29.706 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test started = started 2023-11-08T05:15:29.707 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:15:29.707 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_10 2023-11-08T05:15:29.707 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local local_cluster=cluster1 2023-11-08T05:15:29.707 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:15:29.707 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:15:29.708 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_10 2023-11-08T05:15:29.708 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' snapshot = journal ']' 2023-11-08T05:15:29.708 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' snapshot = snapshot ']' 2023-11-08T05:15:29.708 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_10 2023-11-08T05:15:29.708 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local local_cluster=cluster1 2023-11-08T05:15:29.708 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:15:29.709 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:15:29.709 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_10 2023-11-08T05:15:29.709 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ mkfname cluster2-mirror-image_10.status 2023-11-08T05:15:29.709 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ echo cluster2-mirror-image_10.status 2023-11-08T05:15:29.709 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -e 's|[/ ]|_|g' 2023-11-08T05:15:29.710 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local status_log=/tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_10.status 2023-11-08T05:15:29.710 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ mkfname cluster1-mirror-image_10.status 2023-11-08T05:15:29.710 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ echo cluster1-mirror-image_10.status 2023-11-08T05:15:29.710 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -e 's|[/ ]|_|g' 2023-11-08T05:15:29.711 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local local_status_log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_10.status 2023-11-08T05:15:29.711 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ mirror_image_snapshot cluster2 mirror image_10 2023-11-08T05:15:29.712 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:15:29.712 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:15:29.713 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_10 2023-11-08T05:15:29.713 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_10 2023-11-08T05:15:30.451 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:Snapshot ID: 137 2023-11-08T05:15:30.460 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_10 /tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_10.status 2023-11-08T05:15:30.460 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:15:30.461 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:15:30.461 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_10 2023-11-08T05:15:30.461 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_10.status 2023-11-08T05:15:30.461 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_10 --format xml 2023-11-08T05:15:30.461 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T05:15:30.500 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-11-08T05:15:30.502 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snapshot_id=137 2023-11-08T05:15:30.502 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ true 2023-11-08T05:15:30.503 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-11-08T05:15:30.503 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0.2 2023-11-08T05:15:30.703 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_10 /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_10.status 2023-11-08T05:15:30.704 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:15:30.704 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:15:30.704 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_10 2023-11-08T05:15:30.705 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_10.status 2023-11-08T05:15:30.705 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_10 --format xml 2023-11-08T05:15:30.705 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T05:15:30.744 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ true 2023-11-08T05:15:30.744 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-08T05:15:30.745 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:-:1.1: Document is empty 2023-11-08T05:15:30.746 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: 2023-11-08T05:15:30.746 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:^ 2023-11-08T05:15:30.746 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local primary_snapshot_id= 2023-11-08T05:15:30.746 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test 137 = '' 2023-11-08T05:15:30.746 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-11-08T05:15:30.747 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0.4 2023-11-08T05:15:31.147 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_10 /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_10.status 2023-11-08T05:15:31.148 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:15:31.148 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:15:31.148 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_10 2023-11-08T05:15:31.148 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_10.status 2023-11-08T05:15:31.148 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_10 --format xml 2023-11-08T05:15:31.149 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T05:15:31.190 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ true 2023-11-08T05:15:31.191 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-08T05:15:31.192 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:-:1.1: Document is empty 2023-11-08T05:15:31.192 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: 2023-11-08T05:15:31.192 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:^ 2023-11-08T05:15:31.193 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local primary_snapshot_id= 2023-11-08T05:15:31.193 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test 137 = '' 2023-11-08T05:15:31.194 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-11-08T05:15:31.194 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0.8 2023-11-08T05:15:31.994 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_10 /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_10.status 2023-11-08T05:15:31.995 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:15:31.995 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:15:31.995 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_10 2023-11-08T05:15:31.995 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_10.status 2023-11-08T05:15:31.996 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_10 --format xml 2023-11-08T05:15:31.996 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T05:15:32.049 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-08T05:15:32.051 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local primary_snapshot_id=137 2023-11-08T05:15:32.051 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test 137 = 137 2023-11-08T05:15:32.052 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:15:32.052 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_snap_present cluster1 mirror image_10 snap 2023-11-08T05:15:32.053 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:15:32.053 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:15:32.054 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_10 2023-11-08T05:15:32.054 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap_name=snap 2023-11-08T05:15:32.054 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local s 2023-11-08T05:15:32.055 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-11-08T05:15:32.055 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 1 2023-11-08T05:15:33.052 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 info mirror/image_10@snap 2023-11-08T05:15:33.093 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:rbd image 'image_10': 2023-11-08T05:15:33.093 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: size 128 MiB in 32 objects 2023-11-08T05:15:33.094 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: order 22 (4 MiB objects) 2023-11-08T05:15:33.094 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: snapshot_count: 2 2023-11-08T05:15:33.094 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: id: 11125681323c 2023-11-08T05:15:33.094 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: block_name_prefix: rbd_data.11125681323c 2023-11-08T05:15:33.094 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: format: 2 2023-11-08T05:15:33.095 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: features: layering, exclusive-lock, non-primary 2023-11-08T05:15:33.095 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: op_features: 2023-11-08T05:15:33.095 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: flags: 2023-11-08T05:15:33.095 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: create_timestamp: Wed Nov 8 05:06:05 2023 2023-11-08T05:15:33.096 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: access_timestamp: Wed Nov 8 05:06:05 2023 2023-11-08T05:15:33.096 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: modify_timestamp: Wed Nov 8 05:06:05 2023 2023-11-08T05:15:33.096 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: protected: False 2023-11-08T05:15:33.097 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring state: enabled 2023-11-08T05:15:33.097 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring mode: snapshot 2023-11-08T05:15:33.097 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring global id: ab5153f3-f7f8-4e7b-ad36-9e427596786a 2023-11-08T05:15:33.098 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring primary: false 2023-11-08T05:15:33.099 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:15:33.099 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ compare_image_snaps mirror image_10 snap 2023-11-08T05:15:33.099 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:15:33.100 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_10 2023-11-08T05:15:33.100 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap_name=snap 2023-11-08T05:15:33.100 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local ret=0 2023-11-08T05:15:33.100 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local rmt_export=/tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_10.export 2023-11-08T05:15:33.100 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local loc_export=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_10.export 2023-11-08T05:15:33.101 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rm -f /tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_10.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_10.export 2023-11-08T05:15:33.101 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 -p mirror export image_10@snap /tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_10.export 2023-11-08T05:15:33.273 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-11-08T05:15:33.276 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 -p mirror export image_10@snap /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_10.export 2023-11-08T05:15:33.525 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-11-08T05:15:33.528 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ cmp /tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_10.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_10.export 2023-11-08T05:15:33.584 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rm -f /tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_10.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_10.export 2023-11-08T05:15:33.623 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:15:33.623 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-11-08T05:15:33.623 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image=image_11 2023-11-08T05:15:33.624 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ create_snap cluster2 mirror image_11 snap 2023-11-08T05:15:33.624 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:15:33.624 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:15:33.624 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_11 2023-11-08T05:15:33.625 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap_name=snap 2023-11-08T05:15:33.625 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 -p mirror snap create image_11@snap --debug-rbd=20 --debug-journaler=20 2023-11-08T05:15:35.462 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_image_replay_started cluster1 mirror image_11 2023-11-08T05:15:35.463 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:15:35.463 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:15:35.463 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_11 2023-11-08T05:15:35.463 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_image_replay_state cluster1 mirror image_11 started 2023-11-08T05:15:35.463 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:15:35.463 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:15:35.464 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_11 2023-11-08T05:15:35.464 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local state=started 2023-11-08T05:15:35.464 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local s 2023-11-08T05:15:35.464 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-11-08T05:15:35.464 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 1 2023-11-08T05:15:36.464 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test_image_replay_state cluster1 mirror image_11 started 2023-11-08T05:15:36.464 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:15:36.464 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:15:36.464 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_11 2023-11-08T05:15:36.465 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local test_state=started 2023-11-08T05:15:36.465 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local status_result 2023-11-08T05:15:36.465 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local current_state=stopped 2023-11-08T05:15:36.465 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_11 2023-11-08T05:15:36.465 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_instance=cluster1 2023-11-08T05:15:36.466 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ grep -i state 2023-11-08T05:15:36.466 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T05:15:36.466 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1 2023-11-08T05:15:36.466 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance=cluster1 2023-11-08T05:15:36.466 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local loop_instance 2023-11-08T05:15:36.467 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-11-08T05:15:36.467 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sleep 0 2023-11-08T05:15:36.470 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-11-08T05:15:36.470 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ seq 0 3 2023-11-08T05:15:36.470 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T05:15:36.470 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_11 2023-11-08T05:15:36.471 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1:0 2023-11-08T05:15:36.471 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T05:15:36.471 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance 2023-11-08T05:15:36.471 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-11-08T05:15:36.472 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local val=cluster1:0 2023-11-08T05:15:36.472 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_var_name=cluster 2023-11-08T05:15:36.472 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance_var_name=instance 2023-11-08T05:15:36.472 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cluster=cluster1 2023-11-08T05:15:36.472 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ instance=0 2023-11-08T05:15:36.473 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' 0 = cluster1:0 ']' 2023-11-08T05:15:36.473 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval cluster=cluster1 2023-11-08T05:15:36.473 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:15:36.473 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval instance=0 2023-11-08T05:15:36.474 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=0 2023-11-08T05:15:36.474 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-11-08T05:15:36.474 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local local_cluster=cluster1:0 2023-11-08T05:15:36.474 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster=cluster1 2023-11-08T05:15:36.474 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance 2023-11-08T05:15:36.475 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-11-08T05:15:36.475 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local val=cluster1:0 2023-11-08T05:15:36.475 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster_var_name=local_cluster 2023-11-08T05:15:36.475 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance_var_name=instance 2023-11-08T05:15:36.475 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:15:36.475 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=0 2023-11-08T05:15:36.476 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ '[' 0 = cluster1:0 ']' 2023-11-08T05:15:36.476 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval local_cluster=cluster1 2023-11-08T05:15:36.476 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ local_cluster=cluster1 2023-11-08T05:15:36.476 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval instance=0 2023-11-08T05:15:36.476 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ instance=0 2023-11-08T05:15:36.477 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-11-08T05:15:36.485 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-08T05:15:36.485 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-08T05:15:36.485 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-08T05:15:36.485 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_11 2023-11-08T05:15:36.600 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ return 0 2023-11-08T05:15:36.601 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ status_result=' "state": "Replaying"' 2023-11-08T05:15:36.601 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ echo ' "state": "Replaying"' 2023-11-08T05:15:36.602 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ grep -i Replaying 2023-11-08T05:15:36.602 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: "state": "Replaying" 2023-11-08T05:15:36.604 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ current_state=started 2023-11-08T05:15:36.604 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test started = started 2023-11-08T05:15:36.605 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:15:36.605 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_11 2023-11-08T05:15:36.605 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local local_cluster=cluster1 2023-11-08T05:15:36.605 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:15:36.605 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:15:36.606 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_11 2023-11-08T05:15:36.606 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' snapshot = journal ']' 2023-11-08T05:15:36.606 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' snapshot = snapshot ']' 2023-11-08T05:15:36.606 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_11 2023-11-08T05:15:36.607 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local local_cluster=cluster1 2023-11-08T05:15:36.607 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:15:36.607 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:15:36.607 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_11 2023-11-08T05:15:36.608 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ mkfname cluster2-mirror-image_11.status 2023-11-08T05:15:36.608 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ echo cluster2-mirror-image_11.status 2023-11-08T05:15:36.608 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -e 's|[/ ]|_|g' 2023-11-08T05:15:36.608 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local status_log=/tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_11.status 2023-11-08T05:15:36.608 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ mkfname cluster1-mirror-image_11.status 2023-11-08T05:15:36.609 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ echo cluster1-mirror-image_11.status 2023-11-08T05:15:36.609 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -e 's|[/ ]|_|g' 2023-11-08T05:15:36.611 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local local_status_log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_11.status 2023-11-08T05:15:36.611 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ mirror_image_snapshot cluster2 mirror image_11 2023-11-08T05:15:36.611 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:15:36.611 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:15:36.611 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_11 2023-11-08T05:15:36.612 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_11 2023-11-08T05:15:37.477 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:Snapshot ID: 139 2023-11-08T05:15:37.486 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_11 /tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_11.status 2023-11-08T05:15:37.487 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:15:37.487 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:15:37.487 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_11 2023-11-08T05:15:37.487 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_11.status 2023-11-08T05:15:37.487 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_11 --format xml 2023-11-08T05:15:37.488 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T05:15:37.528 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-11-08T05:15:37.530 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snapshot_id=139 2023-11-08T05:15:37.530 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ true 2023-11-08T05:15:37.530 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-11-08T05:15:37.531 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0.2 2023-11-08T05:15:37.731 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_11 /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_11.status 2023-11-08T05:15:37.732 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:15:37.732 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:15:37.732 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_11 2023-11-08T05:15:37.732 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_11.status 2023-11-08T05:15:37.732 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_11 --format xml 2023-11-08T05:15:37.733 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T05:15:37.780 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-08T05:15:37.781 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local primary_snapshot_id=78 2023-11-08T05:15:37.782 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test 139 = 78 2023-11-08T05:15:37.782 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-11-08T05:15:37.782 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0.4 2023-11-08T05:15:38.183 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_11 /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_11.status 2023-11-08T05:15:38.183 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:15:38.183 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:15:38.183 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_11 2023-11-08T05:15:38.184 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_11.status 2023-11-08T05:15:38.184 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_11 --format xml 2023-11-08T05:15:38.184 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T05:15:38.228 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-08T05:15:38.230 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local primary_snapshot_id=78 2023-11-08T05:15:38.230 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test 139 = 78 2023-11-08T05:15:38.230 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-11-08T05:15:38.231 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0.8 2023-11-08T05:15:39.032 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_11 /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_11.status 2023-11-08T05:15:39.032 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:15:39.032 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:15:39.032 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_11 2023-11-08T05:15:39.033 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_11.status 2023-11-08T05:15:39.033 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_11 --format xml 2023-11-08T05:15:39.033 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T05:15:39.077 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ true 2023-11-08T05:15:39.077 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-08T05:15:39.078 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:-:1.1: Document is empty 2023-11-08T05:15:39.079 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: 2023-11-08T05:15:39.079 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:^ 2023-11-08T05:15:39.080 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local primary_snapshot_id= 2023-11-08T05:15:39.080 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test 139 = '' 2023-11-08T05:15:39.080 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-11-08T05:15:39.080 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 1.6 2023-11-08T05:15:40.681 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_11 /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_11.status 2023-11-08T05:15:40.681 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:15:40.681 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:15:40.682 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_11 2023-11-08T05:15:40.682 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_11.status 2023-11-08T05:15:40.682 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_11 --format xml 2023-11-08T05:15:40.682 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T05:15:40.725 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-08T05:15:40.728 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local primary_snapshot_id=139 2023-11-08T05:15:40.728 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test 139 = 139 2023-11-08T05:15:40.728 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:15:40.728 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_snap_present cluster1 mirror image_11 snap 2023-11-08T05:15:40.729 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:15:40.729 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:15:40.729 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_11 2023-11-08T05:15:40.729 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap_name=snap 2023-11-08T05:15:40.729 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local s 2023-11-08T05:15:40.730 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-11-08T05:15:40.730 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 1 2023-11-08T05:15:41.729 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 info mirror/image_11@snap 2023-11-08T05:15:41.771 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:rbd image 'image_11': 2023-11-08T05:15:41.772 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: size 128 MiB in 32 objects 2023-11-08T05:15:41.772 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: order 22 (4 MiB objects) 2023-11-08T05:15:41.772 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: snapshot_count: 2 2023-11-08T05:15:41.772 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: id: 11d875837cbe 2023-11-08T05:15:41.773 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: block_name_prefix: rbd_data.11d875837cbe 2023-11-08T05:15:41.773 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: format: 2 2023-11-08T05:15:41.773 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: features: layering, exclusive-lock, non-primary 2023-11-08T05:15:41.773 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: op_features: 2023-11-08T05:15:41.773 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: flags: 2023-11-08T05:15:41.774 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: create_timestamp: Wed Nov 8 05:06:11 2023 2023-11-08T05:15:41.774 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: access_timestamp: Wed Nov 8 05:06:11 2023 2023-11-08T05:15:41.774 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: modify_timestamp: Wed Nov 8 05:06:11 2023 2023-11-08T05:15:41.774 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: protected: False 2023-11-08T05:15:41.775 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring state: enabled 2023-11-08T05:15:41.775 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring mode: snapshot 2023-11-08T05:15:41.775 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring global id: b31ed973-6d38-4d25-adab-1da0bc236902 2023-11-08T05:15:41.775 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring primary: false 2023-11-08T05:15:41.776 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:15:41.777 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ compare_image_snaps mirror image_11 snap 2023-11-08T05:15:41.777 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:15:41.777 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_11 2023-11-08T05:15:41.777 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap_name=snap 2023-11-08T05:15:41.777 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local ret=0 2023-11-08T05:15:41.778 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local rmt_export=/tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_11.export 2023-11-08T05:15:41.778 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local loc_export=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_11.export 2023-11-08T05:15:41.778 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rm -f /tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_11.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_11.export 2023-11-08T05:15:41.778 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 -p mirror export image_11@snap /tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_11.export 2023-11-08T05:15:41.957 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-11-08T05:15:41.961 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 -p mirror export image_11@snap /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_11.export 2023-11-08T05:15:42.186 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-11-08T05:15:42.189 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ cmp /tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_11.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_11.export 2023-11-08T05:15:42.246 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rm -f /tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_11.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_11.export 2023-11-08T05:15:42.285 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:15:42.285 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-11-08T05:15:42.285 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image=image_12 2023-11-08T05:15:42.285 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ create_snap cluster2 mirror image_12 snap 2023-11-08T05:15:42.286 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:15:42.286 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:15:42.286 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_12 2023-11-08T05:15:42.286 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap_name=snap 2023-11-08T05:15:42.286 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 -p mirror snap create image_12@snap --debug-rbd=20 --debug-journaler=20 2023-11-08T05:15:43.491 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_image_replay_started cluster1 mirror image_12 2023-11-08T05:15:43.491 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:15:43.491 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:15:43.491 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_12 2023-11-08T05:15:43.492 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_image_replay_state cluster1 mirror image_12 started 2023-11-08T05:15:43.492 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:15:43.492 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:15:43.492 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_12 2023-11-08T05:15:43.492 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local state=started 2023-11-08T05:15:43.493 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local s 2023-11-08T05:15:43.493 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-11-08T05:15:43.493 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 1 2023-11-08T05:15:44.492 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test_image_replay_state cluster1 mirror image_12 started 2023-11-08T05:15:44.492 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:15:44.492 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:15:44.493 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_12 2023-11-08T05:15:44.493 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local test_state=started 2023-11-08T05:15:44.493 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local status_result 2023-11-08T05:15:44.493 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local current_state=stopped 2023-11-08T05:15:44.496 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_12 2023-11-08T05:15:44.496 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ grep -i state 2023-11-08T05:15:44.496 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_instance=cluster1 2023-11-08T05:15:44.496 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T05:15:44.497 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1 2023-11-08T05:15:44.497 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance=cluster1 2023-11-08T05:15:44.497 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local loop_instance 2023-11-08T05:15:44.497 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-11-08T05:15:44.497 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sleep 0 2023-11-08T05:15:44.498 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-11-08T05:15:44.498 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ seq 0 3 2023-11-08T05:15:44.498 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T05:15:44.498 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_12 2023-11-08T05:15:44.498 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1:0 2023-11-08T05:15:44.499 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T05:15:44.499 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance 2023-11-08T05:15:44.499 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-11-08T05:15:44.499 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local val=cluster1:0 2023-11-08T05:15:44.499 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_var_name=cluster 2023-11-08T05:15:44.500 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance_var_name=instance 2023-11-08T05:15:44.500 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cluster=cluster1 2023-11-08T05:15:44.500 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ instance=0 2023-11-08T05:15:44.500 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' 0 = cluster1:0 ']' 2023-11-08T05:15:44.500 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval cluster=cluster1 2023-11-08T05:15:44.501 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:15:44.501 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval instance=0 2023-11-08T05:15:44.501 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=0 2023-11-08T05:15:44.503 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-11-08T05:15:44.503 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local local_cluster=cluster1:0 2023-11-08T05:15:44.503 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster=cluster1 2023-11-08T05:15:44.503 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance 2023-11-08T05:15:44.504 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-11-08T05:15:44.504 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local val=cluster1:0 2023-11-08T05:15:44.504 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster_var_name=local_cluster 2023-11-08T05:15:44.504 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance_var_name=instance 2023-11-08T05:15:44.504 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:15:44.505 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=0 2023-11-08T05:15:44.505 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ '[' 0 = cluster1:0 ']' 2023-11-08T05:15:44.505 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval local_cluster=cluster1 2023-11-08T05:15:44.505 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ local_cluster=cluster1 2023-11-08T05:15:44.505 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval instance=0 2023-11-08T05:15:44.506 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ instance=0 2023-11-08T05:15:44.506 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-11-08T05:15:44.515 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-08T05:15:44.516 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-08T05:15:44.516 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-08T05:15:44.516 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_12 2023-11-08T05:15:44.630 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ return 0 2023-11-08T05:15:44.631 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ status_result=' "state": "Replaying"' 2023-11-08T05:15:44.631 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ echo ' "state": "Replaying"' 2023-11-08T05:15:44.631 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ grep -i Replaying 2023-11-08T05:15:44.632 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: "state": "Replaying" 2023-11-08T05:15:44.634 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ current_state=started 2023-11-08T05:15:44.634 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test started = started 2023-11-08T05:15:44.635 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:15:44.635 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_12 2023-11-08T05:15:44.635 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local local_cluster=cluster1 2023-11-08T05:15:44.635 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:15:44.635 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:15:44.636 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_12 2023-11-08T05:15:44.636 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' snapshot = journal ']' 2023-11-08T05:15:44.636 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' snapshot = snapshot ']' 2023-11-08T05:15:44.636 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_12 2023-11-08T05:15:44.636 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local local_cluster=cluster1 2023-11-08T05:15:44.637 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:15:44.637 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:15:44.637 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_12 2023-11-08T05:15:44.637 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ mkfname cluster2-mirror-image_12.status 2023-11-08T05:15:44.637 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ echo cluster2-mirror-image_12.status 2023-11-08T05:15:44.638 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -e 's|[/ ]|_|g' 2023-11-08T05:15:44.638 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local status_log=/tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_12.status 2023-11-08T05:15:44.638 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ mkfname cluster1-mirror-image_12.status 2023-11-08T05:15:44.638 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ echo cluster1-mirror-image_12.status 2023-11-08T05:15:44.638 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -e 's|[/ ]|_|g' 2023-11-08T05:15:44.639 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local local_status_log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_12.status 2023-11-08T05:15:44.639 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ mirror_image_snapshot cluster2 mirror image_12 2023-11-08T05:15:44.640 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:15:44.640 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:15:44.640 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_12 2023-11-08T05:15:44.640 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_12 2023-11-08T05:15:45.501 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:Snapshot ID: 141 2023-11-08T05:15:45.509 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_12 /tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_12.status 2023-11-08T05:15:45.509 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:15:45.510 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:15:45.510 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_12 2023-11-08T05:15:45.510 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_12.status 2023-11-08T05:15:45.510 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_12 --format xml 2023-11-08T05:15:45.510 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T05:15:45.548 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-11-08T05:15:45.550 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snapshot_id=141 2023-11-08T05:15:45.550 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ true 2023-11-08T05:15:45.551 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-11-08T05:15:45.551 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0.2 2023-11-08T05:15:45.752 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_12 /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_12.status 2023-11-08T05:15:45.753 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:15:45.753 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:15:45.753 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_12 2023-11-08T05:15:45.754 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_12.status 2023-11-08T05:15:45.754 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_12 --format xml 2023-11-08T05:15:45.754 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T05:15:45.792 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-08T05:15:45.794 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local primary_snapshot_id=79 2023-11-08T05:15:45.794 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test 141 = 79 2023-11-08T05:15:45.795 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-11-08T05:15:45.795 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0.4 2023-11-08T05:15:46.196 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_12 /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_12.status 2023-11-08T05:15:46.196 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:15:46.196 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:15:46.197 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_12 2023-11-08T05:15:46.197 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_12.status 2023-11-08T05:15:46.197 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_12 --format xml 2023-11-08T05:15:46.197 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T05:15:46.235 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-08T05:15:46.237 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local primary_snapshot_id=79 2023-11-08T05:15:46.237 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test 141 = 79 2023-11-08T05:15:46.238 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-11-08T05:15:46.238 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0.8 2023-11-08T05:15:47.039 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_12 /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_12.status 2023-11-08T05:15:47.040 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:15:47.040 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:15:47.040 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_12 2023-11-08T05:15:47.040 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_12.status 2023-11-08T05:15:47.041 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_12 --format xml 2023-11-08T05:15:47.041 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T05:15:47.084 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ true 2023-11-08T05:15:47.084 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-08T05:15:47.086 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:-:1.1: Document is empty 2023-11-08T05:15:47.086 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: 2023-11-08T05:15:47.086 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:^ 2023-11-08T05:15:47.087 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local primary_snapshot_id= 2023-11-08T05:15:47.087 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test 141 = '' 2023-11-08T05:15:47.087 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-11-08T05:15:47.088 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 1.6 2023-11-08T05:15:48.688 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_12 /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_12.status 2023-11-08T05:15:48.688 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:15:48.688 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:15:48.689 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_12 2023-11-08T05:15:48.689 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_12.status 2023-11-08T05:15:48.689 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_12 --format xml 2023-11-08T05:15:48.689 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T05:15:48.732 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-08T05:15:48.734 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local primary_snapshot_id=141 2023-11-08T05:15:48.735 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test 141 = 141 2023-11-08T05:15:48.735 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:15:48.735 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_snap_present cluster1 mirror image_12 snap 2023-11-08T05:15:48.735 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:15:48.735 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:15:48.736 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_12 2023-11-08T05:15:48.736 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap_name=snap 2023-11-08T05:15:48.736 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local s 2023-11-08T05:15:48.736 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-11-08T05:15:48.736 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 1 2023-11-08T05:15:49.735 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 info mirror/image_12@snap 2023-11-08T05:15:49.974 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:rbd image 'image_12': 2023-11-08T05:15:49.975 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: size 128 MiB in 32 objects 2023-11-08T05:15:49.975 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: order 22 (4 MiB objects) 2023-11-08T05:15:49.975 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: snapshot_count: 2 2023-11-08T05:15:49.975 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: id: 12872ad26b9c 2023-11-08T05:15:49.975 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: block_name_prefix: rbd_data.12872ad26b9c 2023-11-08T05:15:49.976 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: format: 2 2023-11-08T05:15:49.976 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: features: layering, exclusive-lock, non-primary 2023-11-08T05:15:49.976 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: op_features: 2023-11-08T05:15:49.976 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: flags: 2023-11-08T05:15:49.976 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: create_timestamp: Wed Nov 8 05:06:21 2023 2023-11-08T05:15:49.977 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: access_timestamp: Wed Nov 8 05:06:21 2023 2023-11-08T05:15:49.977 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: modify_timestamp: Wed Nov 8 05:06:21 2023 2023-11-08T05:15:49.977 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: protected: False 2023-11-08T05:15:49.977 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring state: enabled 2023-11-08T05:15:49.977 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring mode: snapshot 2023-11-08T05:15:49.978 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring global id: c28fe69d-285b-4d38-a8a1-5a5367bf07ce 2023-11-08T05:15:49.978 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring primary: false 2023-11-08T05:15:49.979 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:15:49.979 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ compare_image_snaps mirror image_12 snap 2023-11-08T05:15:49.979 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:15:49.980 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_12 2023-11-08T05:15:49.980 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap_name=snap 2023-11-08T05:15:49.980 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local ret=0 2023-11-08T05:15:49.980 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local rmt_export=/tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_12.export 2023-11-08T05:15:49.980 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local loc_export=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_12.export 2023-11-08T05:15:49.981 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rm -f /tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_12.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_12.export 2023-11-08T05:15:49.981 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 -p mirror export image_12@snap /tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_12.export 2023-11-08T05:15:50.162 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-11-08T05:15:50.165 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 -p mirror export image_12@snap /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_12.export 2023-11-08T05:15:50.394 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-11-08T05:15:50.397 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ cmp /tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_12.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_12.export 2023-11-08T05:15:50.454 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rm -f /tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_12.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_12.export 2023-11-08T05:15:50.492 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:15:50.492 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-11-08T05:15:50.493 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image=image_13 2023-11-08T05:15:50.493 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ create_snap cluster2 mirror image_13 snap 2023-11-08T05:15:50.493 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:15:50.493 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:15:50.494 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_13 2023-11-08T05:15:50.494 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap_name=snap 2023-11-08T05:15:50.494 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 -p mirror snap create image_13@snap --debug-rbd=20 --debug-journaler=20 2023-11-08T05:15:52.531 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_image_replay_started cluster1 mirror image_13 2023-11-08T05:15:52.531 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:15:52.531 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:15:52.531 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_13 2023-11-08T05:15:52.532 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_image_replay_state cluster1 mirror image_13 started 2023-11-08T05:15:52.532 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:15:52.532 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:15:52.532 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_13 2023-11-08T05:15:52.533 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local state=started 2023-11-08T05:15:52.533 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local s 2023-11-08T05:15:52.533 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-11-08T05:15:52.533 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 1 2023-11-08T05:15:53.532 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test_image_replay_state cluster1 mirror image_13 started 2023-11-08T05:15:53.532 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:15:53.532 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:15:53.533 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_13 2023-11-08T05:15:53.533 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local test_state=started 2023-11-08T05:15:53.533 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local status_result 2023-11-08T05:15:53.533 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local current_state=stopped 2023-11-08T05:15:53.533 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_13 2023-11-08T05:15:53.533 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_instance=cluster1 2023-11-08T05:15:53.534 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T05:15:53.534 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1 2023-11-08T05:15:53.534 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ grep -i state 2023-11-08T05:15:53.534 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance=cluster1 2023-11-08T05:15:53.535 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local loop_instance 2023-11-08T05:15:53.535 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-11-08T05:15:53.535 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sleep 0 2023-11-08T05:15:53.537 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-11-08T05:15:53.537 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ seq 0 3 2023-11-08T05:15:53.538 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T05:15:53.538 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_13 2023-11-08T05:15:53.538 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1:0 2023-11-08T05:15:53.538 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T05:15:53.538 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance 2023-11-08T05:15:53.539 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-11-08T05:15:53.539 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local val=cluster1:0 2023-11-08T05:15:53.539 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_var_name=cluster 2023-11-08T05:15:53.539 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance_var_name=instance 2023-11-08T05:15:53.539 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cluster=cluster1 2023-11-08T05:15:53.540 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ instance=0 2023-11-08T05:15:53.540 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' 0 = cluster1:0 ']' 2023-11-08T05:15:53.540 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval cluster=cluster1 2023-11-08T05:15:53.540 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:15:53.540 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval instance=0 2023-11-08T05:15:53.541 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=0 2023-11-08T05:15:53.541 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-11-08T05:15:53.541 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local local_cluster=cluster1:0 2023-11-08T05:15:53.541 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster=cluster1 2023-11-08T05:15:53.541 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance 2023-11-08T05:15:53.541 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-11-08T05:15:53.542 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local val=cluster1:0 2023-11-08T05:15:53.543 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster_var_name=local_cluster 2023-11-08T05:15:53.543 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance_var_name=instance 2023-11-08T05:15:53.543 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:15:53.544 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=0 2023-11-08T05:15:53.544 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ '[' 0 = cluster1:0 ']' 2023-11-08T05:15:53.544 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval local_cluster=cluster1 2023-11-08T05:15:53.544 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ local_cluster=cluster1 2023-11-08T05:15:53.544 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval instance=0 2023-11-08T05:15:53.544 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ instance=0 2023-11-08T05:15:53.545 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-11-08T05:15:53.552 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-08T05:15:53.553 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-08T05:15:53.553 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-08T05:15:53.553 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_13 2023-11-08T05:15:53.659 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:Invalid command: unused arguments: ['mirror/image_13'] 2023-11-08T05:15:53.659 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd mirror status : get status for rbd mirror 2023-11-08T05:15:53.660 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:admin_socket: invalid command 2023-11-08T05:15:53.662 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T05:15:53.663 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_13 2023-11-08T05:15:53.663 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1:1 2023-11-08T05:15:53.663 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T05:15:53.663 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance 2023-11-08T05:15:53.664 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-11-08T05:15:53.664 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local val=cluster1:1 2023-11-08T05:15:53.664 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_var_name=cluster 2023-11-08T05:15:53.664 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance_var_name=instance 2023-11-08T05:15:53.665 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cluster=cluster1 2023-11-08T05:15:53.665 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ instance=1 2023-11-08T05:15:53.665 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' 1 = cluster1:1 ']' 2023-11-08T05:15:53.665 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval cluster=cluster1 2023-11-08T05:15:53.665 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:15:53.666 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval instance=1 2023-11-08T05:15:53.666 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=1 2023-11-08T05:15:53.666 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-11-08T05:15:53.666 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local local_cluster=cluster1:1 2023-11-08T05:15:53.667 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster=cluster1 2023-11-08T05:15:53.667 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance 2023-11-08T05:15:53.667 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-11-08T05:15:53.667 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local val=cluster1:1 2023-11-08T05:15:53.667 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster_var_name=local_cluster 2023-11-08T05:15:53.668 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance_var_name=instance 2023-11-08T05:15:53.668 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:15:53.668 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=1 2023-11-08T05:15:53.668 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ '[' 1 = cluster1:1 ']' 2023-11-08T05:15:53.668 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval local_cluster=cluster1 2023-11-08T05:15:53.669 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ local_cluster=cluster1 2023-11-08T05:15:53.669 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval instance=1 2023-11-08T05:15:53.669 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ instance=1 2023-11-08T05:15:53.669 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-11-08T05:15:53.676 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-08T05:15:53.677 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-08T05:15:53.677 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-08T05:15:53.677 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_13 2023-11-08T05:15:53.777 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:Invalid command: unused arguments: ['mirror/image_13'] 2023-11-08T05:15:53.777 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd mirror status : get status for rbd mirror 2023-11-08T05:15:53.777 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:admin_socket: invalid command 2023-11-08T05:15:53.780 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T05:15:53.781 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_13 2023-11-08T05:15:53.781 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1:2 2023-11-08T05:15:53.781 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T05:15:53.781 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance 2023-11-08T05:15:53.781 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-11-08T05:15:53.782 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local val=cluster1:2 2023-11-08T05:15:53.782 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_var_name=cluster 2023-11-08T05:15:53.782 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance_var_name=instance 2023-11-08T05:15:53.782 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cluster=cluster1 2023-11-08T05:15:53.783 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ instance=2 2023-11-08T05:15:53.783 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' 2 = cluster1:2 ']' 2023-11-08T05:15:53.783 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval cluster=cluster1 2023-11-08T05:15:53.783 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:15:53.783 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval instance=2 2023-11-08T05:15:53.784 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=2 2023-11-08T05:15:53.784 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-11-08T05:15:53.784 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local local_cluster=cluster1:2 2023-11-08T05:15:53.784 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster=cluster1 2023-11-08T05:15:53.784 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance 2023-11-08T05:15:53.785 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-11-08T05:15:53.785 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local val=cluster1:2 2023-11-08T05:15:53.785 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster_var_name=local_cluster 2023-11-08T05:15:53.785 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance_var_name=instance 2023-11-08T05:15:53.786 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:15:53.786 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=2 2023-11-08T05:15:53.786 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ '[' 2 = cluster1:2 ']' 2023-11-08T05:15:53.786 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval local_cluster=cluster1 2023-11-08T05:15:53.786 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ local_cluster=cluster1 2023-11-08T05:15:53.787 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval instance=2 2023-11-08T05:15:53.787 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ instance=2 2023-11-08T05:15:53.787 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-11-08T05:15:53.794 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-08T05:15:53.794 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-08T05:15:53.794 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-08T05:15:53.794 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_13 2023-11-08T05:15:53.894 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:Invalid command: unused arguments: ['mirror/image_13'] 2023-11-08T05:15:53.894 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd mirror status : get status for rbd mirror 2023-11-08T05:15:53.894 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:admin_socket: invalid command 2023-11-08T05:15:53.897 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T05:15:53.897 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_13 2023-11-08T05:15:53.897 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1:3 2023-11-08T05:15:53.898 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T05:15:53.898 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance 2023-11-08T05:15:53.898 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ set_cluster_instance cluster1:3 cluster instance 2023-11-08T05:15:53.898 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local val=cluster1:3 2023-11-08T05:15:53.898 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_var_name=cluster 2023-11-08T05:15:53.899 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance_var_name=instance 2023-11-08T05:15:53.899 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cluster=cluster1 2023-11-08T05:15:53.899 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ instance=3 2023-11-08T05:15:53.899 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' 3 = cluster1:3 ']' 2023-11-08T05:15:53.899 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval cluster=cluster1 2023-11-08T05:15:53.900 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:15:53.900 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval instance=3 2023-11-08T05:15:53.900 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=3 2023-11-08T05:15:53.900 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ daemon_asok_file cluster1:3 cluster1 2023-11-08T05:15:53.900 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local local_cluster=cluster1:3 2023-11-08T05:15:53.901 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster=cluster1 2023-11-08T05:15:53.901 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance 2023-11-08T05:15:53.901 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2023-11-08T05:15:53.901 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local val=cluster1:3 2023-11-08T05:15:53.901 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster_var_name=local_cluster 2023-11-08T05:15:53.902 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance_var_name=instance 2023-11-08T05:15:53.902 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:15:53.902 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=3 2023-11-08T05:15:53.902 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ '[' 3 = cluster1:3 ']' 2023-11-08T05:15:53.902 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval local_cluster=cluster1 2023-11-08T05:15:53.903 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ local_cluster=cluster1 2023-11-08T05:15:53.903 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval instance=3 2023-11-08T05:15:53.903 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ instance=3 2023-11-08T05:15:53.903 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2023-11-08T05:15:53.910 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-11-08T05:15:53.911 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-11-08T05:15:53.911 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-11-08T05:15:53.911 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_13 2023-11-08T05:15:54.017 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ return 0 2023-11-08T05:15:54.018 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ status_result=' "state": "Replaying"' 2023-11-08T05:15:54.018 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ echo ' "state": "Replaying"' 2023-11-08T05:15:54.018 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ grep -i Replaying 2023-11-08T05:15:54.019 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: "state": "Replaying" 2023-11-08T05:15:54.021 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ current_state=started 2023-11-08T05:15:54.021 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test started = started 2023-11-08T05:15:54.022 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:15:54.022 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_13 2023-11-08T05:15:54.022 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local local_cluster=cluster1 2023-11-08T05:15:54.022 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:15:54.022 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:15:54.023 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_13 2023-11-08T05:15:54.023 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' snapshot = journal ']' 2023-11-08T05:15:54.023 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' snapshot = snapshot ']' 2023-11-08T05:15:54.023 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_13 2023-11-08T05:15:54.024 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local local_cluster=cluster1 2023-11-08T05:15:54.024 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:15:54.024 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:15:54.024 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_13 2023-11-08T05:15:54.024 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ mkfname cluster2-mirror-image_13.status 2023-11-08T05:15:54.025 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ echo cluster2-mirror-image_13.status 2023-11-08T05:15:54.025 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -e 's|[/ ]|_|g' 2023-11-08T05:15:54.025 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local status_log=/tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_13.status 2023-11-08T05:15:54.025 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ mkfname cluster1-mirror-image_13.status 2023-11-08T05:15:54.025 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ echo cluster1-mirror-image_13.status 2023-11-08T05:15:54.026 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -e 's|[/ ]|_|g' 2023-11-08T05:15:54.027 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local local_status_log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_13.status 2023-11-08T05:15:54.027 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ mirror_image_snapshot cluster2 mirror image_13 2023-11-08T05:15:54.027 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:15:54.027 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:15:54.027 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_13 2023-11-08T05:15:54.028 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_13 2023-11-08T05:15:54.520 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:Snapshot ID: 143 2023-11-08T05:15:54.528 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_13 /tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_13.status 2023-11-08T05:15:54.529 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:15:54.529 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:15:54.529 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_13 2023-11-08T05:15:54.529 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_13.status 2023-11-08T05:15:54.530 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_13 --format xml 2023-11-08T05:15:54.530 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T05:15:54.565 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-11-08T05:15:54.566 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snapshot_id=143 2023-11-08T05:15:54.567 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ true 2023-11-08T05:15:54.567 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-11-08T05:15:54.567 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0.2 2023-11-08T05:15:54.768 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_13 /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_13.status 2023-11-08T05:15:54.768 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:15:54.768 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:15:54.769 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_13 2023-11-08T05:15:54.769 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_13.status 2023-11-08T05:15:54.769 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_13 --format xml 2023-11-08T05:15:54.769 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T05:15:54.808 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-08T05:15:54.809 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local primary_snapshot_id=80 2023-11-08T05:15:54.809 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test 143 = 80 2023-11-08T05:15:54.810 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-11-08T05:15:54.810 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0.4 2023-11-08T05:15:55.210 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_13 /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_13.status 2023-11-08T05:15:55.211 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:15:55.211 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:15:55.211 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_13 2023-11-08T05:15:55.211 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_13.status 2023-11-08T05:15:55.212 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_13 --format xml 2023-11-08T05:15:55.212 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T05:15:55.250 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-08T05:15:55.252 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local primary_snapshot_id=80 2023-11-08T05:15:55.252 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test 143 = 80 2023-11-08T05:15:55.252 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-11-08T05:15:55.252 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0.8 2023-11-08T05:15:56.053 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_13 /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_13.status 2023-11-08T05:15:56.054 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:15:56.054 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:15:56.054 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_13 2023-11-08T05:15:56.054 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_13.status 2023-11-08T05:15:56.055 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_13 --format xml 2023-11-08T05:15:56.055 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T05:15:56.096 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ true 2023-11-08T05:15:56.097 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-08T05:15:56.098 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:-:1.1: Document is empty 2023-11-08T05:15:56.098 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: 2023-11-08T05:15:56.099 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:^ 2023-11-08T05:15:56.099 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local primary_snapshot_id= 2023-11-08T05:15:56.100 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test 143 = '' 2023-11-08T05:15:56.100 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-11-08T05:15:56.100 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 1.6 2023-11-08T05:15:57.700 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_13 /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_13.status 2023-11-08T05:15:57.701 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:15:57.701 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:15:57.701 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_13 2023-11-08T05:15:57.701 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_13.status 2023-11-08T05:15:57.701 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_13 --format xml 2023-11-08T05:15:57.702 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T05:15:57.741 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-08T05:15:57.743 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local primary_snapshot_id=143 2023-11-08T05:15:57.744 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test 143 = 143 2023-11-08T05:15:57.744 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:15:57.744 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_snap_present cluster1 mirror image_13 snap 2023-11-08T05:15:57.744 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:15:57.744 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:15:57.745 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_13 2023-11-08T05:15:57.745 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap_name=snap 2023-11-08T05:15:57.745 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local s 2023-11-08T05:15:57.745 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-11-08T05:15:57.745 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 1 2023-11-08T05:15:58.744 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 info mirror/image_13@snap 2023-11-08T05:15:58.784 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:rbd image 'image_13': 2023-11-08T05:15:58.784 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: size 128 MiB in 32 objects 2023-11-08T05:15:58.784 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: order 22 (4 MiB objects) 2023-11-08T05:15:58.784 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: snapshot_count: 2 2023-11-08T05:15:58.785 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: id: 10e7a0ca72d5 2023-11-08T05:15:58.785 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: block_name_prefix: rbd_data.10e7a0ca72d5 2023-11-08T05:15:58.785 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: format: 2 2023-11-08T05:15:58.785 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: features: layering, exclusive-lock, non-primary 2023-11-08T05:15:58.786 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: op_features: 2023-11-08T05:15:58.786 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: flags: 2023-11-08T05:15:58.786 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: create_timestamp: Wed Nov 8 05:06:33 2023 2023-11-08T05:15:58.786 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: access_timestamp: Wed Nov 8 05:06:33 2023 2023-11-08T05:15:58.786 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: modify_timestamp: Wed Nov 8 05:06:33 2023 2023-11-08T05:15:58.787 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: protected: False 2023-11-08T05:15:58.787 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring state: enabled 2023-11-08T05:15:58.787 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring mode: snapshot 2023-11-08T05:15:58.787 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring global id: cb42c52c-8a81-4992-b8fb-d273572391a5 2023-11-08T05:15:58.787 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring primary: false 2023-11-08T05:15:58.789 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:15:58.789 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ compare_image_snaps mirror image_13 snap 2023-11-08T05:15:58.789 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:15:58.789 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_13 2023-11-08T05:15:58.789 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap_name=snap 2023-11-08T05:15:58.790 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local ret=0 2023-11-08T05:15:58.790 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local rmt_export=/tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_13.export 2023-11-08T05:15:58.790 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local loc_export=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_13.export 2023-11-08T05:15:58.790 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rm -f /tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_13.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_13.export 2023-11-08T05:15:58.790 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 -p mirror export image_13@snap /tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_13.export 2023-11-08T05:15:58.980 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-11-08T05:15:58.983 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 -p mirror export image_13@snap /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_13.export 2023-11-08T05:15:59.247 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-11-08T05:15:59.250 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ cmp /tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_13.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_13.export 2023-11-08T05:15:59.305 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rm -f /tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_13.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_13.export 2023-11-08T05:15:59.344 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:15:59.344 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-11-08T05:15:59.344 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image=image_14 2023-11-08T05:15:59.345 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ create_snap cluster2 mirror image_14 snap 2023-11-08T05:15:59.345 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:15:59.345 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:15:59.346 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_14 2023-11-08T05:15:59.346 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap_name=snap 2023-11-08T05:15:59.346 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 -p mirror snap create image_14@snap --debug-rbd=20 --debug-journaler=20 2023-11-08T05:16:00.537 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_image_replay_started cluster1 mirror image_14 2023-11-08T05:16:00.537 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:16:00.538 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:16:00.538 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_14 2023-11-08T05:16:00.538 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_image_replay_state cluster1 mirror image_14 started 2023-11-08T05:16:00.538 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:16:00.539 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:16:00.539 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_14 2023-11-08T05:16:00.539 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local state=started 2023-11-08T05:16:00.539 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local s 2023-11-08T05:16:00.539 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-11-08T05:16:00.539 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 1 2023-11-08T05:16:01.540 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test_image_replay_state cluster1 mirror image_14 started 2023-11-08T05:16:01.540 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:16:01.540 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:16:01.541 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_14 2023-11-08T05:16:01.541 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local test_state=started 2023-11-08T05:16:01.541 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local status_result 2023-11-08T05:16:01.541 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local current_state=stopped 2023-11-08T05:16:01.541 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_14 2023-11-08T05:16:01.542 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_instance=cluster1 2023-11-08T05:16:01.542 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T05:16:01.542 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1 2023-11-08T05:16:01.542 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ grep -i state 2023-11-08T05:16:01.542 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance=cluster1 2023-11-08T05:16:01.543 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local loop_instance 2023-11-08T05:16:01.543 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-11-08T05:16:01.544 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sleep 0 2023-11-08T05:16:01.547 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-11-08T05:16:01.547 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ seq 0 3 2023-11-08T05:16:01.547 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T05:16:01.547 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_14 2023-11-08T05:16:01.547 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1:0 2023-11-08T05:16:01.548 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T05:16:01.548 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance 2023-11-08T05:16:01.548 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-11-08T05:16:01.548 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local val=cluster1:0 2023-11-08T05:16:01.548 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_var_name=cluster 2023-11-08T05:16:01.549 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance_var_name=instance 2023-11-08T05:16:01.549 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cluster=cluster1 2023-11-08T05:16:01.549 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ instance=0 2023-11-08T05:16:01.549 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' 0 = cluster1:0 ']' 2023-11-08T05:16:01.549 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval cluster=cluster1 2023-11-08T05:16:01.550 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:16:01.550 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval instance=0 2023-11-08T05:16:01.550 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=0 2023-11-08T05:16:01.550 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-11-08T05:16:01.550 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local local_cluster=cluster1:0 2023-11-08T05:16:01.551 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster=cluster1 2023-11-08T05:16:01.551 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance 2023-11-08T05:16:01.551 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-11-08T05:16:01.551 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local val=cluster1:0 2023-11-08T05:16:01.551 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster_var_name=local_cluster 2023-11-08T05:16:01.555 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance_var_name=instance 2023-11-08T05:16:01.555 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:16:01.556 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=0 2023-11-08T05:16:01.556 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ '[' 0 = cluster1:0 ']' 2023-11-08T05:16:01.556 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval local_cluster=cluster1 2023-11-08T05:16:01.556 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ local_cluster=cluster1 2023-11-08T05:16:01.556 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval instance=0 2023-11-08T05:16:01.557 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ instance=0 2023-11-08T05:16:01.557 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-11-08T05:16:01.560 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-08T05:16:01.560 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-08T05:16:01.560 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-08T05:16:01.560 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_14 2023-11-08T05:16:01.667 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:Invalid command: unused arguments: ['mirror/image_14'] 2023-11-08T05:16:01.667 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd mirror status : get status for rbd mirror 2023-11-08T05:16:01.667 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:admin_socket: invalid command 2023-11-08T05:16:01.670 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T05:16:01.670 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_14 2023-11-08T05:16:01.670 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1:1 2023-11-08T05:16:01.671 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T05:16:01.671 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance 2023-11-08T05:16:01.671 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-11-08T05:16:01.671 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local val=cluster1:1 2023-11-08T05:16:01.671 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_var_name=cluster 2023-11-08T05:16:01.672 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance_var_name=instance 2023-11-08T05:16:01.672 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cluster=cluster1 2023-11-08T05:16:01.672 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ instance=1 2023-11-08T05:16:01.672 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' 1 = cluster1:1 ']' 2023-11-08T05:16:01.672 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval cluster=cluster1 2023-11-08T05:16:01.673 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:16:01.673 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval instance=1 2023-11-08T05:16:01.673 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=1 2023-11-08T05:16:01.673 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-11-08T05:16:01.674 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local local_cluster=cluster1:1 2023-11-08T05:16:01.674 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster=cluster1 2023-11-08T05:16:01.674 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance 2023-11-08T05:16:01.674 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-11-08T05:16:01.674 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local val=cluster1:1 2023-11-08T05:16:01.675 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster_var_name=local_cluster 2023-11-08T05:16:01.675 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance_var_name=instance 2023-11-08T05:16:01.675 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:16:01.675 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=1 2023-11-08T05:16:01.675 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ '[' 1 = cluster1:1 ']' 2023-11-08T05:16:01.676 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval local_cluster=cluster1 2023-11-08T05:16:01.676 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ local_cluster=cluster1 2023-11-08T05:16:01.676 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval instance=1 2023-11-08T05:16:01.676 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ instance=1 2023-11-08T05:16:01.676 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-11-08T05:16:01.684 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-08T05:16:01.684 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-08T05:16:01.685 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-08T05:16:01.685 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_14 2023-11-08T05:16:01.784 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:Invalid command: unused arguments: ['mirror/image_14'] 2023-11-08T05:16:01.785 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd mirror status : get status for rbd mirror 2023-11-08T05:16:01.785 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:admin_socket: invalid command 2023-11-08T05:16:01.787 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T05:16:01.788 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_14 2023-11-08T05:16:01.788 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1:2 2023-11-08T05:16:01.788 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T05:16:01.788 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance 2023-11-08T05:16:01.788 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-11-08T05:16:01.789 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local val=cluster1:2 2023-11-08T05:16:01.789 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_var_name=cluster 2023-11-08T05:16:01.789 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance_var_name=instance 2023-11-08T05:16:01.789 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cluster=cluster1 2023-11-08T05:16:01.789 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ instance=2 2023-11-08T05:16:01.790 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' 2 = cluster1:2 ']' 2023-11-08T05:16:01.790 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval cluster=cluster1 2023-11-08T05:16:01.790 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:16:01.790 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval instance=2 2023-11-08T05:16:01.790 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=2 2023-11-08T05:16:01.791 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-11-08T05:16:01.791 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local local_cluster=cluster1:2 2023-11-08T05:16:01.791 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster=cluster1 2023-11-08T05:16:01.791 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance 2023-11-08T05:16:01.791 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-11-08T05:16:01.792 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local val=cluster1:2 2023-11-08T05:16:01.792 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster_var_name=local_cluster 2023-11-08T05:16:01.792 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance_var_name=instance 2023-11-08T05:16:01.792 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:16:01.792 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=2 2023-11-08T05:16:01.792 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ '[' 2 = cluster1:2 ']' 2023-11-08T05:16:01.793 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval local_cluster=cluster1 2023-11-08T05:16:01.793 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ local_cluster=cluster1 2023-11-08T05:16:01.793 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval instance=2 2023-11-08T05:16:01.793 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ instance=2 2023-11-08T05:16:01.793 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-11-08T05:16:01.803 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-08T05:16:01.803 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-08T05:16:01.803 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-08T05:16:01.803 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_14 2023-11-08T05:16:01.910 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ return 0 2023-11-08T05:16:01.911 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ status_result=' "state": "Replaying"' 2023-11-08T05:16:01.911 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ echo ' "state": "Replaying"' 2023-11-08T05:16:01.912 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ grep -i Replaying 2023-11-08T05:16:01.912 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: "state": "Replaying" 2023-11-08T05:16:01.914 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ current_state=started 2023-11-08T05:16:01.914 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test started = started 2023-11-08T05:16:01.914 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:16:01.915 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_14 2023-11-08T05:16:01.915 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local local_cluster=cluster1 2023-11-08T05:16:01.915 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:16:01.915 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:16:01.915 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_14 2023-11-08T05:16:01.916 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' snapshot = journal ']' 2023-11-08T05:16:01.916 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' snapshot = snapshot ']' 2023-11-08T05:16:01.916 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_14 2023-11-08T05:16:01.916 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local local_cluster=cluster1 2023-11-08T05:16:01.917 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:16:01.917 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:16:01.917 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_14 2023-11-08T05:16:01.917 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ mkfname cluster2-mirror-image_14.status 2023-11-08T05:16:01.917 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ echo cluster2-mirror-image_14.status 2023-11-08T05:16:01.917 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -e 's|[/ ]|_|g' 2023-11-08T05:16:01.918 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local status_log=/tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_14.status 2023-11-08T05:16:01.918 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ mkfname cluster1-mirror-image_14.status 2023-11-08T05:16:01.918 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ echo cluster1-mirror-image_14.status 2023-11-08T05:16:01.918 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -e 's|[/ ]|_|g' 2023-11-08T05:16:01.919 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local local_status_log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_14.status 2023-11-08T05:16:01.919 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ mirror_image_snapshot cluster2 mirror image_14 2023-11-08T05:16:01.920 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:16:01.920 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:16:01.920 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_14 2023-11-08T05:16:01.920 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_14 2023-11-08T05:16:02.536 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:Snapshot ID: 145 2023-11-08T05:16:02.545 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_14 /tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_14.status 2023-11-08T05:16:02.545 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:16:02.546 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:16:02.546 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_14 2023-11-08T05:16:02.546 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_14.status 2023-11-08T05:16:02.547 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_14 --format xml 2023-11-08T05:16:02.547 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T05:16:02.581 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-11-08T05:16:02.583 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snapshot_id=145 2023-11-08T05:16:02.583 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ true 2023-11-08T05:16:02.583 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-11-08T05:16:02.583 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0.2 2023-11-08T05:16:02.784 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_14 /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_14.status 2023-11-08T05:16:02.784 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:16:02.785 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:16:02.785 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_14 2023-11-08T05:16:02.785 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_14.status 2023-11-08T05:16:02.785 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_14 --format xml 2023-11-08T05:16:02.785 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T05:16:02.824 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-08T05:16:02.826 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local primary_snapshot_id=81 2023-11-08T05:16:02.826 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test 145 = 81 2023-11-08T05:16:02.826 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-11-08T05:16:02.827 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0.4 2023-11-08T05:16:03.227 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_14 /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_14.status 2023-11-08T05:16:03.228 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:16:03.228 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:16:03.228 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_14 2023-11-08T05:16:03.229 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_14.status 2023-11-08T05:16:03.229 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_14 --format xml 2023-11-08T05:16:03.229 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T05:16:03.277 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-08T05:16:03.278 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local primary_snapshot_id=81 2023-11-08T05:16:03.279 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test 145 = 81 2023-11-08T05:16:03.279 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-11-08T05:16:03.279 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0.8 2023-11-08T05:16:04.080 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_14 /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_14.status 2023-11-08T05:16:04.080 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:16:04.081 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:16:04.081 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_14 2023-11-08T05:16:04.081 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_14.status 2023-11-08T05:16:04.081 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_14 --format xml 2023-11-08T05:16:04.081 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T05:16:04.123 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ true 2023-11-08T05:16:04.124 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-08T05:16:04.125 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:-:1.1: Document is empty 2023-11-08T05:16:04.125 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: 2023-11-08T05:16:04.125 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:^ 2023-11-08T05:16:04.126 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local primary_snapshot_id= 2023-11-08T05:16:04.126 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test 145 = '' 2023-11-08T05:16:04.126 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-11-08T05:16:04.127 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 1.6 2023-11-08T05:16:05.727 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_14 /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_14.status 2023-11-08T05:16:05.727 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:16:05.727 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:16:05.728 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_14 2023-11-08T05:16:05.728 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_14.status 2023-11-08T05:16:05.728 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_14 --format xml 2023-11-08T05:16:05.728 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T05:16:05.771 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-08T05:16:05.773 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local primary_snapshot_id=145 2023-11-08T05:16:05.774 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test 145 = 145 2023-11-08T05:16:05.774 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:16:05.774 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_snap_present cluster1 mirror image_14 snap 2023-11-08T05:16:05.774 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:16:05.775 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:16:05.775 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_14 2023-11-08T05:16:05.775 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap_name=snap 2023-11-08T05:16:05.775 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local s 2023-11-08T05:16:05.776 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-11-08T05:16:05.776 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 1 2023-11-08T05:16:06.774 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 info mirror/image_14@snap 2023-11-08T05:16:06.814 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:rbd image 'image_14': 2023-11-08T05:16:06.814 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: size 128 MiB in 32 objects 2023-11-08T05:16:06.815 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: order 22 (4 MiB objects) 2023-11-08T05:16:06.815 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: snapshot_count: 2 2023-11-08T05:16:06.815 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: id: 11125f783ff4 2023-11-08T05:16:06.815 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: block_name_prefix: rbd_data.11125f783ff4 2023-11-08T05:16:06.815 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: format: 2 2023-11-08T05:16:06.815 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: features: layering, exclusive-lock, non-primary 2023-11-08T05:16:06.816 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: op_features: 2023-11-08T05:16:06.816 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: flags: 2023-11-08T05:16:06.816 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: create_timestamp: Wed Nov 8 05:06:46 2023 2023-11-08T05:16:06.816 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: access_timestamp: Wed Nov 8 05:06:46 2023 2023-11-08T05:16:06.816 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: modify_timestamp: Wed Nov 8 05:06:46 2023 2023-11-08T05:16:06.817 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: protected: False 2023-11-08T05:16:06.817 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring state: enabled 2023-11-08T05:16:06.817 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring mode: snapshot 2023-11-08T05:16:06.817 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring global id: 2e7c6dbb-61d7-4f4a-9510-adb4bfb2cb02 2023-11-08T05:16:06.817 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring primary: false 2023-11-08T05:16:06.819 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:16:06.819 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ compare_image_snaps mirror image_14 snap 2023-11-08T05:16:06.819 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:16:06.819 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_14 2023-11-08T05:16:06.820 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap_name=snap 2023-11-08T05:16:06.820 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local ret=0 2023-11-08T05:16:06.820 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local rmt_export=/tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_14.export 2023-11-08T05:16:06.820 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local loc_export=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_14.export 2023-11-08T05:16:06.820 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rm -f /tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_14.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_14.export 2023-11-08T05:16:06.821 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 -p mirror export image_14@snap /tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_14.export 2023-11-08T05:16:07.009 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-11-08T05:16:07.012 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 -p mirror export image_14@snap /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_14.export 2023-11-08T05:16:07.247 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-11-08T05:16:07.251 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ cmp /tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_14.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_14.export 2023-11-08T05:16:07.309 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rm -f /tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_14.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_14.export 2023-11-08T05:16:07.348 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:16:07.348 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-11-08T05:16:07.348 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image=image_15 2023-11-08T05:16:07.349 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ create_snap cluster2 mirror image_15 snap 2023-11-08T05:16:07.349 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:16:07.349 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:16:07.349 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_15 2023-11-08T05:16:07.350 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap_name=snap 2023-11-08T05:16:07.350 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 -p mirror snap create image_15@snap --debug-rbd=20 --debug-journaler=20 2023-11-08T05:16:08.560 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_image_replay_started cluster1 mirror image_15 2023-11-08T05:16:08.560 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:16:08.560 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:16:08.561 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_15 2023-11-08T05:16:08.561 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_image_replay_state cluster1 mirror image_15 started 2023-11-08T05:16:08.561 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:16:08.561 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:16:08.562 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_15 2023-11-08T05:16:08.562 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local state=started 2023-11-08T05:16:08.562 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local s 2023-11-08T05:16:08.562 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-11-08T05:16:08.562 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 1 2023-11-08T05:16:09.561 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test_image_replay_state cluster1 mirror image_15 started 2023-11-08T05:16:09.561 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:16:09.561 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:16:09.562 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_15 2023-11-08T05:16:09.562 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local test_state=started 2023-11-08T05:16:09.562 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local status_result 2023-11-08T05:16:09.562 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local current_state=stopped 2023-11-08T05:16:09.565 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_15 2023-11-08T05:16:09.565 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ grep -i state 2023-11-08T05:16:09.565 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_instance=cluster1 2023-11-08T05:16:09.566 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T05:16:09.566 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1 2023-11-08T05:16:09.566 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance=cluster1 2023-11-08T05:16:09.566 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local loop_instance 2023-11-08T05:16:09.566 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-11-08T05:16:09.567 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sleep 0 2023-11-08T05:16:09.567 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-11-08T05:16:09.567 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ seq 0 3 2023-11-08T05:16:09.567 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T05:16:09.567 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_15 2023-11-08T05:16:09.568 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1:0 2023-11-08T05:16:09.568 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T05:16:09.568 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance 2023-11-08T05:16:09.568 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-11-08T05:16:09.568 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local val=cluster1:0 2023-11-08T05:16:09.569 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_var_name=cluster 2023-11-08T05:16:09.569 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance_var_name=instance 2023-11-08T05:16:09.569 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cluster=cluster1 2023-11-08T05:16:09.569 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ instance=0 2023-11-08T05:16:09.569 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' 0 = cluster1:0 ']' 2023-11-08T05:16:09.569 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval cluster=cluster1 2023-11-08T05:16:09.570 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:16:09.570 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval instance=0 2023-11-08T05:16:09.570 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=0 2023-11-08T05:16:09.570 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-11-08T05:16:09.570 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local local_cluster=cluster1:0 2023-11-08T05:16:09.571 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster=cluster1 2023-11-08T05:16:09.571 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance 2023-11-08T05:16:09.572 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-11-08T05:16:09.573 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local val=cluster1:0 2023-11-08T05:16:09.573 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster_var_name=local_cluster 2023-11-08T05:16:09.573 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance_var_name=instance 2023-11-08T05:16:09.573 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:16:09.573 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=0 2023-11-08T05:16:09.574 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ '[' 0 = cluster1:0 ']' 2023-11-08T05:16:09.574 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval local_cluster=cluster1 2023-11-08T05:16:09.574 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ local_cluster=cluster1 2023-11-08T05:16:09.574 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval instance=0 2023-11-08T05:16:09.574 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ instance=0 2023-11-08T05:16:09.575 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-11-08T05:16:09.581 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-08T05:16:09.582 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-08T05:16:09.582 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-08T05:16:09.582 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_15 2023-11-08T05:16:09.694 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ return 0 2023-11-08T05:16:09.695 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ status_result=' "state": "Replaying"' 2023-11-08T05:16:09.695 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ echo ' "state": "Replaying"' 2023-11-08T05:16:09.696 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ grep -i Replaying 2023-11-08T05:16:09.696 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: "state": "Replaying" 2023-11-08T05:16:09.698 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ current_state=started 2023-11-08T05:16:09.698 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test started = started 2023-11-08T05:16:09.699 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:16:09.699 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_15 2023-11-08T05:16:09.699 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local local_cluster=cluster1 2023-11-08T05:16:09.699 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:16:09.699 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:16:09.700 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_15 2023-11-08T05:16:09.700 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' snapshot = journal ']' 2023-11-08T05:16:09.700 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' snapshot = snapshot ']' 2023-11-08T05:16:09.700 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_15 2023-11-08T05:16:09.700 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local local_cluster=cluster1 2023-11-08T05:16:09.701 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:16:09.701 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:16:09.701 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_15 2023-11-08T05:16:09.701 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ mkfname cluster2-mirror-image_15.status 2023-11-08T05:16:09.701 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ echo cluster2-mirror-image_15.status 2023-11-08T05:16:09.702 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -e 's|[/ ]|_|g' 2023-11-08T05:16:09.702 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local status_log=/tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_15.status 2023-11-08T05:16:09.702 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ mkfname cluster1-mirror-image_15.status 2023-11-08T05:16:09.702 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ echo cluster1-mirror-image_15.status 2023-11-08T05:16:09.702 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -e 's|[/ ]|_|g' 2023-11-08T05:16:09.703 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local local_status_log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_15.status 2023-11-08T05:16:09.704 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ mirror_image_snapshot cluster2 mirror image_15 2023-11-08T05:16:09.704 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:16:09.704 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:16:09.704 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_15 2023-11-08T05:16:09.704 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_15 2023-11-08T05:16:10.561 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:Snapshot ID: 147 2023-11-08T05:16:10.569 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_15 /tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_15.status 2023-11-08T05:16:10.569 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:16:10.570 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:16:10.570 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_15 2023-11-08T05:16:10.570 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_15.status 2023-11-08T05:16:10.570 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_15 --format xml 2023-11-08T05:16:10.571 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T05:16:10.606 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-11-08T05:16:10.608 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snapshot_id=147 2023-11-08T05:16:10.609 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ true 2023-11-08T05:16:10.609 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-11-08T05:16:10.609 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0.2 2023-11-08T05:16:10.810 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_15 /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_15.status 2023-11-08T05:16:10.810 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:16:10.810 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:16:10.811 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_15 2023-11-08T05:16:10.811 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_15.status 2023-11-08T05:16:10.811 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_15 --format xml 2023-11-08T05:16:10.811 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T05:16:10.849 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-08T05:16:10.850 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local primary_snapshot_id=82 2023-11-08T05:16:10.851 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test 147 = 82 2023-11-08T05:16:10.851 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-11-08T05:16:10.851 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0.4 2023-11-08T05:16:11.252 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_15 /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_15.status 2023-11-08T05:16:11.252 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:16:11.252 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:16:11.253 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_15 2023-11-08T05:16:11.253 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_15.status 2023-11-08T05:16:11.253 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_15 --format xml 2023-11-08T05:16:11.253 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T05:16:11.293 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-08T05:16:11.295 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local primary_snapshot_id=82 2023-11-08T05:16:11.295 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test 147 = 82 2023-11-08T05:16:11.295 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-11-08T05:16:11.295 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0.8 2023-11-08T05:16:12.097 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_15 /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_15.status 2023-11-08T05:16:12.097 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:16:12.097 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:16:12.097 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_15 2023-11-08T05:16:12.098 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_15.status 2023-11-08T05:16:12.098 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_15 --format xml 2023-11-08T05:16:12.098 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T05:16:12.142 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ true 2023-11-08T05:16:12.142 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-08T05:16:12.143 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:-:1.1: Document is empty 2023-11-08T05:16:12.143 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: 2023-11-08T05:16:12.144 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:^ 2023-11-08T05:16:12.144 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local primary_snapshot_id= 2023-11-08T05:16:12.144 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test 147 = '' 2023-11-08T05:16:12.145 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-11-08T05:16:12.145 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 1.6 2023-11-08T05:16:13.746 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_15 /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_15.status 2023-11-08T05:16:13.746 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:16:13.746 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:16:13.746 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_15 2023-11-08T05:16:13.747 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_15.status 2023-11-08T05:16:13.747 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_15 --format xml 2023-11-08T05:16:13.747 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T05:16:13.790 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-08T05:16:13.792 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local primary_snapshot_id=147 2023-11-08T05:16:13.793 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test 147 = 147 2023-11-08T05:16:13.793 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:16:13.793 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_snap_present cluster1 mirror image_15 snap 2023-11-08T05:16:13.793 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:16:13.794 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:16:13.794 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_15 2023-11-08T05:16:13.794 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap_name=snap 2023-11-08T05:16:13.794 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local s 2023-11-08T05:16:13.795 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-11-08T05:16:13.795 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 1 2023-11-08T05:16:14.793 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 info mirror/image_15@snap 2023-11-08T05:16:14.830 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:rbd image 'image_15': 2023-11-08T05:16:14.831 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: size 128 MiB in 32 objects 2023-11-08T05:16:14.831 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: order 22 (4 MiB objects) 2023-11-08T05:16:14.831 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: snapshot_count: 2 2023-11-08T05:16:14.831 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: id: 11d87e083a6f 2023-11-08T05:16:14.831 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: block_name_prefix: rbd_data.11d87e083a6f 2023-11-08T05:16:14.832 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: format: 2 2023-11-08T05:16:14.832 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: features: layering, exclusive-lock, non-primary 2023-11-08T05:16:14.832 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: op_features: 2023-11-08T05:16:14.832 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: flags: 2023-11-08T05:16:14.832 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: create_timestamp: Wed Nov 8 05:07:00 2023 2023-11-08T05:16:14.833 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: access_timestamp: Wed Nov 8 05:07:00 2023 2023-11-08T05:16:14.833 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: modify_timestamp: Wed Nov 8 05:07:00 2023 2023-11-08T05:16:14.833 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: protected: False 2023-11-08T05:16:14.833 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring state: enabled 2023-11-08T05:16:14.833 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring mode: snapshot 2023-11-08T05:16:14.834 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring global id: c5f2bed1-9f45-45f4-aaf6-7a50884f15f8 2023-11-08T05:16:14.834 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring primary: false 2023-11-08T05:16:14.835 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:16:14.835 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ compare_image_snaps mirror image_15 snap 2023-11-08T05:16:14.835 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:16:14.836 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_15 2023-11-08T05:16:14.836 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap_name=snap 2023-11-08T05:16:14.836 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local ret=0 2023-11-08T05:16:14.836 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local rmt_export=/tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_15.export 2023-11-08T05:16:14.836 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local loc_export=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_15.export 2023-11-08T05:16:14.837 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rm -f /tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_15.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_15.export 2023-11-08T05:16:14.837 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 -p mirror export image_15@snap /tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_15.export 2023-11-08T05:16:15.012 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-11-08T05:16:15.015 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 -p mirror export image_15@snap /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_15.export 2023-11-08T05:16:15.243 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-11-08T05:16:15.246 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ cmp /tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_15.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_15.export 2023-11-08T05:16:15.301 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rm -f /tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_15.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_15.export 2023-11-08T05:16:15.340 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:16:15.340 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-11-08T05:16:15.340 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image=image_16 2023-11-08T05:16:15.340 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ create_snap cluster2 mirror image_16 snap 2023-11-08T05:16:15.341 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:16:15.341 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:16:15.341 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_16 2023-11-08T05:16:15.341 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap_name=snap 2023-11-08T05:16:15.341 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 -p mirror snap create image_16@snap --debug-rbd=20 --debug-journaler=20 2023-11-08T05:16:16.582 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_image_replay_started cluster1 mirror image_16 2023-11-08T05:16:16.583 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:16:16.583 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:16:16.583 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_16 2023-11-08T05:16:16.583 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_image_replay_state cluster1 mirror image_16 started 2023-11-08T05:16:16.584 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:16:16.584 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:16:16.584 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_16 2023-11-08T05:16:16.584 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local state=started 2023-11-08T05:16:16.584 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local s 2023-11-08T05:16:16.585 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-11-08T05:16:16.585 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 1 2023-11-08T05:16:17.584 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test_image_replay_state cluster1 mirror image_16 started 2023-11-08T05:16:17.585 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:16:17.585 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:16:17.585 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_16 2023-11-08T05:16:17.585 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local test_state=started 2023-11-08T05:16:17.585 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local status_result 2023-11-08T05:16:17.586 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local current_state=stopped 2023-11-08T05:16:17.586 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_16 2023-11-08T05:16:17.586 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_instance=cluster1 2023-11-08T05:16:17.586 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ grep -i state 2023-11-08T05:16:17.587 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T05:16:17.587 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1 2023-11-08T05:16:17.587 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance=cluster1 2023-11-08T05:16:17.587 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local loop_instance 2023-11-08T05:16:17.587 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-11-08T05:16:17.588 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sleep 0 2023-11-08T05:16:17.592 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-11-08T05:16:17.593 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ seq 0 3 2023-11-08T05:16:17.593 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T05:16:17.594 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_16 2023-11-08T05:16:17.594 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1:0 2023-11-08T05:16:17.594 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T05:16:17.594 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance 2023-11-08T05:16:17.595 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-11-08T05:16:17.595 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local val=cluster1:0 2023-11-08T05:16:17.595 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_var_name=cluster 2023-11-08T05:16:17.596 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance_var_name=instance 2023-11-08T05:16:17.596 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cluster=cluster1 2023-11-08T05:16:17.596 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ instance=0 2023-11-08T05:16:17.596 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' 0 = cluster1:0 ']' 2023-11-08T05:16:17.597 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval cluster=cluster1 2023-11-08T05:16:17.597 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:16:17.597 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval instance=0 2023-11-08T05:16:17.597 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=0 2023-11-08T05:16:17.597 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-11-08T05:16:17.597 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local local_cluster=cluster1:0 2023-11-08T05:16:17.598 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster=cluster1 2023-11-08T05:16:17.598 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance 2023-11-08T05:16:17.598 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-11-08T05:16:17.598 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local val=cluster1:0 2023-11-08T05:16:17.598 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster_var_name=local_cluster 2023-11-08T05:16:17.599 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance_var_name=instance 2023-11-08T05:16:17.599 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:16:17.599 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=0 2023-11-08T05:16:17.599 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ '[' 0 = cluster1:0 ']' 2023-11-08T05:16:17.599 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval local_cluster=cluster1 2023-11-08T05:16:17.600 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ local_cluster=cluster1 2023-11-08T05:16:17.600 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval instance=0 2023-11-08T05:16:17.600 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ instance=0 2023-11-08T05:16:17.600 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-11-08T05:16:17.604 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-08T05:16:17.604 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-08T05:16:17.605 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-08T05:16:17.605 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_16 2023-11-08T05:16:17.718 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ return 0 2023-11-08T05:16:17.719 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ status_result=' "state": "Replaying"' 2023-11-08T05:16:17.720 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ echo ' "state": "Replaying"' 2023-11-08T05:16:17.720 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ grep -i Replaying 2023-11-08T05:16:17.721 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: "state": "Replaying" 2023-11-08T05:16:17.725 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ current_state=started 2023-11-08T05:16:17.725 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test started = started 2023-11-08T05:16:17.725 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:16:17.725 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_16 2023-11-08T05:16:17.726 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local local_cluster=cluster1 2023-11-08T05:16:17.726 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:16:17.726 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:16:17.726 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_16 2023-11-08T05:16:17.727 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' snapshot = journal ']' 2023-11-08T05:16:17.727 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' snapshot = snapshot ']' 2023-11-08T05:16:17.727 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_16 2023-11-08T05:16:17.727 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local local_cluster=cluster1 2023-11-08T05:16:17.727 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:16:17.727 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:16:17.728 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_16 2023-11-08T05:16:17.728 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ mkfname cluster2-mirror-image_16.status 2023-11-08T05:16:17.728 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ echo cluster2-mirror-image_16.status 2023-11-08T05:16:17.728 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -e 's|[/ ]|_|g' 2023-11-08T05:16:17.728 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local status_log=/tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_16.status 2023-11-08T05:16:17.729 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ mkfname cluster1-mirror-image_16.status 2023-11-08T05:16:17.729 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ echo cluster1-mirror-image_16.status 2023-11-08T05:16:17.729 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -e 's|[/ ]|_|g' 2023-11-08T05:16:17.729 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local local_status_log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_16.status 2023-11-08T05:16:17.729 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ mirror_image_snapshot cluster2 mirror image_16 2023-11-08T05:16:17.730 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:16:17.730 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:16:17.730 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_16 2023-11-08T05:16:17.730 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_16 2023-11-08T05:16:18.586 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:Snapshot ID: 149 2023-11-08T05:16:18.596 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_16 /tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_16.status 2023-11-08T05:16:18.596 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:16:18.596 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:16:18.596 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_16 2023-11-08T05:16:18.597 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_16.status 2023-11-08T05:16:18.597 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_16 --format xml 2023-11-08T05:16:18.597 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T05:16:18.635 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-11-08T05:16:18.637 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snapshot_id=149 2023-11-08T05:16:18.637 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ true 2023-11-08T05:16:18.637 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-11-08T05:16:18.638 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0.2 2023-11-08T05:16:18.838 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_16 /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_16.status 2023-11-08T05:16:18.838 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:16:18.839 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:16:18.839 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_16 2023-11-08T05:16:18.839 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_16.status 2023-11-08T05:16:18.839 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_16 --format xml 2023-11-08T05:16:18.840 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T05:16:18.878 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-08T05:16:18.882 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local primary_snapshot_id=83 2023-11-08T05:16:18.882 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test 149 = 83 2023-11-08T05:16:18.882 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-11-08T05:16:18.882 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0.4 2023-11-08T05:16:19.281 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_16 /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_16.status 2023-11-08T05:16:19.282 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:16:19.282 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:16:19.282 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_16 2023-11-08T05:16:19.282 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_16.status 2023-11-08T05:16:19.283 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_16 --format xml 2023-11-08T05:16:19.283 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T05:16:19.321 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-08T05:16:19.323 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local primary_snapshot_id=83 2023-11-08T05:16:19.324 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test 149 = 83 2023-11-08T05:16:19.324 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-11-08T05:16:19.324 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0.8 2023-11-08T05:16:20.125 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_16 /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_16.status 2023-11-08T05:16:20.125 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:16:20.125 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:16:20.125 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_16 2023-11-08T05:16:20.126 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_16.status 2023-11-08T05:16:20.126 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_16 --format xml 2023-11-08T05:16:20.126 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T05:16:20.167 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ true 2023-11-08T05:16:20.168 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-08T05:16:20.169 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:-:1.1: Document is empty 2023-11-08T05:16:20.169 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: 2023-11-08T05:16:20.169 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:^ 2023-11-08T05:16:20.170 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local primary_snapshot_id= 2023-11-08T05:16:20.170 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test 149 = '' 2023-11-08T05:16:20.170 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-11-08T05:16:20.171 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 1.6 2023-11-08T05:16:21.771 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_16 /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_16.status 2023-11-08T05:16:21.771 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:16:21.771 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:16:21.772 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_16 2023-11-08T05:16:21.772 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_16.status 2023-11-08T05:16:21.772 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_16 --format xml 2023-11-08T05:16:21.772 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T05:16:21.819 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-08T05:16:21.821 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local primary_snapshot_id=149 2023-11-08T05:16:21.822 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test 149 = 149 2023-11-08T05:16:21.822 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:16:21.822 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_snap_present cluster1 mirror image_16 snap 2023-11-08T05:16:21.822 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:16:21.822 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:16:21.823 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_16 2023-11-08T05:16:21.823 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap_name=snap 2023-11-08T05:16:21.823 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local s 2023-11-08T05:16:21.823 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-11-08T05:16:21.823 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 1 2023-11-08T05:16:22.822 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 info mirror/image_16@snap 2023-11-08T05:16:22.862 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:rbd image 'image_16': 2023-11-08T05:16:22.863 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: size 128 MiB in 32 objects 2023-11-08T05:16:22.863 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: order 22 (4 MiB objects) 2023-11-08T05:16:22.863 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: snapshot_count: 2 2023-11-08T05:16:22.863 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: id: 128738ea5437 2023-11-08T05:16:22.864 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: block_name_prefix: rbd_data.128738ea5437 2023-11-08T05:16:22.864 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: format: 2 2023-11-08T05:16:22.864 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: features: layering, exclusive-lock, non-primary 2023-11-08T05:16:22.864 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: op_features: 2023-11-08T05:16:22.864 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: flags: 2023-11-08T05:16:22.865 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: create_timestamp: Wed Nov 8 05:07:09 2023 2023-11-08T05:16:22.865 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: access_timestamp: Wed Nov 8 05:07:09 2023 2023-11-08T05:16:22.865 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: modify_timestamp: Wed Nov 8 05:07:09 2023 2023-11-08T05:16:22.865 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: protected: False 2023-11-08T05:16:22.865 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring state: enabled 2023-11-08T05:16:22.865 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring mode: snapshot 2023-11-08T05:16:22.866 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring global id: 6852ffd8-136b-481a-b662-fdde8fec627d 2023-11-08T05:16:22.866 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring primary: false 2023-11-08T05:16:22.867 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:16:22.867 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ compare_image_snaps mirror image_16 snap 2023-11-08T05:16:22.867 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:16:22.868 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_16 2023-11-08T05:16:22.868 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap_name=snap 2023-11-08T05:16:22.868 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local ret=0 2023-11-08T05:16:22.868 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local rmt_export=/tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_16.export 2023-11-08T05:16:22.868 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local loc_export=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_16.export 2023-11-08T05:16:22.869 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rm -f /tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_16.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_16.export 2023-11-08T05:16:22.869 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 -p mirror export image_16@snap /tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_16.export 2023-11-08T05:16:23.046 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-11-08T05:16:23.049 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 -p mirror export image_16@snap /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_16.export 2023-11-08T05:16:23.290 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-11-08T05:16:23.293 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ cmp /tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_16.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_16.export 2023-11-08T05:16:23.349 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rm -f /tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_16.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_16.export 2023-11-08T05:16:23.386 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:16:23.387 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-11-08T05:16:23.387 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image=image_17 2023-11-08T05:16:23.387 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ create_snap cluster2 mirror image_17 snap 2023-11-08T05:16:23.387 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:16:23.387 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:16:23.388 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_17 2023-11-08T05:16:23.388 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap_name=snap 2023-11-08T05:16:23.388 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 -p mirror snap create image_17@snap --debug-rbd=20 --debug-journaler=20 2023-11-08T05:16:25.198 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_image_replay_started cluster1 mirror image_17 2023-11-08T05:16:25.199 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:16:25.199 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:16:25.199 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_17 2023-11-08T05:16:25.199 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_image_replay_state cluster1 mirror image_17 started 2023-11-08T05:16:25.199 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:16:25.200 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:16:25.200 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_17 2023-11-08T05:16:25.200 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local state=started 2023-11-08T05:16:25.200 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local s 2023-11-08T05:16:25.200 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-11-08T05:16:25.201 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 1 2023-11-08T05:16:26.200 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test_image_replay_state cluster1 mirror image_17 started 2023-11-08T05:16:26.200 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:16:26.200 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:16:26.201 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_17 2023-11-08T05:16:26.201 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local test_state=started 2023-11-08T05:16:26.201 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local status_result 2023-11-08T05:16:26.201 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local current_state=stopped 2023-11-08T05:16:26.204 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_17 2023-11-08T05:16:26.204 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ grep -i state 2023-11-08T05:16:26.204 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_instance=cluster1 2023-11-08T05:16:26.204 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T05:16:26.204 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1 2023-11-08T05:16:26.205 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance=cluster1 2023-11-08T05:16:26.205 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local loop_instance 2023-11-08T05:16:26.205 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-11-08T05:16:26.205 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sleep 0 2023-11-08T05:16:26.205 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-11-08T05:16:26.206 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ seq 0 3 2023-11-08T05:16:26.206 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T05:16:26.206 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_17 2023-11-08T05:16:26.206 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1:0 2023-11-08T05:16:26.206 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T05:16:26.207 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance 2023-11-08T05:16:26.207 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-11-08T05:16:26.207 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local val=cluster1:0 2023-11-08T05:16:26.207 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_var_name=cluster 2023-11-08T05:16:26.207 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance_var_name=instance 2023-11-08T05:16:26.207 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cluster=cluster1 2023-11-08T05:16:26.208 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ instance=0 2023-11-08T05:16:26.208 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' 0 = cluster1:0 ']' 2023-11-08T05:16:26.208 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval cluster=cluster1 2023-11-08T05:16:26.208 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:16:26.208 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval instance=0 2023-11-08T05:16:26.209 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=0 2023-11-08T05:16:26.209 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-11-08T05:16:26.210 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local local_cluster=cluster1:0 2023-11-08T05:16:26.211 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster=cluster1 2023-11-08T05:16:26.211 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance 2023-11-08T05:16:26.211 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-11-08T05:16:26.211 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local val=cluster1:0 2023-11-08T05:16:26.212 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster_var_name=local_cluster 2023-11-08T05:16:26.212 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance_var_name=instance 2023-11-08T05:16:26.212 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:16:26.212 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=0 2023-11-08T05:16:26.212 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ '[' 0 = cluster1:0 ']' 2023-11-08T05:16:26.212 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval local_cluster=cluster1 2023-11-08T05:16:26.213 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ local_cluster=cluster1 2023-11-08T05:16:26.213 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval instance=0 2023-11-08T05:16:26.213 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ instance=0 2023-11-08T05:16:26.213 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-11-08T05:16:26.221 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-08T05:16:26.222 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-08T05:16:26.222 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-08T05:16:26.222 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_17 2023-11-08T05:16:26.331 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:Invalid command: unused arguments: ['mirror/image_17'] 2023-11-08T05:16:26.331 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd mirror status : get status for rbd mirror 2023-11-08T05:16:26.332 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:admin_socket: invalid command 2023-11-08T05:16:26.334 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T05:16:26.335 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_17 2023-11-08T05:16:26.335 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1:1 2023-11-08T05:16:26.335 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T05:16:26.335 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance 2023-11-08T05:16:26.335 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-11-08T05:16:26.336 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local val=cluster1:1 2023-11-08T05:16:26.336 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_var_name=cluster 2023-11-08T05:16:26.336 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance_var_name=instance 2023-11-08T05:16:26.336 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cluster=cluster1 2023-11-08T05:16:26.336 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ instance=1 2023-11-08T05:16:26.337 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' 1 = cluster1:1 ']' 2023-11-08T05:16:26.337 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval cluster=cluster1 2023-11-08T05:16:26.337 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:16:26.337 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval instance=1 2023-11-08T05:16:26.337 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=1 2023-11-08T05:16:26.338 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-11-08T05:16:26.338 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local local_cluster=cluster1:1 2023-11-08T05:16:26.338 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster=cluster1 2023-11-08T05:16:26.338 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance 2023-11-08T05:16:26.338 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-11-08T05:16:26.339 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local val=cluster1:1 2023-11-08T05:16:26.339 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster_var_name=local_cluster 2023-11-08T05:16:26.339 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance_var_name=instance 2023-11-08T05:16:26.339 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:16:26.339 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=1 2023-11-08T05:16:26.340 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ '[' 1 = cluster1:1 ']' 2023-11-08T05:16:26.340 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval local_cluster=cluster1 2023-11-08T05:16:26.340 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ local_cluster=cluster1 2023-11-08T05:16:26.340 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval instance=1 2023-11-08T05:16:26.340 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ instance=1 2023-11-08T05:16:26.341 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-11-08T05:16:26.348 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-08T05:16:26.349 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-08T05:16:26.349 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-08T05:16:26.349 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_17 2023-11-08T05:16:26.449 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:Invalid command: unused arguments: ['mirror/image_17'] 2023-11-08T05:16:26.449 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd mirror status : get status for rbd mirror 2023-11-08T05:16:26.450 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:admin_socket: invalid command 2023-11-08T05:16:26.452 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T05:16:26.452 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_17 2023-11-08T05:16:26.453 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1:2 2023-11-08T05:16:26.453 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T05:16:26.453 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance 2023-11-08T05:16:26.453 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-11-08T05:16:26.453 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local val=cluster1:2 2023-11-08T05:16:26.454 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_var_name=cluster 2023-11-08T05:16:26.454 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance_var_name=instance 2023-11-08T05:16:26.454 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cluster=cluster1 2023-11-08T05:16:26.454 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ instance=2 2023-11-08T05:16:26.454 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' 2 = cluster1:2 ']' 2023-11-08T05:16:26.455 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval cluster=cluster1 2023-11-08T05:16:26.455 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:16:26.455 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval instance=2 2023-11-08T05:16:26.455 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=2 2023-11-08T05:16:26.455 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-11-08T05:16:26.456 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local local_cluster=cluster1:2 2023-11-08T05:16:26.456 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster=cluster1 2023-11-08T05:16:26.456 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance 2023-11-08T05:16:26.456 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-11-08T05:16:26.456 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local val=cluster1:2 2023-11-08T05:16:26.456 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster_var_name=local_cluster 2023-11-08T05:16:26.457 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance_var_name=instance 2023-11-08T05:16:26.457 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:16:26.457 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=2 2023-11-08T05:16:26.457 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ '[' 2 = cluster1:2 ']' 2023-11-08T05:16:26.457 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval local_cluster=cluster1 2023-11-08T05:16:26.458 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ local_cluster=cluster1 2023-11-08T05:16:26.458 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval instance=2 2023-11-08T05:16:26.458 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ instance=2 2023-11-08T05:16:26.458 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-11-08T05:16:26.466 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-08T05:16:26.467 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-08T05:16:26.467 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-08T05:16:26.467 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_17 2023-11-08T05:16:26.569 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:Invalid command: unused arguments: ['mirror/image_17'] 2023-11-08T05:16:26.569 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd mirror status : get status for rbd mirror 2023-11-08T05:16:26.569 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:admin_socket: invalid command 2023-11-08T05:16:26.572 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T05:16:26.572 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_17 2023-11-08T05:16:26.573 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1:3 2023-11-08T05:16:26.573 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T05:16:26.573 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance 2023-11-08T05:16:26.573 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ set_cluster_instance cluster1:3 cluster instance 2023-11-08T05:16:26.573 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local val=cluster1:3 2023-11-08T05:16:26.574 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_var_name=cluster 2023-11-08T05:16:26.574 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance_var_name=instance 2023-11-08T05:16:26.574 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cluster=cluster1 2023-11-08T05:16:26.574 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ instance=3 2023-11-08T05:16:26.574 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' 3 = cluster1:3 ']' 2023-11-08T05:16:26.575 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval cluster=cluster1 2023-11-08T05:16:26.575 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:16:26.575 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval instance=3 2023-11-08T05:16:26.575 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=3 2023-11-08T05:16:26.575 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ daemon_asok_file cluster1:3 cluster1 2023-11-08T05:16:26.576 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local local_cluster=cluster1:3 2023-11-08T05:16:26.576 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster=cluster1 2023-11-08T05:16:26.576 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance 2023-11-08T05:16:26.576 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2023-11-08T05:16:26.576 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local val=cluster1:3 2023-11-08T05:16:26.577 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster_var_name=local_cluster 2023-11-08T05:16:26.577 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance_var_name=instance 2023-11-08T05:16:26.577 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:16:26.577 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=3 2023-11-08T05:16:26.577 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ '[' 3 = cluster1:3 ']' 2023-11-08T05:16:26.578 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval local_cluster=cluster1 2023-11-08T05:16:26.578 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ local_cluster=cluster1 2023-11-08T05:16:26.578 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval instance=3 2023-11-08T05:16:26.578 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ instance=3 2023-11-08T05:16:26.579 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2023-11-08T05:16:26.587 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-11-08T05:16:26.587 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-11-08T05:16:26.587 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-11-08T05:16:26.588 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_17 2023-11-08T05:16:26.694 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ return 0 2023-11-08T05:16:26.695 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ status_result=' "state": "Replaying"' 2023-11-08T05:16:26.695 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ echo ' "state": "Replaying"' 2023-11-08T05:16:26.696 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ grep -i Replaying 2023-11-08T05:16:26.696 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: "state": "Replaying" 2023-11-08T05:16:26.698 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ current_state=started 2023-11-08T05:16:26.698 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test started = started 2023-11-08T05:16:26.699 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:16:26.699 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_17 2023-11-08T05:16:26.699 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local local_cluster=cluster1 2023-11-08T05:16:26.699 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:16:26.699 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:16:26.700 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_17 2023-11-08T05:16:26.700 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' snapshot = journal ']' 2023-11-08T05:16:26.700 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' snapshot = snapshot ']' 2023-11-08T05:16:26.700 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_17 2023-11-08T05:16:26.700 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local local_cluster=cluster1 2023-11-08T05:16:26.700 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:16:26.701 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:16:26.701 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_17 2023-11-08T05:16:26.701 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ mkfname cluster2-mirror-image_17.status 2023-11-08T05:16:26.701 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ echo cluster2-mirror-image_17.status 2023-11-08T05:16:26.701 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -e 's|[/ ]|_|g' 2023-11-08T05:16:26.702 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local status_log=/tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_17.status 2023-11-08T05:16:26.702 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ mkfname cluster1-mirror-image_17.status 2023-11-08T05:16:26.702 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ echo cluster1-mirror-image_17.status 2023-11-08T05:16:26.702 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -e 's|[/ ]|_|g' 2023-11-08T05:16:26.703 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local local_status_log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_17.status 2023-11-08T05:16:26.703 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ mirror_image_snapshot cluster2 mirror image_17 2023-11-08T05:16:26.704 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:16:26.704 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:16:26.704 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_17 2023-11-08T05:16:26.704 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_17 2023-11-08T05:16:27.201 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:Snapshot ID: 151 2023-11-08T05:16:27.210 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_17 /tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_17.status 2023-11-08T05:16:27.210 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:16:27.211 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:16:27.211 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_17 2023-11-08T05:16:27.211 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_17.status 2023-11-08T05:16:27.211 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_17 --format xml 2023-11-08T05:16:27.211 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T05:16:27.248 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-11-08T05:16:27.250 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snapshot_id=151 2023-11-08T05:16:27.250 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ true 2023-11-08T05:16:27.250 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-11-08T05:16:27.250 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0.2 2023-11-08T05:16:27.451 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_17 /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_17.status 2023-11-08T05:16:27.452 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:16:27.452 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:16:27.452 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_17 2023-11-08T05:16:27.452 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_17.status 2023-11-08T05:16:27.452 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_17 --format xml 2023-11-08T05:16:27.453 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T05:16:27.490 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ true 2023-11-08T05:16:27.490 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-08T05:16:27.491 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:-:1.1: Document is empty 2023-11-08T05:16:27.491 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: 2023-11-08T05:16:27.492 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:^ 2023-11-08T05:16:27.492 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local primary_snapshot_id= 2023-11-08T05:16:27.493 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test 151 = '' 2023-11-08T05:16:27.493 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-11-08T05:16:27.493 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0.4 2023-11-08T05:16:27.893 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_17 /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_17.status 2023-11-08T05:16:27.894 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:16:27.894 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:16:27.894 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_17 2023-11-08T05:16:27.894 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_17.status 2023-11-08T05:16:27.895 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_17 --format xml 2023-11-08T05:16:27.895 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T05:16:27.933 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ true 2023-11-08T05:16:27.934 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-08T05:16:27.935 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:-:1.1: Document is empty 2023-11-08T05:16:27.935 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: 2023-11-08T05:16:27.935 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:^ 2023-11-08T05:16:27.936 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local primary_snapshot_id= 2023-11-08T05:16:27.936 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test 151 = '' 2023-11-08T05:16:27.936 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-11-08T05:16:27.936 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0.8 2023-11-08T05:16:28.737 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_17 /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_17.status 2023-11-08T05:16:28.737 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:16:28.737 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:16:28.738 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_17 2023-11-08T05:16:28.738 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_17.status 2023-11-08T05:16:28.738 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_17 --format xml 2023-11-08T05:16:28.738 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T05:16:28.826 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ true 2023-11-08T05:16:28.826 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-08T05:16:28.827 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:-:1.1: Document is empty 2023-11-08T05:16:28.827 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: 2023-11-08T05:16:28.828 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:^ 2023-11-08T05:16:28.828 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local primary_snapshot_id= 2023-11-08T05:16:28.829 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test 151 = '' 2023-11-08T05:16:28.829 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-11-08T05:16:28.829 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 1.6 2023-11-08T05:16:30.429 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_17 /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_17.status 2023-11-08T05:16:30.430 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:16:30.430 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:16:30.430 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_17 2023-11-08T05:16:30.430 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_17.status 2023-11-08T05:16:30.431 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_17 --format xml 2023-11-08T05:16:30.431 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T05:16:30.476 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-08T05:16:30.478 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local primary_snapshot_id=151 2023-11-08T05:16:30.478 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test 151 = 151 2023-11-08T05:16:30.479 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:16:30.479 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_snap_present cluster1 mirror image_17 snap 2023-11-08T05:16:30.479 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:16:30.479 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:16:30.479 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_17 2023-11-08T05:16:30.480 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap_name=snap 2023-11-08T05:16:30.480 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local s 2023-11-08T05:16:30.480 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-11-08T05:16:30.480 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 1 2023-11-08T05:16:31.479 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 info mirror/image_17@snap 2023-11-08T05:16:31.517 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:rbd image 'image_17': 2023-11-08T05:16:31.517 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: size 128 MiB in 32 objects 2023-11-08T05:16:31.518 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: order 22 (4 MiB objects) 2023-11-08T05:16:31.518 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: snapshot_count: 2 2023-11-08T05:16:31.518 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: id: 10e71e2e10d1 2023-11-08T05:16:31.518 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: block_name_prefix: rbd_data.10e71e2e10d1 2023-11-08T05:16:31.518 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: format: 2 2023-11-08T05:16:31.519 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: features: layering, exclusive-lock, non-primary 2023-11-08T05:16:31.519 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: op_features: 2023-11-08T05:16:31.519 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: flags: 2023-11-08T05:16:31.519 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: create_timestamp: Wed Nov 8 05:07:22 2023 2023-11-08T05:16:31.519 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: access_timestamp: Wed Nov 8 05:07:22 2023 2023-11-08T05:16:31.520 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: modify_timestamp: Wed Nov 8 05:07:22 2023 2023-11-08T05:16:31.520 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: protected: False 2023-11-08T05:16:31.520 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring state: enabled 2023-11-08T05:16:31.520 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring mode: snapshot 2023-11-08T05:16:31.520 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring global id: 7b62298f-cd5f-402a-b9f9-d2211585565d 2023-11-08T05:16:31.521 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring primary: false 2023-11-08T05:16:31.522 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:16:31.522 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ compare_image_snaps mirror image_17 snap 2023-11-08T05:16:31.522 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:16:31.522 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_17 2023-11-08T05:16:31.523 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap_name=snap 2023-11-08T05:16:31.523 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local ret=0 2023-11-08T05:16:31.523 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local rmt_export=/tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_17.export 2023-11-08T05:16:31.523 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local loc_export=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_17.export 2023-11-08T05:16:31.523 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rm -f /tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_17.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_17.export 2023-11-08T05:16:31.524 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 -p mirror export image_17@snap /tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_17.export 2023-11-08T05:16:31.643 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-11-08T05:16:31.646 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 -p mirror export image_17@snap /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_17.export 2023-11-08T05:16:31.830 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-11-08T05:16:31.833 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ cmp /tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_17.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_17.export 2023-11-08T05:16:31.933 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rm -f /tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_17.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_17.export 2023-11-08T05:16:31.949 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:16:31.949 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-11-08T05:16:31.949 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image=image_18 2023-11-08T05:16:31.949 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ create_snap cluster2 mirror image_18 snap 2023-11-08T05:16:31.950 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:16:31.950 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:16:31.950 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_18 2023-11-08T05:16:31.950 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap_name=snap 2023-11-08T05:16:31.950 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 -p mirror snap create image_18@snap --debug-rbd=20 --debug-journaler=20 2023-11-08T05:16:33.217 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_image_replay_started cluster1 mirror image_18 2023-11-08T05:16:33.217 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:16:33.218 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:16:33.218 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_18 2023-11-08T05:16:33.218 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_image_replay_state cluster1 mirror image_18 started 2023-11-08T05:16:33.218 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:16:33.218 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:16:33.219 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_18 2023-11-08T05:16:33.219 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local state=started 2023-11-08T05:16:33.219 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local s 2023-11-08T05:16:33.219 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-11-08T05:16:33.219 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 1 2023-11-08T05:16:34.218 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test_image_replay_state cluster1 mirror image_18 started 2023-11-08T05:16:34.219 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:16:34.219 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:16:34.219 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_18 2023-11-08T05:16:34.219 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local test_state=started 2023-11-08T05:16:34.220 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local status_result 2023-11-08T05:16:34.220 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local current_state=stopped 2023-11-08T05:16:34.222 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_18 2023-11-08T05:16:34.223 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ grep -i state 2023-11-08T05:16:34.223 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_instance=cluster1 2023-11-08T05:16:34.223 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T05:16:34.223 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1 2023-11-08T05:16:34.224 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance=cluster1 2023-11-08T05:16:34.224 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local loop_instance 2023-11-08T05:16:34.224 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-11-08T05:16:34.224 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sleep 0 2023-11-08T05:16:34.224 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-11-08T05:16:34.225 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ seq 0 3 2023-11-08T05:16:34.225 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T05:16:34.225 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_18 2023-11-08T05:16:34.225 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1:0 2023-11-08T05:16:34.226 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T05:16:34.226 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance 2023-11-08T05:16:34.226 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-11-08T05:16:34.226 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local val=cluster1:0 2023-11-08T05:16:34.226 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_var_name=cluster 2023-11-08T05:16:34.227 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance_var_name=instance 2023-11-08T05:16:34.227 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cluster=cluster1 2023-11-08T05:16:34.227 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ instance=0 2023-11-08T05:16:34.227 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' 0 = cluster1:0 ']' 2023-11-08T05:16:34.227 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval cluster=cluster1 2023-11-08T05:16:34.228 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:16:34.228 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval instance=0 2023-11-08T05:16:34.228 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=0 2023-11-08T05:16:34.228 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-11-08T05:16:34.228 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local local_cluster=cluster1:0 2023-11-08T05:16:34.228 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster=cluster1 2023-11-08T05:16:34.229 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance 2023-11-08T05:16:34.229 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-11-08T05:16:34.229 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local val=cluster1:0 2023-11-08T05:16:34.229 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster_var_name=local_cluster 2023-11-08T05:16:34.229 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance_var_name=instance 2023-11-08T05:16:34.231 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:16:34.231 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=0 2023-11-08T05:16:34.231 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ '[' 0 = cluster1:0 ']' 2023-11-08T05:16:34.231 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval local_cluster=cluster1 2023-11-08T05:16:34.231 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ local_cluster=cluster1 2023-11-08T05:16:34.232 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval instance=0 2023-11-08T05:16:34.232 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ instance=0 2023-11-08T05:16:34.232 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-11-08T05:16:34.242 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-08T05:16:34.242 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-08T05:16:34.242 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-08T05:16:34.243 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_18 2023-11-08T05:16:34.350 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:Invalid command: unused arguments: ['mirror/image_18'] 2023-11-08T05:16:34.351 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd mirror status : get status for rbd mirror 2023-11-08T05:16:34.351 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:admin_socket: invalid command 2023-11-08T05:16:34.353 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T05:16:34.354 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_18 2023-11-08T05:16:34.354 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1:1 2023-11-08T05:16:34.354 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T05:16:34.354 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance 2023-11-08T05:16:34.354 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-11-08T05:16:34.355 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local val=cluster1:1 2023-11-08T05:16:34.355 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_var_name=cluster 2023-11-08T05:16:34.355 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance_var_name=instance 2023-11-08T05:16:34.355 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cluster=cluster1 2023-11-08T05:16:34.355 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ instance=1 2023-11-08T05:16:34.356 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' 1 = cluster1:1 ']' 2023-11-08T05:16:34.356 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval cluster=cluster1 2023-11-08T05:16:34.356 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:16:34.356 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval instance=1 2023-11-08T05:16:34.356 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=1 2023-11-08T05:16:34.357 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-11-08T05:16:34.357 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local local_cluster=cluster1:1 2023-11-08T05:16:34.357 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster=cluster1 2023-11-08T05:16:34.357 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance 2023-11-08T05:16:34.357 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-11-08T05:16:34.358 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local val=cluster1:1 2023-11-08T05:16:34.358 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster_var_name=local_cluster 2023-11-08T05:16:34.358 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance_var_name=instance 2023-11-08T05:16:34.358 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:16:34.358 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=1 2023-11-08T05:16:34.359 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ '[' 1 = cluster1:1 ']' 2023-11-08T05:16:34.359 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval local_cluster=cluster1 2023-11-08T05:16:34.359 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ local_cluster=cluster1 2023-11-08T05:16:34.359 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval instance=1 2023-11-08T05:16:34.359 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ instance=1 2023-11-08T05:16:34.360 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-11-08T05:16:34.369 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-08T05:16:34.369 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-08T05:16:34.369 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-08T05:16:34.369 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_18 2023-11-08T05:16:34.470 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:Invalid command: unused arguments: ['mirror/image_18'] 2023-11-08T05:16:34.471 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd mirror status : get status for rbd mirror 2023-11-08T05:16:34.471 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:admin_socket: invalid command 2023-11-08T05:16:34.474 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T05:16:34.474 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_18 2023-11-08T05:16:34.474 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1:2 2023-11-08T05:16:34.474 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T05:16:34.475 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance 2023-11-08T05:16:34.475 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-11-08T05:16:34.475 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local val=cluster1:2 2023-11-08T05:16:34.475 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_var_name=cluster 2023-11-08T05:16:34.475 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance_var_name=instance 2023-11-08T05:16:34.476 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cluster=cluster1 2023-11-08T05:16:34.476 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ instance=2 2023-11-08T05:16:34.476 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' 2 = cluster1:2 ']' 2023-11-08T05:16:34.476 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval cluster=cluster1 2023-11-08T05:16:34.476 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:16:34.476 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval instance=2 2023-11-08T05:16:34.477 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=2 2023-11-08T05:16:34.477 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-11-08T05:16:34.477 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local local_cluster=cluster1:2 2023-11-08T05:16:34.477 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster=cluster1 2023-11-08T05:16:34.477 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance 2023-11-08T05:16:34.478 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-11-08T05:16:34.478 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local val=cluster1:2 2023-11-08T05:16:34.478 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster_var_name=local_cluster 2023-11-08T05:16:34.478 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance_var_name=instance 2023-11-08T05:16:34.478 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:16:34.479 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=2 2023-11-08T05:16:34.479 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ '[' 2 = cluster1:2 ']' 2023-11-08T05:16:34.479 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval local_cluster=cluster1 2023-11-08T05:16:34.479 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ local_cluster=cluster1 2023-11-08T05:16:34.479 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval instance=2 2023-11-08T05:16:34.480 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ instance=2 2023-11-08T05:16:34.480 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-11-08T05:16:34.488 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-08T05:16:34.489 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-08T05:16:34.489 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-08T05:16:34.489 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_18 2023-11-08T05:16:34.599 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ return 0 2023-11-08T05:16:34.599 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ status_result=' "state": "Replaying"' 2023-11-08T05:16:34.600 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ echo ' "state": "Replaying"' 2023-11-08T05:16:34.600 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ grep -i Replaying 2023-11-08T05:16:34.600 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: "state": "Replaying" 2023-11-08T05:16:34.602 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ current_state=started 2023-11-08T05:16:34.603 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test started = started 2023-11-08T05:16:34.603 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:16:34.603 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_18 2023-11-08T05:16:34.603 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local local_cluster=cluster1 2023-11-08T05:16:34.603 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:16:34.604 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:16:34.604 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_18 2023-11-08T05:16:34.604 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' snapshot = journal ']' 2023-11-08T05:16:34.604 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' snapshot = snapshot ']' 2023-11-08T05:16:34.604 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_18 2023-11-08T05:16:34.605 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local local_cluster=cluster1 2023-11-08T05:16:34.605 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:16:34.605 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:16:34.605 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_18 2023-11-08T05:16:34.605 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ mkfname cluster2-mirror-image_18.status 2023-11-08T05:16:34.606 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ echo cluster2-mirror-image_18.status 2023-11-08T05:16:34.606 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -e 's|[/ ]|_|g' 2023-11-08T05:16:34.606 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local status_log=/tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_18.status 2023-11-08T05:16:34.606 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ mkfname cluster1-mirror-image_18.status 2023-11-08T05:16:34.606 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ echo cluster1-mirror-image_18.status 2023-11-08T05:16:34.607 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -e 's|[/ ]|_|g' 2023-11-08T05:16:34.608 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local local_status_log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_18.status 2023-11-08T05:16:34.608 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ mirror_image_snapshot cluster2 mirror image_18 2023-11-08T05:16:34.608 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:16:34.608 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:16:34.609 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_18 2023-11-08T05:16:34.609 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_18 2023-11-08T05:16:35.219 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:Snapshot ID: 153 2023-11-08T05:16:35.229 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_18 /tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_18.status 2023-11-08T05:16:35.229 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:16:35.229 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:16:35.229 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_18 2023-11-08T05:16:35.230 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_18.status 2023-11-08T05:16:35.230 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_18 --format xml 2023-11-08T05:16:35.230 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T05:16:35.266 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-11-08T05:16:35.268 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snapshot_id=153 2023-11-08T05:16:35.268 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ true 2023-11-08T05:16:35.268 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-11-08T05:16:35.268 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0.2 2023-11-08T05:16:35.469 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_18 /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_18.status 2023-11-08T05:16:35.469 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:16:35.469 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:16:35.470 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_18 2023-11-08T05:16:35.470 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_18.status 2023-11-08T05:16:35.470 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_18 --format xml 2023-11-08T05:16:35.470 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T05:16:35.509 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-08T05:16:35.510 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local primary_snapshot_id=85 2023-11-08T05:16:35.511 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test 153 = 85 2023-11-08T05:16:35.511 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-11-08T05:16:35.511 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0.4 2023-11-08T05:16:35.912 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_18 /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_18.status 2023-11-08T05:16:35.912 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:16:35.913 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:16:35.913 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_18 2023-11-08T05:16:35.913 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_18.status 2023-11-08T05:16:35.913 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_18 --format xml 2023-11-08T05:16:35.913 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T05:16:35.954 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ true 2023-11-08T05:16:35.954 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-08T05:16:35.955 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:-:1.1: Document is empty 2023-11-08T05:16:35.955 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: 2023-11-08T05:16:35.955 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:^ 2023-11-08T05:16:35.956 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local primary_snapshot_id= 2023-11-08T05:16:35.956 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test 153 = '' 2023-11-08T05:16:35.957 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-11-08T05:16:35.957 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0.8 2023-11-08T05:16:36.757 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_18 /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_18.status 2023-11-08T05:16:36.757 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:16:36.758 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:16:36.758 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_18 2023-11-08T05:16:36.758 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_18.status 2023-11-08T05:16:36.758 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_18 --format xml 2023-11-08T05:16:36.758 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T05:16:36.815 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ true 2023-11-08T05:16:36.815 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-08T05:16:36.816 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:-:1.1: Document is empty 2023-11-08T05:16:36.817 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: 2023-11-08T05:16:36.817 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:^ 2023-11-08T05:16:36.818 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local primary_snapshot_id= 2023-11-08T05:16:36.818 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test 153 = '' 2023-11-08T05:16:36.818 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-11-08T05:16:36.818 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 1.6 2023-11-08T05:16:38.419 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_18 /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_18.status 2023-11-08T05:16:38.419 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:16:38.419 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:16:38.420 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_18 2023-11-08T05:16:38.420 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_18.status 2023-11-08T05:16:38.420 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_18 --format xml 2023-11-08T05:16:38.420 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T05:16:38.462 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-08T05:16:38.465 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local primary_snapshot_id=153 2023-11-08T05:16:38.465 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test 153 = 153 2023-11-08T05:16:38.465 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:16:38.465 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_snap_present cluster1 mirror image_18 snap 2023-11-08T05:16:38.465 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:16:38.466 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:16:38.466 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_18 2023-11-08T05:16:38.466 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap_name=snap 2023-11-08T05:16:38.466 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local s 2023-11-08T05:16:38.467 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-11-08T05:16:38.467 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 1 2023-11-08T05:16:39.465 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 info mirror/image_18@snap 2023-11-08T05:16:39.506 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:rbd image 'image_18': 2023-11-08T05:16:39.506 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: size 128 MiB in 32 objects 2023-11-08T05:16:39.506 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: order 22 (4 MiB objects) 2023-11-08T05:16:39.507 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: snapshot_count: 2 2023-11-08T05:16:39.507 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: id: 11128a8f7adb 2023-11-08T05:16:39.507 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: block_name_prefix: rbd_data.11128a8f7adb 2023-11-08T05:16:39.507 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: format: 2 2023-11-08T05:16:39.507 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: features: layering, exclusive-lock, non-primary 2023-11-08T05:16:39.508 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: op_features: 2023-11-08T05:16:39.508 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: flags: 2023-11-08T05:16:39.508 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: create_timestamp: Wed Nov 8 05:07:37 2023 2023-11-08T05:16:39.508 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: access_timestamp: Wed Nov 8 05:07:37 2023 2023-11-08T05:16:39.508 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: modify_timestamp: Wed Nov 8 05:07:37 2023 2023-11-08T05:16:39.509 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: protected: False 2023-11-08T05:16:39.509 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring state: enabled 2023-11-08T05:16:39.509 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring mode: snapshot 2023-11-08T05:16:39.509 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring global id: 2645822b-c23b-4b5d-bdd0-8114af333323 2023-11-08T05:16:39.509 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring primary: false 2023-11-08T05:16:39.510 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:16:39.511 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ compare_image_snaps mirror image_18 snap 2023-11-08T05:16:39.511 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:16:39.511 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_18 2023-11-08T05:16:39.511 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap_name=snap 2023-11-08T05:16:39.511 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local ret=0 2023-11-08T05:16:39.512 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local rmt_export=/tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_18.export 2023-11-08T05:16:39.512 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local loc_export=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_18.export 2023-11-08T05:16:39.512 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rm -f /tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_18.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_18.export 2023-11-08T05:16:39.512 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 -p mirror export image_18@snap /tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_18.export 2023-11-08T05:16:39.693 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-11-08T05:16:39.696 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 -p mirror export image_18@snap /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_18.export 2023-11-08T05:16:39.944 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-11-08T05:16:39.946 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ cmp /tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_18.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_18.export 2023-11-08T05:16:40.001 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rm -f /tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_18.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_18.export 2023-11-08T05:16:40.038 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:16:40.039 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-11-08T05:16:40.039 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image=image_19 2023-11-08T05:16:40.039 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ create_snap cluster2 mirror image_19 snap 2023-11-08T05:16:40.039 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:16:40.040 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:16:40.040 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_19 2023-11-08T05:16:40.040 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap_name=snap 2023-11-08T05:16:40.040 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 -p mirror snap create image_19@snap --debug-rbd=20 --debug-journaler=20 2023-11-08T05:16:41.237 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_image_replay_started cluster1 mirror image_19 2023-11-08T05:16:41.238 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:16:41.238 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:16:41.238 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_19 2023-11-08T05:16:41.238 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_image_replay_state cluster1 mirror image_19 started 2023-11-08T05:16:41.239 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:16:41.239 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:16:41.239 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_19 2023-11-08T05:16:41.239 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local state=started 2023-11-08T05:16:41.240 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local s 2023-11-08T05:16:41.240 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-11-08T05:16:41.240 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 1 2023-11-08T05:16:42.238 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test_image_replay_state cluster1 mirror image_19 started 2023-11-08T05:16:42.239 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:16:42.239 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:16:42.239 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_19 2023-11-08T05:16:42.239 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local test_state=started 2023-11-08T05:16:42.240 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local status_result 2023-11-08T05:16:42.240 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local current_state=stopped 2023-11-08T05:16:42.240 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_19 2023-11-08T05:16:42.240 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_instance=cluster1 2023-11-08T05:16:42.240 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T05:16:42.241 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ grep -i state 2023-11-08T05:16:42.241 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1 2023-11-08T05:16:42.241 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance=cluster1 2023-11-08T05:16:42.241 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local loop_instance 2023-11-08T05:16:42.241 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-11-08T05:16:42.241 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sleep 0 2023-11-08T05:16:42.244 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-11-08T05:16:42.244 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ seq 0 3 2023-11-08T05:16:42.244 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T05:16:42.244 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_19 2023-11-08T05:16:42.244 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1:0 2023-11-08T05:16:42.245 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T05:16:42.245 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance 2023-11-08T05:16:42.245 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-11-08T05:16:42.245 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local val=cluster1:0 2023-11-08T05:16:42.245 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_var_name=cluster 2023-11-08T05:16:42.246 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance_var_name=instance 2023-11-08T05:16:42.246 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cluster=cluster1 2023-11-08T05:16:42.246 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ instance=0 2023-11-08T05:16:42.246 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' 0 = cluster1:0 ']' 2023-11-08T05:16:42.247 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval cluster=cluster1 2023-11-08T05:16:42.247 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:16:42.247 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval instance=0 2023-11-08T05:16:42.247 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=0 2023-11-08T05:16:42.247 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-11-08T05:16:42.248 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local local_cluster=cluster1:0 2023-11-08T05:16:42.248 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster=cluster1 2023-11-08T05:16:42.248 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance 2023-11-08T05:16:42.248 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-11-08T05:16:42.248 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local val=cluster1:0 2023-11-08T05:16:42.249 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster_var_name=local_cluster 2023-11-08T05:16:42.249 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance_var_name=instance 2023-11-08T05:16:42.250 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:16:42.250 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=0 2023-11-08T05:16:42.250 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ '[' 0 = cluster1:0 ']' 2023-11-08T05:16:42.251 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval local_cluster=cluster1 2023-11-08T05:16:42.251 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ local_cluster=cluster1 2023-11-08T05:16:42.251 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval instance=0 2023-11-08T05:16:42.251 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ instance=0 2023-11-08T05:16:42.251 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-11-08T05:16:42.261 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-08T05:16:42.261 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-08T05:16:42.262 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-08T05:16:42.262 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_19 2023-11-08T05:16:42.373 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:Invalid command: unused arguments: ['mirror/image_19'] 2023-11-08T05:16:42.374 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd mirror status : get status for rbd mirror 2023-11-08T05:16:42.374 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:admin_socket: invalid command 2023-11-08T05:16:42.377 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T05:16:42.377 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_19 2023-11-08T05:16:42.377 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1:1 2023-11-08T05:16:42.378 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T05:16:42.378 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance 2023-11-08T05:16:42.378 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-11-08T05:16:42.378 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local val=cluster1:1 2023-11-08T05:16:42.379 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_var_name=cluster 2023-11-08T05:16:42.379 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance_var_name=instance 2023-11-08T05:16:42.379 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cluster=cluster1 2023-11-08T05:16:42.379 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ instance=1 2023-11-08T05:16:42.379 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' 1 = cluster1:1 ']' 2023-11-08T05:16:42.380 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval cluster=cluster1 2023-11-08T05:16:42.380 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:16:42.380 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval instance=1 2023-11-08T05:16:42.380 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=1 2023-11-08T05:16:42.380 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-11-08T05:16:42.381 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local local_cluster=cluster1:1 2023-11-08T05:16:42.381 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster=cluster1 2023-11-08T05:16:42.381 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance 2023-11-08T05:16:42.381 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-11-08T05:16:42.382 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local val=cluster1:1 2023-11-08T05:16:42.382 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster_var_name=local_cluster 2023-11-08T05:16:42.382 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance_var_name=instance 2023-11-08T05:16:42.382 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:16:42.382 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=1 2023-11-08T05:16:42.383 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ '[' 1 = cluster1:1 ']' 2023-11-08T05:16:42.383 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval local_cluster=cluster1 2023-11-08T05:16:42.383 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ local_cluster=cluster1 2023-11-08T05:16:42.383 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval instance=1 2023-11-08T05:16:42.383 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ instance=1 2023-11-08T05:16:42.384 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-11-08T05:16:42.391 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-08T05:16:42.392 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-08T05:16:42.392 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-08T05:16:42.392 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_19 2023-11-08T05:16:42.494 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:Invalid command: unused arguments: ['mirror/image_19'] 2023-11-08T05:16:42.494 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd mirror status : get status for rbd mirror 2023-11-08T05:16:42.494 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:admin_socket: invalid command 2023-11-08T05:16:42.497 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T05:16:42.498 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_19 2023-11-08T05:16:42.498 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1:2 2023-11-08T05:16:42.498 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T05:16:42.498 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance 2023-11-08T05:16:42.499 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-11-08T05:16:42.499 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local val=cluster1:2 2023-11-08T05:16:42.499 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_var_name=cluster 2023-11-08T05:16:42.499 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance_var_name=instance 2023-11-08T05:16:42.500 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cluster=cluster1 2023-11-08T05:16:42.500 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ instance=2 2023-11-08T05:16:42.500 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' 2 = cluster1:2 ']' 2023-11-08T05:16:42.500 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval cluster=cluster1 2023-11-08T05:16:42.500 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:16:42.500 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval instance=2 2023-11-08T05:16:42.501 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=2 2023-11-08T05:16:42.501 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-11-08T05:16:42.501 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local local_cluster=cluster1:2 2023-11-08T05:16:42.501 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster=cluster1 2023-11-08T05:16:42.502 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance 2023-11-08T05:16:42.502 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-11-08T05:16:42.502 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local val=cluster1:2 2023-11-08T05:16:42.502 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster_var_name=local_cluster 2023-11-08T05:16:42.502 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance_var_name=instance 2023-11-08T05:16:42.502 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:16:42.503 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=2 2023-11-08T05:16:42.503 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ '[' 2 = cluster1:2 ']' 2023-11-08T05:16:42.503 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval local_cluster=cluster1 2023-11-08T05:16:42.503 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ local_cluster=cluster1 2023-11-08T05:16:42.503 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval instance=2 2023-11-08T05:16:42.504 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ instance=2 2023-11-08T05:16:42.504 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-11-08T05:16:42.512 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-08T05:16:42.512 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-08T05:16:42.512 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-08T05:16:42.512 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_19 2023-11-08T05:16:42.611 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:Invalid command: unused arguments: ['mirror/image_19'] 2023-11-08T05:16:42.611 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd mirror status : get status for rbd mirror 2023-11-08T05:16:42.611 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:admin_socket: invalid command 2023-11-08T05:16:42.614 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T05:16:42.614 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_19 2023-11-08T05:16:42.614 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1:3 2023-11-08T05:16:42.614 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T05:16:42.614 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance 2023-11-08T05:16:42.615 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ set_cluster_instance cluster1:3 cluster instance 2023-11-08T05:16:42.615 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local val=cluster1:3 2023-11-08T05:16:42.615 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_var_name=cluster 2023-11-08T05:16:42.615 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance_var_name=instance 2023-11-08T05:16:42.615 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cluster=cluster1 2023-11-08T05:16:42.616 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ instance=3 2023-11-08T05:16:42.616 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' 3 = cluster1:3 ']' 2023-11-08T05:16:42.616 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval cluster=cluster1 2023-11-08T05:16:42.616 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:16:42.616 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval instance=3 2023-11-08T05:16:42.616 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=3 2023-11-08T05:16:42.617 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ daemon_asok_file cluster1:3 cluster1 2023-11-08T05:16:42.617 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local local_cluster=cluster1:3 2023-11-08T05:16:42.617 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster=cluster1 2023-11-08T05:16:42.617 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance 2023-11-08T05:16:42.617 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2023-11-08T05:16:42.618 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local val=cluster1:3 2023-11-08T05:16:42.618 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster_var_name=local_cluster 2023-11-08T05:16:42.618 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance_var_name=instance 2023-11-08T05:16:42.618 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:16:42.618 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=3 2023-11-08T05:16:42.619 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ '[' 3 = cluster1:3 ']' 2023-11-08T05:16:42.619 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval local_cluster=cluster1 2023-11-08T05:16:42.619 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ local_cluster=cluster1 2023-11-08T05:16:42.619 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval instance=3 2023-11-08T05:16:42.619 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ instance=3 2023-11-08T05:16:42.620 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2023-11-08T05:16:42.628 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-11-08T05:16:42.628 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-11-08T05:16:42.628 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-11-08T05:16:42.629 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_19 2023-11-08T05:16:42.737 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ return 0 2023-11-08T05:16:42.738 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ status_result=' "state": "Replaying"' 2023-11-08T05:16:42.739 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ echo ' "state": "Replaying"' 2023-11-08T05:16:42.739 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ grep -i Replaying 2023-11-08T05:16:42.739 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: "state": "Replaying" 2023-11-08T05:16:42.742 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ current_state=started 2023-11-08T05:16:42.742 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test started = started 2023-11-08T05:16:42.742 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:16:42.743 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_19 2023-11-08T05:16:42.743 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local local_cluster=cluster1 2023-11-08T05:16:42.743 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:16:42.743 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:16:42.743 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_19 2023-11-08T05:16:42.744 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' snapshot = journal ']' 2023-11-08T05:16:42.744 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' snapshot = snapshot ']' 2023-11-08T05:16:42.744 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_19 2023-11-08T05:16:42.744 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local local_cluster=cluster1 2023-11-08T05:16:42.744 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:16:42.745 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:16:42.745 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_19 2023-11-08T05:16:42.745 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ mkfname cluster2-mirror-image_19.status 2023-11-08T05:16:42.745 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ echo cluster2-mirror-image_19.status 2023-11-08T05:16:42.745 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -e 's|[/ ]|_|g' 2023-11-08T05:16:42.745 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local status_log=/tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_19.status 2023-11-08T05:16:42.746 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ mkfname cluster1-mirror-image_19.status 2023-11-08T05:16:42.746 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ echo cluster1-mirror-image_19.status 2023-11-08T05:16:42.746 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -e 's|[/ ]|_|g' 2023-11-08T05:16:42.747 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local local_status_log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_19.status 2023-11-08T05:16:42.747 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ mirror_image_snapshot cluster2 mirror image_19 2023-11-08T05:16:42.747 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:16:42.748 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:16:42.748 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_19 2023-11-08T05:16:42.748 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_19 2023-11-08T05:16:43.242 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:Snapshot ID: 155 2023-11-08T05:16:43.250 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_19 /tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_19.status 2023-11-08T05:16:43.251 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:16:43.251 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:16:43.251 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_19 2023-11-08T05:16:43.251 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_19.status 2023-11-08T05:16:43.252 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_19 --format xml 2023-11-08T05:16:43.252 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T05:16:43.288 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-11-08T05:16:43.290 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snapshot_id=155 2023-11-08T05:16:43.290 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ true 2023-11-08T05:16:43.290 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-11-08T05:16:43.291 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0.2 2023-11-08T05:16:43.491 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_19 /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_19.status 2023-11-08T05:16:43.492 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:16:43.493 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:16:43.493 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_19 2023-11-08T05:16:43.494 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_19.status 2023-11-08T05:16:43.494 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_19 --format xml 2023-11-08T05:16:43.494 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T05:16:43.531 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-08T05:16:43.532 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local primary_snapshot_id=86 2023-11-08T05:16:43.533 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test 155 = 86 2023-11-08T05:16:43.533 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-11-08T05:16:43.533 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0.4 2023-11-08T05:16:43.934 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_19 /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_19.status 2023-11-08T05:16:43.934 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:16:43.935 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:16:43.935 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_19 2023-11-08T05:16:43.935 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_19.status 2023-11-08T05:16:43.935 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_19 --format xml 2023-11-08T05:16:43.935 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T05:16:43.973 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ true 2023-11-08T05:16:43.973 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-08T05:16:43.974 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:-:1.1: Document is empty 2023-11-08T05:16:43.974 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: 2023-11-08T05:16:43.975 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:^ 2023-11-08T05:16:43.975 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local primary_snapshot_id= 2023-11-08T05:16:43.976 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test 155 = '' 2023-11-08T05:16:43.976 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-11-08T05:16:43.976 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0.8 2023-11-08T05:16:44.200 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.0 2023-11-08T05:16:44.200 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Sent signal 15 2023-11-08T05:16:44.201 INFO:tasks.rbd_mirror_thrash:waiting for 2 secs before reviving daemons 2023-11-08T05:16:44.201 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi087.stderr:2023-11-08T05:16:44.198+0000 7fb2e9e10700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 134988) UID: 1000 2023-11-08T05:16:44.402 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi087.stderr:daemon-helper: command failed with exit status 1 2023-11-08T05:16:44.776 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_19 /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_19.status 2023-11-08T05:16:44.777 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:16:44.777 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:16:44.777 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_19 2023-11-08T05:16:44.777 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_19.status 2023-11-08T05:16:44.777 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_19 --format xml 2023-11-08T05:16:44.778 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T05:16:44.820 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ true 2023-11-08T05:16:44.820 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-08T05:16:44.821 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:-:1.1: Document is empty 2023-11-08T05:16:44.821 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: 2023-11-08T05:16:44.821 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:^ 2023-11-08T05:16:44.822 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local primary_snapshot_id= 2023-11-08T05:16:44.822 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test 155 = '' 2023-11-08T05:16:44.823 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-11-08T05:16:44.823 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 1.6 2023-11-08T05:16:46.202 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.0 2023-11-08T05:16:46.202 INFO:teuthology.orchestra.run:waiting for 600 2023-11-08T05:16:46.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-08T05:16:46.203 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.0 2023-11-08T05:16:46.203 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2023-11-08T05:16:46.204 DEBUG:teuthology.orchestra.run.smithi087:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2023-11-08T05:16:46.206 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2023-11-08T05:16:46.207 INFO:tasks.rbd_mirror_thrash:waiting for 62 secs before thrashing 2023-11-08T05:16:46.423 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_19 /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_19.status 2023-11-08T05:16:46.423 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:16:46.423 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:16:46.424 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_19 2023-11-08T05:16:46.424 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_19.status 2023-11-08T05:16:46.424 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_19 --format xml 2023-11-08T05:16:46.424 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T05:16:46.465 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-08T05:16:46.467 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local primary_snapshot_id=155 2023-11-08T05:16:46.467 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test 155 = 155 2023-11-08T05:16:46.467 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:16:46.467 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_snap_present cluster1 mirror image_19 snap 2023-11-08T05:16:46.468 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:16:46.468 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:16:46.468 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_19 2023-11-08T05:16:46.468 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap_name=snap 2023-11-08T05:16:46.468 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local s 2023-11-08T05:16:46.469 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-11-08T05:16:46.469 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 1 2023-11-08T05:16:47.467 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 info mirror/image_19@snap 2023-11-08T05:16:47.510 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:rbd image 'image_19': 2023-11-08T05:16:47.511 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: size 128 MiB in 32 objects 2023-11-08T05:16:47.511 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: order 22 (4 MiB objects) 2023-11-08T05:16:47.511 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: snapshot_count: 2 2023-11-08T05:16:47.511 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: id: 11d826309c7f 2023-11-08T05:16:47.512 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: block_name_prefix: rbd_data.11d826309c7f 2023-11-08T05:16:47.512 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: format: 2 2023-11-08T05:16:47.512 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: features: layering, exclusive-lock, non-primary 2023-11-08T05:16:47.512 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: op_features: 2023-11-08T05:16:47.512 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: flags: 2023-11-08T05:16:47.513 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: create_timestamp: Wed Nov 8 05:07:48 2023 2023-11-08T05:16:47.513 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: access_timestamp: Wed Nov 8 05:07:48 2023 2023-11-08T05:16:47.513 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: modify_timestamp: Wed Nov 8 05:07:48 2023 2023-11-08T05:16:47.513 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: protected: False 2023-11-08T05:16:47.513 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring state: enabled 2023-11-08T05:16:47.514 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring mode: snapshot 2023-11-08T05:16:47.514 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring global id: 783a1447-f0d7-471d-9452-7cb8dece9780 2023-11-08T05:16:47.514 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring primary: false 2023-11-08T05:16:47.515 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:16:47.516 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ compare_image_snaps mirror image_19 snap 2023-11-08T05:16:47.516 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:16:47.516 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_19 2023-11-08T05:16:47.516 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap_name=snap 2023-11-08T05:16:47.517 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local ret=0 2023-11-08T05:16:47.517 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local rmt_export=/tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_19.export 2023-11-08T05:16:47.517 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local loc_export=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_19.export 2023-11-08T05:16:47.517 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rm -f /tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_19.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_19.export 2023-11-08T05:16:47.517 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 -p mirror export image_19@snap /tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_19.export 2023-11-08T05:16:47.692 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-11-08T05:16:47.695 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 -p mirror export image_19@snap /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_19.export 2023-11-08T05:16:47.901 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-11-08T05:16:47.904 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ cmp /tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_19.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_19.export 2023-11-08T05:16:47.959 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rm -f /tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_19.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_19.export 2023-11-08T05:16:47.996 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:16:47.997 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-11-08T05:16:47.997 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image=image_20 2023-11-08T05:16:47.997 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ create_snap cluster2 mirror image_20 snap 2023-11-08T05:16:47.997 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:16:47.998 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:16:47.998 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_20 2023-11-08T05:16:47.998 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap_name=snap 2023-11-08T05:16:47.998 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 -p mirror snap create image_20@snap --debug-rbd=20 --debug-journaler=20 2023-11-08T05:16:49.255 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_image_replay_started cluster1 mirror image_20 2023-11-08T05:16:49.255 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:16:49.256 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:16:49.256 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_20 2023-11-08T05:16:49.256 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_image_replay_state cluster1 mirror image_20 started 2023-11-08T05:16:49.256 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:16:49.256 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:16:49.257 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_20 2023-11-08T05:16:49.257 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local state=started 2023-11-08T05:16:49.257 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local s 2023-11-08T05:16:49.257 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-11-08T05:16:49.257 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 1 2023-11-08T05:16:50.256 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test_image_replay_state cluster1 mirror image_20 started 2023-11-08T05:16:50.256 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:16:50.256 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:16:50.257 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_20 2023-11-08T05:16:50.257 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local test_state=started 2023-11-08T05:16:50.257 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local status_result 2023-11-08T05:16:50.257 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local current_state=stopped 2023-11-08T05:16:50.261 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_20 2023-11-08T05:16:50.261 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_instance=cluster1 2023-11-08T05:16:50.261 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ grep -i state 2023-11-08T05:16:50.262 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T05:16:50.262 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1 2023-11-08T05:16:50.262 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance=cluster1 2023-11-08T05:16:50.262 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local loop_instance 2023-11-08T05:16:50.262 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-11-08T05:16:50.263 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sleep 0 2023-11-08T05:16:50.263 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-11-08T05:16:50.263 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ seq 0 3 2023-11-08T05:16:50.263 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T05:16:50.263 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_20 2023-11-08T05:16:50.264 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1:0 2023-11-08T05:16:50.264 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T05:16:50.264 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance 2023-11-08T05:16:50.264 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-11-08T05:16:50.265 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local val=cluster1:0 2023-11-08T05:16:50.265 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_var_name=cluster 2023-11-08T05:16:50.265 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance_var_name=instance 2023-11-08T05:16:50.265 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cluster=cluster1 2023-11-08T05:16:50.265 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ instance=0 2023-11-08T05:16:50.265 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' 0 = cluster1:0 ']' 2023-11-08T05:16:50.266 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval cluster=cluster1 2023-11-08T05:16:50.266 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:16:50.266 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval instance=0 2023-11-08T05:16:50.266 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=0 2023-11-08T05:16:50.266 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-11-08T05:16:50.267 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local local_cluster=cluster1:0 2023-11-08T05:16:50.267 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster=cluster1 2023-11-08T05:16:50.267 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance 2023-11-08T05:16:50.267 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-11-08T05:16:50.267 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local val=cluster1:0 2023-11-08T05:16:50.268 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster_var_name=local_cluster 2023-11-08T05:16:50.268 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance_var_name=instance 2023-11-08T05:16:50.268 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:16:50.268 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=0 2023-11-08T05:16:50.268 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ '[' 0 = cluster1:0 ']' 2023-11-08T05:16:50.269 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval local_cluster=cluster1 2023-11-08T05:16:50.269 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ local_cluster=cluster1 2023-11-08T05:16:50.269 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval instance=0 2023-11-08T05:16:50.269 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ instance=0 2023-11-08T05:16:50.269 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-11-08T05:16:50.278 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-08T05:16:50.278 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-08T05:16:50.278 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-08T05:16:50.278 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_20 2023-11-08T05:16:50.377 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:Invalid command: unused arguments: ['mirror/image_20'] 2023-11-08T05:16:50.378 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd mirror status : get status for rbd mirror 2023-11-08T05:16:50.378 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:admin_socket: invalid command 2023-11-08T05:16:50.383 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T05:16:50.384 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_20 2023-11-08T05:16:50.384 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1:1 2023-11-08T05:16:50.384 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T05:16:50.385 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance 2023-11-08T05:16:50.385 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-11-08T05:16:50.386 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local val=cluster1:1 2023-11-08T05:16:50.386 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_var_name=cluster 2023-11-08T05:16:50.386 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance_var_name=instance 2023-11-08T05:16:50.387 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cluster=cluster1 2023-11-08T05:16:50.387 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ instance=1 2023-11-08T05:16:50.387 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' 1 = cluster1:1 ']' 2023-11-08T05:16:50.387 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval cluster=cluster1 2023-11-08T05:16:50.388 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:16:50.388 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval instance=1 2023-11-08T05:16:50.388 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=1 2023-11-08T05:16:50.388 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-11-08T05:16:50.388 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local local_cluster=cluster1:1 2023-11-08T05:16:50.389 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster=cluster1 2023-11-08T05:16:50.389 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance 2023-11-08T05:16:50.389 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-11-08T05:16:50.389 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local val=cluster1:1 2023-11-08T05:16:50.390 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster_var_name=local_cluster 2023-11-08T05:16:50.390 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance_var_name=instance 2023-11-08T05:16:50.390 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:16:50.390 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=1 2023-11-08T05:16:50.390 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ '[' 1 = cluster1:1 ']' 2023-11-08T05:16:50.391 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval local_cluster=cluster1 2023-11-08T05:16:50.391 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ local_cluster=cluster1 2023-11-08T05:16:50.391 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval instance=1 2023-11-08T05:16:50.391 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ instance=1 2023-11-08T05:16:50.392 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-11-08T05:16:50.396 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-08T05:16:50.396 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-08T05:16:50.396 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-08T05:16:50.396 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_20 2023-11-08T05:16:50.505 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ return 0 2023-11-08T05:16:50.506 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ status_result=' "state": "Replaying"' 2023-11-08T05:16:50.506 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ echo ' "state": "Replaying"' 2023-11-08T05:16:50.507 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ grep -i Replaying 2023-11-08T05:16:50.507 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: "state": "Replaying" 2023-11-08T05:16:50.509 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ current_state=started 2023-11-08T05:16:50.510 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test started = started 2023-11-08T05:16:50.510 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:16:50.510 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_20 2023-11-08T05:16:50.510 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local local_cluster=cluster1 2023-11-08T05:16:50.510 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:16:50.511 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:16:50.511 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_20 2023-11-08T05:16:50.511 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' snapshot = journal ']' 2023-11-08T05:16:50.511 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' snapshot = snapshot ']' 2023-11-08T05:16:50.511 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_20 2023-11-08T05:16:50.512 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local local_cluster=cluster1 2023-11-08T05:16:50.512 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:16:50.512 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:16:50.512 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_20 2023-11-08T05:16:50.512 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ mkfname cluster2-mirror-image_20.status 2023-11-08T05:16:50.513 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ echo cluster2-mirror-image_20.status 2023-11-08T05:16:50.513 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -e 's|[/ ]|_|g' 2023-11-08T05:16:50.513 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local status_log=/tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_20.status 2023-11-08T05:16:50.513 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ mkfname cluster1-mirror-image_20.status 2023-11-08T05:16:50.513 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ echo cluster1-mirror-image_20.status 2023-11-08T05:16:50.514 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -e 's|[/ ]|_|g' 2023-11-08T05:16:50.515 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local local_status_log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_20.status 2023-11-08T05:16:50.515 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ mirror_image_snapshot cluster2 mirror image_20 2023-11-08T05:16:50.515 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:16:50.515 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:16:50.515 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_20 2023-11-08T05:16:50.516 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_20 2023-11-08T05:16:51.267 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:Snapshot ID: 157 2023-11-08T05:16:51.276 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_20 /tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_20.status 2023-11-08T05:16:51.277 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:16:51.277 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:16:51.277 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_20 2023-11-08T05:16:51.278 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_20.status 2023-11-08T05:16:51.278 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_20 --format xml 2023-11-08T05:16:51.278 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T05:16:51.316 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-11-08T05:16:51.318 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snapshot_id=157 2023-11-08T05:16:51.318 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ true 2023-11-08T05:16:51.319 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-11-08T05:16:51.319 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0.2 2023-11-08T05:16:51.520 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_20 /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_20.status 2023-11-08T05:16:51.520 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:16:51.520 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:16:51.520 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_20 2023-11-08T05:16:51.521 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_20.status 2023-11-08T05:16:51.521 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_20 --format xml 2023-11-08T05:16:51.521 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T05:16:51.560 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-08T05:16:51.562 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local primary_snapshot_id=87 2023-11-08T05:16:51.563 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test 157 = 87 2023-11-08T05:16:51.563 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-11-08T05:16:51.563 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0.4 2023-11-08T05:16:51.964 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_20 /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_20.status 2023-11-08T05:16:51.964 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:16:51.964 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:16:51.965 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_20 2023-11-08T05:16:51.965 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_20.status 2023-11-08T05:16:51.965 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_20 --format xml 2023-11-08T05:16:51.965 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T05:16:52.007 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ true 2023-11-08T05:16:52.007 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-08T05:16:52.008 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:-:1.1: Document is empty 2023-11-08T05:16:52.008 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: 2023-11-08T05:16:52.009 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:^ 2023-11-08T05:16:52.009 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local primary_snapshot_id= 2023-11-08T05:16:52.010 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test 157 = '' 2023-11-08T05:16:52.010 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-11-08T05:16:52.010 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0.8 2023-11-08T05:16:52.810 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_20 /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_20.status 2023-11-08T05:16:52.811 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:16:52.811 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:16:52.811 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_20 2023-11-08T05:16:52.811 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_20.status 2023-11-08T05:16:52.812 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_20 --format xml 2023-11-08T05:16:52.812 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T05:16:52.850 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-08T05:16:52.852 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local primary_snapshot_id=157 2023-11-08T05:16:52.852 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test 157 = 157 2023-11-08T05:16:52.852 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:16:52.852 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_snap_present cluster1 mirror image_20 snap 2023-11-08T05:16:52.853 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:16:52.853 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:16:52.853 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_20 2023-11-08T05:16:52.853 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap_name=snap 2023-11-08T05:16:52.853 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local s 2023-11-08T05:16:52.854 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-11-08T05:16:52.854 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 1 2023-11-08T05:16:53.853 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 info mirror/image_20@snap 2023-11-08T05:16:53.890 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:rbd image 'image_20': 2023-11-08T05:16:53.891 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: size 128 MiB in 32 objects 2023-11-08T05:16:53.891 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: order 22 (4 MiB objects) 2023-11-08T05:16:53.891 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: snapshot_count: 2 2023-11-08T05:16:53.892 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: id: 128792070bc1 2023-11-08T05:16:53.892 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: block_name_prefix: rbd_data.128792070bc1 2023-11-08T05:16:53.892 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: format: 2 2023-11-08T05:16:53.893 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: features: layering, exclusive-lock, non-primary 2023-11-08T05:16:53.893 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: op_features: 2023-11-08T05:16:53.893 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: flags: 2023-11-08T05:16:53.894 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: create_timestamp: Wed Nov 8 05:07:56 2023 2023-11-08T05:16:53.894 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: access_timestamp: Wed Nov 8 05:07:56 2023 2023-11-08T05:16:53.895 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: modify_timestamp: Wed Nov 8 05:07:56 2023 2023-11-08T05:16:53.895 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: protected: False 2023-11-08T05:16:53.895 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring state: enabled 2023-11-08T05:16:53.895 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring mode: snapshot 2023-11-08T05:16:53.895 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring global id: d1ee9201-2afc-4853-a96a-0fbcc08cbfe5 2023-11-08T05:16:53.896 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring primary: false 2023-11-08T05:16:53.897 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:16:53.897 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ compare_image_snaps mirror image_20 snap 2023-11-08T05:16:53.898 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:16:53.898 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_20 2023-11-08T05:16:53.898 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap_name=snap 2023-11-08T05:16:53.898 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local ret=0 2023-11-08T05:16:53.899 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local rmt_export=/tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_20.export 2023-11-08T05:16:53.899 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local loc_export=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_20.export 2023-11-08T05:16:53.899 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rm -f /tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_20.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_20.export 2023-11-08T05:16:53.899 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 -p mirror export image_20@snap /tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_20.export 2023-11-08T05:16:54.072 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-11-08T05:16:54.075 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 -p mirror export image_20@snap /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_20.export 2023-11-08T05:16:54.259 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-11-08T05:16:54.262 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ cmp /tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_20.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_20.export 2023-11-08T05:16:54.317 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rm -f /tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_20.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_20.export 2023-11-08T05:16:54.354 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:16:54.354 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-11-08T05:16:54.354 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image=image_21 2023-11-08T05:16:54.355 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ create_snap cluster2 mirror image_21 snap 2023-11-08T05:16:54.355 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:16:54.355 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:16:54.355 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_21 2023-11-08T05:16:54.355 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap_name=snap 2023-11-08T05:16:54.356 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 -p mirror snap create image_21@snap --debug-rbd=20 --debug-journaler=20 2023-11-08T05:16:56.278 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_image_replay_started cluster1 mirror image_21 2023-11-08T05:16:56.279 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:16:56.279 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:16:56.279 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_21 2023-11-08T05:16:56.280 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_image_replay_state cluster1 mirror image_21 started 2023-11-08T05:16:56.280 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:16:56.280 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:16:56.280 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_21 2023-11-08T05:16:56.280 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local state=started 2023-11-08T05:16:56.281 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local s 2023-11-08T05:16:56.281 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-11-08T05:16:56.281 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 1 2023-11-08T05:16:57.280 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test_image_replay_state cluster1 mirror image_21 started 2023-11-08T05:16:57.280 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:16:57.280 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:16:57.280 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_21 2023-11-08T05:16:57.281 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local test_state=started 2023-11-08T05:16:57.281 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local status_result 2023-11-08T05:16:57.281 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local current_state=stopped 2023-11-08T05:16:57.282 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_21 2023-11-08T05:16:57.282 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_instance=cluster1 2023-11-08T05:16:57.283 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ grep -i state 2023-11-08T05:16:57.283 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T05:16:57.283 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1 2023-11-08T05:16:57.283 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance=cluster1 2023-11-08T05:16:57.284 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local loop_instance 2023-11-08T05:16:57.284 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-11-08T05:16:57.284 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sleep 0 2023-11-08T05:16:57.285 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-11-08T05:16:57.285 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ seq 0 3 2023-11-08T05:16:57.288 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T05:16:57.288 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_21 2023-11-08T05:16:57.289 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1:0 2023-11-08T05:16:57.289 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T05:16:57.289 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance 2023-11-08T05:16:57.289 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-11-08T05:16:57.289 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local val=cluster1:0 2023-11-08T05:16:57.290 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_var_name=cluster 2023-11-08T05:16:57.290 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance_var_name=instance 2023-11-08T05:16:57.290 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cluster=cluster1 2023-11-08T05:16:57.290 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ instance=0 2023-11-08T05:16:57.291 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' 0 = cluster1:0 ']' 2023-11-08T05:16:57.291 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval cluster=cluster1 2023-11-08T05:16:57.291 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:16:57.291 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval instance=0 2023-11-08T05:16:57.292 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=0 2023-11-08T05:16:57.292 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-11-08T05:16:57.292 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local local_cluster=cluster1:0 2023-11-08T05:16:57.293 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster=cluster1 2023-11-08T05:16:57.293 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance 2023-11-08T05:16:57.293 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-11-08T05:16:57.293 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local val=cluster1:0 2023-11-08T05:16:57.293 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster_var_name=local_cluster 2023-11-08T05:16:57.294 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance_var_name=instance 2023-11-08T05:16:57.294 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:16:57.294 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=0 2023-11-08T05:16:57.294 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ '[' 0 = cluster1:0 ']' 2023-11-08T05:16:57.295 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval local_cluster=cluster1 2023-11-08T05:16:57.295 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ local_cluster=cluster1 2023-11-08T05:16:57.295 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval instance=0 2023-11-08T05:16:57.296 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ instance=0 2023-11-08T05:16:57.296 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-11-08T05:16:57.302 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-08T05:16:57.302 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-08T05:16:57.303 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-08T05:16:57.303 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_21 2023-11-08T05:16:57.402 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:Invalid command: unused arguments: ['mirror/image_21'] 2023-11-08T05:16:57.403 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd mirror status : get status for rbd mirror 2023-11-08T05:16:57.403 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:admin_socket: invalid command 2023-11-08T05:16:57.406 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T05:16:57.407 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_21 2023-11-08T05:16:57.407 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1:1 2023-11-08T05:16:57.407 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T05:16:57.407 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance 2023-11-08T05:16:57.408 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-11-08T05:16:57.408 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local val=cluster1:1 2023-11-08T05:16:57.408 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_var_name=cluster 2023-11-08T05:16:57.409 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance_var_name=instance 2023-11-08T05:16:57.409 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cluster=cluster1 2023-11-08T05:16:57.409 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ instance=1 2023-11-08T05:16:57.409 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' 1 = cluster1:1 ']' 2023-11-08T05:16:57.409 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval cluster=cluster1 2023-11-08T05:16:57.410 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:16:57.410 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval instance=1 2023-11-08T05:16:57.410 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=1 2023-11-08T05:16:57.411 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-11-08T05:16:57.411 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local local_cluster=cluster1:1 2023-11-08T05:16:57.411 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster=cluster1 2023-11-08T05:16:57.412 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance 2023-11-08T05:16:57.412 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-11-08T05:16:57.412 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local val=cluster1:1 2023-11-08T05:16:57.413 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster_var_name=local_cluster 2023-11-08T05:16:57.413 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance_var_name=instance 2023-11-08T05:16:57.413 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:16:57.414 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=1 2023-11-08T05:16:57.414 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ '[' 1 = cluster1:1 ']' 2023-11-08T05:16:57.415 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval local_cluster=cluster1 2023-11-08T05:16:57.415 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ local_cluster=cluster1 2023-11-08T05:16:57.415 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval instance=1 2023-11-08T05:16:57.416 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ instance=1 2023-11-08T05:16:57.416 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-11-08T05:16:57.420 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-08T05:16:57.420 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-08T05:16:57.421 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-08T05:16:57.421 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_21 2023-11-08T05:16:57.522 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:Invalid command: unused arguments: ['mirror/image_21'] 2023-11-08T05:16:57.522 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd mirror status : get status for rbd mirror 2023-11-08T05:16:57.522 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:admin_socket: invalid command 2023-11-08T05:16:57.526 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T05:16:57.527 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_21 2023-11-08T05:16:57.527 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1:2 2023-11-08T05:16:57.527 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T05:16:57.527 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance 2023-11-08T05:16:57.528 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-11-08T05:16:57.528 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local val=cluster1:2 2023-11-08T05:16:57.528 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_var_name=cluster 2023-11-08T05:16:57.529 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance_var_name=instance 2023-11-08T05:16:57.529 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cluster=cluster1 2023-11-08T05:16:57.529 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ instance=2 2023-11-08T05:16:57.529 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' 2 = cluster1:2 ']' 2023-11-08T05:16:57.529 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval cluster=cluster1 2023-11-08T05:16:57.529 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:16:57.530 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval instance=2 2023-11-08T05:16:57.530 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=2 2023-11-08T05:16:57.530 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-11-08T05:16:57.530 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local local_cluster=cluster1:2 2023-11-08T05:16:57.530 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster=cluster1 2023-11-08T05:16:57.531 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance 2023-11-08T05:16:57.531 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-11-08T05:16:57.531 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local val=cluster1:2 2023-11-08T05:16:57.531 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster_var_name=local_cluster 2023-11-08T05:16:57.531 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance_var_name=instance 2023-11-08T05:16:57.532 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:16:57.532 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=2 2023-11-08T05:16:57.532 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ '[' 2 = cluster1:2 ']' 2023-11-08T05:16:57.532 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval local_cluster=cluster1 2023-11-08T05:16:57.533 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ local_cluster=cluster1 2023-11-08T05:16:57.533 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval instance=2 2023-11-08T05:16:57.533 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ instance=2 2023-11-08T05:16:57.533 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-11-08T05:16:57.539 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-08T05:16:57.540 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-08T05:16:57.540 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-08T05:16:57.540 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_21 2023-11-08T05:16:57.646 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ return 0 2023-11-08T05:16:57.647 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ status_result=' "state": "Replaying"' 2023-11-08T05:16:57.647 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ echo ' "state": "Replaying"' 2023-11-08T05:16:57.648 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ grep -i Replaying 2023-11-08T05:16:57.648 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: "state": "Replaying" 2023-11-08T05:16:57.650 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ current_state=started 2023-11-08T05:16:57.650 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test started = started 2023-11-08T05:16:57.651 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:16:57.651 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_21 2023-11-08T05:16:57.651 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local local_cluster=cluster1 2023-11-08T05:16:57.651 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:16:57.652 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:16:57.652 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_21 2023-11-08T05:16:57.652 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' snapshot = journal ']' 2023-11-08T05:16:57.652 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' snapshot = snapshot ']' 2023-11-08T05:16:57.652 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_21 2023-11-08T05:16:57.653 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local local_cluster=cluster1 2023-11-08T05:16:57.653 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:16:57.653 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:16:57.653 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_21 2023-11-08T05:16:57.653 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ mkfname cluster2-mirror-image_21.status 2023-11-08T05:16:57.653 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ echo cluster2-mirror-image_21.status 2023-11-08T05:16:57.654 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -e 's|[/ ]|_|g' 2023-11-08T05:16:57.654 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local status_log=/tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_21.status 2023-11-08T05:16:57.654 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ mkfname cluster1-mirror-image_21.status 2023-11-08T05:16:57.654 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ echo cluster1-mirror-image_21.status 2023-11-08T05:16:57.654 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -e 's|[/ ]|_|g' 2023-11-08T05:16:57.655 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local local_status_log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_21.status 2023-11-08T05:16:57.656 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ mirror_image_snapshot cluster2 mirror image_21 2023-11-08T05:16:57.656 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:16:57.656 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:16:57.656 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_21 2023-11-08T05:16:57.656 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_21 2023-11-08T05:16:58.286 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:Snapshot ID: 159 2023-11-08T05:16:58.295 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_21 /tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_21.status 2023-11-08T05:16:58.295 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:16:58.296 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:16:58.296 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_21 2023-11-08T05:16:58.296 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_21.status 2023-11-08T05:16:58.296 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_21 --format xml 2023-11-08T05:16:58.296 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T05:16:58.334 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-11-08T05:16:58.336 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snapshot_id=159 2023-11-08T05:16:58.336 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ true 2023-11-08T05:16:58.336 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-11-08T05:16:58.337 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0.2 2023-11-08T05:16:58.537 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_21 /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_21.status 2023-11-08T05:16:58.538 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:16:58.538 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:16:58.538 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_21 2023-11-08T05:16:58.538 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_21.status 2023-11-08T05:16:58.538 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_21 --format xml 2023-11-08T05:16:58.539 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T05:16:58.576 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ true 2023-11-08T05:16:58.577 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-08T05:16:58.578 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:-:1.1: Document is empty 2023-11-08T05:16:58.578 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: 2023-11-08T05:16:58.578 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:^ 2023-11-08T05:16:58.579 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local primary_snapshot_id= 2023-11-08T05:16:58.579 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test 159 = '' 2023-11-08T05:16:58.579 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-11-08T05:16:58.580 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0.4 2023-11-08T05:16:58.980 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_21 /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_21.status 2023-11-08T05:16:58.980 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:16:58.980 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:16:58.980 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_21 2023-11-08T05:16:58.981 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_21.status 2023-11-08T05:16:58.981 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_21 --format xml 2023-11-08T05:16:58.981 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T05:16:59.019 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ true 2023-11-08T05:16:59.019 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-08T05:16:59.020 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:-:1.1: Document is empty 2023-11-08T05:16:59.020 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: 2023-11-08T05:16:59.020 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:^ 2023-11-08T05:16:59.021 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local primary_snapshot_id= 2023-11-08T05:16:59.021 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test 159 = '' 2023-11-08T05:16:59.022 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-11-08T05:16:59.022 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0.8 2023-11-08T05:16:59.822 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_21 /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_21.status 2023-11-08T05:16:59.823 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:16:59.823 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:16:59.823 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_21 2023-11-08T05:16:59.823 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_21.status 2023-11-08T05:16:59.823 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_21 --format xml 2023-11-08T05:16:59.823 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T05:16:59.860 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-08T05:16:59.862 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local primary_snapshot_id=159 2023-11-08T05:16:59.863 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test 159 = 159 2023-11-08T05:16:59.863 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:16:59.863 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_snap_present cluster1 mirror image_21 snap 2023-11-08T05:16:59.863 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:16:59.863 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:16:59.864 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_21 2023-11-08T05:16:59.864 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap_name=snap 2023-11-08T05:16:59.864 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local s 2023-11-08T05:16:59.864 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-11-08T05:16:59.864 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 1 2023-11-08T05:17:00.863 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 info mirror/image_21@snap 2023-11-08T05:17:00.903 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:rbd image 'image_21': 2023-11-08T05:17:00.903 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: size 128 MiB in 32 objects 2023-11-08T05:17:00.903 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: order 22 (4 MiB objects) 2023-11-08T05:17:00.903 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: snapshot_count: 2 2023-11-08T05:17:00.904 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: id: 10e7b8e37f93 2023-11-08T05:17:00.904 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: block_name_prefix: rbd_data.10e7b8e37f93 2023-11-08T05:17:00.904 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: format: 2 2023-11-08T05:17:00.904 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: features: layering, exclusive-lock, non-primary 2023-11-08T05:17:00.904 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: op_features: 2023-11-08T05:17:00.905 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: flags: 2023-11-08T05:17:00.905 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: create_timestamp: Wed Nov 8 05:08:06 2023 2023-11-08T05:17:00.905 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: access_timestamp: Wed Nov 8 05:08:06 2023 2023-11-08T05:17:00.905 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: modify_timestamp: Wed Nov 8 05:08:06 2023 2023-11-08T05:17:00.905 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: protected: False 2023-11-08T05:17:00.906 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring state: enabled 2023-11-08T05:17:00.906 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring mode: snapshot 2023-11-08T05:17:00.906 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring global id: 1d89653a-15af-4c63-92d4-196f96656cd3 2023-11-08T05:17:00.906 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring primary: false 2023-11-08T05:17:00.908 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:17:00.908 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ compare_image_snaps mirror image_21 snap 2023-11-08T05:17:00.908 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:17:00.908 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_21 2023-11-08T05:17:00.908 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap_name=snap 2023-11-08T05:17:00.909 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local ret=0 2023-11-08T05:17:00.909 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local rmt_export=/tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_21.export 2023-11-08T05:17:00.909 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local loc_export=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_21.export 2023-11-08T05:17:00.909 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rm -f /tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_21.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_21.export 2023-11-08T05:17:00.910 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 -p mirror export image_21@snap /tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_21.export 2023-11-08T05:17:01.080 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-11-08T05:17:01.083 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 -p mirror export image_21@snap /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_21.export 2023-11-08T05:17:01.314 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-11-08T05:17:01.317 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ cmp /tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_21.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_21.export 2023-11-08T05:17:01.372 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rm -f /tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_21.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_21.export 2023-11-08T05:17:01.410 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:17:01.410 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-11-08T05:17:01.410 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image=image_22 2023-11-08T05:17:01.411 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ create_snap cluster2 mirror image_22 snap 2023-11-08T05:17:01.411 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:17:01.411 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:17:01.411 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_22 2023-11-08T05:17:01.412 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap_name=snap 2023-11-08T05:17:01.412 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 -p mirror snap create image_22@snap --debug-rbd=20 --debug-journaler=20 2023-11-08T05:17:03.293 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_image_replay_started cluster1 mirror image_22 2023-11-08T05:17:03.294 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:17:03.294 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:17:03.294 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_22 2023-11-08T05:17:03.294 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_image_replay_state cluster1 mirror image_22 started 2023-11-08T05:17:03.295 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:17:03.295 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:17:03.295 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_22 2023-11-08T05:17:03.295 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local state=started 2023-11-08T05:17:03.296 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local s 2023-11-08T05:17:03.296 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-11-08T05:17:03.296 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 1 2023-11-08T05:17:04.294 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test_image_replay_state cluster1 mirror image_22 started 2023-11-08T05:17:04.294 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:17:04.295 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:17:04.295 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_22 2023-11-08T05:17:04.295 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local test_state=started 2023-11-08T05:17:04.295 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local status_result 2023-11-08T05:17:04.295 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local current_state=stopped 2023-11-08T05:17:04.297 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_22 2023-11-08T05:17:04.297 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_instance=cluster1 2023-11-08T05:17:04.297 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ grep -i state 2023-11-08T05:17:04.297 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T05:17:04.298 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1 2023-11-08T05:17:04.298 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance=cluster1 2023-11-08T05:17:04.298 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local loop_instance 2023-11-08T05:17:04.299 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-11-08T05:17:04.299 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sleep 0 2023-11-08T05:17:04.299 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-11-08T05:17:04.300 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ seq 0 3 2023-11-08T05:17:04.302 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T05:17:04.302 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_22 2023-11-08T05:17:04.302 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1:0 2023-11-08T05:17:04.302 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T05:17:04.303 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance 2023-11-08T05:17:04.303 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-11-08T05:17:04.303 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local val=cluster1:0 2023-11-08T05:17:04.303 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_var_name=cluster 2023-11-08T05:17:04.303 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance_var_name=instance 2023-11-08T05:17:04.304 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cluster=cluster1 2023-11-08T05:17:04.304 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ instance=0 2023-11-08T05:17:04.304 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' 0 = cluster1:0 ']' 2023-11-08T05:17:04.304 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval cluster=cluster1 2023-11-08T05:17:04.304 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:17:04.304 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval instance=0 2023-11-08T05:17:04.305 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=0 2023-11-08T05:17:04.305 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-11-08T05:17:04.305 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local local_cluster=cluster1:0 2023-11-08T05:17:04.305 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster=cluster1 2023-11-08T05:17:04.305 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance 2023-11-08T05:17:04.306 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-11-08T05:17:04.306 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local val=cluster1:0 2023-11-08T05:17:04.306 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster_var_name=local_cluster 2023-11-08T05:17:04.307 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance_var_name=instance 2023-11-08T05:17:04.307 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:17:04.308 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=0 2023-11-08T05:17:04.308 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ '[' 0 = cluster1:0 ']' 2023-11-08T05:17:04.308 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval local_cluster=cluster1 2023-11-08T05:17:04.308 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ local_cluster=cluster1 2023-11-08T05:17:04.308 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval instance=0 2023-11-08T05:17:04.309 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ instance=0 2023-11-08T05:17:04.309 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-11-08T05:17:04.316 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-08T05:17:04.316 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-08T05:17:04.316 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-08T05:17:04.317 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_22 2023-11-08T05:17:04.414 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:Invalid command: unused arguments: ['mirror/image_22'] 2023-11-08T05:17:04.414 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd mirror status : get status for rbd mirror 2023-11-08T05:17:04.415 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:admin_socket: invalid command 2023-11-08T05:17:04.418 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T05:17:04.419 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_22 2023-11-08T05:17:04.419 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1:1 2023-11-08T05:17:04.420 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T05:17:04.420 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance 2023-11-08T05:17:04.420 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-11-08T05:17:04.421 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local val=cluster1:1 2023-11-08T05:17:04.421 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_var_name=cluster 2023-11-08T05:17:04.421 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance_var_name=instance 2023-11-08T05:17:04.421 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cluster=cluster1 2023-11-08T05:17:04.422 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ instance=1 2023-11-08T05:17:04.422 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' 1 = cluster1:1 ']' 2023-11-08T05:17:04.422 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval cluster=cluster1 2023-11-08T05:17:04.423 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:17:04.423 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval instance=1 2023-11-08T05:17:04.423 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=1 2023-11-08T05:17:04.424 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-11-08T05:17:04.424 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local local_cluster=cluster1:1 2023-11-08T05:17:04.424 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster=cluster1 2023-11-08T05:17:04.424 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance 2023-11-08T05:17:04.425 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-11-08T05:17:04.425 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local val=cluster1:1 2023-11-08T05:17:04.425 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster_var_name=local_cluster 2023-11-08T05:17:04.426 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance_var_name=instance 2023-11-08T05:17:04.426 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:17:04.426 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=1 2023-11-08T05:17:04.426 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ '[' 1 = cluster1:1 ']' 2023-11-08T05:17:04.427 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval local_cluster=cluster1 2023-11-08T05:17:04.427 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ local_cluster=cluster1 2023-11-08T05:17:04.427 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval instance=1 2023-11-08T05:17:04.428 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ instance=1 2023-11-08T05:17:04.428 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-11-08T05:17:04.431 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-08T05:17:04.432 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-08T05:17:04.432 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-08T05:17:04.432 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_22 2023-11-08T05:17:04.529 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:Invalid command: unused arguments: ['mirror/image_22'] 2023-11-08T05:17:04.529 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd mirror status : get status for rbd mirror 2023-11-08T05:17:04.529 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:admin_socket: invalid command 2023-11-08T05:17:04.533 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T05:17:04.533 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_22 2023-11-08T05:17:04.534 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1:2 2023-11-08T05:17:04.534 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T05:17:04.534 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance 2023-11-08T05:17:04.534 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-11-08T05:17:04.534 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local val=cluster1:2 2023-11-08T05:17:04.535 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_var_name=cluster 2023-11-08T05:17:04.535 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance_var_name=instance 2023-11-08T05:17:04.535 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cluster=cluster1 2023-11-08T05:17:04.535 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ instance=2 2023-11-08T05:17:04.536 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' 2 = cluster1:2 ']' 2023-11-08T05:17:04.536 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval cluster=cluster1 2023-11-08T05:17:04.536 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:17:04.536 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval instance=2 2023-11-08T05:17:04.537 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=2 2023-11-08T05:17:04.537 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-11-08T05:17:04.537 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local local_cluster=cluster1:2 2023-11-08T05:17:04.537 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster=cluster1 2023-11-08T05:17:04.537 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance 2023-11-08T05:17:04.538 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-11-08T05:17:04.538 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local val=cluster1:2 2023-11-08T05:17:04.538 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster_var_name=local_cluster 2023-11-08T05:17:04.538 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance_var_name=instance 2023-11-08T05:17:04.538 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:17:04.539 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=2 2023-11-08T05:17:04.539 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ '[' 2 = cluster1:2 ']' 2023-11-08T05:17:04.539 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval local_cluster=cluster1 2023-11-08T05:17:04.539 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ local_cluster=cluster1 2023-11-08T05:17:04.539 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval instance=2 2023-11-08T05:17:04.540 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ instance=2 2023-11-08T05:17:04.540 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-11-08T05:17:04.545 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-08T05:17:04.546 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-08T05:17:04.546 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-08T05:17:04.546 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_22 2023-11-08T05:17:04.645 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:Invalid command: unused arguments: ['mirror/image_22'] 2023-11-08T05:17:04.645 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd mirror status : get status for rbd mirror 2023-11-08T05:17:04.645 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:admin_socket: invalid command 2023-11-08T05:17:04.648 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T05:17:04.648 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_22 2023-11-08T05:17:04.648 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1:3 2023-11-08T05:17:04.649 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T05:17:04.649 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance 2023-11-08T05:17:04.649 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ set_cluster_instance cluster1:3 cluster instance 2023-11-08T05:17:04.649 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local val=cluster1:3 2023-11-08T05:17:04.649 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_var_name=cluster 2023-11-08T05:17:04.650 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance_var_name=instance 2023-11-08T05:17:04.650 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cluster=cluster1 2023-11-08T05:17:04.650 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ instance=3 2023-11-08T05:17:04.650 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' 3 = cluster1:3 ']' 2023-11-08T05:17:04.650 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval cluster=cluster1 2023-11-08T05:17:04.651 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:17:04.651 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval instance=3 2023-11-08T05:17:04.651 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=3 2023-11-08T05:17:04.651 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ daemon_asok_file cluster1:3 cluster1 2023-11-08T05:17:04.651 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local local_cluster=cluster1:3 2023-11-08T05:17:04.652 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster=cluster1 2023-11-08T05:17:04.652 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance 2023-11-08T05:17:04.652 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2023-11-08T05:17:04.652 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local val=cluster1:3 2023-11-08T05:17:04.652 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster_var_name=local_cluster 2023-11-08T05:17:04.653 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance_var_name=instance 2023-11-08T05:17:04.653 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:17:04.653 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=3 2023-11-08T05:17:04.653 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ '[' 3 = cluster1:3 ']' 2023-11-08T05:17:04.653 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval local_cluster=cluster1 2023-11-08T05:17:04.654 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ local_cluster=cluster1 2023-11-08T05:17:04.654 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval instance=3 2023-11-08T05:17:04.654 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ instance=3 2023-11-08T05:17:04.654 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2023-11-08T05:17:04.661 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-11-08T05:17:04.662 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-11-08T05:17:04.662 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-11-08T05:17:04.662 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_22 2023-11-08T05:17:04.770 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ return 0 2023-11-08T05:17:04.771 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ status_result=' "state": "Replaying"' 2023-11-08T05:17:04.771 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ echo ' "state": "Replaying"' 2023-11-08T05:17:04.771 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ grep -i Replaying 2023-11-08T05:17:04.772 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: "state": "Replaying" 2023-11-08T05:17:04.774 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ current_state=started 2023-11-08T05:17:04.774 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test started = started 2023-11-08T05:17:04.774 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:17:04.774 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_22 2023-11-08T05:17:04.775 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local local_cluster=cluster1 2023-11-08T05:17:04.775 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:17:04.775 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:17:04.775 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_22 2023-11-08T05:17:04.775 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' snapshot = journal ']' 2023-11-08T05:17:04.775 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' snapshot = snapshot ']' 2023-11-08T05:17:04.776 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_22 2023-11-08T05:17:04.776 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local local_cluster=cluster1 2023-11-08T05:17:04.776 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:17:04.776 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:17:04.776 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_22 2023-11-08T05:17:04.777 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ mkfname cluster2-mirror-image_22.status 2023-11-08T05:17:04.777 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ echo cluster2-mirror-image_22.status 2023-11-08T05:17:04.777 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -e 's|[/ ]|_|g' 2023-11-08T05:17:04.777 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local status_log=/tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_22.status 2023-11-08T05:17:04.777 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ mkfname cluster1-mirror-image_22.status 2023-11-08T05:17:04.778 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ echo cluster1-mirror-image_22.status 2023-11-08T05:17:04.778 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -e 's|[/ ]|_|g' 2023-11-08T05:17:04.779 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local local_status_log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_22.status 2023-11-08T05:17:04.779 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ mirror_image_snapshot cluster2 mirror image_22 2023-11-08T05:17:04.779 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:17:04.779 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:17:04.779 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_22 2023-11-08T05:17:04.780 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_22 2023-11-08T05:17:05.299 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:Snapshot ID: 161 2023-11-08T05:17:05.308 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_22 /tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_22.status 2023-11-08T05:17:05.309 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:17:05.309 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:17:05.309 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_22 2023-11-08T05:17:05.309 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_22.status 2023-11-08T05:17:05.309 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_22 --format xml 2023-11-08T05:17:05.310 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T05:17:05.348 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-11-08T05:17:05.350 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snapshot_id=161 2023-11-08T05:17:05.350 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ true 2023-11-08T05:17:05.351 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-11-08T05:17:05.351 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0.2 2023-11-08T05:17:05.552 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_22 /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_22.status 2023-11-08T05:17:05.552 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:17:05.552 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:17:05.553 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_22 2023-11-08T05:17:05.553 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_22.status 2023-11-08T05:17:05.553 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_22 --format xml 2023-11-08T05:17:05.553 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T05:17:05.592 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ true 2023-11-08T05:17:05.593 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-08T05:17:05.593 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:-:1.1: Document is empty 2023-11-08T05:17:05.594 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: 2023-11-08T05:17:05.594 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:^ 2023-11-08T05:17:05.596 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local primary_snapshot_id= 2023-11-08T05:17:05.596 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test 161 = '' 2023-11-08T05:17:05.596 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-11-08T05:17:05.597 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0.4 2023-11-08T05:17:05.996 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_22 /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_22.status 2023-11-08T05:17:05.996 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:17:05.996 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:17:05.997 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_22 2023-11-08T05:17:05.997 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_22.status 2023-11-08T05:17:05.997 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_22 --format xml 2023-11-08T05:17:05.997 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T05:17:06.038 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ true 2023-11-08T05:17:06.038 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-08T05:17:06.039 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:-:1.1: Document is empty 2023-11-08T05:17:06.039 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: 2023-11-08T05:17:06.039 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:^ 2023-11-08T05:17:06.040 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local primary_snapshot_id= 2023-11-08T05:17:06.040 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test 161 = '' 2023-11-08T05:17:06.041 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-11-08T05:17:06.041 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0.8 2023-11-08T05:17:06.841 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_22 /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_22.status 2023-11-08T05:17:06.841 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:17:06.842 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:17:06.842 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_22 2023-11-08T05:17:06.842 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_22.status 2023-11-08T05:17:06.842 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_22 --format xml 2023-11-08T05:17:06.843 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T05:17:06.880 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-08T05:17:06.882 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local primary_snapshot_id=161 2023-11-08T05:17:06.882 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test 161 = 161 2023-11-08T05:17:06.882 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:17:06.883 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_snap_present cluster1 mirror image_22 snap 2023-11-08T05:17:06.883 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:17:06.883 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:17:06.883 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_22 2023-11-08T05:17:06.883 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap_name=snap 2023-11-08T05:17:06.884 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local s 2023-11-08T05:17:06.884 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-11-08T05:17:06.884 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 1 2023-11-08T05:17:07.883 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 info mirror/image_22@snap 2023-11-08T05:17:07.919 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:rbd image 'image_22': 2023-11-08T05:17:07.919 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: size 128 MiB in 32 objects 2023-11-08T05:17:07.919 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: order 22 (4 MiB objects) 2023-11-08T05:17:07.919 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: snapshot_count: 2 2023-11-08T05:17:07.920 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: id: 11122e04051 2023-11-08T05:17:07.920 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: block_name_prefix: rbd_data.11122e04051 2023-11-08T05:17:07.920 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: format: 2 2023-11-08T05:17:07.920 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: features: layering, exclusive-lock, non-primary 2023-11-08T05:17:07.920 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: op_features: 2023-11-08T05:17:07.921 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: flags: 2023-11-08T05:17:07.921 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: create_timestamp: Wed Nov 8 05:08:19 2023 2023-11-08T05:17:07.921 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: access_timestamp: Wed Nov 8 05:08:19 2023 2023-11-08T05:17:07.921 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: modify_timestamp: Wed Nov 8 05:08:19 2023 2023-11-08T05:17:07.922 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: protected: False 2023-11-08T05:17:07.922 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring state: enabled 2023-11-08T05:17:07.922 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring mode: snapshot 2023-11-08T05:17:07.922 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring global id: b8cf32bf-f325-43e1-a374-ee759365a5a1 2023-11-08T05:17:07.922 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring primary: false 2023-11-08T05:17:07.924 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:17:07.924 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ compare_image_snaps mirror image_22 snap 2023-11-08T05:17:07.925 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:17:07.925 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_22 2023-11-08T05:17:07.925 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap_name=snap 2023-11-08T05:17:07.925 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local ret=0 2023-11-08T05:17:07.926 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local rmt_export=/tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_22.export 2023-11-08T05:17:07.926 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local loc_export=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_22.export 2023-11-08T05:17:07.926 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rm -f /tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_22.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_22.export 2023-11-08T05:17:07.926 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 -p mirror export image_22@snap /tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_22.export 2023-11-08T05:17:08.101 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-11-08T05:17:08.103 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 -p mirror export image_22@snap /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_22.export 2023-11-08T05:17:08.338 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-11-08T05:17:08.341 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ cmp /tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_22.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_22.export 2023-11-08T05:17:08.396 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rm -f /tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_22.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_22.export 2023-11-08T05:17:08.433 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:17:08.434 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-11-08T05:17:08.434 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image=image_23 2023-11-08T05:17:08.435 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ create_snap cluster2 mirror image_23 snap 2023-11-08T05:17:08.435 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:17:08.435 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:17:08.436 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_23 2023-11-08T05:17:08.436 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap_name=snap 2023-11-08T05:17:08.436 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 -p mirror snap create image_23@snap --debug-rbd=20 --debug-journaler=20 2023-11-08T05:17:10.314 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_image_replay_started cluster1 mirror image_23 2023-11-08T05:17:10.315 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:17:10.315 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:17:10.315 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_23 2023-11-08T05:17:10.315 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_image_replay_state cluster1 mirror image_23 started 2023-11-08T05:17:10.316 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:17:10.316 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:17:10.316 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_23 2023-11-08T05:17:10.316 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local state=started 2023-11-08T05:17:10.316 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local s 2023-11-08T05:17:10.317 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-11-08T05:17:10.317 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 1 2023-11-08T05:17:11.316 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test_image_replay_state cluster1 mirror image_23 started 2023-11-08T05:17:11.316 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:17:11.317 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:17:11.317 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_23 2023-11-08T05:17:11.317 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local test_state=started 2023-11-08T05:17:11.317 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local status_result 2023-11-08T05:17:11.318 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local current_state=stopped 2023-11-08T05:17:11.318 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_23 2023-11-08T05:17:11.318 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_instance=cluster1 2023-11-08T05:17:11.318 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T05:17:11.318 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1 2023-11-08T05:17:11.318 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance=cluster1 2023-11-08T05:17:11.319 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ grep -i state 2023-11-08T05:17:11.319 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local loop_instance 2023-11-08T05:17:11.319 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-11-08T05:17:11.319 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sleep 0 2023-11-08T05:17:11.322 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-11-08T05:17:11.323 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ seq 0 3 2023-11-08T05:17:11.323 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T05:17:11.323 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_23 2023-11-08T05:17:11.323 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1:0 2023-11-08T05:17:11.324 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T05:17:11.324 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance 2023-11-08T05:17:11.324 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-11-08T05:17:11.324 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local val=cluster1:0 2023-11-08T05:17:11.324 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_var_name=cluster 2023-11-08T05:17:11.325 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance_var_name=instance 2023-11-08T05:17:11.325 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cluster=cluster1 2023-11-08T05:17:11.325 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ instance=0 2023-11-08T05:17:11.325 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' 0 = cluster1:0 ']' 2023-11-08T05:17:11.325 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval cluster=cluster1 2023-11-08T05:17:11.326 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:17:11.326 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval instance=0 2023-11-08T05:17:11.326 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=0 2023-11-08T05:17:11.326 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-11-08T05:17:11.327 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local local_cluster=cluster1:0 2023-11-08T05:17:11.327 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster=cluster1 2023-11-08T05:17:11.327 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance 2023-11-08T05:17:11.327 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-11-08T05:17:11.327 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local val=cluster1:0 2023-11-08T05:17:11.328 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster_var_name=local_cluster 2023-11-08T05:17:11.328 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance_var_name=instance 2023-11-08T05:17:11.328 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:17:11.328 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=0 2023-11-08T05:17:11.328 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ '[' 0 = cluster1:0 ']' 2023-11-08T05:17:11.329 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval local_cluster=cluster1 2023-11-08T05:17:11.329 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ local_cluster=cluster1 2023-11-08T05:17:11.329 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval instance=0 2023-11-08T05:17:11.329 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ instance=0 2023-11-08T05:17:11.329 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-11-08T05:17:11.338 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-08T05:17:11.338 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-08T05:17:11.338 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-08T05:17:11.338 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_23 2023-11-08T05:17:11.436 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:Invalid command: unused arguments: ['mirror/image_23'] 2023-11-08T05:17:11.436 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd mirror status : get status for rbd mirror 2023-11-08T05:17:11.437 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:admin_socket: invalid command 2023-11-08T05:17:11.440 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T05:17:11.440 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_23 2023-11-08T05:17:11.440 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1:1 2023-11-08T05:17:11.441 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T05:17:11.441 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance 2023-11-08T05:17:11.441 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-11-08T05:17:11.441 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local val=cluster1:1 2023-11-08T05:17:11.441 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_var_name=cluster 2023-11-08T05:17:11.442 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance_var_name=instance 2023-11-08T05:17:11.442 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cluster=cluster1 2023-11-08T05:17:11.442 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ instance=1 2023-11-08T05:17:11.442 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' 1 = cluster1:1 ']' 2023-11-08T05:17:11.443 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval cluster=cluster1 2023-11-08T05:17:11.443 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:17:11.443 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval instance=1 2023-11-08T05:17:11.443 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=1 2023-11-08T05:17:11.443 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-11-08T05:17:11.444 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local local_cluster=cluster1:1 2023-11-08T05:17:11.444 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster=cluster1 2023-11-08T05:17:11.444 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance 2023-11-08T05:17:11.444 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-11-08T05:17:11.444 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local val=cluster1:1 2023-11-08T05:17:11.445 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster_var_name=local_cluster 2023-11-08T05:17:11.445 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance_var_name=instance 2023-11-08T05:17:11.445 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:17:11.445 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=1 2023-11-08T05:17:11.445 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ '[' 1 = cluster1:1 ']' 2023-11-08T05:17:11.446 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval local_cluster=cluster1 2023-11-08T05:17:11.446 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ local_cluster=cluster1 2023-11-08T05:17:11.446 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval instance=1 2023-11-08T05:17:11.446 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ instance=1 2023-11-08T05:17:11.446 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-11-08T05:17:11.454 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-08T05:17:11.455 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-08T05:17:11.455 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-08T05:17:11.455 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_23 2023-11-08T05:17:11.552 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:Invalid command: unused arguments: ['mirror/image_23'] 2023-11-08T05:17:11.552 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd mirror status : get status for rbd mirror 2023-11-08T05:17:11.553 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:admin_socket: invalid command 2023-11-08T05:17:11.556 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T05:17:11.556 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_23 2023-11-08T05:17:11.556 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1:2 2023-11-08T05:17:11.556 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T05:17:11.556 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance 2023-11-08T05:17:11.557 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-11-08T05:17:11.557 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local val=cluster1:2 2023-11-08T05:17:11.557 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_var_name=cluster 2023-11-08T05:17:11.557 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance_var_name=instance 2023-11-08T05:17:11.557 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cluster=cluster1 2023-11-08T05:17:11.558 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ instance=2 2023-11-08T05:17:11.558 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' 2 = cluster1:2 ']' 2023-11-08T05:17:11.558 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval cluster=cluster1 2023-11-08T05:17:11.558 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:17:11.558 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval instance=2 2023-11-08T05:17:11.559 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=2 2023-11-08T05:17:11.559 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-11-08T05:17:11.559 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local local_cluster=cluster1:2 2023-11-08T05:17:11.559 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster=cluster1 2023-11-08T05:17:11.560 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance 2023-11-08T05:17:11.560 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-11-08T05:17:11.560 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local val=cluster1:2 2023-11-08T05:17:11.560 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster_var_name=local_cluster 2023-11-08T05:17:11.560 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance_var_name=instance 2023-11-08T05:17:11.561 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:17:11.561 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=2 2023-11-08T05:17:11.561 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ '[' 2 = cluster1:2 ']' 2023-11-08T05:17:11.561 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval local_cluster=cluster1 2023-11-08T05:17:11.561 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ local_cluster=cluster1 2023-11-08T05:17:11.562 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval instance=2 2023-11-08T05:17:11.562 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ instance=2 2023-11-08T05:17:11.562 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-11-08T05:17:11.569 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-08T05:17:11.570 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-08T05:17:11.570 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-08T05:17:11.570 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_23 2023-11-08T05:17:11.671 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:Invalid command: unused arguments: ['mirror/image_23'] 2023-11-08T05:17:11.672 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd mirror status : get status for rbd mirror 2023-11-08T05:17:11.672 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:admin_socket: invalid command 2023-11-08T05:17:11.675 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T05:17:11.675 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_23 2023-11-08T05:17:11.676 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1:3 2023-11-08T05:17:11.676 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T05:17:11.676 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance 2023-11-08T05:17:11.676 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ set_cluster_instance cluster1:3 cluster instance 2023-11-08T05:17:11.676 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local val=cluster1:3 2023-11-08T05:17:11.677 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_var_name=cluster 2023-11-08T05:17:11.677 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance_var_name=instance 2023-11-08T05:17:11.677 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cluster=cluster1 2023-11-08T05:17:11.677 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ instance=3 2023-11-08T05:17:11.677 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' 3 = cluster1:3 ']' 2023-11-08T05:17:11.678 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval cluster=cluster1 2023-11-08T05:17:11.678 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:17:11.678 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval instance=3 2023-11-08T05:17:11.678 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=3 2023-11-08T05:17:11.678 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ daemon_asok_file cluster1:3 cluster1 2023-11-08T05:17:11.679 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local local_cluster=cluster1:3 2023-11-08T05:17:11.679 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster=cluster1 2023-11-08T05:17:11.679 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance 2023-11-08T05:17:11.679 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2023-11-08T05:17:11.679 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local val=cluster1:3 2023-11-08T05:17:11.680 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster_var_name=local_cluster 2023-11-08T05:17:11.680 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance_var_name=instance 2023-11-08T05:17:11.680 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:17:11.680 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=3 2023-11-08T05:17:11.680 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ '[' 3 = cluster1:3 ']' 2023-11-08T05:17:11.681 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval local_cluster=cluster1 2023-11-08T05:17:11.681 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ local_cluster=cluster1 2023-11-08T05:17:11.681 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval instance=3 2023-11-08T05:17:11.681 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ instance=3 2023-11-08T05:17:11.681 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2023-11-08T05:17:11.689 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-11-08T05:17:11.689 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-11-08T05:17:11.690 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-11-08T05:17:11.690 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_23 2023-11-08T05:17:11.797 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ return 0 2023-11-08T05:17:11.797 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ status_result=' "state": "Replaying"' 2023-11-08T05:17:11.797 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ echo ' "state": "Replaying"' 2023-11-08T05:17:11.798 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ grep -i Replaying 2023-11-08T05:17:11.798 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: "state": "Replaying" 2023-11-08T05:17:11.801 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ current_state=started 2023-11-08T05:17:11.801 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test started = started 2023-11-08T05:17:11.801 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:17:11.801 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_23 2023-11-08T05:17:11.802 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local local_cluster=cluster1 2023-11-08T05:17:11.802 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:17:11.802 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:17:11.802 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_23 2023-11-08T05:17:11.803 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' snapshot = journal ']' 2023-11-08T05:17:11.803 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' snapshot = snapshot ']' 2023-11-08T05:17:11.803 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_23 2023-11-08T05:17:11.803 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local local_cluster=cluster1 2023-11-08T05:17:11.803 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:17:11.804 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:17:11.804 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_23 2023-11-08T05:17:11.804 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ mkfname cluster2-mirror-image_23.status 2023-11-08T05:17:11.804 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ echo cluster2-mirror-image_23.status 2023-11-08T05:17:11.804 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -e 's|[/ ]|_|g' 2023-11-08T05:17:11.805 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local status_log=/tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_23.status 2023-11-08T05:17:11.805 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ mkfname cluster1-mirror-image_23.status 2023-11-08T05:17:11.805 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ echo cluster1-mirror-image_23.status 2023-11-08T05:17:11.805 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -e 's|[/ ]|_|g' 2023-11-08T05:17:11.806 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local local_status_log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_23.status 2023-11-08T05:17:11.806 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ mirror_image_snapshot cluster2 mirror image_23 2023-11-08T05:17:11.807 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:17:11.807 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:17:11.807 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_23 2023-11-08T05:17:11.807 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_23 2023-11-08T05:17:12.322 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:Snapshot ID: 163 2023-11-08T05:17:12.332 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_23 /tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_23.status 2023-11-08T05:17:12.332 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:17:12.332 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:17:12.333 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_23 2023-11-08T05:17:12.333 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_23.status 2023-11-08T05:17:12.333 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_23 --format xml 2023-11-08T05:17:12.333 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T05:17:12.372 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-11-08T05:17:12.374 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snapshot_id=163 2023-11-08T05:17:12.375 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ true 2023-11-08T05:17:12.375 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-11-08T05:17:12.375 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0.2 2023-11-08T05:17:12.576 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_23 /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_23.status 2023-11-08T05:17:12.576 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:17:12.576 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:17:12.577 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_23 2023-11-08T05:17:12.577 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_23.status 2023-11-08T05:17:12.577 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_23 --format xml 2023-11-08T05:17:12.577 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T05:17:12.618 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ true 2023-11-08T05:17:12.618 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-08T05:17:12.619 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:-:1.1: Document is empty 2023-11-08T05:17:12.619 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: 2023-11-08T05:17:12.620 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:^ 2023-11-08T05:17:12.620 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local primary_snapshot_id= 2023-11-08T05:17:12.621 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test 163 = '' 2023-11-08T05:17:12.621 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-11-08T05:17:12.621 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0.4 2023-11-08T05:17:13.021 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_23 /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_23.status 2023-11-08T05:17:13.022 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:17:13.022 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:17:13.022 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_23 2023-11-08T05:17:13.023 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_23.status 2023-11-08T05:17:13.023 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_23 --format xml 2023-11-08T05:17:13.023 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T05:17:13.061 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ true 2023-11-08T05:17:13.062 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-08T05:17:13.065 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:-:1.1: Document is empty 2023-11-08T05:17:13.066 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: 2023-11-08T05:17:13.066 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:^ 2023-11-08T05:17:13.066 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local primary_snapshot_id= 2023-11-08T05:17:13.066 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test 163 = '' 2023-11-08T05:17:13.067 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-11-08T05:17:13.067 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0.8 2023-11-08T05:17:13.865 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_23 /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_23.status 2023-11-08T05:17:13.865 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:17:13.865 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:17:13.865 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_23 2023-11-08T05:17:13.866 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_23.status 2023-11-08T05:17:13.866 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_23 --format xml 2023-11-08T05:17:13.866 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T05:17:13.903 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-08T05:17:13.906 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local primary_snapshot_id=163 2023-11-08T05:17:13.906 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test 163 = 163 2023-11-08T05:17:13.906 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:17:13.907 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_snap_present cluster1 mirror image_23 snap 2023-11-08T05:17:13.907 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:17:13.908 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:17:13.908 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_23 2023-11-08T05:17:13.908 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap_name=snap 2023-11-08T05:17:13.908 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local s 2023-11-08T05:17:13.909 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-11-08T05:17:13.909 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 1 2023-11-08T05:17:14.906 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 info mirror/image_23@snap 2023-11-08T05:17:14.939 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:rbd image 'image_23': 2023-11-08T05:17:14.939 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: size 128 MiB in 32 objects 2023-11-08T05:17:14.939 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: order 22 (4 MiB objects) 2023-11-08T05:17:14.940 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: snapshot_count: 2 2023-11-08T05:17:14.940 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: id: 11d83585ce58 2023-11-08T05:17:14.940 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: block_name_prefix: rbd_data.11d83585ce58 2023-11-08T05:17:14.940 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: format: 2 2023-11-08T05:17:14.940 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: features: layering, exclusive-lock, non-primary 2023-11-08T05:17:14.941 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: op_features: 2023-11-08T05:17:14.941 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: flags: 2023-11-08T05:17:14.941 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: create_timestamp: Wed Nov 8 05:08:30 2023 2023-11-08T05:17:14.941 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: access_timestamp: Wed Nov 8 05:08:30 2023 2023-11-08T05:17:14.941 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: modify_timestamp: Wed Nov 8 05:08:30 2023 2023-11-08T05:17:14.942 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: protected: False 2023-11-08T05:17:14.942 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring state: enabled 2023-11-08T05:17:14.942 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring mode: snapshot 2023-11-08T05:17:14.942 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring global id: b0c3eb50-dd81-436f-b795-2235c773f092 2023-11-08T05:17:14.942 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring primary: false 2023-11-08T05:17:14.944 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:17:14.944 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ compare_image_snaps mirror image_23 snap 2023-11-08T05:17:14.944 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:17:14.944 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_23 2023-11-08T05:17:14.945 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap_name=snap 2023-11-08T05:17:14.945 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local ret=0 2023-11-08T05:17:14.945 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local rmt_export=/tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_23.export 2023-11-08T05:17:14.945 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local loc_export=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_23.export 2023-11-08T05:17:14.945 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rm -f /tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_23.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_23.export 2023-11-08T05:17:14.946 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 -p mirror export image_23@snap /tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_23.export 2023-11-08T05:17:15.100 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.891302905090491) > weight (0.25) 2023-11-08T05:17:15.100 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.1 2023-11-08T05:17:15.101 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Sent signal 15 2023-11-08T05:17:15.101 INFO:tasks.rbd_mirror_thrash:waiting for 3 secs before reviving daemons 2023-11-08T05:17:15.101 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi087.stderr:2023-11-08T05:17:15.100+0000 7f3fb71e6700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 (PID: 136217) UID: 1000 2023-11-08T05:17:15.121 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-11-08T05:17:15.125 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 -p mirror export image_23@snap /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_23.export 2023-11-08T05:17:15.302 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi087.stderr:daemon-helper: command failed with exit status 1 2023-11-08T05:17:15.351 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-11-08T05:17:15.354 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ cmp /tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_23.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_23.export 2023-11-08T05:17:15.412 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rm -f /tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_23.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_23.export 2023-11-08T05:17:15.450 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:17:15.450 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-11-08T05:17:15.450 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image=image_24 2023-11-08T05:17:15.451 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ create_snap cluster2 mirror image_24 snap 2023-11-08T05:17:15.451 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:17:15.451 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:17:15.451 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_24 2023-11-08T05:17:15.451 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap_name=snap 2023-11-08T05:17:15.452 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 -p mirror snap create image_24@snap --debug-rbd=20 --debug-journaler=20 2023-11-08T05:17:17.335 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_image_replay_started cluster1 mirror image_24 2023-11-08T05:17:17.336 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:17:17.336 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:17:17.336 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_24 2023-11-08T05:17:17.336 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_image_replay_state cluster1 mirror image_24 started 2023-11-08T05:17:17.336 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:17:17.337 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:17:17.337 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_24 2023-11-08T05:17:17.337 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local state=started 2023-11-08T05:17:17.337 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local s 2023-11-08T05:17:17.337 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-11-08T05:17:17.338 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 1 2023-11-08T05:17:18.102 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.1 2023-11-08T05:17:18.102 INFO:teuthology.orchestra.run:waiting for 600 2023-11-08T05:17:18.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-08T05:17:18.103 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.1 2023-11-08T05:17:18.103 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Restarting daemon 2023-11-08T05:17:18.104 DEBUG:teuthology.orchestra.run.smithi087:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 2023-11-08T05:17:18.107 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Started 2023-11-08T05:17:18.107 INFO:tasks.rbd_mirror_thrash:waiting for 119 secs before thrashing 2023-11-08T05:17:18.337 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test_image_replay_state cluster1 mirror image_24 started 2023-11-08T05:17:18.337 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:17:18.337 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:17:18.338 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_24 2023-11-08T05:17:18.338 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local test_state=started 2023-11-08T05:17:18.338 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local status_result 2023-11-08T05:17:18.338 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local current_state=stopped 2023-11-08T05:17:18.338 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_24 2023-11-08T05:17:18.339 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_instance=cluster1 2023-11-08T05:17:18.339 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T05:17:18.339 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ grep -i state 2023-11-08T05:17:18.339 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1 2023-11-08T05:17:18.340 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance=cluster1 2023-11-08T05:17:18.340 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local loop_instance 2023-11-08T05:17:18.340 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-11-08T05:17:18.341 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sleep 0 2023-11-08T05:17:18.343 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-11-08T05:17:18.343 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ seq 0 3 2023-11-08T05:17:18.343 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T05:17:18.343 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_24 2023-11-08T05:17:18.344 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1:0 2023-11-08T05:17:18.344 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T05:17:18.344 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance 2023-11-08T05:17:18.344 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-11-08T05:17:18.344 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local val=cluster1:0 2023-11-08T05:17:18.345 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_var_name=cluster 2023-11-08T05:17:18.345 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance_var_name=instance 2023-11-08T05:17:18.345 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cluster=cluster1 2023-11-08T05:17:18.345 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ instance=0 2023-11-08T05:17:18.345 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' 0 = cluster1:0 ']' 2023-11-08T05:17:18.345 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval cluster=cluster1 2023-11-08T05:17:18.346 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:17:18.346 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval instance=0 2023-11-08T05:17:18.346 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=0 2023-11-08T05:17:18.346 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-11-08T05:17:18.346 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local local_cluster=cluster1:0 2023-11-08T05:17:18.348 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster=cluster1 2023-11-08T05:17:18.348 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance 2023-11-08T05:17:18.348 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-11-08T05:17:18.349 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local val=cluster1:0 2023-11-08T05:17:18.349 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster_var_name=local_cluster 2023-11-08T05:17:18.349 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance_var_name=instance 2023-11-08T05:17:18.349 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:17:18.350 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=0 2023-11-08T05:17:18.350 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ '[' 0 = cluster1:0 ']' 2023-11-08T05:17:18.350 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval local_cluster=cluster1 2023-11-08T05:17:18.350 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ local_cluster=cluster1 2023-11-08T05:17:18.350 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval instance=0 2023-11-08T05:17:18.351 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ instance=0 2023-11-08T05:17:18.351 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-11-08T05:17:18.354 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-08T05:17:18.355 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-08T05:17:18.355 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-08T05:17:18.355 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_24 2023-11-08T05:17:18.462 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ return 0 2023-11-08T05:17:18.463 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ status_result=' "state": "Replaying"' 2023-11-08T05:17:18.463 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ echo ' "state": "Replaying"' 2023-11-08T05:17:18.464 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ grep -i Replaying 2023-11-08T05:17:18.464 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: "state": "Replaying" 2023-11-08T05:17:18.466 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ current_state=started 2023-11-08T05:17:18.467 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test started = started 2023-11-08T05:17:18.467 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:17:18.467 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_24 2023-11-08T05:17:18.467 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local local_cluster=cluster1 2023-11-08T05:17:18.467 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:17:18.468 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:17:18.468 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_24 2023-11-08T05:17:18.468 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' snapshot = journal ']' 2023-11-08T05:17:18.468 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' snapshot = snapshot ']' 2023-11-08T05:17:18.468 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_24 2023-11-08T05:17:18.469 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local local_cluster=cluster1 2023-11-08T05:17:18.469 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:17:18.469 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:17:18.469 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_24 2023-11-08T05:17:18.469 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ mkfname cluster2-mirror-image_24.status 2023-11-08T05:17:18.469 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ echo cluster2-mirror-image_24.status 2023-11-08T05:17:18.470 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -e 's|[/ ]|_|g' 2023-11-08T05:17:18.470 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local status_log=/tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_24.status 2023-11-08T05:17:18.470 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ mkfname cluster1-mirror-image_24.status 2023-11-08T05:17:18.470 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ echo cluster1-mirror-image_24.status 2023-11-08T05:17:18.471 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -e 's|[/ ]|_|g' 2023-11-08T05:17:18.472 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local local_status_log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_24.status 2023-11-08T05:17:18.472 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ mirror_image_snapshot cluster2 mirror image_24 2023-11-08T05:17:18.472 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:17:18.472 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:17:18.472 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_24 2023-11-08T05:17:18.473 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_24 2023-11-08T05:17:19.345 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:Snapshot ID: 165 2023-11-08T05:17:19.353 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_24 /tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_24.status 2023-11-08T05:17:19.353 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:17:19.354 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:17:19.354 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_24 2023-11-08T05:17:19.354 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_24.status 2023-11-08T05:17:19.354 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_24 --format xml 2023-11-08T05:17:19.354 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T05:17:19.392 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-11-08T05:17:19.394 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snapshot_id=165 2023-11-08T05:17:19.395 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ true 2023-11-08T05:17:19.395 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-11-08T05:17:19.395 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0.2 2023-11-08T05:17:19.596 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_24 /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_24.status 2023-11-08T05:17:19.596 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:17:19.596 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:17:19.596 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_24 2023-11-08T05:17:19.597 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_24.status 2023-11-08T05:17:19.597 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_24 --format xml 2023-11-08T05:17:19.597 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T05:17:19.637 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ true 2023-11-08T05:17:19.638 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-08T05:17:19.639 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:-:1.1: Document is empty 2023-11-08T05:17:19.639 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: 2023-11-08T05:17:19.639 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:^ 2023-11-08T05:17:19.640 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local primary_snapshot_id= 2023-11-08T05:17:19.640 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test 165 = '' 2023-11-08T05:17:19.640 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-11-08T05:17:19.640 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0.4 2023-11-08T05:17:20.041 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_24 /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_24.status 2023-11-08T05:17:20.041 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:17:20.041 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:17:20.042 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_24 2023-11-08T05:17:20.042 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_24.status 2023-11-08T05:17:20.042 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_24 --format xml 2023-11-08T05:17:20.042 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T05:17:20.085 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ true 2023-11-08T05:17:20.085 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-08T05:17:20.086 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:-:1.1: Document is empty 2023-11-08T05:17:20.086 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: 2023-11-08T05:17:20.087 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:^ 2023-11-08T05:17:20.087 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local primary_snapshot_id= 2023-11-08T05:17:20.088 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test 165 = '' 2023-11-08T05:17:20.088 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-11-08T05:17:20.088 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0.8 2023-11-08T05:17:20.888 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_24 /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_24.status 2023-11-08T05:17:20.889 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:17:20.889 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:17:20.889 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_24 2023-11-08T05:17:20.889 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_24.status 2023-11-08T05:17:20.890 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_24 --format xml 2023-11-08T05:17:20.890 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T05:17:20.926 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-08T05:17:20.928 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local primary_snapshot_id=165 2023-11-08T05:17:20.929 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test 165 = 165 2023-11-08T05:17:20.929 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:17:20.929 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_snap_present cluster1 mirror image_24 snap 2023-11-08T05:17:20.929 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:17:20.929 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:17:20.930 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_24 2023-11-08T05:17:20.930 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap_name=snap 2023-11-08T05:17:20.930 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local s 2023-11-08T05:17:20.930 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-11-08T05:17:20.931 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 1 2023-11-08T05:17:21.929 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 info mirror/image_24@snap 2023-11-08T05:17:21.965 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:rbd image 'image_24': 2023-11-08T05:17:21.965 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: size 128 MiB in 32 objects 2023-11-08T05:17:21.966 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: order 22 (4 MiB objects) 2023-11-08T05:17:21.966 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: snapshot_count: 2 2023-11-08T05:17:21.966 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: id: 12879c5cc0e7 2023-11-08T05:17:21.966 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: block_name_prefix: rbd_data.12879c5cc0e7 2023-11-08T05:17:21.966 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: format: 2 2023-11-08T05:17:21.967 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: features: layering, exclusive-lock, non-primary 2023-11-08T05:17:21.967 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: op_features: 2023-11-08T05:17:21.967 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: flags: 2023-11-08T05:17:21.967 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: create_timestamp: Wed Nov 8 05:08:36 2023 2023-11-08T05:17:21.968 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: access_timestamp: Wed Nov 8 05:08:36 2023 2023-11-08T05:17:21.968 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: modify_timestamp: Wed Nov 8 05:08:36 2023 2023-11-08T05:17:21.968 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: protected: False 2023-11-08T05:17:21.968 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring state: enabled 2023-11-08T05:17:21.968 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring mode: snapshot 2023-11-08T05:17:21.969 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring global id: 4bf12ca9-9a70-496b-8c0e-da3173089291 2023-11-08T05:17:21.969 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring primary: false 2023-11-08T05:17:21.970 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:17:21.970 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ compare_image_snaps mirror image_24 snap 2023-11-08T05:17:21.971 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:17:21.971 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_24 2023-11-08T05:17:21.971 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap_name=snap 2023-11-08T05:17:21.971 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local ret=0 2023-11-08T05:17:21.971 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local rmt_export=/tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_24.export 2023-11-08T05:17:21.971 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local loc_export=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_24.export 2023-11-08T05:17:21.972 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rm -f /tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_24.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_24.export 2023-11-08T05:17:21.972 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 -p mirror export image_24@snap /tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_24.export 2023-11-08T05:17:22.156 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-11-08T05:17:22.159 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 -p mirror export image_24@snap /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_24.export 2023-11-08T05:17:22.393 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-11-08T05:17:22.396 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ cmp /tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_24.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_24.export 2023-11-08T05:17:22.452 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rm -f /tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_24.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_24.export 2023-11-08T05:17:22.492 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:17:22.492 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-11-08T05:17:22.492 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image=image_25 2023-11-08T05:17:22.493 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ create_snap cluster2 mirror image_25 snap 2023-11-08T05:17:22.493 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:17:22.493 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:17:22.493 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_25 2023-11-08T05:17:22.493 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap_name=snap 2023-11-08T05:17:22.494 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 -p mirror snap create image_25@snap --debug-rbd=20 --debug-journaler=20 2023-11-08T05:17:24.356 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_image_replay_started cluster1 mirror image_25 2023-11-08T05:17:24.357 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:17:24.357 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:17:24.357 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_25 2023-11-08T05:17:24.357 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_image_replay_state cluster1 mirror image_25 started 2023-11-08T05:17:24.357 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:17:24.358 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:17:24.358 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_25 2023-11-08T05:17:24.358 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local state=started 2023-11-08T05:17:24.358 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local s 2023-11-08T05:17:24.358 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-11-08T05:17:24.359 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 1 2023-11-08T05:17:25.358 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test_image_replay_state cluster1 mirror image_25 started 2023-11-08T05:17:25.358 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:17:25.358 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:17:25.358 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_25 2023-11-08T05:17:25.359 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local test_state=started 2023-11-08T05:17:25.359 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local status_result 2023-11-08T05:17:25.359 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local current_state=stopped 2023-11-08T05:17:25.364 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_25 2023-11-08T05:17:25.364 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_instance=cluster1 2023-11-08T05:17:25.364 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T05:17:25.364 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ grep -i state 2023-11-08T05:17:25.365 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1 2023-11-08T05:17:25.365 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance=cluster1 2023-11-08T05:17:25.365 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local loop_instance 2023-11-08T05:17:25.365 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-11-08T05:17:25.365 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sleep 0 2023-11-08T05:17:25.366 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-11-08T05:17:25.366 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ seq 0 3 2023-11-08T05:17:25.366 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T05:17:25.366 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_25 2023-11-08T05:17:25.366 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1:0 2023-11-08T05:17:25.367 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T05:17:25.367 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance 2023-11-08T05:17:25.367 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-11-08T05:17:25.367 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local val=cluster1:0 2023-11-08T05:17:25.367 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_var_name=cluster 2023-11-08T05:17:25.368 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance_var_name=instance 2023-11-08T05:17:25.368 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cluster=cluster1 2023-11-08T05:17:25.368 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ instance=0 2023-11-08T05:17:25.368 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' 0 = cluster1:0 ']' 2023-11-08T05:17:25.368 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval cluster=cluster1 2023-11-08T05:17:25.369 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:17:25.369 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval instance=0 2023-11-08T05:17:25.369 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=0 2023-11-08T05:17:25.369 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-11-08T05:17:25.370 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local local_cluster=cluster1:0 2023-11-08T05:17:25.370 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster=cluster1 2023-11-08T05:17:25.370 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance 2023-11-08T05:17:25.371 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-11-08T05:17:25.371 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local val=cluster1:0 2023-11-08T05:17:25.371 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster_var_name=local_cluster 2023-11-08T05:17:25.371 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance_var_name=instance 2023-11-08T05:17:25.371 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:17:25.372 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=0 2023-11-08T05:17:25.372 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ '[' 0 = cluster1:0 ']' 2023-11-08T05:17:25.372 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval local_cluster=cluster1 2023-11-08T05:17:25.373 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ local_cluster=cluster1 2023-11-08T05:17:25.373 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval instance=0 2023-11-08T05:17:25.373 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ instance=0 2023-11-08T05:17:25.373 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-11-08T05:17:25.379 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-08T05:17:25.379 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-08T05:17:25.380 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-08T05:17:25.380 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_25 2023-11-08T05:17:25.487 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:Invalid command: unused arguments: ['mirror/image_25'] 2023-11-08T05:17:25.487 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd mirror status : get status for rbd mirror 2023-11-08T05:17:25.487 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:admin_socket: invalid command 2023-11-08T05:17:25.491 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T05:17:25.491 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_25 2023-11-08T05:17:25.491 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1:1 2023-11-08T05:17:25.491 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T05:17:25.492 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance 2023-11-08T05:17:25.492 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-11-08T05:17:25.492 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local val=cluster1:1 2023-11-08T05:17:25.492 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_var_name=cluster 2023-11-08T05:17:25.492 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance_var_name=instance 2023-11-08T05:17:25.493 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cluster=cluster1 2023-11-08T05:17:25.493 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ instance=1 2023-11-08T05:17:25.493 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' 1 = cluster1:1 ']' 2023-11-08T05:17:25.493 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval cluster=cluster1 2023-11-08T05:17:25.493 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:17:25.494 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval instance=1 2023-11-08T05:17:25.494 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=1 2023-11-08T05:17:25.494 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-11-08T05:17:25.494 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local local_cluster=cluster1:1 2023-11-08T05:17:25.495 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster=cluster1 2023-11-08T05:17:25.495 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance 2023-11-08T05:17:25.495 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-11-08T05:17:25.496 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local val=cluster1:1 2023-11-08T05:17:25.496 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster_var_name=local_cluster 2023-11-08T05:17:25.496 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance_var_name=instance 2023-11-08T05:17:25.496 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:17:25.496 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=1 2023-11-08T05:17:25.496 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ '[' 1 = cluster1:1 ']' 2023-11-08T05:17:25.497 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval local_cluster=cluster1 2023-11-08T05:17:25.497 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ local_cluster=cluster1 2023-11-08T05:17:25.497 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval instance=1 2023-11-08T05:17:25.497 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ instance=1 2023-11-08T05:17:25.497 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-11-08T05:17:25.504 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-08T05:17:25.505 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-08T05:17:25.505 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-08T05:17:25.505 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_25 2023-11-08T05:17:25.606 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:Invalid command: unused arguments: ['mirror/image_25'] 2023-11-08T05:17:25.607 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd mirror status : get status for rbd mirror 2023-11-08T05:17:25.607 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:admin_socket: invalid command 2023-11-08T05:17:25.610 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T05:17:25.610 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_25 2023-11-08T05:17:25.611 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1:2 2023-11-08T05:17:25.611 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T05:17:25.611 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance 2023-11-08T05:17:25.611 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-11-08T05:17:25.611 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local val=cluster1:2 2023-11-08T05:17:25.612 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_var_name=cluster 2023-11-08T05:17:25.612 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance_var_name=instance 2023-11-08T05:17:25.612 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cluster=cluster1 2023-11-08T05:17:25.612 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ instance=2 2023-11-08T05:17:25.612 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' 2 = cluster1:2 ']' 2023-11-08T05:17:25.613 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval cluster=cluster1 2023-11-08T05:17:25.613 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:17:25.613 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval instance=2 2023-11-08T05:17:25.613 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=2 2023-11-08T05:17:25.613 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-11-08T05:17:25.614 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local local_cluster=cluster1:2 2023-11-08T05:17:25.614 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster=cluster1 2023-11-08T05:17:25.614 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance 2023-11-08T05:17:25.614 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-11-08T05:17:25.614 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local val=cluster1:2 2023-11-08T05:17:25.615 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster_var_name=local_cluster 2023-11-08T05:17:25.615 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance_var_name=instance 2023-11-08T05:17:25.615 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:17:25.615 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=2 2023-11-08T05:17:25.615 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ '[' 2 = cluster1:2 ']' 2023-11-08T05:17:25.616 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval local_cluster=cluster1 2023-11-08T05:17:25.616 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ local_cluster=cluster1 2023-11-08T05:17:25.616 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval instance=2 2023-11-08T05:17:25.616 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ instance=2 2023-11-08T05:17:25.616 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-11-08T05:17:25.624 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-08T05:17:25.624 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-08T05:17:25.624 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-08T05:17:25.624 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_25 2023-11-08T05:17:25.722 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:Invalid command: unused arguments: ['mirror/image_25'] 2023-11-08T05:17:25.723 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd mirror status : get status for rbd mirror 2023-11-08T05:17:25.723 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:admin_socket: invalid command 2023-11-08T05:17:25.729 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T05:17:25.730 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_25 2023-11-08T05:17:25.730 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1:3 2023-11-08T05:17:25.730 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T05:17:25.731 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance 2023-11-08T05:17:25.731 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ set_cluster_instance cluster1:3 cluster instance 2023-11-08T05:17:25.731 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local val=cluster1:3 2023-11-08T05:17:25.732 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_var_name=cluster 2023-11-08T05:17:25.732 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance_var_name=instance 2023-11-08T05:17:25.732 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cluster=cluster1 2023-11-08T05:17:25.733 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ instance=3 2023-11-08T05:17:25.733 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' 3 = cluster1:3 ']' 2023-11-08T05:17:25.733 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval cluster=cluster1 2023-11-08T05:17:25.734 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:17:25.734 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval instance=3 2023-11-08T05:17:25.734 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=3 2023-11-08T05:17:25.734 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ daemon_asok_file cluster1:3 cluster1 2023-11-08T05:17:25.735 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local local_cluster=cluster1:3 2023-11-08T05:17:25.735 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster=cluster1 2023-11-08T05:17:25.735 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance 2023-11-08T05:17:25.736 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2023-11-08T05:17:25.736 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local val=cluster1:3 2023-11-08T05:17:25.736 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster_var_name=local_cluster 2023-11-08T05:17:25.736 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance_var_name=instance 2023-11-08T05:17:25.736 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:17:25.737 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=3 2023-11-08T05:17:25.737 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ '[' 3 = cluster1:3 ']' 2023-11-08T05:17:25.737 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval local_cluster=cluster1 2023-11-08T05:17:25.737 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ local_cluster=cluster1 2023-11-08T05:17:25.737 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval instance=3 2023-11-08T05:17:25.738 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ instance=3 2023-11-08T05:17:25.738 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2023-11-08T05:17:25.739 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-11-08T05:17:25.740 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-11-08T05:17:25.740 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-11-08T05:17:25.740 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_25 2023-11-08T05:17:25.846 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ return 0 2023-11-08T05:17:25.847 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ status_result=' "state": "Replaying"' 2023-11-08T05:17:25.848 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ echo ' "state": "Replaying"' 2023-11-08T05:17:25.848 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ grep -i Replaying 2023-11-08T05:17:25.848 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: "state": "Replaying" 2023-11-08T05:17:25.851 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ current_state=started 2023-11-08T05:17:25.851 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test started = started 2023-11-08T05:17:25.851 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:17:25.852 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_25 2023-11-08T05:17:25.852 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local local_cluster=cluster1 2023-11-08T05:17:25.852 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:17:25.852 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:17:25.852 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_25 2023-11-08T05:17:25.853 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' snapshot = journal ']' 2023-11-08T05:17:25.853 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' snapshot = snapshot ']' 2023-11-08T05:17:25.853 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_25 2023-11-08T05:17:25.853 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local local_cluster=cluster1 2023-11-08T05:17:25.853 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:17:25.854 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:17:25.854 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_25 2023-11-08T05:17:25.854 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ mkfname cluster2-mirror-image_25.status 2023-11-08T05:17:25.854 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ echo cluster2-mirror-image_25.status 2023-11-08T05:17:25.854 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -e 's|[/ ]|_|g' 2023-11-08T05:17:25.855 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local status_log=/tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_25.status 2023-11-08T05:17:25.855 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ mkfname cluster1-mirror-image_25.status 2023-11-08T05:17:25.855 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ echo cluster1-mirror-image_25.status 2023-11-08T05:17:25.855 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -e 's|[/ ]|_|g' 2023-11-08T05:17:25.856 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local local_status_log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_25.status 2023-11-08T05:17:25.856 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ mirror_image_snapshot cluster2 mirror image_25 2023-11-08T05:17:25.857 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:17:25.857 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:17:25.857 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_25 2023-11-08T05:17:25.857 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_25 2023-11-08T05:17:26.364 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:Snapshot ID: 167 2023-11-08T05:17:26.372 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_25 /tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_25.status 2023-11-08T05:17:26.373 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:17:26.373 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:17:26.373 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_25 2023-11-08T05:17:26.373 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_25.status 2023-11-08T05:17:26.373 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_25 --format xml 2023-11-08T05:17:26.374 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T05:17:26.410 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-11-08T05:17:26.414 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snapshot_id=167 2023-11-08T05:17:26.415 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ true 2023-11-08T05:17:26.415 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-11-08T05:17:26.415 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0.2 2023-11-08T05:17:26.613 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_25 /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_25.status 2023-11-08T05:17:26.613 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:17:26.614 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:17:26.614 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_25 2023-11-08T05:17:26.614 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_25.status 2023-11-08T05:17:26.614 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_25 --format xml 2023-11-08T05:17:26.614 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T05:17:26.655 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ true 2023-11-08T05:17:26.655 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-08T05:17:26.656 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:-:1.1: Document is empty 2023-11-08T05:17:26.656 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: 2023-11-08T05:17:26.656 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:^ 2023-11-08T05:17:26.657 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local primary_snapshot_id= 2023-11-08T05:17:26.657 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test 167 = '' 2023-11-08T05:17:26.657 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-11-08T05:17:26.658 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0.4 2023-11-08T05:17:27.058 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_25 /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_25.status 2023-11-08T05:17:27.058 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:17:27.058 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:17:27.059 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_25 2023-11-08T05:17:27.059 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_25.status 2023-11-08T05:17:27.059 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_25 --format xml 2023-11-08T05:17:27.059 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T05:17:27.100 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ true 2023-11-08T05:17:27.101 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-08T05:17:27.102 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:-:1.1: Document is empty 2023-11-08T05:17:27.102 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: 2023-11-08T05:17:27.102 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:^ 2023-11-08T05:17:27.103 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local primary_snapshot_id= 2023-11-08T05:17:27.103 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test 167 = '' 2023-11-08T05:17:27.103 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-11-08T05:17:27.104 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0.8 2023-11-08T05:17:27.904 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_25 /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_25.status 2023-11-08T05:17:27.904 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:17:27.905 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:17:27.905 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_25 2023-11-08T05:17:27.905 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_25.status 2023-11-08T05:17:27.906 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_25 --format xml 2023-11-08T05:17:27.906 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T05:17:27.944 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-08T05:17:27.946 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local primary_snapshot_id=167 2023-11-08T05:17:27.946 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test 167 = 167 2023-11-08T05:17:27.946 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:17:27.946 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_snap_present cluster1 mirror image_25 snap 2023-11-08T05:17:27.947 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:17:27.947 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:17:27.947 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_25 2023-11-08T05:17:27.947 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap_name=snap 2023-11-08T05:17:27.947 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local s 2023-11-08T05:17:27.948 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-11-08T05:17:27.948 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 1 2023-11-08T05:17:28.947 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 info mirror/image_25@snap 2023-11-08T05:17:28.992 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:rbd image 'image_25': 2023-11-08T05:17:28.992 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: size 128 MiB in 32 objects 2023-11-08T05:17:28.992 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: order 22 (4 MiB objects) 2023-11-08T05:17:28.992 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: snapshot_count: 2 2023-11-08T05:17:28.993 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: id: 10e73f836a57 2023-11-08T05:17:28.993 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: block_name_prefix: rbd_data.10e73f836a57 2023-11-08T05:17:28.993 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: format: 2 2023-11-08T05:17:28.993 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: features: layering, exclusive-lock, non-primary 2023-11-08T05:17:28.993 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: op_features: 2023-11-08T05:17:28.994 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: flags: 2023-11-08T05:17:28.994 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: create_timestamp: Wed Nov 8 05:08:48 2023 2023-11-08T05:17:28.994 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: access_timestamp: Wed Nov 8 05:08:48 2023 2023-11-08T05:17:28.994 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: modify_timestamp: Wed Nov 8 05:08:48 2023 2023-11-08T05:17:28.994 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: protected: False 2023-11-08T05:17:28.995 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring state: enabled 2023-11-08T05:17:28.995 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring mode: snapshot 2023-11-08T05:17:28.995 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring global id: 4f72c9b1-e028-4538-ade8-ae09b2818aa9 2023-11-08T05:17:28.995 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring primary: false 2023-11-08T05:17:28.996 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:17:28.997 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ compare_image_snaps mirror image_25 snap 2023-11-08T05:17:28.997 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:17:28.997 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_25 2023-11-08T05:17:28.997 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap_name=snap 2023-11-08T05:17:28.997 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local ret=0 2023-11-08T05:17:28.998 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local rmt_export=/tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_25.export 2023-11-08T05:17:28.998 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local loc_export=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_25.export 2023-11-08T05:17:28.998 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rm -f /tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_25.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_25.export 2023-11-08T05:17:28.998 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 -p mirror export image_25@snap /tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_25.export 2023-11-08T05:17:29.180 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-11-08T05:17:29.183 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 -p mirror export image_25@snap /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_25.export 2023-11-08T05:17:29.397 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-11-08T05:17:29.400 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ cmp /tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_25.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_25.export 2023-11-08T05:17:29.455 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rm -f /tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_25.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_25.export 2023-11-08T05:17:29.493 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:17:29.493 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-11-08T05:17:29.493 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image=image_26 2023-11-08T05:17:29.494 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ create_snap cluster2 mirror image_26 snap 2023-11-08T05:17:29.494 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:17:29.494 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:17:29.494 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_26 2023-11-08T05:17:29.494 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap_name=snap 2023-11-08T05:17:29.495 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 -p mirror snap create image_26@snap --debug-rbd=20 --debug-journaler=20 2023-11-08T05:17:31.373 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_image_replay_started cluster1 mirror image_26 2023-11-08T05:17:31.374 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:17:31.374 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:17:31.374 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_26 2023-11-08T05:17:31.375 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_image_replay_state cluster1 mirror image_26 started 2023-11-08T05:17:31.375 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:17:31.375 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:17:31.375 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_26 2023-11-08T05:17:31.375 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local state=started 2023-11-08T05:17:31.375 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local s 2023-11-08T05:17:31.376 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-11-08T05:17:31.376 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 1 2023-11-08T05:17:32.375 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test_image_replay_state cluster1 mirror image_26 started 2023-11-08T05:17:32.375 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:17:32.375 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:17:32.376 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_26 2023-11-08T05:17:32.376 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local test_state=started 2023-11-08T05:17:32.376 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local status_result 2023-11-08T05:17:32.377 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local current_state=stopped 2023-11-08T05:17:32.380 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_26 2023-11-08T05:17:32.380 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_instance=cluster1 2023-11-08T05:17:32.381 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ grep -i state 2023-11-08T05:17:32.381 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T05:17:32.381 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1 2023-11-08T05:17:32.381 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance=cluster1 2023-11-08T05:17:32.382 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local loop_instance 2023-11-08T05:17:32.382 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-11-08T05:17:32.382 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sleep 0 2023-11-08T05:17:32.382 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-11-08T05:17:32.382 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ seq 0 3 2023-11-08T05:17:32.383 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T05:17:32.383 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_26 2023-11-08T05:17:32.383 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1:0 2023-11-08T05:17:32.383 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T05:17:32.383 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance 2023-11-08T05:17:32.384 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-11-08T05:17:32.384 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local val=cluster1:0 2023-11-08T05:17:32.384 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_var_name=cluster 2023-11-08T05:17:32.384 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance_var_name=instance 2023-11-08T05:17:32.384 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cluster=cluster1 2023-11-08T05:17:32.384 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ instance=0 2023-11-08T05:17:32.385 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' 0 = cluster1:0 ']' 2023-11-08T05:17:32.385 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval cluster=cluster1 2023-11-08T05:17:32.385 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:17:32.385 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval instance=0 2023-11-08T05:17:32.385 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=0 2023-11-08T05:17:32.386 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-11-08T05:17:32.386 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local local_cluster=cluster1:0 2023-11-08T05:17:32.386 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster=cluster1 2023-11-08T05:17:32.386 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance 2023-11-08T05:17:32.386 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-11-08T05:17:32.387 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local val=cluster1:0 2023-11-08T05:17:32.387 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster_var_name=local_cluster 2023-11-08T05:17:32.387 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance_var_name=instance 2023-11-08T05:17:32.387 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:17:32.387 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=0 2023-11-08T05:17:32.388 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ '[' 0 = cluster1:0 ']' 2023-11-08T05:17:32.388 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval local_cluster=cluster1 2023-11-08T05:17:32.388 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ local_cluster=cluster1 2023-11-08T05:17:32.388 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval instance=0 2023-11-08T05:17:32.388 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ instance=0 2023-11-08T05:17:32.389 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-11-08T05:17:32.398 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-08T05:17:32.398 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-08T05:17:32.398 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-08T05:17:32.398 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_26 2023-11-08T05:17:32.503 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:Invalid command: unused arguments: ['mirror/image_26'] 2023-11-08T05:17:32.503 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd mirror status : get status for rbd mirror 2023-11-08T05:17:32.503 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:admin_socket: invalid command 2023-11-08T05:17:32.506 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T05:17:32.506 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_26 2023-11-08T05:17:32.506 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1:1 2023-11-08T05:17:32.507 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T05:17:32.507 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance 2023-11-08T05:17:32.507 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-11-08T05:17:32.507 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local val=cluster1:1 2023-11-08T05:17:32.507 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_var_name=cluster 2023-11-08T05:17:32.508 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance_var_name=instance 2023-11-08T05:17:32.508 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cluster=cluster1 2023-11-08T05:17:32.508 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ instance=1 2023-11-08T05:17:32.508 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' 1 = cluster1:1 ']' 2023-11-08T05:17:32.508 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval cluster=cluster1 2023-11-08T05:17:32.509 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:17:32.509 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval instance=1 2023-11-08T05:17:32.509 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=1 2023-11-08T05:17:32.509 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-11-08T05:17:32.509 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local local_cluster=cluster1:1 2023-11-08T05:17:32.510 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster=cluster1 2023-11-08T05:17:32.510 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance 2023-11-08T05:17:32.510 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-11-08T05:17:32.510 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local val=cluster1:1 2023-11-08T05:17:32.510 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster_var_name=local_cluster 2023-11-08T05:17:32.510 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance_var_name=instance 2023-11-08T05:17:32.511 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:17:32.511 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=1 2023-11-08T05:17:32.511 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ '[' 1 = cluster1:1 ']' 2023-11-08T05:17:32.511 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval local_cluster=cluster1 2023-11-08T05:17:32.511 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ local_cluster=cluster1 2023-11-08T05:17:32.512 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval instance=1 2023-11-08T05:17:32.512 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ instance=1 2023-11-08T05:17:32.512 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-11-08T05:17:32.521 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-08T05:17:32.521 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-08T05:17:32.521 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-08T05:17:32.522 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_26 2023-11-08T05:17:32.629 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ return 0 2023-11-08T05:17:32.629 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ status_result=' "state": "Replaying"' 2023-11-08T05:17:32.630 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ echo ' "state": "Replaying"' 2023-11-08T05:17:32.630 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ grep -i Replaying 2023-11-08T05:17:32.630 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: "state": "Replaying" 2023-11-08T05:17:32.632 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ current_state=started 2023-11-08T05:17:32.633 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test started = started 2023-11-08T05:17:32.634 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:17:32.634 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_26 2023-11-08T05:17:32.634 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local local_cluster=cluster1 2023-11-08T05:17:32.634 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:17:32.634 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:17:32.635 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_26 2023-11-08T05:17:32.635 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' snapshot = journal ']' 2023-11-08T05:17:32.635 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' snapshot = snapshot ']' 2023-11-08T05:17:32.635 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_26 2023-11-08T05:17:32.635 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local local_cluster=cluster1 2023-11-08T05:17:32.636 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:17:32.636 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:17:32.636 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_26 2023-11-08T05:17:32.636 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ mkfname cluster2-mirror-image_26.status 2023-11-08T05:17:32.636 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ echo cluster2-mirror-image_26.status 2023-11-08T05:17:32.636 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -e 's|[/ ]|_|g' 2023-11-08T05:17:32.637 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local status_log=/tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_26.status 2023-11-08T05:17:32.637 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ mkfname cluster1-mirror-image_26.status 2023-11-08T05:17:32.637 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ echo cluster1-mirror-image_26.status 2023-11-08T05:17:32.637 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -e 's|[/ ]|_|g' 2023-11-08T05:17:32.638 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local local_status_log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_26.status 2023-11-08T05:17:32.638 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ mirror_image_snapshot cluster2 mirror image_26 2023-11-08T05:17:32.639 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:17:32.639 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:17:32.639 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_26 2023-11-08T05:17:32.639 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_26 2023-11-08T05:17:33.381 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:Snapshot ID: 169 2023-11-08T05:17:33.390 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_26 /tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_26.status 2023-11-08T05:17:33.390 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:17:33.390 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:17:33.391 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_26 2023-11-08T05:17:33.391 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_26.status 2023-11-08T05:17:33.391 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_26 --format xml 2023-11-08T05:17:33.391 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T05:17:33.426 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-11-08T05:17:33.428 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snapshot_id=169 2023-11-08T05:17:33.428 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ true 2023-11-08T05:17:33.429 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-11-08T05:17:33.429 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0.2 2023-11-08T05:17:33.630 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_26 /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_26.status 2023-11-08T05:17:33.630 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:17:33.630 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:17:33.631 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_26 2023-11-08T05:17:33.631 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_26.status 2023-11-08T05:17:33.631 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_26 --format xml 2023-11-08T05:17:33.631 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T05:17:33.668 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ true 2023-11-08T05:17:33.669 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-08T05:17:33.670 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:-:1.1: Document is empty 2023-11-08T05:17:33.670 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: 2023-11-08T05:17:33.670 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:^ 2023-11-08T05:17:33.671 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local primary_snapshot_id= 2023-11-08T05:17:33.671 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test 169 = '' 2023-11-08T05:17:33.671 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-11-08T05:17:33.672 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0.4 2023-11-08T05:17:34.072 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_26 /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_26.status 2023-11-08T05:17:34.072 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:17:34.072 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:17:34.072 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_26 2023-11-08T05:17:34.073 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_26.status 2023-11-08T05:17:34.073 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_26 --format xml 2023-11-08T05:17:34.073 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T05:17:34.113 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ true 2023-11-08T05:17:34.113 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-08T05:17:34.115 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:-:1.1: Document is empty 2023-11-08T05:17:34.115 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: 2023-11-08T05:17:34.115 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:^ 2023-11-08T05:17:34.116 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local primary_snapshot_id= 2023-11-08T05:17:34.116 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test 169 = '' 2023-11-08T05:17:34.116 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-11-08T05:17:34.116 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0.8 2023-11-08T05:17:34.917 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_26 /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_26.status 2023-11-08T05:17:34.917 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:17:34.917 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:17:34.917 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_26 2023-11-08T05:17:34.918 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_26.status 2023-11-08T05:17:34.918 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_26 --format xml 2023-11-08T05:17:34.918 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T05:17:34.955 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-08T05:17:34.957 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local primary_snapshot_id=169 2023-11-08T05:17:34.957 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test 169 = 169 2023-11-08T05:17:34.957 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:17:34.958 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_snap_present cluster1 mirror image_26 snap 2023-11-08T05:17:34.958 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:17:34.958 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:17:34.958 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_26 2023-11-08T05:17:34.958 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap_name=snap 2023-11-08T05:17:34.959 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local s 2023-11-08T05:17:34.959 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-11-08T05:17:34.959 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 1 2023-11-08T05:17:35.958 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 info mirror/image_26@snap 2023-11-08T05:17:35.995 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:rbd image 'image_26': 2023-11-08T05:17:35.995 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: size 128 MiB in 32 objects 2023-11-08T05:17:35.995 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: order 22 (4 MiB objects) 2023-11-08T05:17:35.995 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: snapshot_count: 2 2023-11-08T05:17:35.996 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: id: 12a9684f2e32 2023-11-08T05:17:35.996 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: block_name_prefix: rbd_data.12a9684f2e32 2023-11-08T05:17:35.996 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: format: 2 2023-11-08T05:17:35.996 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: features: layering, exclusive-lock, non-primary 2023-11-08T05:17:35.996 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: op_features: 2023-11-08T05:17:35.997 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: flags: 2023-11-08T05:17:35.997 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: create_timestamp: Wed Nov 8 05:09:22 2023 2023-11-08T05:17:35.997 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: access_timestamp: Wed Nov 8 05:09:22 2023 2023-11-08T05:17:35.997 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: modify_timestamp: Wed Nov 8 05:09:22 2023 2023-11-08T05:17:35.998 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: protected: False 2023-11-08T05:17:35.998 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring state: enabled 2023-11-08T05:17:35.998 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring mode: snapshot 2023-11-08T05:17:35.999 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring global id: fff00620-df68-4ab5-8679-1bb60d01c79a 2023-11-08T05:17:35.999 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring primary: false 2023-11-08T05:17:36.000 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:17:36.000 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ compare_image_snaps mirror image_26 snap 2023-11-08T05:17:36.000 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:17:36.001 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_26 2023-11-08T05:17:36.001 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap_name=snap 2023-11-08T05:17:36.001 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local ret=0 2023-11-08T05:17:36.001 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local rmt_export=/tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_26.export 2023-11-08T05:17:36.001 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local loc_export=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_26.export 2023-11-08T05:17:36.002 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rm -f /tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_26.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_26.export 2023-11-08T05:17:36.002 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 -p mirror export image_26@snap /tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_26.export 2023-11-08T05:17:36.175 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-11-08T05:17:36.179 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 -p mirror export image_26@snap /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_26.export 2023-11-08T05:17:36.368 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-11-08T05:17:36.371 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ cmp /tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_26.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_26.export 2023-11-08T05:17:36.427 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rm -f /tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_26.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_26.export 2023-11-08T05:17:36.465 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:17:36.465 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-11-08T05:17:36.466 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image=image_27 2023-11-08T05:17:36.466 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ create_snap cluster2 mirror image_27 snap 2023-11-08T05:17:36.466 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:17:36.466 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:17:36.466 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_27 2023-11-08T05:17:36.467 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap_name=snap 2023-11-08T05:17:36.467 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 -p mirror snap create image_27@snap --debug-rbd=20 --debug-journaler=20 2023-11-08T05:17:38.325 INFO:tasks.ceph.cluster2.osd.2.smithi087.stderr:2023-11-08T05:17:38.323+0000 7fa243317700 -1 reset not still connected to 0x5613ba5e01a0 2023-11-08T05:17:38.398 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_image_replay_started cluster1 mirror image_27 2023-11-08T05:17:38.398 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:17:38.399 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:17:38.399 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_27 2023-11-08T05:17:38.399 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_image_replay_state cluster1 mirror image_27 started 2023-11-08T05:17:38.399 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:17:38.399 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:17:38.400 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_27 2023-11-08T05:17:38.400 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local state=started 2023-11-08T05:17:38.400 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local s 2023-11-08T05:17:38.400 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-11-08T05:17:38.400 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 1 2023-11-08T05:17:39.400 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test_image_replay_state cluster1 mirror image_27 started 2023-11-08T05:17:39.400 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:17:39.401 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:17:39.401 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_27 2023-11-08T05:17:39.401 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local test_state=started 2023-11-08T05:17:39.401 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local status_result 2023-11-08T05:17:39.401 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local current_state=stopped 2023-11-08T05:17:39.402 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_27 2023-11-08T05:17:39.402 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ grep -i state 2023-11-08T05:17:39.402 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_instance=cluster1 2023-11-08T05:17:39.402 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T05:17:39.402 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1 2023-11-08T05:17:39.403 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance=cluster1 2023-11-08T05:17:39.403 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local loop_instance 2023-11-08T05:17:39.403 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-11-08T05:17:39.403 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sleep 0 2023-11-08T05:17:39.405 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-11-08T05:17:39.405 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ seq 0 3 2023-11-08T05:17:39.405 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T05:17:39.406 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_27 2023-11-08T05:17:39.406 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1:0 2023-11-08T05:17:39.406 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T05:17:39.406 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance 2023-11-08T05:17:39.406 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-11-08T05:17:39.407 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local val=cluster1:0 2023-11-08T05:17:39.407 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_var_name=cluster 2023-11-08T05:17:39.407 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance_var_name=instance 2023-11-08T05:17:39.407 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cluster=cluster1 2023-11-08T05:17:39.407 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ instance=0 2023-11-08T05:17:39.408 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' 0 = cluster1:0 ']' 2023-11-08T05:17:39.408 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval cluster=cluster1 2023-11-08T05:17:39.408 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:17:39.408 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval instance=0 2023-11-08T05:17:39.408 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=0 2023-11-08T05:17:39.409 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-11-08T05:17:39.410 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local local_cluster=cluster1:0 2023-11-08T05:17:39.410 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster=cluster1 2023-11-08T05:17:39.411 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance 2023-11-08T05:17:39.411 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-11-08T05:17:39.411 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local val=cluster1:0 2023-11-08T05:17:39.411 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster_var_name=local_cluster 2023-11-08T05:17:39.411 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance_var_name=instance 2023-11-08T05:17:39.412 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:17:39.412 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=0 2023-11-08T05:17:39.412 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ '[' 0 = cluster1:0 ']' 2023-11-08T05:17:39.412 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval local_cluster=cluster1 2023-11-08T05:17:39.412 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ local_cluster=cluster1 2023-11-08T05:17:39.413 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval instance=0 2023-11-08T05:17:39.413 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ instance=0 2023-11-08T05:17:39.413 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-11-08T05:17:39.422 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-08T05:17:39.422 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-08T05:17:39.422 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-08T05:17:39.423 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_27 2023-11-08T05:17:39.526 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:Invalid command: unused arguments: ['mirror/image_27'] 2023-11-08T05:17:39.527 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd mirror status : get status for rbd mirror 2023-11-08T05:17:39.527 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:admin_socket: invalid command 2023-11-08T05:17:39.530 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T05:17:39.530 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_27 2023-11-08T05:17:39.530 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1:1 2023-11-08T05:17:39.530 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T05:17:39.530 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance 2023-11-08T05:17:39.531 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-11-08T05:17:39.531 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local val=cluster1:1 2023-11-08T05:17:39.531 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_var_name=cluster 2023-11-08T05:17:39.531 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance_var_name=instance 2023-11-08T05:17:39.531 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cluster=cluster1 2023-11-08T05:17:39.532 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ instance=1 2023-11-08T05:17:39.532 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' 1 = cluster1:1 ']' 2023-11-08T05:17:39.532 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval cluster=cluster1 2023-11-08T05:17:39.532 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:17:39.532 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval instance=1 2023-11-08T05:17:39.533 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=1 2023-11-08T05:17:39.533 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-11-08T05:17:39.533 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local local_cluster=cluster1:1 2023-11-08T05:17:39.533 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster=cluster1 2023-11-08T05:17:39.533 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance 2023-11-08T05:17:39.534 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-11-08T05:17:39.534 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local val=cluster1:1 2023-11-08T05:17:39.534 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster_var_name=local_cluster 2023-11-08T05:17:39.534 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance_var_name=instance 2023-11-08T05:17:39.534 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:17:39.535 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=1 2023-11-08T05:17:39.535 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ '[' 1 = cluster1:1 ']' 2023-11-08T05:17:39.535 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval local_cluster=cluster1 2023-11-08T05:17:39.535 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ local_cluster=cluster1 2023-11-08T05:17:39.535 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval instance=1 2023-11-08T05:17:39.535 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ instance=1 2023-11-08T05:17:39.536 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-11-08T05:17:39.544 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-08T05:17:39.544 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-08T05:17:39.545 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-08T05:17:39.545 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_27 2023-11-08T05:17:39.644 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:Invalid command: unused arguments: ['mirror/image_27'] 2023-11-08T05:17:39.644 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd mirror status : get status for rbd mirror 2023-11-08T05:17:39.645 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:admin_socket: invalid command 2023-11-08T05:17:39.647 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T05:17:39.648 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_27 2023-11-08T05:17:39.648 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1:2 2023-11-08T05:17:39.648 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T05:17:39.648 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance 2023-11-08T05:17:39.648 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-11-08T05:17:39.649 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local val=cluster1:2 2023-11-08T05:17:39.649 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_var_name=cluster 2023-11-08T05:17:39.649 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance_var_name=instance 2023-11-08T05:17:39.649 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cluster=cluster1 2023-11-08T05:17:39.650 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ instance=2 2023-11-08T05:17:39.650 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' 2 = cluster1:2 ']' 2023-11-08T05:17:39.650 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval cluster=cluster1 2023-11-08T05:17:39.650 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:17:39.650 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval instance=2 2023-11-08T05:17:39.651 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=2 2023-11-08T05:17:39.651 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-11-08T05:17:39.651 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local local_cluster=cluster1:2 2023-11-08T05:17:39.651 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster=cluster1 2023-11-08T05:17:39.651 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance 2023-11-08T05:17:39.652 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-11-08T05:17:39.652 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local val=cluster1:2 2023-11-08T05:17:39.652 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster_var_name=local_cluster 2023-11-08T05:17:39.652 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance_var_name=instance 2023-11-08T05:17:39.652 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:17:39.653 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=2 2023-11-08T05:17:39.653 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ '[' 2 = cluster1:2 ']' 2023-11-08T05:17:39.653 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval local_cluster=cluster1 2023-11-08T05:17:39.653 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ local_cluster=cluster1 2023-11-08T05:17:39.653 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval instance=2 2023-11-08T05:17:39.654 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ instance=2 2023-11-08T05:17:39.654 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-11-08T05:17:39.661 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-08T05:17:39.661 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-08T05:17:39.662 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-08T05:17:39.662 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_27 2023-11-08T05:17:39.761 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:Invalid command: unused arguments: ['mirror/image_27'] 2023-11-08T05:17:39.761 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd mirror status : get status for rbd mirror 2023-11-08T05:17:39.761 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:admin_socket: invalid command 2023-11-08T05:17:39.764 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T05:17:39.765 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_27 2023-11-08T05:17:39.765 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1:3 2023-11-08T05:17:39.765 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T05:17:39.765 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance 2023-11-08T05:17:39.765 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ set_cluster_instance cluster1:3 cluster instance 2023-11-08T05:17:39.766 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local val=cluster1:3 2023-11-08T05:17:39.766 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_var_name=cluster 2023-11-08T05:17:39.766 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance_var_name=instance 2023-11-08T05:17:39.766 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cluster=cluster1 2023-11-08T05:17:39.766 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ instance=3 2023-11-08T05:17:39.767 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' 3 = cluster1:3 ']' 2023-11-08T05:17:39.767 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval cluster=cluster1 2023-11-08T05:17:39.767 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:17:39.767 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval instance=3 2023-11-08T05:17:39.767 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=3 2023-11-08T05:17:39.768 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ daemon_asok_file cluster1:3 cluster1 2023-11-08T05:17:39.768 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local local_cluster=cluster1:3 2023-11-08T05:17:39.768 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster=cluster1 2023-11-08T05:17:39.768 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance 2023-11-08T05:17:39.768 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2023-11-08T05:17:39.769 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local val=cluster1:3 2023-11-08T05:17:39.769 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster_var_name=local_cluster 2023-11-08T05:17:39.769 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance_var_name=instance 2023-11-08T05:17:39.769 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:17:39.769 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=3 2023-11-08T05:17:39.770 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ '[' 3 = cluster1:3 ']' 2023-11-08T05:17:39.770 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval local_cluster=cluster1 2023-11-08T05:17:39.770 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ local_cluster=cluster1 2023-11-08T05:17:39.770 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval instance=3 2023-11-08T05:17:39.771 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ instance=3 2023-11-08T05:17:39.771 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2023-11-08T05:17:39.780 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-11-08T05:17:39.780 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-11-08T05:17:39.780 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-11-08T05:17:39.781 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_27 2023-11-08T05:17:39.887 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ return 0 2023-11-08T05:17:39.887 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ status_result=' "state": "Replaying"' 2023-11-08T05:17:39.888 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ echo ' "state": "Replaying"' 2023-11-08T05:17:39.888 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ grep -i Replaying 2023-11-08T05:17:39.888 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: "state": "Replaying" 2023-11-08T05:17:39.891 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ current_state=started 2023-11-08T05:17:39.891 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test started = started 2023-11-08T05:17:39.891 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:17:39.891 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_27 2023-11-08T05:17:39.892 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local local_cluster=cluster1 2023-11-08T05:17:39.892 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:17:39.892 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:17:39.892 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_27 2023-11-08T05:17:39.893 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' snapshot = journal ']' 2023-11-08T05:17:39.893 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' snapshot = snapshot ']' 2023-11-08T05:17:39.893 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_27 2023-11-08T05:17:39.893 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local local_cluster=cluster1 2023-11-08T05:17:39.893 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:17:39.894 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:17:39.894 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_27 2023-11-08T05:17:39.894 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ mkfname cluster2-mirror-image_27.status 2023-11-08T05:17:39.894 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ echo cluster2-mirror-image_27.status 2023-11-08T05:17:39.894 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -e 's|[/ ]|_|g' 2023-11-08T05:17:39.895 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local status_log=/tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_27.status 2023-11-08T05:17:39.895 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ mkfname cluster1-mirror-image_27.status 2023-11-08T05:17:39.895 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ echo cluster1-mirror-image_27.status 2023-11-08T05:17:39.895 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -e 's|[/ ]|_|g' 2023-11-08T05:17:39.896 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local local_status_log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_27.status 2023-11-08T05:17:39.896 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ mirror_image_snapshot cluster2 mirror image_27 2023-11-08T05:17:39.897 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:17:39.897 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:17:39.897 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_27 2023-11-08T05:17:39.897 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_27 2023-11-08T05:17:40.401 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:Snapshot ID: 171 2023-11-08T05:17:40.410 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_27 /tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_27.status 2023-11-08T05:17:40.410 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:17:40.411 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:17:40.411 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_27 2023-11-08T05:17:40.411 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_27.status 2023-11-08T05:17:40.411 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_27 --format xml 2023-11-08T05:17:40.411 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T05:17:40.649 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-11-08T05:17:40.651 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snapshot_id=171 2023-11-08T05:17:40.651 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ true 2023-11-08T05:17:40.652 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-11-08T05:17:40.652 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0.2 2023-11-08T05:17:40.853 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_27 /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_27.status 2023-11-08T05:17:40.853 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:17:40.854 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:17:40.854 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_27 2023-11-08T05:17:40.854 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_27.status 2023-11-08T05:17:40.854 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_27 --format xml 2023-11-08T05:17:40.855 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T05:17:40.894 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ true 2023-11-08T05:17:40.894 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-08T05:17:40.895 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:-:1.1: Document is empty 2023-11-08T05:17:40.895 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: 2023-11-08T05:17:40.896 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:^ 2023-11-08T05:17:40.897 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local primary_snapshot_id= 2023-11-08T05:17:40.897 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test 171 = '' 2023-11-08T05:17:40.897 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-11-08T05:17:40.897 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0.4 2023-11-08T05:17:41.297 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_27 /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_27.status 2023-11-08T05:17:41.297 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:17:41.297 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:17:41.298 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_27 2023-11-08T05:17:41.298 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_27.status 2023-11-08T05:17:41.298 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_27 --format xml 2023-11-08T05:17:41.298 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T05:17:41.343 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ true 2023-11-08T05:17:41.343 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-08T05:17:41.344 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:-:1.1: Document is empty 2023-11-08T05:17:41.345 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: 2023-11-08T05:17:41.345 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:^ 2023-11-08T05:17:41.346 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local primary_snapshot_id= 2023-11-08T05:17:41.346 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test 171 = '' 2023-11-08T05:17:41.346 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-11-08T05:17:41.346 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0.8 2023-11-08T05:17:42.147 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_27 /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_27.status 2023-11-08T05:17:42.147 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:17:42.147 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:17:42.148 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_27 2023-11-08T05:17:42.148 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_27.status 2023-11-08T05:17:42.148 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_27 --format xml 2023-11-08T05:17:42.148 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T05:17:42.188 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-08T05:17:42.191 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local primary_snapshot_id=171 2023-11-08T05:17:42.191 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test 171 = 171 2023-11-08T05:17:42.192 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:17:42.192 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_snap_present cluster1 mirror image_27 snap 2023-11-08T05:17:42.192 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:17:42.193 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:17:42.193 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_27 2023-11-08T05:17:42.193 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap_name=snap 2023-11-08T05:17:42.194 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local s 2023-11-08T05:17:42.194 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-11-08T05:17:42.194 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 1 2023-11-08T05:17:43.191 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 info mirror/image_27@snap 2023-11-08T05:17:43.231 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:rbd image 'image_27': 2023-11-08T05:17:43.231 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: size 128 MiB in 32 objects 2023-11-08T05:17:43.232 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: order 22 (4 MiB objects) 2023-11-08T05:17:43.232 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: snapshot_count: 2 2023-11-08T05:17:43.232 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: id: 12a98de117e7 2023-11-08T05:17:43.232 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: block_name_prefix: rbd_data.12a98de117e7 2023-11-08T05:17:43.233 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: format: 2 2023-11-08T05:17:43.233 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: features: layering, exclusive-lock, non-primary 2023-11-08T05:17:43.233 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: op_features: 2023-11-08T05:17:43.234 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: flags: 2023-11-08T05:17:43.234 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: create_timestamp: Wed Nov 8 05:09:03 2023 2023-11-08T05:17:43.234 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: access_timestamp: Wed Nov 8 05:09:03 2023 2023-11-08T05:17:43.235 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: modify_timestamp: Wed Nov 8 05:09:03 2023 2023-11-08T05:17:43.235 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: protected: False 2023-11-08T05:17:43.236 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring state: enabled 2023-11-08T05:17:43.236 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring mode: snapshot 2023-11-08T05:17:43.236 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring global id: ebbdc03f-1275-4b8e-b5a6-94b379cd1440 2023-11-08T05:17:43.237 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring primary: false 2023-11-08T05:17:43.239 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:17:43.239 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ compare_image_snaps mirror image_27 snap 2023-11-08T05:17:43.239 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:17:43.240 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_27 2023-11-08T05:17:43.240 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap_name=snap 2023-11-08T05:17:43.240 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local ret=0 2023-11-08T05:17:43.240 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local rmt_export=/tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_27.export 2023-11-08T05:17:43.240 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local loc_export=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_27.export 2023-11-08T05:17:43.241 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rm -f /tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_27.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_27.export 2023-11-08T05:17:43.241 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 -p mirror export image_27@snap /tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_27.export 2023-11-08T05:17:43.414 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-11-08T05:17:43.417 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 -p mirror export image_27@snap /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_27.export 2023-11-08T05:17:43.631 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-11-08T05:17:43.634 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ cmp /tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_27.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_27.export 2023-11-08T05:17:43.688 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rm -f /tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_27.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_27.export 2023-11-08T05:17:43.725 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:17:43.725 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-11-08T05:17:43.725 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image=image_28 2023-11-08T05:17:43.726 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ create_snap cluster2 mirror image_28 snap 2023-11-08T05:17:43.726 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:17:43.726 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:17:43.726 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_28 2023-11-08T05:17:43.726 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap_name=snap 2023-11-08T05:17:43.727 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 -p mirror snap create image_28@snap --debug-rbd=20 --debug-journaler=20 2023-11-08T05:17:45.422 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_image_replay_started cluster1 mirror image_28 2023-11-08T05:17:45.422 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:17:45.423 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:17:45.423 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_28 2023-11-08T05:17:45.423 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_image_replay_state cluster1 mirror image_28 started 2023-11-08T05:17:45.424 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:17:45.424 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:17:45.424 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_28 2023-11-08T05:17:45.424 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local state=started 2023-11-08T05:17:45.425 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local s 2023-11-08T05:17:45.425 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-11-08T05:17:45.425 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 1 2023-11-08T05:17:46.422 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test_image_replay_state cluster1 mirror image_28 started 2023-11-08T05:17:46.422 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:17:46.423 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:17:46.423 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_28 2023-11-08T05:17:46.423 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local test_state=started 2023-11-08T05:17:46.423 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local status_result 2023-11-08T05:17:46.423 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local current_state=stopped 2023-11-08T05:17:46.425 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_28 2023-11-08T05:17:46.425 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_instance=cluster1 2023-11-08T05:17:46.425 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T05:17:46.425 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ grep -i state 2023-11-08T05:17:46.426 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1 2023-11-08T05:17:46.426 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance=cluster1 2023-11-08T05:17:46.426 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local loop_instance 2023-11-08T05:17:46.426 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-11-08T05:17:46.427 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sleep 0 2023-11-08T05:17:46.427 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-11-08T05:17:46.427 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ seq 0 3 2023-11-08T05:17:46.430 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T05:17:46.430 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_28 2023-11-08T05:17:46.431 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1:0 2023-11-08T05:17:46.431 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T05:17:46.431 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance 2023-11-08T05:17:46.431 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-11-08T05:17:46.431 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local val=cluster1:0 2023-11-08T05:17:46.432 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_var_name=cluster 2023-11-08T05:17:46.432 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance_var_name=instance 2023-11-08T05:17:46.432 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cluster=cluster1 2023-11-08T05:17:46.432 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ instance=0 2023-11-08T05:17:46.432 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' 0 = cluster1:0 ']' 2023-11-08T05:17:46.433 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval cluster=cluster1 2023-11-08T05:17:46.433 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:17:46.433 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval instance=0 2023-11-08T05:17:46.433 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=0 2023-11-08T05:17:46.433 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-11-08T05:17:46.434 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local local_cluster=cluster1:0 2023-11-08T05:17:46.434 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster=cluster1 2023-11-08T05:17:46.434 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance 2023-11-08T05:17:46.434 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-11-08T05:17:46.434 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local val=cluster1:0 2023-11-08T05:17:46.435 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster_var_name=local_cluster 2023-11-08T05:17:46.435 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance_var_name=instance 2023-11-08T05:17:46.435 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:17:46.435 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=0 2023-11-08T05:17:46.435 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ '[' 0 = cluster1:0 ']' 2023-11-08T05:17:46.436 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval local_cluster=cluster1 2023-11-08T05:17:46.436 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ local_cluster=cluster1 2023-11-08T05:17:46.436 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval instance=0 2023-11-08T05:17:46.436 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ instance=0 2023-11-08T05:17:46.436 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-11-08T05:17:46.445 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-08T05:17:46.445 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-08T05:17:46.445 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-08T05:17:46.445 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_28 2023-11-08T05:17:46.556 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:Invalid command: unused arguments: ['mirror/image_28'] 2023-11-08T05:17:46.556 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd mirror status : get status for rbd mirror 2023-11-08T05:17:46.556 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:admin_socket: invalid command 2023-11-08T05:17:46.559 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T05:17:46.559 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_28 2023-11-08T05:17:46.559 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1:1 2023-11-08T05:17:46.560 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T05:17:46.560 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance 2023-11-08T05:17:46.560 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-11-08T05:17:46.560 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local val=cluster1:1 2023-11-08T05:17:46.560 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_var_name=cluster 2023-11-08T05:17:46.561 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance_var_name=instance 2023-11-08T05:17:46.561 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cluster=cluster1 2023-11-08T05:17:46.561 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ instance=1 2023-11-08T05:17:46.561 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' 1 = cluster1:1 ']' 2023-11-08T05:17:46.561 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval cluster=cluster1 2023-11-08T05:17:46.562 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:17:46.562 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval instance=1 2023-11-08T05:17:46.562 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=1 2023-11-08T05:17:46.562 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-11-08T05:17:46.562 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local local_cluster=cluster1:1 2023-11-08T05:17:46.563 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster=cluster1 2023-11-08T05:17:46.563 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance 2023-11-08T05:17:46.563 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-11-08T05:17:46.563 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local val=cluster1:1 2023-11-08T05:17:46.563 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster_var_name=local_cluster 2023-11-08T05:17:46.564 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance_var_name=instance 2023-11-08T05:17:46.564 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:17:46.564 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=1 2023-11-08T05:17:46.564 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ '[' 1 = cluster1:1 ']' 2023-11-08T05:17:46.564 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval local_cluster=cluster1 2023-11-08T05:17:46.565 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ local_cluster=cluster1 2023-11-08T05:17:46.565 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval instance=1 2023-11-08T05:17:46.565 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ instance=1 2023-11-08T05:17:46.566 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-11-08T05:17:46.573 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-08T05:17:46.574 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-08T05:17:46.574 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-08T05:17:46.574 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_28 2023-11-08T05:17:46.672 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:Invalid command: unused arguments: ['mirror/image_28'] 2023-11-08T05:17:46.672 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd mirror status : get status for rbd mirror 2023-11-08T05:17:46.672 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:admin_socket: invalid command 2023-11-08T05:17:46.676 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T05:17:46.676 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_28 2023-11-08T05:17:46.677 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1:2 2023-11-08T05:17:46.677 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T05:17:46.677 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance 2023-11-08T05:17:46.677 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-11-08T05:17:46.678 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local val=cluster1:2 2023-11-08T05:17:46.678 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_var_name=cluster 2023-11-08T05:17:46.678 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance_var_name=instance 2023-11-08T05:17:46.678 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cluster=cluster1 2023-11-08T05:17:46.678 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ instance=2 2023-11-08T05:17:46.679 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' 2 = cluster1:2 ']' 2023-11-08T05:17:46.679 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval cluster=cluster1 2023-11-08T05:17:46.679 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:17:46.679 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval instance=2 2023-11-08T05:17:46.680 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=2 2023-11-08T05:17:46.680 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-11-08T05:17:46.680 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local local_cluster=cluster1:2 2023-11-08T05:17:46.681 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster=cluster1 2023-11-08T05:17:46.681 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance 2023-11-08T05:17:46.681 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-11-08T05:17:46.681 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local val=cluster1:2 2023-11-08T05:17:46.681 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster_var_name=local_cluster 2023-11-08T05:17:46.682 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance_var_name=instance 2023-11-08T05:17:46.682 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:17:46.682 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=2 2023-11-08T05:17:46.682 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ '[' 2 = cluster1:2 ']' 2023-11-08T05:17:46.683 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval local_cluster=cluster1 2023-11-08T05:17:46.683 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ local_cluster=cluster1 2023-11-08T05:17:46.683 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval instance=2 2023-11-08T05:17:46.683 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ instance=2 2023-11-08T05:17:46.683 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-11-08T05:17:46.689 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-08T05:17:46.689 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-08T05:17:46.689 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-08T05:17:46.690 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_28 2023-11-08T05:17:46.796 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ return 0 2023-11-08T05:17:46.797 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ status_result=' "state": "Replaying"' 2023-11-08T05:17:46.797 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ echo ' "state": "Replaying"' 2023-11-08T05:17:46.797 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ grep -i Replaying 2023-11-08T05:17:46.798 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: "state": "Replaying" 2023-11-08T05:17:46.800 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ current_state=started 2023-11-08T05:17:46.800 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test started = started 2023-11-08T05:17:46.801 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:17:46.801 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_28 2023-11-08T05:17:46.802 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local local_cluster=cluster1 2023-11-08T05:17:46.802 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:17:46.802 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:17:46.802 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_28 2023-11-08T05:17:46.803 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' snapshot = journal ']' 2023-11-08T05:17:46.803 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' snapshot = snapshot ']' 2023-11-08T05:17:46.803 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_28 2023-11-08T05:17:46.803 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local local_cluster=cluster1 2023-11-08T05:17:46.803 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:17:46.804 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:17:46.804 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_28 2023-11-08T05:17:46.804 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ mkfname cluster2-mirror-image_28.status 2023-11-08T05:17:46.804 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ echo cluster2-mirror-image_28.status 2023-11-08T05:17:46.804 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -e 's|[/ ]|_|g' 2023-11-08T05:17:46.804 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local status_log=/tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_28.status 2023-11-08T05:17:46.805 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ mkfname cluster1-mirror-image_28.status 2023-11-08T05:17:46.805 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ echo cluster1-mirror-image_28.status 2023-11-08T05:17:46.805 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -e 's|[/ ]|_|g' 2023-11-08T05:17:46.806 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local local_status_log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_28.status 2023-11-08T05:17:46.806 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ mirror_image_snapshot cluster2 mirror image_28 2023-11-08T05:17:46.806 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:17:46.807 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:17:46.807 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_28 2023-11-08T05:17:46.807 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_28 2023-11-08T05:17:47.425 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:Snapshot ID: 173 2023-11-08T05:17:47.434 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_28 /tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_28.status 2023-11-08T05:17:47.434 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:17:47.435 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:17:47.435 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_28 2023-11-08T05:17:47.435 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_28.status 2023-11-08T05:17:47.435 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_28 --format xml 2023-11-08T05:17:47.435 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T05:17:47.473 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-11-08T05:17:47.475 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snapshot_id=173 2023-11-08T05:17:47.476 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ true 2023-11-08T05:17:47.476 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-11-08T05:17:47.476 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0.2 2023-11-08T05:17:47.677 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_28 /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_28.status 2023-11-08T05:17:47.677 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:17:47.677 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:17:47.678 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_28 2023-11-08T05:17:47.678 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_28.status 2023-11-08T05:17:47.678 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_28 --format xml 2023-11-08T05:17:47.678 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T05:17:47.717 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ true 2023-11-08T05:17:47.717 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-08T05:17:47.718 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:-:1.1: Document is empty 2023-11-08T05:17:47.718 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: 2023-11-08T05:17:47.719 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:^ 2023-11-08T05:17:47.720 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local primary_snapshot_id= 2023-11-08T05:17:47.720 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test 173 = '' 2023-11-08T05:17:47.720 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-11-08T05:17:47.720 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0.4 2023-11-08T05:17:48.120 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_28 /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_28.status 2023-11-08T05:17:48.120 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:17:48.121 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:17:48.121 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_28 2023-11-08T05:17:48.121 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_28.status 2023-11-08T05:17:48.121 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_28 --format xml 2023-11-08T05:17:48.122 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T05:17:48.162 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ true 2023-11-08T05:17:48.162 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-08T05:17:48.163 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:-:1.1: Document is empty 2023-11-08T05:17:48.163 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: 2023-11-08T05:17:48.164 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:^ 2023-11-08T05:17:48.164 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local primary_snapshot_id= 2023-11-08T05:17:48.165 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test 173 = '' 2023-11-08T05:17:48.165 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-11-08T05:17:48.165 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0.8 2023-11-08T05:17:48.207 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.8492437229338945) > weight (0.25) 2023-11-08T05:17:48.208 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.8875004454497903) > weight (0.25) 2023-11-08T05:17:48.208 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.4869715830034326) > weight (0.25) 2023-11-08T05:17:48.208 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.3 2023-11-08T05:17:48.209 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Sent signal 15 2023-11-08T05:17:48.209 INFO:tasks.rbd_mirror_thrash:waiting for 5 secs before reviving daemons 2023-11-08T05:17:48.210 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi087.stderr:2023-11-08T05:17:48.208+0000 7efed2fb1700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.3 (PID: 135877) UID: 1000 2023-11-08T05:17:48.221 INFO:tasks.ceph.cluster1.osd.1.smithi077.stderr:2023-11-08T05:17:48.218+0000 7f63db900700 -1 reset not still connected to 0x5634df5cf6c0 2023-11-08T05:17:48.221 INFO:tasks.ceph.cluster1.osd.1.smithi077.stderr:2023-11-08T05:17:48.218+0000 7f63db900700 -1 reset not still connected to 0x5634dfccc680 2023-11-08T05:17:48.410 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi087.stderr:daemon-helper: command failed with exit status 1 2023-11-08T05:17:48.965 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_28 /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_28.status 2023-11-08T05:17:48.966 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:17:48.966 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:17:48.966 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_28 2023-11-08T05:17:48.966 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_28.status 2023-11-08T05:17:48.966 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_28 --format xml 2023-11-08T05:17:48.967 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T05:17:49.003 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-08T05:17:49.005 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local primary_snapshot_id=173 2023-11-08T05:17:49.005 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test 173 = 173 2023-11-08T05:17:49.005 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:17:49.005 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_snap_present cluster1 mirror image_28 snap 2023-11-08T05:17:49.006 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:17:49.006 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:17:49.006 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_28 2023-11-08T05:17:49.006 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap_name=snap 2023-11-08T05:17:49.006 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local s 2023-11-08T05:17:49.007 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-11-08T05:17:49.007 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 1 2023-11-08T05:17:50.005 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 info mirror/image_28@snap 2023-11-08T05:17:50.038 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:rbd image 'image_28': 2023-11-08T05:17:50.039 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: size 128 MiB in 32 objects 2023-11-08T05:17:50.039 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: order 22 (4 MiB objects) 2023-11-08T05:17:50.039 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: snapshot_count: 2 2023-11-08T05:17:50.039 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: id: 12a9fdf64f04 2023-11-08T05:17:50.040 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: block_name_prefix: rbd_data.12a9fdf64f04 2023-11-08T05:17:50.040 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: format: 2 2023-11-08T05:17:50.040 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: features: layering, exclusive-lock, non-primary 2023-11-08T05:17:50.040 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: op_features: 2023-11-08T05:17:50.041 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: flags: 2023-11-08T05:17:50.041 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: create_timestamp: Wed Nov 8 05:09:17 2023 2023-11-08T05:17:50.041 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: access_timestamp: Wed Nov 8 05:09:17 2023 2023-11-08T05:17:50.041 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: modify_timestamp: Wed Nov 8 05:09:17 2023 2023-11-08T05:17:50.041 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: protected: False 2023-11-08T05:17:50.042 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring state: enabled 2023-11-08T05:17:50.042 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring mode: snapshot 2023-11-08T05:17:50.042 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring global id: 06978a7d-19c8-42c0-8df8-1e3a18982290 2023-11-08T05:17:50.042 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring primary: false 2023-11-08T05:17:50.043 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:17:50.044 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ compare_image_snaps mirror image_28 snap 2023-11-08T05:17:50.044 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:17:50.044 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_28 2023-11-08T05:17:50.044 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap_name=snap 2023-11-08T05:17:50.045 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local ret=0 2023-11-08T05:17:50.045 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local rmt_export=/tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_28.export 2023-11-08T05:17:50.045 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local loc_export=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_28.export 2023-11-08T05:17:50.045 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rm -f /tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_28.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_28.export 2023-11-08T05:17:50.045 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 -p mirror export image_28@snap /tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_28.export 2023-11-08T05:17:50.206 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-11-08T05:17:50.209 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 -p mirror export image_28@snap /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_28.export 2023-11-08T05:17:50.462 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-11-08T05:17:50.464 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ cmp /tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_28.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_28.export 2023-11-08T05:17:50.519 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rm -f /tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_28.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_28.export 2023-11-08T05:17:50.557 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:17:50.558 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-11-08T05:17:50.558 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image=image_29 2023-11-08T05:17:50.559 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ create_snap cluster2 mirror image_29 snap 2023-11-08T05:17:50.559 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:17:50.559 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:17:50.560 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_29 2023-11-08T05:17:50.560 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap_name=snap 2023-11-08T05:17:50.561 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 -p mirror snap create image_29@snap --debug-rbd=20 --debug-journaler=20 2023-11-08T05:17:52.437 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_image_replay_started cluster1 mirror image_29 2023-11-08T05:17:52.438 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:17:52.438 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:17:52.438 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_29 2023-11-08T05:17:52.439 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_image_replay_state cluster1 mirror image_29 started 2023-11-08T05:17:52.441 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:17:52.442 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:17:52.442 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_29 2023-11-08T05:17:52.442 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local state=started 2023-11-08T05:17:52.442 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local s 2023-11-08T05:17:52.442 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-11-08T05:17:52.443 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 1 2023-11-08T05:17:53.210 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.3 2023-11-08T05:17:53.211 INFO:teuthology.orchestra.run:waiting for 600 2023-11-08T05:17:53.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-08T05:17:53.211 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.3 2023-11-08T05:17:53.212 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Restarting daemon 2023-11-08T05:17:53.212 DEBUG:teuthology.orchestra.run.smithi087:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.3 2023-11-08T05:17:53.215 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Started 2023-11-08T05:17:53.215 INFO:tasks.rbd_mirror_thrash:waiting for 94 secs before thrashing 2023-11-08T05:17:53.439 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test_image_replay_state cluster1 mirror image_29 started 2023-11-08T05:17:53.440 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:17:53.440 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:17:53.440 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_29 2023-11-08T05:17:53.440 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local test_state=started 2023-11-08T05:17:53.441 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local status_result 2023-11-08T05:17:53.441 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local current_state=stopped 2023-11-08T05:17:53.441 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_29 2023-11-08T05:17:53.441 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_instance=cluster1 2023-11-08T05:17:53.441 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T05:17:53.442 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ grep -i state 2023-11-08T05:17:53.442 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1 2023-11-08T05:17:53.442 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance=cluster1 2023-11-08T05:17:53.442 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local loop_instance 2023-11-08T05:17:53.442 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-11-08T05:17:53.443 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sleep 0 2023-11-08T05:17:53.444 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-11-08T05:17:53.444 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ seq 0 3 2023-11-08T05:17:53.445 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T05:17:53.445 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_29 2023-11-08T05:17:53.445 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1:0 2023-11-08T05:17:53.445 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T05:17:53.445 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance 2023-11-08T05:17:53.446 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-11-08T05:17:53.446 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local val=cluster1:0 2023-11-08T05:17:53.446 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_var_name=cluster 2023-11-08T05:17:53.446 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance_var_name=instance 2023-11-08T05:17:53.446 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cluster=cluster1 2023-11-08T05:17:53.447 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ instance=0 2023-11-08T05:17:53.447 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' 0 = cluster1:0 ']' 2023-11-08T05:17:53.447 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval cluster=cluster1 2023-11-08T05:17:53.448 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:17:53.450 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval instance=0 2023-11-08T05:17:53.450 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=0 2023-11-08T05:17:53.450 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-11-08T05:17:53.450 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local local_cluster=cluster1:0 2023-11-08T05:17:53.451 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster=cluster1 2023-11-08T05:17:53.451 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance 2023-11-08T05:17:53.451 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-11-08T05:17:53.451 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local val=cluster1:0 2023-11-08T05:17:53.451 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster_var_name=local_cluster 2023-11-08T05:17:53.452 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance_var_name=instance 2023-11-08T05:17:53.452 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:17:53.452 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=0 2023-11-08T05:17:53.452 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ '[' 0 = cluster1:0 ']' 2023-11-08T05:17:53.452 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval local_cluster=cluster1 2023-11-08T05:17:53.453 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ local_cluster=cluster1 2023-11-08T05:17:53.453 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval instance=0 2023-11-08T05:17:53.453 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ instance=0 2023-11-08T05:17:53.453 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-11-08T05:17:53.458 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-08T05:17:53.458 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-08T05:17:53.458 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-08T05:17:53.459 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_29 2023-11-08T05:17:53.562 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:Invalid command: unused arguments: ['mirror/image_29'] 2023-11-08T05:17:53.562 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd mirror status : get status for rbd mirror 2023-11-08T05:17:53.563 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:admin_socket: invalid command 2023-11-08T05:17:53.565 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T05:17:53.565 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_29 2023-11-08T05:17:53.565 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1:1 2023-11-08T05:17:53.566 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T05:17:53.566 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance 2023-11-08T05:17:53.566 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-11-08T05:17:53.566 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local val=cluster1:1 2023-11-08T05:17:53.567 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_var_name=cluster 2023-11-08T05:17:53.567 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance_var_name=instance 2023-11-08T05:17:53.567 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cluster=cluster1 2023-11-08T05:17:53.567 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ instance=1 2023-11-08T05:17:53.567 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' 1 = cluster1:1 ']' 2023-11-08T05:17:53.568 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval cluster=cluster1 2023-11-08T05:17:53.568 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:17:53.568 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval instance=1 2023-11-08T05:17:53.568 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=1 2023-11-08T05:17:53.568 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-11-08T05:17:53.569 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local local_cluster=cluster1:1 2023-11-08T05:17:53.569 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster=cluster1 2023-11-08T05:17:53.569 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance 2023-11-08T05:17:53.569 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-11-08T05:17:53.569 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local val=cluster1:1 2023-11-08T05:17:53.570 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster_var_name=local_cluster 2023-11-08T05:17:53.570 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance_var_name=instance 2023-11-08T05:17:53.570 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:17:53.570 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=1 2023-11-08T05:17:53.570 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ '[' 1 = cluster1:1 ']' 2023-11-08T05:17:53.571 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval local_cluster=cluster1 2023-11-08T05:17:53.571 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ local_cluster=cluster1 2023-11-08T05:17:53.571 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval instance=1 2023-11-08T05:17:53.571 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ instance=1 2023-11-08T05:17:53.571 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-11-08T05:17:53.582 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-08T05:17:53.582 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-08T05:17:53.583 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-08T05:17:53.583 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_29 2023-11-08T05:17:53.685 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:Invalid command: unused arguments: ['mirror/image_29'] 2023-11-08T05:17:53.686 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd mirror status : get status for rbd mirror 2023-11-08T05:17:53.686 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:admin_socket: invalid command 2023-11-08T05:17:53.689 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T05:17:53.689 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_29 2023-11-08T05:17:53.689 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1:2 2023-11-08T05:17:53.690 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T05:17:53.690 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance 2023-11-08T05:17:53.690 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-11-08T05:17:53.690 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local val=cluster1:2 2023-11-08T05:17:53.690 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_var_name=cluster 2023-11-08T05:17:53.691 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance_var_name=instance 2023-11-08T05:17:53.691 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cluster=cluster1 2023-11-08T05:17:53.691 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ instance=2 2023-11-08T05:17:53.691 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' 2 = cluster1:2 ']' 2023-11-08T05:17:53.691 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval cluster=cluster1 2023-11-08T05:17:53.692 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:17:53.692 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval instance=2 2023-11-08T05:17:53.692 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=2 2023-11-08T05:17:53.692 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-11-08T05:17:53.692 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local local_cluster=cluster1:2 2023-11-08T05:17:53.693 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster=cluster1 2023-11-08T05:17:53.693 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance 2023-11-08T05:17:53.693 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-11-08T05:17:53.693 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local val=cluster1:2 2023-11-08T05:17:53.693 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster_var_name=local_cluster 2023-11-08T05:17:53.694 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance_var_name=instance 2023-11-08T05:17:53.694 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:17:53.694 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=2 2023-11-08T05:17:53.694 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ '[' 2 = cluster1:2 ']' 2023-11-08T05:17:53.695 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval local_cluster=cluster1 2023-11-08T05:17:53.695 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ local_cluster=cluster1 2023-11-08T05:17:53.695 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval instance=2 2023-11-08T05:17:53.695 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ instance=2 2023-11-08T05:17:53.695 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-11-08T05:17:53.703 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-08T05:17:53.703 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-08T05:17:53.703 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-08T05:17:53.703 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_29 2023-11-08T05:17:53.810 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ return 0 2023-11-08T05:17:53.811 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ status_result=' "state": "Replaying"' 2023-11-08T05:17:53.812 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ echo ' "state": "Replaying"' 2023-11-08T05:17:53.812 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ grep -i Replaying 2023-11-08T05:17:53.812 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: "state": "Replaying" 2023-11-08T05:17:53.814 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ current_state=started 2023-11-08T05:17:53.815 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test started = started 2023-11-08T05:17:53.815 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:17:53.815 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_29 2023-11-08T05:17:53.815 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local local_cluster=cluster1 2023-11-08T05:17:53.815 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:17:53.816 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:17:53.816 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_29 2023-11-08T05:17:53.816 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' snapshot = journal ']' 2023-11-08T05:17:53.816 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' snapshot = snapshot ']' 2023-11-08T05:17:53.816 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_29 2023-11-08T05:17:53.817 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local local_cluster=cluster1 2023-11-08T05:17:53.817 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:17:53.817 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:17:53.817 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_29 2023-11-08T05:17:53.817 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ mkfname cluster2-mirror-image_29.status 2023-11-08T05:17:53.818 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ echo cluster2-mirror-image_29.status 2023-11-08T05:17:53.818 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -e 's|[/ ]|_|g' 2023-11-08T05:17:53.818 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local status_log=/tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_29.status 2023-11-08T05:17:53.818 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ mkfname cluster1-mirror-image_29.status 2023-11-08T05:17:53.818 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ echo cluster1-mirror-image_29.status 2023-11-08T05:17:53.818 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -e 's|[/ ]|_|g' 2023-11-08T05:17:53.819 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local local_status_log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_29.status 2023-11-08T05:17:53.820 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ mirror_image_snapshot cluster2 mirror image_29 2023-11-08T05:17:53.820 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:17:53.820 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:17:53.820 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_29 2023-11-08T05:17:53.820 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_29 2023-11-08T05:17:54.444 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:Snapshot ID: 175 2023-11-08T05:17:54.452 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_29 /tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_29.status 2023-11-08T05:17:54.452 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:17:54.452 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:17:54.453 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_29 2023-11-08T05:17:54.453 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_29.status 2023-11-08T05:17:54.453 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_29 --format xml 2023-11-08T05:17:54.454 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T05:17:54.490 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-11-08T05:17:54.492 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snapshot_id=175 2023-11-08T05:17:54.493 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ true 2023-11-08T05:17:54.493 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-11-08T05:17:54.493 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0.2 2023-11-08T05:17:54.693 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_29 /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_29.status 2023-11-08T05:17:54.694 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:17:54.694 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:17:54.694 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_29 2023-11-08T05:17:54.694 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_29.status 2023-11-08T05:17:54.695 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_29 --format xml 2023-11-08T05:17:54.695 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T05:17:54.935 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ true 2023-11-08T05:17:54.935 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-08T05:17:54.936 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:-:1.1: Document is empty 2023-11-08T05:17:54.936 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: 2023-11-08T05:17:54.936 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:^ 2023-11-08T05:17:54.937 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local primary_snapshot_id= 2023-11-08T05:17:54.937 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test 175 = '' 2023-11-08T05:17:54.938 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-11-08T05:17:54.938 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0.4 2023-11-08T05:17:55.338 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_29 /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_29.status 2023-11-08T05:17:55.338 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:17:55.339 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:17:55.339 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_29 2023-11-08T05:17:55.339 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_29.status 2023-11-08T05:17:55.339 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_29 --format xml 2023-11-08T05:17:55.339 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T05:17:55.378 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ true 2023-11-08T05:17:55.379 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-08T05:17:55.380 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:-:1.1: Document is empty 2023-11-08T05:17:55.380 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: 2023-11-08T05:17:55.380 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:^ 2023-11-08T05:17:55.381 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local primary_snapshot_id= 2023-11-08T05:17:55.381 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test 175 = '' 2023-11-08T05:17:55.381 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-11-08T05:17:55.382 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0.8 2023-11-08T05:17:56.182 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_29 /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_29.status 2023-11-08T05:17:56.182 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:17:56.182 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:17:56.183 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_29 2023-11-08T05:17:56.183 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_29.status 2023-11-08T05:17:56.183 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_29 --format xml 2023-11-08T05:17:56.183 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T05:17:56.220 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-08T05:17:56.223 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local primary_snapshot_id=175 2023-11-08T05:17:56.223 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test 175 = 175 2023-11-08T05:17:56.223 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:17:56.224 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_snap_present cluster1 mirror image_29 snap 2023-11-08T05:17:56.224 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:17:56.224 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:17:56.224 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_29 2023-11-08T05:17:56.224 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap_name=snap 2023-11-08T05:17:56.225 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local s 2023-11-08T05:17:56.225 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-11-08T05:17:56.225 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 1 2023-11-08T05:17:57.224 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 info mirror/image_29@snap 2023-11-08T05:17:57.261 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:rbd image 'image_29': 2023-11-08T05:17:57.262 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: size 128 MiB in 32 objects 2023-11-08T05:17:57.262 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: order 22 (4 MiB objects) 2023-11-08T05:17:57.262 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: snapshot_count: 2 2023-11-08T05:17:57.262 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: id: 10e7c5d79e84 2023-11-08T05:17:57.263 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: block_name_prefix: rbd_data.10e7c5d79e84 2023-11-08T05:17:57.263 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: format: 2 2023-11-08T05:17:57.263 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: features: layering, exclusive-lock, non-primary 2023-11-08T05:17:57.263 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: op_features: 2023-11-08T05:17:57.263 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: flags: 2023-11-08T05:17:57.264 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: create_timestamp: Wed Nov 8 05:09:31 2023 2023-11-08T05:17:57.264 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: access_timestamp: Wed Nov 8 05:09:31 2023 2023-11-08T05:17:57.264 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: modify_timestamp: Wed Nov 8 05:09:31 2023 2023-11-08T05:17:57.264 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: protected: False 2023-11-08T05:17:57.265 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring state: enabled 2023-11-08T05:17:57.265 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring mode: snapshot 2023-11-08T05:17:57.265 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring global id: 224b796b-9a1e-47f0-a771-84babb70a923 2023-11-08T05:17:57.265 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring primary: false 2023-11-08T05:17:57.267 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:17:57.267 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ compare_image_snaps mirror image_29 snap 2023-11-08T05:17:57.267 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:17:57.267 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_29 2023-11-08T05:17:57.268 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap_name=snap 2023-11-08T05:17:57.268 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local ret=0 2023-11-08T05:17:57.268 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local rmt_export=/tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_29.export 2023-11-08T05:17:57.268 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local loc_export=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_29.export 2023-11-08T05:17:57.268 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rm -f /tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_29.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_29.export 2023-11-08T05:17:57.269 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 -p mirror export image_29@snap /tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_29.export 2023-11-08T05:17:57.449 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-11-08T05:17:57.452 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 -p mirror export image_29@snap /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_29.export 2023-11-08T05:17:57.706 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-11-08T05:17:57.709 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ cmp /tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_29.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_29.export 2023-11-08T05:17:57.764 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rm -f /tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_29.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_29.export 2023-11-08T05:17:57.801 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:17:57.802 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-11-08T05:17:57.802 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image=image_30 2023-11-08T05:17:57.802 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ create_snap cluster2 mirror image_30 snap 2023-11-08T05:17:57.802 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:17:57.803 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:17:57.803 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_30 2023-11-08T05:17:57.803 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap_name=snap 2023-11-08T05:17:57.803 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 -p mirror snap create image_30@snap --debug-rbd=20 --debug-journaler=20 2023-11-08T05:17:59.450 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_image_replay_started cluster1 mirror image_30 2023-11-08T05:17:59.450 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:17:59.450 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:17:59.451 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_30 2023-11-08T05:17:59.451 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_image_replay_state cluster1 mirror image_30 started 2023-11-08T05:17:59.451 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:17:59.451 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:17:59.451 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_30 2023-11-08T05:17:59.452 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local state=started 2023-11-08T05:17:59.452 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local s 2023-11-08T05:17:59.452 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-11-08T05:17:59.452 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 1 2023-11-08T05:18:00.452 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test_image_replay_state cluster1 mirror image_30 started 2023-11-08T05:18:00.452 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:18:00.452 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:18:00.452 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_30 2023-11-08T05:18:00.453 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local test_state=started 2023-11-08T05:18:00.453 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local status_result 2023-11-08T05:18:00.453 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local current_state=stopped 2023-11-08T05:18:00.453 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_30 2023-11-08T05:18:00.453 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_instance=cluster1 2023-11-08T05:18:00.454 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T05:18:00.454 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ grep -i state 2023-11-08T05:18:00.454 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1 2023-11-08T05:18:00.454 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance=cluster1 2023-11-08T05:18:00.454 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local loop_instance 2023-11-08T05:18:00.455 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-11-08T05:18:00.455 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sleep 0 2023-11-08T05:18:00.457 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-11-08T05:18:00.457 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ seq 0 3 2023-11-08T05:18:00.457 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T05:18:00.458 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_30 2023-11-08T05:18:00.458 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1:0 2023-11-08T05:18:00.458 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T05:18:00.458 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance 2023-11-08T05:18:00.458 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-11-08T05:18:00.459 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local val=cluster1:0 2023-11-08T05:18:00.459 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_var_name=cluster 2023-11-08T05:18:00.459 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance_var_name=instance 2023-11-08T05:18:00.459 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cluster=cluster1 2023-11-08T05:18:00.459 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ instance=0 2023-11-08T05:18:00.460 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' 0 = cluster1:0 ']' 2023-11-08T05:18:00.460 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval cluster=cluster1 2023-11-08T05:18:00.460 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:18:00.460 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval instance=0 2023-11-08T05:18:00.460 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=0 2023-11-08T05:18:00.460 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-11-08T05:18:00.461 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local local_cluster=cluster1:0 2023-11-08T05:18:00.462 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster=cluster1 2023-11-08T05:18:00.463 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance 2023-11-08T05:18:00.463 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-11-08T05:18:00.463 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local val=cluster1:0 2023-11-08T05:18:00.463 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster_var_name=local_cluster 2023-11-08T05:18:00.463 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance_var_name=instance 2023-11-08T05:18:00.464 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:18:00.464 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=0 2023-11-08T05:18:00.464 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ '[' 0 = cluster1:0 ']' 2023-11-08T05:18:00.464 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval local_cluster=cluster1 2023-11-08T05:18:00.464 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ local_cluster=cluster1 2023-11-08T05:18:00.465 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval instance=0 2023-11-08T05:18:00.465 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ instance=0 2023-11-08T05:18:00.465 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-11-08T05:18:00.472 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-08T05:18:00.473 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-08T05:18:00.473 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-08T05:18:00.473 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_30 2023-11-08T05:18:00.579 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:Invalid command: unused arguments: ['mirror/image_30'] 2023-11-08T05:18:00.580 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd mirror status : get status for rbd mirror 2023-11-08T05:18:00.580 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:admin_socket: invalid command 2023-11-08T05:18:00.583 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T05:18:00.583 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_30 2023-11-08T05:18:00.583 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1:1 2023-11-08T05:18:00.584 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T05:18:00.584 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance 2023-11-08T05:18:00.584 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-11-08T05:18:00.584 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local val=cluster1:1 2023-11-08T05:18:00.584 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_var_name=cluster 2023-11-08T05:18:00.585 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance_var_name=instance 2023-11-08T05:18:00.585 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cluster=cluster1 2023-11-08T05:18:00.585 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ instance=1 2023-11-08T05:18:00.585 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' 1 = cluster1:1 ']' 2023-11-08T05:18:00.585 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval cluster=cluster1 2023-11-08T05:18:00.585 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:18:00.586 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval instance=1 2023-11-08T05:18:00.586 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=1 2023-11-08T05:18:00.586 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-11-08T05:18:00.586 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local local_cluster=cluster1:1 2023-11-08T05:18:00.586 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster=cluster1 2023-11-08T05:18:00.587 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance 2023-11-08T05:18:00.587 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-11-08T05:18:00.587 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local val=cluster1:1 2023-11-08T05:18:00.587 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster_var_name=local_cluster 2023-11-08T05:18:00.587 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance_var_name=instance 2023-11-08T05:18:00.587 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:18:00.588 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=1 2023-11-08T05:18:00.588 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ '[' 1 = cluster1:1 ']' 2023-11-08T05:18:00.588 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval local_cluster=cluster1 2023-11-08T05:18:00.588 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ local_cluster=cluster1 2023-11-08T05:18:00.588 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval instance=1 2023-11-08T05:18:00.589 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ instance=1 2023-11-08T05:18:00.589 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-11-08T05:18:00.598 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-08T05:18:00.598 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-08T05:18:00.598 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-08T05:18:00.598 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_30 2023-11-08T05:18:00.696 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:Invalid command: unused arguments: ['mirror/image_30'] 2023-11-08T05:18:00.696 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd mirror status : get status for rbd mirror 2023-11-08T05:18:00.696 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:admin_socket: invalid command 2023-11-08T05:18:00.699 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T05:18:00.699 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_30 2023-11-08T05:18:00.700 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1:2 2023-11-08T05:18:00.700 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T05:18:00.700 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance 2023-11-08T05:18:00.700 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-11-08T05:18:00.701 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local val=cluster1:2 2023-11-08T05:18:00.701 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_var_name=cluster 2023-11-08T05:18:00.701 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance_var_name=instance 2023-11-08T05:18:00.701 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cluster=cluster1 2023-11-08T05:18:00.701 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ instance=2 2023-11-08T05:18:00.702 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' 2 = cluster1:2 ']' 2023-11-08T05:18:00.702 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval cluster=cluster1 2023-11-08T05:18:00.702 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:18:00.702 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval instance=2 2023-11-08T05:18:00.702 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=2 2023-11-08T05:18:00.703 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-11-08T05:18:00.703 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local local_cluster=cluster1:2 2023-11-08T05:18:00.703 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster=cluster1 2023-11-08T05:18:00.703 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance 2023-11-08T05:18:00.704 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-11-08T05:18:00.705 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local val=cluster1:2 2023-11-08T05:18:00.705 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster_var_name=local_cluster 2023-11-08T05:18:00.705 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance_var_name=instance 2023-11-08T05:18:00.705 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:18:00.705 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=2 2023-11-08T05:18:00.706 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ '[' 2 = cluster1:2 ']' 2023-11-08T05:18:00.706 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval local_cluster=cluster1 2023-11-08T05:18:00.706 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ local_cluster=cluster1 2023-11-08T05:18:00.706 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval instance=2 2023-11-08T05:18:00.706 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ instance=2 2023-11-08T05:18:00.707 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-11-08T05:18:00.713 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-08T05:18:00.713 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-08T05:18:00.714 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-08T05:18:00.714 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_30 2023-11-08T05:18:00.811 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:Invalid command: unused arguments: ['mirror/image_30'] 2023-11-08T05:18:00.811 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd mirror status : get status for rbd mirror 2023-11-08T05:18:00.812 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:admin_socket: invalid command 2023-11-08T05:18:00.815 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T05:18:00.815 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_30 2023-11-08T05:18:00.816 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1:3 2023-11-08T05:18:00.816 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T05:18:00.816 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance 2023-11-08T05:18:00.816 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ set_cluster_instance cluster1:3 cluster instance 2023-11-08T05:18:00.816 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local val=cluster1:3 2023-11-08T05:18:00.817 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_var_name=cluster 2023-11-08T05:18:00.817 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance_var_name=instance 2023-11-08T05:18:00.817 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cluster=cluster1 2023-11-08T05:18:00.817 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ instance=3 2023-11-08T05:18:00.818 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' 3 = cluster1:3 ']' 2023-11-08T05:18:00.818 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval cluster=cluster1 2023-11-08T05:18:00.818 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:18:00.818 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval instance=3 2023-11-08T05:18:00.818 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=3 2023-11-08T05:18:00.819 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ daemon_asok_file cluster1:3 cluster1 2023-11-08T05:18:00.819 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local local_cluster=cluster1:3 2023-11-08T05:18:00.819 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster=cluster1 2023-11-08T05:18:00.819 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance 2023-11-08T05:18:00.820 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2023-11-08T05:18:00.820 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local val=cluster1:3 2023-11-08T05:18:00.820 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster_var_name=local_cluster 2023-11-08T05:18:00.820 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance_var_name=instance 2023-11-08T05:18:00.820 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:18:00.821 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=3 2023-11-08T05:18:00.821 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ '[' 3 = cluster1:3 ']' 2023-11-08T05:18:00.821 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval local_cluster=cluster1 2023-11-08T05:18:00.822 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ local_cluster=cluster1 2023-11-08T05:18:00.822 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval instance=3 2023-11-08T05:18:00.822 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ instance=3 2023-11-08T05:18:00.823 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2023-11-08T05:18:00.829 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-11-08T05:18:00.829 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-11-08T05:18:00.829 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-11-08T05:18:00.829 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_30 2023-11-08T05:18:00.917 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:Invalid command: unused arguments: ['mirror/image_30'] 2023-11-08T05:18:00.918 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd mirror status : get status for rbd mirror 2023-11-08T05:18:00.918 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:admin_socket: invalid command 2023-11-08T05:18:00.919 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-11-08T05:18:00.919 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sleep 1 2023-11-08T05:18:01.919 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-11-08T05:18:01.920 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ seq 0 3 2023-11-08T05:18:01.923 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T05:18:01.923 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_30 2023-11-08T05:18:01.923 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1:0 2023-11-08T05:18:01.924 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T05:18:01.924 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance 2023-11-08T05:18:01.924 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-11-08T05:18:01.924 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local val=cluster1:0 2023-11-08T05:18:01.924 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_var_name=cluster 2023-11-08T05:18:01.925 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance_var_name=instance 2023-11-08T05:18:01.925 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cluster=cluster1 2023-11-08T05:18:01.925 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ instance=0 2023-11-08T05:18:01.925 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' 0 = cluster1:0 ']' 2023-11-08T05:18:01.925 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval cluster=cluster1 2023-11-08T05:18:01.926 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:18:01.926 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval instance=0 2023-11-08T05:18:01.926 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=0 2023-11-08T05:18:01.926 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-11-08T05:18:01.926 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local local_cluster=cluster1:0 2023-11-08T05:18:01.927 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster=cluster1 2023-11-08T05:18:01.927 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance 2023-11-08T05:18:01.927 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-11-08T05:18:01.927 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local val=cluster1:0 2023-11-08T05:18:01.927 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster_var_name=local_cluster 2023-11-08T05:18:01.927 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance_var_name=instance 2023-11-08T05:18:01.928 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:18:01.928 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=0 2023-11-08T05:18:01.928 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ '[' 0 = cluster1:0 ']' 2023-11-08T05:18:01.928 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval local_cluster=cluster1 2023-11-08T05:18:01.928 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ local_cluster=cluster1 2023-11-08T05:18:01.929 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval instance=0 2023-11-08T05:18:01.929 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ instance=0 2023-11-08T05:18:01.929 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-11-08T05:18:01.940 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-08T05:18:01.940 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-08T05:18:01.940 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-08T05:18:01.940 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_30 2023-11-08T05:18:02.044 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:Invalid command: unused arguments: ['mirror/image_30'] 2023-11-08T05:18:02.044 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd mirror status : get status for rbd mirror 2023-11-08T05:18:02.044 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:admin_socket: invalid command 2023-11-08T05:18:02.047 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T05:18:02.047 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_30 2023-11-08T05:18:02.047 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1:1 2023-11-08T05:18:02.048 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T05:18:02.048 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance 2023-11-08T05:18:02.048 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-11-08T05:18:02.048 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local val=cluster1:1 2023-11-08T05:18:02.048 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_var_name=cluster 2023-11-08T05:18:02.049 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance_var_name=instance 2023-11-08T05:18:02.049 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cluster=cluster1 2023-11-08T05:18:02.049 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ instance=1 2023-11-08T05:18:02.049 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' 1 = cluster1:1 ']' 2023-11-08T05:18:02.049 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval cluster=cluster1 2023-11-08T05:18:02.050 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:18:02.050 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval instance=1 2023-11-08T05:18:02.050 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=1 2023-11-08T05:18:02.050 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-11-08T05:18:02.050 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local local_cluster=cluster1:1 2023-11-08T05:18:02.051 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster=cluster1 2023-11-08T05:18:02.051 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance 2023-11-08T05:18:02.051 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-11-08T05:18:02.051 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local val=cluster1:1 2023-11-08T05:18:02.051 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster_var_name=local_cluster 2023-11-08T05:18:02.052 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance_var_name=instance 2023-11-08T05:18:02.052 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:18:02.052 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=1 2023-11-08T05:18:02.052 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ '[' 1 = cluster1:1 ']' 2023-11-08T05:18:02.052 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval local_cluster=cluster1 2023-11-08T05:18:02.053 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ local_cluster=cluster1 2023-11-08T05:18:02.053 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval instance=1 2023-11-08T05:18:02.053 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ instance=1 2023-11-08T05:18:02.053 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-11-08T05:18:02.061 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-08T05:18:02.062 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-08T05:18:02.062 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-08T05:18:02.062 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_30 2023-11-08T05:18:02.159 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:Invalid command: unused arguments: ['mirror/image_30'] 2023-11-08T05:18:02.160 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd mirror status : get status for rbd mirror 2023-11-08T05:18:02.160 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:admin_socket: invalid command 2023-11-08T05:18:02.163 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T05:18:02.163 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_30 2023-11-08T05:18:02.163 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1:2 2023-11-08T05:18:02.164 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T05:18:02.164 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance 2023-11-08T05:18:02.164 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-11-08T05:18:02.164 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local val=cluster1:2 2023-11-08T05:18:02.164 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_var_name=cluster 2023-11-08T05:18:02.164 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance_var_name=instance 2023-11-08T05:18:02.165 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cluster=cluster1 2023-11-08T05:18:02.165 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ instance=2 2023-11-08T05:18:02.165 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' 2 = cluster1:2 ']' 2023-11-08T05:18:02.166 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval cluster=cluster1 2023-11-08T05:18:02.166 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:18:02.166 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval instance=2 2023-11-08T05:18:02.166 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=2 2023-11-08T05:18:02.167 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-11-08T05:18:02.167 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local local_cluster=cluster1:2 2023-11-08T05:18:02.167 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster=cluster1 2023-11-08T05:18:02.167 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance 2023-11-08T05:18:02.167 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-11-08T05:18:02.168 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local val=cluster1:2 2023-11-08T05:18:02.168 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster_var_name=local_cluster 2023-11-08T05:18:02.168 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance_var_name=instance 2023-11-08T05:18:02.168 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:18:02.168 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=2 2023-11-08T05:18:02.169 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ '[' 2 = cluster1:2 ']' 2023-11-08T05:18:02.169 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval local_cluster=cluster1 2023-11-08T05:18:02.169 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ local_cluster=cluster1 2023-11-08T05:18:02.169 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval instance=2 2023-11-08T05:18:02.169 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ instance=2 2023-11-08T05:18:02.170 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-11-08T05:18:02.176 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-08T05:18:02.177 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-08T05:18:02.177 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-08T05:18:02.177 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_30 2023-11-08T05:18:02.275 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:Invalid command: unused arguments: ['mirror/image_30'] 2023-11-08T05:18:02.276 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd mirror status : get status for rbd mirror 2023-11-08T05:18:02.276 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:admin_socket: invalid command 2023-11-08T05:18:02.279 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T05:18:02.279 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_30 2023-11-08T05:18:02.279 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1:3 2023-11-08T05:18:02.279 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T05:18:02.279 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance 2023-11-08T05:18:02.280 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ set_cluster_instance cluster1:3 cluster instance 2023-11-08T05:18:02.280 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local val=cluster1:3 2023-11-08T05:18:02.280 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_var_name=cluster 2023-11-08T05:18:02.280 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance_var_name=instance 2023-11-08T05:18:02.281 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cluster=cluster1 2023-11-08T05:18:02.281 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ instance=3 2023-11-08T05:18:02.281 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' 3 = cluster1:3 ']' 2023-11-08T05:18:02.281 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval cluster=cluster1 2023-11-08T05:18:02.281 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:18:02.282 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval instance=3 2023-11-08T05:18:02.282 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=3 2023-11-08T05:18:02.282 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ daemon_asok_file cluster1:3 cluster1 2023-11-08T05:18:02.282 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local local_cluster=cluster1:3 2023-11-08T05:18:02.282 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster=cluster1 2023-11-08T05:18:02.283 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance 2023-11-08T05:18:02.283 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2023-11-08T05:18:02.283 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local val=cluster1:3 2023-11-08T05:18:02.283 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster_var_name=local_cluster 2023-11-08T05:18:02.283 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance_var_name=instance 2023-11-08T05:18:02.284 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:18:02.284 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=3 2023-11-08T05:18:02.284 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ '[' 3 = cluster1:3 ']' 2023-11-08T05:18:02.284 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval local_cluster=cluster1 2023-11-08T05:18:02.284 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ local_cluster=cluster1 2023-11-08T05:18:02.285 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval instance=3 2023-11-08T05:18:02.285 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ instance=3 2023-11-08T05:18:02.285 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2023-11-08T05:18:02.294 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-11-08T05:18:02.294 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-11-08T05:18:02.294 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-11-08T05:18:02.295 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_30 2023-11-08T05:18:02.385 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:Invalid command: unused arguments: ['mirror/image_30'] 2023-11-08T05:18:02.386 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd mirror status : get status for rbd mirror 2023-11-08T05:18:02.386 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:admin_socket: invalid command 2023-11-08T05:18:02.387 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-11-08T05:18:02.387 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sleep 2 2023-11-08T05:18:04.388 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-11-08T05:18:04.388 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ seq 0 3 2023-11-08T05:18:04.392 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T05:18:04.392 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_30 2023-11-08T05:18:04.392 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1:0 2023-11-08T05:18:04.392 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T05:18:04.393 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance 2023-11-08T05:18:04.393 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-11-08T05:18:04.393 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local val=cluster1:0 2023-11-08T05:18:04.393 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_var_name=cluster 2023-11-08T05:18:04.393 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance_var_name=instance 2023-11-08T05:18:04.394 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cluster=cluster1 2023-11-08T05:18:04.394 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ instance=0 2023-11-08T05:18:04.394 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' 0 = cluster1:0 ']' 2023-11-08T05:18:04.394 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval cluster=cluster1 2023-11-08T05:18:04.394 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:18:04.395 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval instance=0 2023-11-08T05:18:04.395 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=0 2023-11-08T05:18:04.395 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-11-08T05:18:04.395 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local local_cluster=cluster1:0 2023-11-08T05:18:04.395 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster=cluster1 2023-11-08T05:18:04.396 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance 2023-11-08T05:18:04.396 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-11-08T05:18:04.396 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local val=cluster1:0 2023-11-08T05:18:04.396 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster_var_name=local_cluster 2023-11-08T05:18:04.396 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance_var_name=instance 2023-11-08T05:18:04.397 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:18:04.397 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=0 2023-11-08T05:18:04.397 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ '[' 0 = cluster1:0 ']' 2023-11-08T05:18:04.397 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval local_cluster=cluster1 2023-11-08T05:18:04.397 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ local_cluster=cluster1 2023-11-08T05:18:04.397 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval instance=0 2023-11-08T05:18:04.398 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ instance=0 2023-11-08T05:18:04.398 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-11-08T05:18:04.408 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-08T05:18:04.408 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-08T05:18:04.408 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-08T05:18:04.409 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_30 2023-11-08T05:18:04.515 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:Invalid command: unused arguments: ['mirror/image_30'] 2023-11-08T05:18:04.515 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd mirror status : get status for rbd mirror 2023-11-08T05:18:04.516 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:admin_socket: invalid command 2023-11-08T05:18:04.518 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T05:18:04.518 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_30 2023-11-08T05:18:04.519 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1:1 2023-11-08T05:18:04.519 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T05:18:04.519 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance 2023-11-08T05:18:04.519 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-11-08T05:18:04.519 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local val=cluster1:1 2023-11-08T05:18:04.520 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_var_name=cluster 2023-11-08T05:18:04.520 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance_var_name=instance 2023-11-08T05:18:04.520 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cluster=cluster1 2023-11-08T05:18:04.520 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ instance=1 2023-11-08T05:18:04.521 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' 1 = cluster1:1 ']' 2023-11-08T05:18:04.521 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval cluster=cluster1 2023-11-08T05:18:04.521 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:18:04.521 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval instance=1 2023-11-08T05:18:04.521 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=1 2023-11-08T05:18:04.521 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-11-08T05:18:04.522 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local local_cluster=cluster1:1 2023-11-08T05:18:04.522 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster=cluster1 2023-11-08T05:18:04.522 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance 2023-11-08T05:18:04.522 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-11-08T05:18:04.522 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local val=cluster1:1 2023-11-08T05:18:04.523 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster_var_name=local_cluster 2023-11-08T05:18:04.523 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance_var_name=instance 2023-11-08T05:18:04.523 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:18:04.523 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=1 2023-11-08T05:18:04.523 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ '[' 1 = cluster1:1 ']' 2023-11-08T05:18:04.524 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval local_cluster=cluster1 2023-11-08T05:18:04.524 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ local_cluster=cluster1 2023-11-08T05:18:04.524 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval instance=1 2023-11-08T05:18:04.524 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ instance=1 2023-11-08T05:18:04.524 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-11-08T05:18:04.533 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-08T05:18:04.533 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-08T05:18:04.533 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-08T05:18:04.533 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_30 2023-11-08T05:18:04.630 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:Invalid command: unused arguments: ['mirror/image_30'] 2023-11-08T05:18:04.631 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd mirror status : get status for rbd mirror 2023-11-08T05:18:04.631 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:admin_socket: invalid command 2023-11-08T05:18:04.634 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T05:18:04.634 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_30 2023-11-08T05:18:04.634 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1:2 2023-11-08T05:18:04.635 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T05:18:04.635 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance 2023-11-08T05:18:04.635 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-11-08T05:18:04.635 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local val=cluster1:2 2023-11-08T05:18:04.636 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_var_name=cluster 2023-11-08T05:18:04.636 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance_var_name=instance 2023-11-08T05:18:04.636 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cluster=cluster1 2023-11-08T05:18:04.636 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ instance=2 2023-11-08T05:18:04.636 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' 2 = cluster1:2 ']' 2023-11-08T05:18:04.637 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval cluster=cluster1 2023-11-08T05:18:04.637 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:18:04.638 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval instance=2 2023-11-08T05:18:04.638 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=2 2023-11-08T05:18:04.639 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-11-08T05:18:04.639 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local local_cluster=cluster1:2 2023-11-08T05:18:04.639 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster=cluster1 2023-11-08T05:18:04.639 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance 2023-11-08T05:18:04.639 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-11-08T05:18:04.640 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local val=cluster1:2 2023-11-08T05:18:04.640 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster_var_name=local_cluster 2023-11-08T05:18:04.640 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance_var_name=instance 2023-11-08T05:18:04.640 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:18:04.641 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=2 2023-11-08T05:18:04.641 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ '[' 2 = cluster1:2 ']' 2023-11-08T05:18:04.641 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval local_cluster=cluster1 2023-11-08T05:18:04.641 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ local_cluster=cluster1 2023-11-08T05:18:04.642 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval instance=2 2023-11-08T05:18:04.642 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ instance=2 2023-11-08T05:18:04.643 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-11-08T05:18:04.648 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-08T05:18:04.648 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-08T05:18:04.648 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-08T05:18:04.648 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_30 2023-11-08T05:18:04.747 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:Invalid command: unused arguments: ['mirror/image_30'] 2023-11-08T05:18:04.747 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd mirror status : get status for rbd mirror 2023-11-08T05:18:04.747 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:admin_socket: invalid command 2023-11-08T05:18:04.750 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T05:18:04.750 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_30 2023-11-08T05:18:04.751 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1:3 2023-11-08T05:18:04.751 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T05:18:04.751 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance 2023-11-08T05:18:04.751 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ set_cluster_instance cluster1:3 cluster instance 2023-11-08T05:18:04.751 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local val=cluster1:3 2023-11-08T05:18:04.752 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_var_name=cluster 2023-11-08T05:18:04.752 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance_var_name=instance 2023-11-08T05:18:04.752 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cluster=cluster1 2023-11-08T05:18:04.752 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ instance=3 2023-11-08T05:18:04.752 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' 3 = cluster1:3 ']' 2023-11-08T05:18:04.753 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval cluster=cluster1 2023-11-08T05:18:04.753 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:18:04.753 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval instance=3 2023-11-08T05:18:04.753 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=3 2023-11-08T05:18:04.753 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ daemon_asok_file cluster1:3 cluster1 2023-11-08T05:18:04.753 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local local_cluster=cluster1:3 2023-11-08T05:18:04.754 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster=cluster1 2023-11-08T05:18:04.754 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance 2023-11-08T05:18:04.754 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2023-11-08T05:18:04.754 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local val=cluster1:3 2023-11-08T05:18:04.754 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster_var_name=local_cluster 2023-11-08T05:18:04.755 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance_var_name=instance 2023-11-08T05:18:04.755 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:18:04.755 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=3 2023-11-08T05:18:04.755 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ '[' 3 = cluster1:3 ']' 2023-11-08T05:18:04.755 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval local_cluster=cluster1 2023-11-08T05:18:04.756 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ local_cluster=cluster1 2023-11-08T05:18:04.756 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval instance=3 2023-11-08T05:18:04.756 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ instance=3 2023-11-08T05:18:04.756 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2023-11-08T05:18:04.764 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-11-08T05:18:04.765 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-11-08T05:18:04.765 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-11-08T05:18:04.765 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_30 2023-11-08T05:18:04.856 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:Invalid command: unused arguments: ['mirror/image_30'] 2023-11-08T05:18:04.856 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd mirror status : get status for rbd mirror 2023-11-08T05:18:04.856 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:admin_socket: invalid command 2023-11-08T05:18:04.857 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-11-08T05:18:04.857 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sleep 4 2023-11-08T05:18:08.858 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-11-08T05:18:08.858 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ seq 0 3 2023-11-08T05:18:08.861 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T05:18:08.862 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_30 2023-11-08T05:18:08.862 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1:0 2023-11-08T05:18:08.862 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T05:18:08.862 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance 2023-11-08T05:18:08.863 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-11-08T05:18:08.863 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local val=cluster1:0 2023-11-08T05:18:08.863 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_var_name=cluster 2023-11-08T05:18:08.863 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance_var_name=instance 2023-11-08T05:18:08.863 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cluster=cluster1 2023-11-08T05:18:08.864 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ instance=0 2023-11-08T05:18:08.864 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' 0 = cluster1:0 ']' 2023-11-08T05:18:08.864 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval cluster=cluster1 2023-11-08T05:18:08.864 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:18:08.864 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval instance=0 2023-11-08T05:18:08.865 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=0 2023-11-08T05:18:08.865 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-11-08T05:18:08.865 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local local_cluster=cluster1:0 2023-11-08T05:18:08.865 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster=cluster1 2023-11-08T05:18:08.865 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance 2023-11-08T05:18:08.865 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-11-08T05:18:08.866 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local val=cluster1:0 2023-11-08T05:18:08.866 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster_var_name=local_cluster 2023-11-08T05:18:08.866 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance_var_name=instance 2023-11-08T05:18:08.866 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:18:08.866 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=0 2023-11-08T05:18:08.867 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ '[' 0 = cluster1:0 ']' 2023-11-08T05:18:08.867 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval local_cluster=cluster1 2023-11-08T05:18:08.867 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ local_cluster=cluster1 2023-11-08T05:18:08.867 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval instance=0 2023-11-08T05:18:08.867 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ instance=0 2023-11-08T05:18:08.868 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-11-08T05:18:08.879 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-08T05:18:08.879 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-08T05:18:08.879 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-08T05:18:08.880 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_30 2023-11-08T05:18:08.985 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:Invalid command: unused arguments: ['mirror/image_30'] 2023-11-08T05:18:08.985 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd mirror status : get status for rbd mirror 2023-11-08T05:18:08.985 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:admin_socket: invalid command 2023-11-08T05:18:08.989 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T05:18:08.989 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_30 2023-11-08T05:18:08.989 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1:1 2023-11-08T05:18:08.989 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T05:18:08.989 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance 2023-11-08T05:18:08.990 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-11-08T05:18:08.990 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local val=cluster1:1 2023-11-08T05:18:08.990 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_var_name=cluster 2023-11-08T05:18:08.990 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance_var_name=instance 2023-11-08T05:18:08.991 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cluster=cluster1 2023-11-08T05:18:08.991 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ instance=1 2023-11-08T05:18:08.991 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' 1 = cluster1:1 ']' 2023-11-08T05:18:08.991 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval cluster=cluster1 2023-11-08T05:18:08.991 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:18:08.991 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval instance=1 2023-11-08T05:18:08.992 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=1 2023-11-08T05:18:08.992 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-11-08T05:18:08.992 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local local_cluster=cluster1:1 2023-11-08T05:18:08.992 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster=cluster1 2023-11-08T05:18:08.993 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance 2023-11-08T05:18:08.993 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-11-08T05:18:08.993 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local val=cluster1:1 2023-11-08T05:18:08.993 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster_var_name=local_cluster 2023-11-08T05:18:08.994 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance_var_name=instance 2023-11-08T05:18:08.994 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:18:08.994 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=1 2023-11-08T05:18:08.994 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ '[' 1 = cluster1:1 ']' 2023-11-08T05:18:08.994 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval local_cluster=cluster1 2023-11-08T05:18:08.995 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ local_cluster=cluster1 2023-11-08T05:18:08.995 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval instance=1 2023-11-08T05:18:08.995 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ instance=1 2023-11-08T05:18:08.995 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-11-08T05:18:09.003 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-08T05:18:09.003 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-08T05:18:09.004 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-08T05:18:09.004 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_30 2023-11-08T05:18:09.100 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:Invalid command: unused arguments: ['mirror/image_30'] 2023-11-08T05:18:09.101 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd mirror status : get status for rbd mirror 2023-11-08T05:18:09.101 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:admin_socket: invalid command 2023-11-08T05:18:09.104 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T05:18:09.104 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_30 2023-11-08T05:18:09.104 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1:2 2023-11-08T05:18:09.105 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T05:18:09.105 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance 2023-11-08T05:18:09.105 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-11-08T05:18:09.105 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local val=cluster1:2 2023-11-08T05:18:09.105 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_var_name=cluster 2023-11-08T05:18:09.106 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance_var_name=instance 2023-11-08T05:18:09.106 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cluster=cluster1 2023-11-08T05:18:09.106 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ instance=2 2023-11-08T05:18:09.106 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' 2 = cluster1:2 ']' 2023-11-08T05:18:09.106 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval cluster=cluster1 2023-11-08T05:18:09.107 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:18:09.107 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval instance=2 2023-11-08T05:18:09.107 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=2 2023-11-08T05:18:09.107 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-11-08T05:18:09.107 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local local_cluster=cluster1:2 2023-11-08T05:18:09.107 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster=cluster1 2023-11-08T05:18:09.108 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance 2023-11-08T05:18:09.108 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-11-08T05:18:09.108 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local val=cluster1:2 2023-11-08T05:18:09.108 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster_var_name=local_cluster 2023-11-08T05:18:09.108 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance_var_name=instance 2023-11-08T05:18:09.109 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:18:09.109 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=2 2023-11-08T05:18:09.109 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ '[' 2 = cluster1:2 ']' 2023-11-08T05:18:09.109 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval local_cluster=cluster1 2023-11-08T05:18:09.109 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ local_cluster=cluster1 2023-11-08T05:18:09.110 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval instance=2 2023-11-08T05:18:09.110 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ instance=2 2023-11-08T05:18:09.110 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-11-08T05:18:09.118 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-08T05:18:09.119 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-08T05:18:09.119 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-08T05:18:09.119 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_30 2023-11-08T05:18:09.217 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:Invalid command: unused arguments: ['mirror/image_30'] 2023-11-08T05:18:09.218 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd mirror status : get status for rbd mirror 2023-11-08T05:18:09.218 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:admin_socket: invalid command 2023-11-08T05:18:09.221 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T05:18:09.221 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_30 2023-11-08T05:18:09.221 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1:3 2023-11-08T05:18:09.221 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T05:18:09.221 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance 2023-11-08T05:18:09.222 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ set_cluster_instance cluster1:3 cluster instance 2023-11-08T05:18:09.222 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local val=cluster1:3 2023-11-08T05:18:09.222 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_var_name=cluster 2023-11-08T05:18:09.222 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance_var_name=instance 2023-11-08T05:18:09.222 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cluster=cluster1 2023-11-08T05:18:09.223 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ instance=3 2023-11-08T05:18:09.223 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' 3 = cluster1:3 ']' 2023-11-08T05:18:09.223 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval cluster=cluster1 2023-11-08T05:18:09.223 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:18:09.223 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval instance=3 2023-11-08T05:18:09.224 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=3 2023-11-08T05:18:09.224 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ daemon_asok_file cluster1:3 cluster1 2023-11-08T05:18:09.224 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local local_cluster=cluster1:3 2023-11-08T05:18:09.224 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster=cluster1 2023-11-08T05:18:09.224 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance 2023-11-08T05:18:09.225 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2023-11-08T05:18:09.225 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local val=cluster1:3 2023-11-08T05:18:09.225 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster_var_name=local_cluster 2023-11-08T05:18:09.225 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance_var_name=instance 2023-11-08T05:18:09.226 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:18:09.226 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=3 2023-11-08T05:18:09.226 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ '[' 3 = cluster1:3 ']' 2023-11-08T05:18:09.226 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval local_cluster=cluster1 2023-11-08T05:18:09.226 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ local_cluster=cluster1 2023-11-08T05:18:09.227 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval instance=3 2023-11-08T05:18:09.227 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ instance=3 2023-11-08T05:18:09.227 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2023-11-08T05:18:09.234 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-11-08T05:18:09.235 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-11-08T05:18:09.235 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-11-08T05:18:09.235 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_30 2023-11-08T05:18:09.325 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:Invalid command: unused arguments: ['mirror/image_30'] 2023-11-08T05:18:09.326 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd mirror status : get status for rbd mirror 2023-11-08T05:18:09.326 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:admin_socket: invalid command 2023-11-08T05:18:09.326 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-11-08T05:18:09.327 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sleep 8 2023-11-08T05:18:17.328 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-11-08T05:18:17.328 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ seq 0 3 2023-11-08T05:18:17.331 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T05:18:17.331 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_30 2023-11-08T05:18:17.332 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1:0 2023-11-08T05:18:17.332 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T05:18:17.332 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance 2023-11-08T05:18:17.332 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-11-08T05:18:17.332 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local val=cluster1:0 2023-11-08T05:18:17.333 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_var_name=cluster 2023-11-08T05:18:17.333 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance_var_name=instance 2023-11-08T05:18:17.333 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cluster=cluster1 2023-11-08T05:18:17.333 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ instance=0 2023-11-08T05:18:17.333 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' 0 = cluster1:0 ']' 2023-11-08T05:18:17.334 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval cluster=cluster1 2023-11-08T05:18:17.334 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:18:17.334 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval instance=0 2023-11-08T05:18:17.334 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=0 2023-11-08T05:18:17.334 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-11-08T05:18:17.335 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local local_cluster=cluster1:0 2023-11-08T05:18:17.335 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster=cluster1 2023-11-08T05:18:17.335 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance 2023-11-08T05:18:17.335 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-11-08T05:18:17.335 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local val=cluster1:0 2023-11-08T05:18:17.335 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster_var_name=local_cluster 2023-11-08T05:18:17.336 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance_var_name=instance 2023-11-08T05:18:17.336 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:18:17.336 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=0 2023-11-08T05:18:17.336 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ '[' 0 = cluster1:0 ']' 2023-11-08T05:18:17.336 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval local_cluster=cluster1 2023-11-08T05:18:17.337 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ local_cluster=cluster1 2023-11-08T05:18:17.337 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval instance=0 2023-11-08T05:18:17.337 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ instance=0 2023-11-08T05:18:17.338 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-11-08T05:18:17.348 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-08T05:18:17.349 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-08T05:18:17.349 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-08T05:18:17.349 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_30 2023-11-08T05:18:17.457 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:Invalid command: unused arguments: ['mirror/image_30'] 2023-11-08T05:18:17.457 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd mirror status : get status for rbd mirror 2023-11-08T05:18:17.457 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:admin_socket: invalid command 2023-11-08T05:18:17.460 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T05:18:17.461 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_30 2023-11-08T05:18:17.461 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1:1 2023-11-08T05:18:17.461 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T05:18:17.461 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance 2023-11-08T05:18:17.461 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-11-08T05:18:17.462 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local val=cluster1:1 2023-11-08T05:18:17.462 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_var_name=cluster 2023-11-08T05:18:17.462 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance_var_name=instance 2023-11-08T05:18:17.462 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cluster=cluster1 2023-11-08T05:18:17.462 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ instance=1 2023-11-08T05:18:17.463 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' 1 = cluster1:1 ']' 2023-11-08T05:18:17.463 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval cluster=cluster1 2023-11-08T05:18:17.463 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:18:17.463 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval instance=1 2023-11-08T05:18:17.463 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=1 2023-11-08T05:18:17.464 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-11-08T05:18:17.464 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local local_cluster=cluster1:1 2023-11-08T05:18:17.464 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster=cluster1 2023-11-08T05:18:17.464 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance 2023-11-08T05:18:17.464 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-11-08T05:18:17.465 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local val=cluster1:1 2023-11-08T05:18:17.465 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster_var_name=local_cluster 2023-11-08T05:18:17.465 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance_var_name=instance 2023-11-08T05:18:17.465 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:18:17.465 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=1 2023-11-08T05:18:17.466 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ '[' 1 = cluster1:1 ']' 2023-11-08T05:18:17.466 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval local_cluster=cluster1 2023-11-08T05:18:17.466 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ local_cluster=cluster1 2023-11-08T05:18:17.466 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval instance=1 2023-11-08T05:18:17.466 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ instance=1 2023-11-08T05:18:17.467 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-11-08T05:18:17.475 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-08T05:18:17.475 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-08T05:18:17.475 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-08T05:18:17.476 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_30 2023-11-08T05:18:17.576 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:Invalid command: unused arguments: ['mirror/image_30'] 2023-11-08T05:18:17.576 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd mirror status : get status for rbd mirror 2023-11-08T05:18:17.577 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:admin_socket: invalid command 2023-11-08T05:18:17.580 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T05:18:17.580 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_30 2023-11-08T05:18:17.580 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1:2 2023-11-08T05:18:17.580 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T05:18:17.581 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance 2023-11-08T05:18:17.581 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-11-08T05:18:17.581 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local val=cluster1:2 2023-11-08T05:18:17.581 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_var_name=cluster 2023-11-08T05:18:17.582 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance_var_name=instance 2023-11-08T05:18:17.582 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cluster=cluster1 2023-11-08T05:18:17.582 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ instance=2 2023-11-08T05:18:17.582 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' 2 = cluster1:2 ']' 2023-11-08T05:18:17.582 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval cluster=cluster1 2023-11-08T05:18:17.583 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:18:17.583 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval instance=2 2023-11-08T05:18:17.583 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=2 2023-11-08T05:18:17.583 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-11-08T05:18:17.584 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local local_cluster=cluster1:2 2023-11-08T05:18:17.584 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster=cluster1 2023-11-08T05:18:17.584 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance 2023-11-08T05:18:17.584 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-11-08T05:18:17.584 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local val=cluster1:2 2023-11-08T05:18:17.585 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster_var_name=local_cluster 2023-11-08T05:18:17.585 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance_var_name=instance 2023-11-08T05:18:17.585 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:18:17.585 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=2 2023-11-08T05:18:17.585 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ '[' 2 = cluster1:2 ']' 2023-11-08T05:18:17.586 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval local_cluster=cluster1 2023-11-08T05:18:17.586 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ local_cluster=cluster1 2023-11-08T05:18:17.586 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval instance=2 2023-11-08T05:18:17.586 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ instance=2 2023-11-08T05:18:17.586 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-11-08T05:18:17.594 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-08T05:18:17.594 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-08T05:18:17.594 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-08T05:18:17.595 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_30 2023-11-08T05:18:17.693 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:Invalid command: unused arguments: ['mirror/image_30'] 2023-11-08T05:18:17.693 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd mirror status : get status for rbd mirror 2023-11-08T05:18:17.693 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:admin_socket: invalid command 2023-11-08T05:18:17.696 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T05:18:17.696 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_30 2023-11-08T05:18:17.697 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1:3 2023-11-08T05:18:17.697 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T05:18:17.697 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance 2023-11-08T05:18:17.697 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ set_cluster_instance cluster1:3 cluster instance 2023-11-08T05:18:17.697 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local val=cluster1:3 2023-11-08T05:18:17.697 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_var_name=cluster 2023-11-08T05:18:17.698 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance_var_name=instance 2023-11-08T05:18:17.698 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cluster=cluster1 2023-11-08T05:18:17.698 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ instance=3 2023-11-08T05:18:17.698 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' 3 = cluster1:3 ']' 2023-11-08T05:18:17.699 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval cluster=cluster1 2023-11-08T05:18:17.699 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:18:17.699 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval instance=3 2023-11-08T05:18:17.699 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=3 2023-11-08T05:18:17.699 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ daemon_asok_file cluster1:3 cluster1 2023-11-08T05:18:17.700 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local local_cluster=cluster1:3 2023-11-08T05:18:17.700 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster=cluster1 2023-11-08T05:18:17.700 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance 2023-11-08T05:18:17.700 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2023-11-08T05:18:17.700 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local val=cluster1:3 2023-11-08T05:18:17.701 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster_var_name=local_cluster 2023-11-08T05:18:17.701 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance_var_name=instance 2023-11-08T05:18:17.701 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:18:17.701 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=3 2023-11-08T05:18:17.701 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ '[' 3 = cluster1:3 ']' 2023-11-08T05:18:17.701 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval local_cluster=cluster1 2023-11-08T05:18:17.702 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ local_cluster=cluster1 2023-11-08T05:18:17.702 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval instance=3 2023-11-08T05:18:17.702 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ instance=3 2023-11-08T05:18:17.702 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2023-11-08T05:18:17.711 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-11-08T05:18:17.711 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-11-08T05:18:17.711 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-11-08T05:18:17.711 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_30 2023-11-08T05:18:17.802 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:Invalid command: unused arguments: ['mirror/image_30'] 2023-11-08T05:18:17.803 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd mirror status : get status for rbd mirror 2023-11-08T05:18:17.803 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:admin_socket: invalid command 2023-11-08T05:18:17.803 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-11-08T05:18:17.804 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sleep 8 2023-11-08T05:18:25.805 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-11-08T05:18:25.805 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ seq 0 3 2023-11-08T05:18:25.809 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T05:18:25.809 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_30 2023-11-08T05:18:25.809 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1:0 2023-11-08T05:18:25.809 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T05:18:25.810 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance 2023-11-08T05:18:25.810 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-11-08T05:18:25.810 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local val=cluster1:0 2023-11-08T05:18:25.810 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_var_name=cluster 2023-11-08T05:18:25.810 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance_var_name=instance 2023-11-08T05:18:25.811 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cluster=cluster1 2023-11-08T05:18:25.811 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ instance=0 2023-11-08T05:18:25.811 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' 0 = cluster1:0 ']' 2023-11-08T05:18:25.811 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval cluster=cluster1 2023-11-08T05:18:25.811 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:18:25.812 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval instance=0 2023-11-08T05:18:25.812 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=0 2023-11-08T05:18:25.812 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-11-08T05:18:25.812 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local local_cluster=cluster1:0 2023-11-08T05:18:25.812 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster=cluster1 2023-11-08T05:18:25.813 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance 2023-11-08T05:18:25.813 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-11-08T05:18:25.813 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local val=cluster1:0 2023-11-08T05:18:25.814 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster_var_name=local_cluster 2023-11-08T05:18:25.814 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance_var_name=instance 2023-11-08T05:18:25.814 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:18:25.814 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=0 2023-11-08T05:18:25.814 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ '[' 0 = cluster1:0 ']' 2023-11-08T05:18:25.815 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval local_cluster=cluster1 2023-11-08T05:18:25.815 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ local_cluster=cluster1 2023-11-08T05:18:25.815 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval instance=0 2023-11-08T05:18:25.815 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ instance=0 2023-11-08T05:18:25.815 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-11-08T05:18:25.825 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-08T05:18:25.825 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-08T05:18:25.825 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-08T05:18:25.825 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_30 2023-11-08T05:18:25.931 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:Invalid command: unused arguments: ['mirror/image_30'] 2023-11-08T05:18:25.931 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd mirror status : get status for rbd mirror 2023-11-08T05:18:25.932 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:admin_socket: invalid command 2023-11-08T05:18:25.937 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T05:18:25.937 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_30 2023-11-08T05:18:25.937 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1:1 2023-11-08T05:18:25.938 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T05:18:25.938 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance 2023-11-08T05:18:25.938 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-11-08T05:18:25.939 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local val=cluster1:1 2023-11-08T05:18:25.939 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_var_name=cluster 2023-11-08T05:18:25.939 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance_var_name=instance 2023-11-08T05:18:25.940 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cluster=cluster1 2023-11-08T05:18:25.940 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ instance=1 2023-11-08T05:18:25.940 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' 1 = cluster1:1 ']' 2023-11-08T05:18:25.941 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval cluster=cluster1 2023-11-08T05:18:25.941 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:18:25.941 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval instance=1 2023-11-08T05:18:25.941 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=1 2023-11-08T05:18:25.941 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-11-08T05:18:25.942 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local local_cluster=cluster1:1 2023-11-08T05:18:25.942 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster=cluster1 2023-11-08T05:18:25.942 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance 2023-11-08T05:18:25.943 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-11-08T05:18:25.943 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local val=cluster1:1 2023-11-08T05:18:25.943 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster_var_name=local_cluster 2023-11-08T05:18:25.944 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance_var_name=instance 2023-11-08T05:18:25.944 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:18:25.944 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=1 2023-11-08T05:18:25.945 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ '[' 1 = cluster1:1 ']' 2023-11-08T05:18:25.945 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval local_cluster=cluster1 2023-11-08T05:18:25.946 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ local_cluster=cluster1 2023-11-08T05:18:25.946 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval instance=1 2023-11-08T05:18:25.946 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ instance=1 2023-11-08T05:18:25.947 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-11-08T05:18:25.948 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-08T05:18:25.949 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-08T05:18:25.949 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-08T05:18:25.949 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_30 2023-11-08T05:18:26.048 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:Invalid command: unused arguments: ['mirror/image_30'] 2023-11-08T05:18:26.049 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd mirror status : get status for rbd mirror 2023-11-08T05:18:26.049 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:admin_socket: invalid command 2023-11-08T05:18:26.054 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T05:18:26.054 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_30 2023-11-08T05:18:26.055 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1:2 2023-11-08T05:18:26.055 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T05:18:26.055 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance 2023-11-08T05:18:26.056 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-11-08T05:18:26.056 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local val=cluster1:2 2023-11-08T05:18:26.057 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_var_name=cluster 2023-11-08T05:18:26.057 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance_var_name=instance 2023-11-08T05:18:26.057 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cluster=cluster1 2023-11-08T05:18:26.058 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ instance=2 2023-11-08T05:18:26.058 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' 2 = cluster1:2 ']' 2023-11-08T05:18:26.058 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval cluster=cluster1 2023-11-08T05:18:26.059 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:18:26.059 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval instance=2 2023-11-08T05:18:26.059 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=2 2023-11-08T05:18:26.060 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-11-08T05:18:26.060 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local local_cluster=cluster1:2 2023-11-08T05:18:26.060 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster=cluster1 2023-11-08T05:18:26.060 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance 2023-11-08T05:18:26.061 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-11-08T05:18:26.061 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local val=cluster1:2 2023-11-08T05:18:26.061 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster_var_name=local_cluster 2023-11-08T05:18:26.062 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance_var_name=instance 2023-11-08T05:18:26.062 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:18:26.062 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=2 2023-11-08T05:18:26.063 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ '[' 2 = cluster1:2 ']' 2023-11-08T05:18:26.063 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval local_cluster=cluster1 2023-11-08T05:18:26.063 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ local_cluster=cluster1 2023-11-08T05:18:26.063 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval instance=2 2023-11-08T05:18:26.063 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ instance=2 2023-11-08T05:18:26.064 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-11-08T05:18:26.065 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-08T05:18:26.066 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-08T05:18:26.066 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-08T05:18:26.066 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_30 2023-11-08T05:18:26.165 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:Invalid command: unused arguments: ['mirror/image_30'] 2023-11-08T05:18:26.166 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd mirror status : get status for rbd mirror 2023-11-08T05:18:26.166 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:admin_socket: invalid command 2023-11-08T05:18:26.168 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T05:18:26.169 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_30 2023-11-08T05:18:26.169 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1:3 2023-11-08T05:18:26.169 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T05:18:26.169 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance 2023-11-08T05:18:26.170 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ set_cluster_instance cluster1:3 cluster instance 2023-11-08T05:18:26.170 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local val=cluster1:3 2023-11-08T05:18:26.170 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_var_name=cluster 2023-11-08T05:18:26.170 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance_var_name=instance 2023-11-08T05:18:26.170 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cluster=cluster1 2023-11-08T05:18:26.171 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ instance=3 2023-11-08T05:18:26.171 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' 3 = cluster1:3 ']' 2023-11-08T05:18:26.171 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval cluster=cluster1 2023-11-08T05:18:26.172 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:18:26.172 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval instance=3 2023-11-08T05:18:26.172 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=3 2023-11-08T05:18:26.172 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ daemon_asok_file cluster1:3 cluster1 2023-11-08T05:18:26.172 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local local_cluster=cluster1:3 2023-11-08T05:18:26.172 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster=cluster1 2023-11-08T05:18:26.173 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance 2023-11-08T05:18:26.174 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2023-11-08T05:18:26.174 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local val=cluster1:3 2023-11-08T05:18:26.174 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster_var_name=local_cluster 2023-11-08T05:18:26.174 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance_var_name=instance 2023-11-08T05:18:26.175 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:18:26.175 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=3 2023-11-08T05:18:26.175 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ '[' 3 = cluster1:3 ']' 2023-11-08T05:18:26.175 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval local_cluster=cluster1 2023-11-08T05:18:26.175 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ local_cluster=cluster1 2023-11-08T05:18:26.176 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval instance=3 2023-11-08T05:18:26.176 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ instance=3 2023-11-08T05:18:26.176 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2023-11-08T05:18:26.182 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-11-08T05:18:26.183 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-11-08T05:18:26.183 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-11-08T05:18:26.183 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_30 2023-11-08T05:18:26.289 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ return 0 2023-11-08T05:18:26.290 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ status_result=' "state": "Replaying"' 2023-11-08T05:18:26.290 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ echo ' "state": "Replaying"' 2023-11-08T05:18:26.290 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ grep -i Replaying 2023-11-08T05:18:26.291 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: "state": "Replaying" 2023-11-08T05:18:26.293 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ current_state=started 2023-11-08T05:18:26.293 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test started = started 2023-11-08T05:18:26.293 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:18:26.293 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_30 2023-11-08T05:18:26.294 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local local_cluster=cluster1 2023-11-08T05:18:26.294 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:18:26.294 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:18:26.294 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_30 2023-11-08T05:18:26.294 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' snapshot = journal ']' 2023-11-08T05:18:26.295 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' snapshot = snapshot ']' 2023-11-08T05:18:26.295 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_30 2023-11-08T05:18:26.295 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local local_cluster=cluster1 2023-11-08T05:18:26.295 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:18:26.295 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:18:26.296 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_30 2023-11-08T05:18:26.296 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ mkfname cluster2-mirror-image_30.status 2023-11-08T05:18:26.296 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ echo cluster2-mirror-image_30.status 2023-11-08T05:18:26.296 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -e 's|[/ ]|_|g' 2023-11-08T05:18:26.296 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local status_log=/tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_30.status 2023-11-08T05:18:26.297 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ mkfname cluster1-mirror-image_30.status 2023-11-08T05:18:26.297 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ echo cluster1-mirror-image_30.status 2023-11-08T05:18:26.297 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -e 's|[/ ]|_|g' 2023-11-08T05:18:26.298 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local local_status_log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_30.status 2023-11-08T05:18:26.298 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ mirror_image_snapshot cluster2 mirror image_30 2023-11-08T05:18:26.298 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:18:26.299 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:18:26.299 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_30 2023-11-08T05:18:26.299 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_30 2023-11-08T05:18:27.225 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:Snapshot ID: 177 2023-11-08T05:18:27.233 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_30 /tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_30.status 2023-11-08T05:18:27.233 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:18:27.233 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:18:27.234 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_30 2023-11-08T05:18:27.234 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_30.status 2023-11-08T05:18:27.234 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_30 --format xml 2023-11-08T05:18:27.235 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T05:18:27.272 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-11-08T05:18:27.274 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snapshot_id=177 2023-11-08T05:18:27.275 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ true 2023-11-08T05:18:27.275 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-11-08T05:18:27.275 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0.2 2023-11-08T05:18:27.476 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_30 /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_30.status 2023-11-08T05:18:27.476 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:18:27.477 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:18:27.477 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_30 2023-11-08T05:18:27.477 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_30.status 2023-11-08T05:18:27.478 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_30 --format xml 2023-11-08T05:18:27.478 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T05:18:27.517 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ true 2023-11-08T05:18:27.517 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-08T05:18:27.519 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:-:1.1: Document is empty 2023-11-08T05:18:27.519 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: 2023-11-08T05:18:27.519 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:^ 2023-11-08T05:18:27.520 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local primary_snapshot_id= 2023-11-08T05:18:27.520 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test 177 = '' 2023-11-08T05:18:27.520 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-11-08T05:18:27.520 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0.4 2023-11-08T05:18:27.920 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_30 /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_30.status 2023-11-08T05:18:27.921 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:18:27.921 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:18:27.921 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_30 2023-11-08T05:18:27.921 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_30.status 2023-11-08T05:18:27.921 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_30 --format xml 2023-11-08T05:18:27.922 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T05:18:27.961 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ true 2023-11-08T05:18:27.961 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-08T05:18:27.962 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:-:1.1: Document is empty 2023-11-08T05:18:27.962 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: 2023-11-08T05:18:27.962 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:^ 2023-11-08T05:18:27.963 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local primary_snapshot_id= 2023-11-08T05:18:27.963 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test 177 = '' 2023-11-08T05:18:27.964 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-11-08T05:18:27.964 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0.8 2023-11-08T05:18:28.764 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_30 /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_30.status 2023-11-08T05:18:28.764 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:18:28.764 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:18:28.765 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_30 2023-11-08T05:18:28.765 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_30.status 2023-11-08T05:18:28.765 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_30 --format xml 2023-11-08T05:18:28.765 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T05:18:28.811 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ true 2023-11-08T05:18:28.812 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-08T05:18:28.813 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:-:1.1: Document is empty 2023-11-08T05:18:28.813 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: 2023-11-08T05:18:28.813 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:^ 2023-11-08T05:18:28.814 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local primary_snapshot_id= 2023-11-08T05:18:28.814 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test 177 = '' 2023-11-08T05:18:28.814 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-11-08T05:18:28.815 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 1.6 2023-11-08T05:18:30.415 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_30 /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_30.status 2023-11-08T05:18:30.416 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:18:30.416 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:18:30.416 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_30 2023-11-08T05:18:30.416 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_30.status 2023-11-08T05:18:30.417 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_30 --format xml 2023-11-08T05:18:30.417 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T05:18:30.460 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-08T05:18:30.462 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local primary_snapshot_id=177 2023-11-08T05:18:30.463 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test 177 = 177 2023-11-08T05:18:30.463 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:18:30.463 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_snap_present cluster1 mirror image_30 snap 2023-11-08T05:18:30.463 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:18:30.463 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:18:30.464 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_30 2023-11-08T05:18:30.464 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap_name=snap 2023-11-08T05:18:30.464 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local s 2023-11-08T05:18:30.464 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-11-08T05:18:30.464 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 1 2023-11-08T05:18:31.463 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 info mirror/image_30@snap 2023-11-08T05:18:31.501 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:rbd image 'image_30': 2023-11-08T05:18:31.501 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: size 128 MiB in 32 objects 2023-11-08T05:18:31.501 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: order 22 (4 MiB objects) 2023-11-08T05:18:31.501 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: snapshot_count: 2 2023-11-08T05:18:31.502 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: id: 11d8c4116169 2023-11-08T05:18:31.502 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: block_name_prefix: rbd_data.11d8c4116169 2023-11-08T05:18:31.502 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: format: 2 2023-11-08T05:18:31.502 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: features: layering, exclusive-lock, non-primary 2023-11-08T05:18:31.503 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: op_features: 2023-11-08T05:18:31.503 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: flags: 2023-11-08T05:18:31.503 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: create_timestamp: Wed Nov 8 05:09:46 2023 2023-11-08T05:18:31.503 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: access_timestamp: Wed Nov 8 05:09:46 2023 2023-11-08T05:18:31.503 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: modify_timestamp: Wed Nov 8 05:09:46 2023 2023-11-08T05:18:31.504 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: protected: False 2023-11-08T05:18:31.504 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring state: enabled 2023-11-08T05:18:31.504 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring mode: snapshot 2023-11-08T05:18:31.504 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring global id: c78f7ab4-c752-49b4-8cd9-bc194f1db211 2023-11-08T05:18:31.504 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring primary: false 2023-11-08T05:18:31.506 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:18:31.506 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ compare_image_snaps mirror image_30 snap 2023-11-08T05:18:31.506 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:18:31.506 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_30 2023-11-08T05:18:31.507 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap_name=snap 2023-11-08T05:18:31.507 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local ret=0 2023-11-08T05:18:31.507 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local rmt_export=/tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_30.export 2023-11-08T05:18:31.507 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local loc_export=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_30.export 2023-11-08T05:18:31.507 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rm -f /tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_30.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_30.export 2023-11-08T05:18:31.508 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 -p mirror export image_30@snap /tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_30.export 2023-11-08T05:18:31.683 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-11-08T05:18:31.686 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 -p mirror export image_30@snap /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_30.export 2023-11-08T05:18:31.899 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-11-08T05:18:31.903 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ cmp /tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_30.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_30.export 2023-11-08T05:18:31.958 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rm -f /tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_30.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_30.export 2023-11-08T05:18:31.995 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:18:31.995 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-11-08T05:18:31.996 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image=image_31 2023-11-08T05:18:31.996 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ create_snap cluster2 mirror image_31 snap 2023-11-08T05:18:31.996 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:18:31.996 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:18:31.996 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_31 2023-11-08T05:18:31.997 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap_name=snap 2023-11-08T05:18:31.997 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 -p mirror snap create image_31@snap --debug-rbd=20 --debug-journaler=20 2023-11-08T05:18:33.248 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_image_replay_started cluster1 mirror image_31 2023-11-08T05:18:33.249 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:18:33.249 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:18:33.249 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_31 2023-11-08T05:18:33.249 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_image_replay_state cluster1 mirror image_31 started 2023-11-08T05:18:33.249 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:18:33.250 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:18:33.250 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_31 2023-11-08T05:18:33.250 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local state=started 2023-11-08T05:18:33.250 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local s 2023-11-08T05:18:33.250 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-11-08T05:18:33.251 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 1 2023-11-08T05:18:34.250 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test_image_replay_state cluster1 mirror image_31 started 2023-11-08T05:18:34.251 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:18:34.251 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:18:34.251 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_31 2023-11-08T05:18:34.251 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local test_state=started 2023-11-08T05:18:34.251 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local status_result 2023-11-08T05:18:34.252 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local current_state=stopped 2023-11-08T05:18:34.252 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_31 2023-11-08T05:18:34.252 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_instance=cluster1 2023-11-08T05:18:34.252 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ grep -i state 2023-11-08T05:18:34.252 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T05:18:34.253 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1 2023-11-08T05:18:34.253 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance=cluster1 2023-11-08T05:18:34.253 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local loop_instance 2023-11-08T05:18:34.253 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-11-08T05:18:34.253 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sleep 0 2023-11-08T05:18:34.256 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-11-08T05:18:34.257 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ seq 0 3 2023-11-08T05:18:34.257 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T05:18:34.257 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_31 2023-11-08T05:18:34.257 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1:0 2023-11-08T05:18:34.258 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T05:18:34.258 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance 2023-11-08T05:18:34.258 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-11-08T05:18:34.258 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local val=cluster1:0 2023-11-08T05:18:34.258 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_var_name=cluster 2023-11-08T05:18:34.259 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance_var_name=instance 2023-11-08T05:18:34.259 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cluster=cluster1 2023-11-08T05:18:34.259 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ instance=0 2023-11-08T05:18:34.259 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' 0 = cluster1:0 ']' 2023-11-08T05:18:34.259 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval cluster=cluster1 2023-11-08T05:18:34.259 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:18:34.260 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval instance=0 2023-11-08T05:18:34.260 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=0 2023-11-08T05:18:34.260 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-11-08T05:18:34.260 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local local_cluster=cluster1:0 2023-11-08T05:18:34.260 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster=cluster1 2023-11-08T05:18:34.261 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance 2023-11-08T05:18:34.261 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-11-08T05:18:34.261 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local val=cluster1:0 2023-11-08T05:18:34.261 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster_var_name=local_cluster 2023-11-08T05:18:34.261 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance_var_name=instance 2023-11-08T05:18:34.262 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:18:34.262 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=0 2023-11-08T05:18:34.262 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ '[' 0 = cluster1:0 ']' 2023-11-08T05:18:34.262 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval local_cluster=cluster1 2023-11-08T05:18:34.262 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ local_cluster=cluster1 2023-11-08T05:18:34.262 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval instance=0 2023-11-08T05:18:34.263 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ instance=0 2023-11-08T05:18:34.263 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-11-08T05:18:34.273 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-08T05:18:34.273 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-08T05:18:34.273 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-08T05:18:34.273 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_31 2023-11-08T05:18:34.388 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ return 0 2023-11-08T05:18:34.389 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ status_result=' "state": "Replaying"' 2023-11-08T05:18:34.389 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ echo ' "state": "Replaying"' 2023-11-08T05:18:34.389 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ grep -i Replaying 2023-11-08T05:18:34.390 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: "state": "Replaying" 2023-11-08T05:18:34.392 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ current_state=started 2023-11-08T05:18:34.392 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test started = started 2023-11-08T05:18:34.392 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:18:34.392 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_31 2023-11-08T05:18:34.392 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local local_cluster=cluster1 2023-11-08T05:18:34.393 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:18:34.393 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:18:34.393 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_31 2023-11-08T05:18:34.393 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' snapshot = journal ']' 2023-11-08T05:18:34.393 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' snapshot = snapshot ']' 2023-11-08T05:18:34.394 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_31 2023-11-08T05:18:34.394 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local local_cluster=cluster1 2023-11-08T05:18:34.394 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:18:34.394 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:18:34.394 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_31 2023-11-08T05:18:34.395 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ mkfname cluster2-mirror-image_31.status 2023-11-08T05:18:34.395 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ echo cluster2-mirror-image_31.status 2023-11-08T05:18:34.395 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -e 's|[/ ]|_|g' 2023-11-08T05:18:34.395 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local status_log=/tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_31.status 2023-11-08T05:18:34.395 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ mkfname cluster1-mirror-image_31.status 2023-11-08T05:18:34.395 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ echo cluster1-mirror-image_31.status 2023-11-08T05:18:34.396 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -e 's|[/ ]|_|g' 2023-11-08T05:18:34.396 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local local_status_log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_31.status 2023-11-08T05:18:34.397 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ mirror_image_snapshot cluster2 mirror image_31 2023-11-08T05:18:34.397 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:18:34.397 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:18:34.397 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_31 2023-11-08T05:18:34.398 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_31 2023-11-08T05:18:35.259 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:Snapshot ID: 179 2023-11-08T05:18:35.268 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_31 /tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_31.status 2023-11-08T05:18:35.268 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:18:35.268 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:18:35.268 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_31 2023-11-08T05:18:35.269 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_31.status 2023-11-08T05:18:35.269 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_31 --format xml 2023-11-08T05:18:35.269 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T05:18:35.307 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-11-08T05:18:35.309 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snapshot_id=179 2023-11-08T05:18:35.309 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ true 2023-11-08T05:18:35.310 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-11-08T05:18:35.310 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0.2 2023-11-08T05:18:35.511 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_31 /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_31.status 2023-11-08T05:18:35.511 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:18:35.511 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:18:35.512 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_31 2023-11-08T05:18:35.512 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_31.status 2023-11-08T05:18:35.512 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_31 --format xml 2023-11-08T05:18:35.512 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T05:18:35.551 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ true 2023-11-08T05:18:35.552 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-08T05:18:35.553 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:-:1.1: Document is empty 2023-11-08T05:18:35.553 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: 2023-11-08T05:18:35.553 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:^ 2023-11-08T05:18:35.554 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local primary_snapshot_id= 2023-11-08T05:18:35.554 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test 179 = '' 2023-11-08T05:18:35.554 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-11-08T05:18:35.554 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0.4 2023-11-08T05:18:35.955 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_31 /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_31.status 2023-11-08T05:18:35.955 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:18:35.955 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:18:35.955 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_31 2023-11-08T05:18:35.956 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_31.status 2023-11-08T05:18:35.956 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_31 --format xml 2023-11-08T05:18:35.956 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T05:18:36.000 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ true 2023-11-08T05:18:36.001 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-08T05:18:36.002 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:-:1.1: Document is empty 2023-11-08T05:18:36.002 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: 2023-11-08T05:18:36.002 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:^ 2023-11-08T05:18:36.003 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local primary_snapshot_id= 2023-11-08T05:18:36.003 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test 179 = '' 2023-11-08T05:18:36.003 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-11-08T05:18:36.004 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0.8 2023-11-08T05:18:36.804 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_31 /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_31.status 2023-11-08T05:18:36.804 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:18:36.804 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:18:36.805 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_31 2023-11-08T05:18:36.805 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_31.status 2023-11-08T05:18:36.805 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_31 --format xml 2023-11-08T05:18:36.805 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T05:18:36.846 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-08T05:18:36.848 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local primary_snapshot_id=179 2023-11-08T05:18:36.848 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test 179 = 179 2023-11-08T05:18:36.848 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:18:36.848 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_snap_present cluster1 mirror image_31 snap 2023-11-08T05:18:36.848 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:18:36.849 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:18:36.849 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_31 2023-11-08T05:18:36.849 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap_name=snap 2023-11-08T05:18:36.849 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local s 2023-11-08T05:18:36.849 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-11-08T05:18:36.850 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 1 2023-11-08T05:18:37.848 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 info mirror/image_31@snap 2023-11-08T05:18:37.886 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:rbd image 'image_31': 2023-11-08T05:18:37.886 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: size 128 MiB in 32 objects 2023-11-08T05:18:37.886 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: order 22 (4 MiB objects) 2023-11-08T05:18:37.887 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: snapshot_count: 2 2023-11-08T05:18:37.887 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: id: 128746c4cdaf 2023-11-08T05:18:37.887 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: block_name_prefix: rbd_data.128746c4cdaf 2023-11-08T05:18:37.887 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: format: 2 2023-11-08T05:18:37.887 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: features: layering, exclusive-lock, non-primary 2023-11-08T05:18:37.888 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: op_features: 2023-11-08T05:18:37.888 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: flags: 2023-11-08T05:18:37.888 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: create_timestamp: Wed Nov 8 05:09:57 2023 2023-11-08T05:18:37.888 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: access_timestamp: Wed Nov 8 05:09:57 2023 2023-11-08T05:18:37.889 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: modify_timestamp: Wed Nov 8 05:09:57 2023 2023-11-08T05:18:37.889 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: protected: False 2023-11-08T05:18:37.889 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring state: enabled 2023-11-08T05:18:37.889 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring mode: snapshot 2023-11-08T05:18:37.889 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring global id: 50f6deef-3c6f-490a-bab8-f906a3136c2a 2023-11-08T05:18:37.890 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring primary: false 2023-11-08T05:18:37.891 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:18:37.891 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ compare_image_snaps mirror image_31 snap 2023-11-08T05:18:37.891 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:18:37.892 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_31 2023-11-08T05:18:37.892 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap_name=snap 2023-11-08T05:18:37.892 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local ret=0 2023-11-08T05:18:37.892 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local rmt_export=/tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_31.export 2023-11-08T05:18:37.892 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local loc_export=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_31.export 2023-11-08T05:18:37.893 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rm -f /tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_31.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_31.export 2023-11-08T05:18:37.893 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 -p mirror export image_31@snap /tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_31.export 2023-11-08T05:18:38.069 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-11-08T05:18:38.072 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 -p mirror export image_31@snap /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_31.export 2023-11-08T05:18:38.255 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-11-08T05:18:38.258 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ cmp /tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_31.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_31.export 2023-11-08T05:18:38.314 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rm -f /tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_31.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_31.export 2023-11-08T05:18:38.354 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:18:38.355 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-11-08T05:18:38.355 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image=image_32 2023-11-08T05:18:38.355 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ create_snap cluster2 mirror image_32 snap 2023-11-08T05:18:38.355 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:18:38.356 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:18:38.356 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_32 2023-11-08T05:18:38.356 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap_name=snap 2023-11-08T05:18:38.356 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 -p mirror snap create image_32@snap --debug-rbd=20 --debug-journaler=20 2023-11-08T05:18:40.277 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_image_replay_started cluster1 mirror image_32 2023-11-08T05:18:40.278 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:18:40.278 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:18:40.278 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_32 2023-11-08T05:18:40.278 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_image_replay_state cluster1 mirror image_32 started 2023-11-08T05:18:40.278 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:18:40.279 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:18:40.279 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_32 2023-11-08T05:18:40.279 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local state=started 2023-11-08T05:18:40.279 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local s 2023-11-08T05:18:40.279 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-11-08T05:18:40.280 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 1 2023-11-08T05:18:41.278 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test_image_replay_state cluster1 mirror image_32 started 2023-11-08T05:18:41.279 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:18:41.279 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:18:41.279 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_32 2023-11-08T05:18:41.279 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local test_state=started 2023-11-08T05:18:41.279 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local status_result 2023-11-08T05:18:41.280 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local current_state=stopped 2023-11-08T05:18:41.281 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_32 2023-11-08T05:18:41.281 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ grep -i state 2023-11-08T05:18:41.281 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_instance=cluster1 2023-11-08T05:18:41.281 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T05:18:41.282 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1 2023-11-08T05:18:41.282 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance=cluster1 2023-11-08T05:18:41.282 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local loop_instance 2023-11-08T05:18:41.282 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-11-08T05:18:41.283 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sleep 0 2023-11-08T05:18:41.283 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-11-08T05:18:41.283 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ seq 0 3 2023-11-08T05:18:41.289 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T05:18:41.289 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_32 2023-11-08T05:18:41.289 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1:0 2023-11-08T05:18:41.290 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T05:18:41.290 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance 2023-11-08T05:18:41.290 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-11-08T05:18:41.291 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local val=cluster1:0 2023-11-08T05:18:41.291 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_var_name=cluster 2023-11-08T05:18:41.291 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance_var_name=instance 2023-11-08T05:18:41.292 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cluster=cluster1 2023-11-08T05:18:41.292 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ instance=0 2023-11-08T05:18:41.293 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' 0 = cluster1:0 ']' 2023-11-08T05:18:41.293 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval cluster=cluster1 2023-11-08T05:18:41.293 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:18:41.294 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval instance=0 2023-11-08T05:18:41.294 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=0 2023-11-08T05:18:41.294 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-11-08T05:18:41.294 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local local_cluster=cluster1:0 2023-11-08T05:18:41.295 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster=cluster1 2023-11-08T05:18:41.295 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance 2023-11-08T05:18:41.295 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-11-08T05:18:41.295 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local val=cluster1:0 2023-11-08T05:18:41.295 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster_var_name=local_cluster 2023-11-08T05:18:41.296 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance_var_name=instance 2023-11-08T05:18:41.296 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:18:41.296 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=0 2023-11-08T05:18:41.296 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ '[' 0 = cluster1:0 ']' 2023-11-08T05:18:41.296 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval local_cluster=cluster1 2023-11-08T05:18:41.297 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ local_cluster=cluster1 2023-11-08T05:18:41.297 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval instance=0 2023-11-08T05:18:41.297 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ instance=0 2023-11-08T05:18:41.297 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-11-08T05:18:41.300 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-08T05:18:41.301 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-08T05:18:41.301 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-08T05:18:41.301 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_32 2023-11-08T05:18:41.410 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:Invalid command: unused arguments: ['mirror/image_32'] 2023-11-08T05:18:41.410 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd mirror status : get status for rbd mirror 2023-11-08T05:18:41.410 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:admin_socket: invalid command 2023-11-08T05:18:41.413 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T05:18:41.413 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_32 2023-11-08T05:18:41.413 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1:1 2023-11-08T05:18:41.414 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T05:18:41.414 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance 2023-11-08T05:18:41.414 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-11-08T05:18:41.414 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local val=cluster1:1 2023-11-08T05:18:41.414 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_var_name=cluster 2023-11-08T05:18:41.415 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance_var_name=instance 2023-11-08T05:18:41.415 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cluster=cluster1 2023-11-08T05:18:41.415 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ instance=1 2023-11-08T05:18:41.415 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' 1 = cluster1:1 ']' 2023-11-08T05:18:41.415 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval cluster=cluster1 2023-11-08T05:18:41.416 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:18:41.416 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval instance=1 2023-11-08T05:18:41.416 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=1 2023-11-08T05:18:41.416 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-11-08T05:18:41.416 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local local_cluster=cluster1:1 2023-11-08T05:18:41.416 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster=cluster1 2023-11-08T05:18:41.417 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance 2023-11-08T05:18:41.417 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-11-08T05:18:41.417 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local val=cluster1:1 2023-11-08T05:18:41.417 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster_var_name=local_cluster 2023-11-08T05:18:41.417 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance_var_name=instance 2023-11-08T05:18:41.418 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:18:41.418 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=1 2023-11-08T05:18:41.418 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ '[' 1 = cluster1:1 ']' 2023-11-08T05:18:41.418 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval local_cluster=cluster1 2023-11-08T05:18:41.418 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ local_cluster=cluster1 2023-11-08T05:18:41.418 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval instance=1 2023-11-08T05:18:41.419 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ instance=1 2023-11-08T05:18:41.419 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-11-08T05:18:41.427 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-08T05:18:41.428 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-08T05:18:41.428 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-08T05:18:41.428 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_32 2023-11-08T05:18:41.541 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ return 0 2023-11-08T05:18:41.542 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ status_result=' "state": "Replaying"' 2023-11-08T05:18:41.542 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ echo ' "state": "Replaying"' 2023-11-08T05:18:41.543 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ grep -i Replaying 2023-11-08T05:18:41.543 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: "state": "Replaying" 2023-11-08T05:18:41.545 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ current_state=started 2023-11-08T05:18:41.545 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test started = started 2023-11-08T05:18:41.546 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:18:41.546 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_32 2023-11-08T05:18:41.546 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local local_cluster=cluster1 2023-11-08T05:18:41.546 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:18:41.546 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:18:41.547 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_32 2023-11-08T05:18:41.547 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' snapshot = journal ']' 2023-11-08T05:18:41.547 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' snapshot = snapshot ']' 2023-11-08T05:18:41.547 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_32 2023-11-08T05:18:41.547 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local local_cluster=cluster1 2023-11-08T05:18:41.548 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:18:41.548 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:18:41.548 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_32 2023-11-08T05:18:41.548 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ mkfname cluster2-mirror-image_32.status 2023-11-08T05:18:41.548 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ echo cluster2-mirror-image_32.status 2023-11-08T05:18:41.549 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -e 's|[/ ]|_|g' 2023-11-08T05:18:41.549 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local status_log=/tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_32.status 2023-11-08T05:18:41.549 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ mkfname cluster1-mirror-image_32.status 2023-11-08T05:18:41.549 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ echo cluster1-mirror-image_32.status 2023-11-08T05:18:41.549 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -e 's|[/ ]|_|g' 2023-11-08T05:18:41.550 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local local_status_log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_32.status 2023-11-08T05:18:41.551 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ mirror_image_snapshot cluster2 mirror image_32 2023-11-08T05:18:41.551 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:18:41.551 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:18:41.551 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_32 2023-11-08T05:18:41.552 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_32 2023-11-08T05:18:42.278 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:Snapshot ID: 181 2023-11-08T05:18:42.287 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_32 /tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_32.status 2023-11-08T05:18:42.287 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:18:42.287 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:18:42.288 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_32 2023-11-08T05:18:42.288 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_32.status 2023-11-08T05:18:42.288 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_32 --format xml 2023-11-08T05:18:42.288 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T05:18:42.327 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-11-08T05:18:42.329 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snapshot_id=181 2023-11-08T05:18:42.330 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ true 2023-11-08T05:18:42.330 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-11-08T05:18:42.330 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0.2 2023-11-08T05:18:42.530 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_32 /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_32.status 2023-11-08T05:18:42.531 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:18:42.531 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:18:42.531 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_32 2023-11-08T05:18:42.531 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_32.status 2023-11-08T05:18:42.531 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_32 --format xml 2023-11-08T05:18:42.532 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T05:18:42.570 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ true 2023-11-08T05:18:42.571 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-08T05:18:42.572 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:-:1.1: Document is empty 2023-11-08T05:18:42.572 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: 2023-11-08T05:18:42.573 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:^ 2023-11-08T05:18:42.573 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local primary_snapshot_id= 2023-11-08T05:18:42.573 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test 181 = '' 2023-11-08T05:18:42.573 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-11-08T05:18:42.574 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0.4 2023-11-08T05:18:42.974 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_32 /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_32.status 2023-11-08T05:18:42.974 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:18:42.974 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:18:42.975 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_32 2023-11-08T05:18:42.975 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_32.status 2023-11-08T05:18:42.975 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_32 --format xml 2023-11-08T05:18:42.975 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T05:18:43.013 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ true 2023-11-08T05:18:43.013 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-08T05:18:43.014 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:-:1.1: Document is empty 2023-11-08T05:18:43.014 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: 2023-11-08T05:18:43.015 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:^ 2023-11-08T05:18:43.015 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local primary_snapshot_id= 2023-11-08T05:18:43.016 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test 181 = '' 2023-11-08T05:18:43.016 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-11-08T05:18:43.016 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0.8 2023-11-08T05:18:43.816 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_32 /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_32.status 2023-11-08T05:18:43.817 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:18:43.817 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:18:43.817 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_32 2023-11-08T05:18:43.817 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_32.status 2023-11-08T05:18:43.818 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_32 --format xml 2023-11-08T05:18:43.818 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T05:18:43.861 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-08T05:18:43.863 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local primary_snapshot_id=181 2023-11-08T05:18:43.863 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test 181 = 181 2023-11-08T05:18:43.863 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:18:43.863 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_snap_present cluster1 mirror image_32 snap 2023-11-08T05:18:43.864 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:18:43.864 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:18:43.864 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_32 2023-11-08T05:18:43.864 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap_name=snap 2023-11-08T05:18:43.864 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local s 2023-11-08T05:18:43.865 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-11-08T05:18:43.865 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 1 2023-11-08T05:18:44.864 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 info mirror/image_32@snap 2023-11-08T05:18:44.904 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:rbd image 'image_32': 2023-11-08T05:18:44.905 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: size 128 MiB in 32 objects 2023-11-08T05:18:44.905 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: order 22 (4 MiB objects) 2023-11-08T05:18:44.905 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: snapshot_count: 2 2023-11-08T05:18:44.905 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: id: 12a9ab01b867 2023-11-08T05:18:44.905 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: block_name_prefix: rbd_data.12a9ab01b867 2023-11-08T05:18:44.906 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: format: 2 2023-11-08T05:18:44.906 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: features: layering, exclusive-lock, non-primary 2023-11-08T05:18:44.906 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: op_features: 2023-11-08T05:18:44.906 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: flags: 2023-11-08T05:18:44.906 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: create_timestamp: Wed Nov 8 05:10:03 2023 2023-11-08T05:18:44.907 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: access_timestamp: Wed Nov 8 05:10:03 2023 2023-11-08T05:18:44.907 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: modify_timestamp: Wed Nov 8 05:10:03 2023 2023-11-08T05:18:44.907 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: protected: False 2023-11-08T05:18:44.907 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring state: enabled 2023-11-08T05:18:44.907 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring mode: snapshot 2023-11-08T05:18:44.908 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring global id: e6ec2241-9334-4ae3-ae69-78e6eb1a509b 2023-11-08T05:18:44.908 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring primary: false 2023-11-08T05:18:44.909 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:18:44.909 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ compare_image_snaps mirror image_32 snap 2023-11-08T05:18:44.909 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:18:44.910 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_32 2023-11-08T05:18:44.910 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap_name=snap 2023-11-08T05:18:44.910 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local ret=0 2023-11-08T05:18:44.910 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local rmt_export=/tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_32.export 2023-11-08T05:18:44.910 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local loc_export=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_32.export 2023-11-08T05:18:44.911 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rm -f /tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_32.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_32.export 2023-11-08T05:18:44.911 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 -p mirror export image_32@snap /tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_32.export 2023-11-08T05:18:45.090 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-11-08T05:18:45.093 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 -p mirror export image_32@snap /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_32.export 2023-11-08T05:18:45.314 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-11-08T05:18:45.316 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ cmp /tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_32.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_32.export 2023-11-08T05:18:45.371 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rm -f /tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_32.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_32.export 2023-11-08T05:18:45.409 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:18:45.409 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-11-08T05:18:45.409 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image=image_33 2023-11-08T05:18:45.409 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ create_snap cluster2 mirror image_33 snap 2023-11-08T05:18:45.410 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:18:45.410 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:18:45.410 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_33 2023-11-08T05:18:45.410 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap_name=snap 2023-11-08T05:18:45.410 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 -p mirror snap create image_33@snap --debug-rbd=20 --debug-journaler=20 2023-11-08T05:18:47.283 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_image_replay_started cluster1 mirror image_33 2023-11-08T05:18:47.283 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:18:47.283 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:18:47.284 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_33 2023-11-08T05:18:47.284 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_image_replay_state cluster1 mirror image_33 started 2023-11-08T05:18:47.284 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:18:47.284 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:18:47.284 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_33 2023-11-08T05:18:47.285 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local state=started 2023-11-08T05:18:47.285 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local s 2023-11-08T05:18:47.285 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-11-08T05:18:47.285 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 1 2023-11-08T05:18:48.284 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test_image_replay_state cluster1 mirror image_33 started 2023-11-08T05:18:48.285 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:18:48.285 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:18:48.285 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_33 2023-11-08T05:18:48.285 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local test_state=started 2023-11-08T05:18:48.286 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local status_result 2023-11-08T05:18:48.286 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local current_state=stopped 2023-11-08T05:18:48.286 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_33 2023-11-08T05:18:48.286 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_instance=cluster1 2023-11-08T05:18:48.286 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T05:18:48.287 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ grep -i state 2023-11-08T05:18:48.287 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1 2023-11-08T05:18:48.287 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance=cluster1 2023-11-08T05:18:48.287 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local loop_instance 2023-11-08T05:18:48.287 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-11-08T05:18:48.288 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sleep 0 2023-11-08T05:18:48.289 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-11-08T05:18:48.290 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ seq 0 3 2023-11-08T05:18:48.290 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T05:18:48.290 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_33 2023-11-08T05:18:48.290 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1:0 2023-11-08T05:18:48.290 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T05:18:48.291 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance 2023-11-08T05:18:48.291 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-11-08T05:18:48.291 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local val=cluster1:0 2023-11-08T05:18:48.291 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_var_name=cluster 2023-11-08T05:18:48.292 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance_var_name=instance 2023-11-08T05:18:48.292 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cluster=cluster1 2023-11-08T05:18:48.292 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ instance=0 2023-11-08T05:18:48.292 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' 0 = cluster1:0 ']' 2023-11-08T05:18:48.292 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval cluster=cluster1 2023-11-08T05:18:48.293 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:18:48.293 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval instance=0 2023-11-08T05:18:48.293 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=0 2023-11-08T05:18:48.295 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-11-08T05:18:48.295 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local local_cluster=cluster1:0 2023-11-08T05:18:48.295 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster=cluster1 2023-11-08T05:18:48.295 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance 2023-11-08T05:18:48.296 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-11-08T05:18:48.296 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local val=cluster1:0 2023-11-08T05:18:48.296 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster_var_name=local_cluster 2023-11-08T05:18:48.296 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance_var_name=instance 2023-11-08T05:18:48.296 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:18:48.297 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=0 2023-11-08T05:18:48.297 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ '[' 0 = cluster1:0 ']' 2023-11-08T05:18:48.297 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval local_cluster=cluster1 2023-11-08T05:18:48.297 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ local_cluster=cluster1 2023-11-08T05:18:48.297 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval instance=0 2023-11-08T05:18:48.298 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ instance=0 2023-11-08T05:18:48.298 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-11-08T05:18:48.306 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-08T05:18:48.306 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-08T05:18:48.306 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-08T05:18:48.306 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_33 2023-11-08T05:18:48.413 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:Invalid command: unused arguments: ['mirror/image_33'] 2023-11-08T05:18:48.414 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd mirror status : get status for rbd mirror 2023-11-08T05:18:48.414 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:admin_socket: invalid command 2023-11-08T05:18:48.419 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T05:18:48.419 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_33 2023-11-08T05:18:48.420 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1:1 2023-11-08T05:18:48.421 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T05:18:48.421 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance 2023-11-08T05:18:48.421 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-11-08T05:18:48.421 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local val=cluster1:1 2023-11-08T05:18:48.422 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_var_name=cluster 2023-11-08T05:18:48.422 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance_var_name=instance 2023-11-08T05:18:48.422 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cluster=cluster1 2023-11-08T05:18:48.422 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ instance=1 2023-11-08T05:18:48.422 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' 1 = cluster1:1 ']' 2023-11-08T05:18:48.423 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval cluster=cluster1 2023-11-08T05:18:48.423 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:18:48.423 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval instance=1 2023-11-08T05:18:48.423 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=1 2023-11-08T05:18:48.423 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-11-08T05:18:48.423 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local local_cluster=cluster1:1 2023-11-08T05:18:48.424 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster=cluster1 2023-11-08T05:18:48.424 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance 2023-11-08T05:18:48.424 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-11-08T05:18:48.424 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local val=cluster1:1 2023-11-08T05:18:48.424 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster_var_name=local_cluster 2023-11-08T05:18:48.425 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance_var_name=instance 2023-11-08T05:18:48.425 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:18:48.425 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=1 2023-11-08T05:18:48.425 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ '[' 1 = cluster1:1 ']' 2023-11-08T05:18:48.425 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval local_cluster=cluster1 2023-11-08T05:18:48.425 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ local_cluster=cluster1 2023-11-08T05:18:48.426 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval instance=1 2023-11-08T05:18:48.426 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ instance=1 2023-11-08T05:18:48.426 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-11-08T05:18:48.430 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-08T05:18:48.431 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-08T05:18:48.431 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-08T05:18:48.431 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_33 2023-11-08T05:18:48.531 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:Invalid command: unused arguments: ['mirror/image_33'] 2023-11-08T05:18:48.531 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd mirror status : get status for rbd mirror 2023-11-08T05:18:48.531 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:admin_socket: invalid command 2023-11-08T05:18:48.534 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T05:18:48.534 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_33 2023-11-08T05:18:48.534 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1:2 2023-11-08T05:18:48.535 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T05:18:48.535 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance 2023-11-08T05:18:48.535 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-11-08T05:18:48.535 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local val=cluster1:2 2023-11-08T05:18:48.535 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_var_name=cluster 2023-11-08T05:18:48.536 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance_var_name=instance 2023-11-08T05:18:48.536 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cluster=cluster1 2023-11-08T05:18:48.536 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ instance=2 2023-11-08T05:18:48.536 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' 2 = cluster1:2 ']' 2023-11-08T05:18:48.536 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval cluster=cluster1 2023-11-08T05:18:48.537 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:18:48.537 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval instance=2 2023-11-08T05:18:48.537 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=2 2023-11-08T05:18:48.537 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-11-08T05:18:48.537 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local local_cluster=cluster1:2 2023-11-08T05:18:48.538 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster=cluster1 2023-11-08T05:18:48.538 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance 2023-11-08T05:18:48.538 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-11-08T05:18:48.538 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local val=cluster1:2 2023-11-08T05:18:48.538 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster_var_name=local_cluster 2023-11-08T05:18:48.539 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance_var_name=instance 2023-11-08T05:18:48.539 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:18:48.539 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=2 2023-11-08T05:18:48.539 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ '[' 2 = cluster1:2 ']' 2023-11-08T05:18:48.539 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval local_cluster=cluster1 2023-11-08T05:18:48.540 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ local_cluster=cluster1 2023-11-08T05:18:48.540 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval instance=2 2023-11-08T05:18:48.540 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ instance=2 2023-11-08T05:18:48.541 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-11-08T05:18:48.548 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-08T05:18:48.548 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-08T05:18:48.548 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-08T05:18:48.549 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_33 2023-11-08T05:18:48.656 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ return 0 2023-11-08T05:18:48.657 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ status_result=' "state": "Replaying"' 2023-11-08T05:18:48.657 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ echo ' "state": "Replaying"' 2023-11-08T05:18:48.658 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ grep -i Replaying 2023-11-08T05:18:48.658 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: "state": "Replaying" 2023-11-08T05:18:48.660 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ current_state=started 2023-11-08T05:18:48.660 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test started = started 2023-11-08T05:18:48.661 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:18:48.661 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_33 2023-11-08T05:18:48.661 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local local_cluster=cluster1 2023-11-08T05:18:48.661 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:18:48.661 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:18:48.662 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_33 2023-11-08T05:18:48.662 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' snapshot = journal ']' 2023-11-08T05:18:48.662 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' snapshot = snapshot ']' 2023-11-08T05:18:48.662 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_33 2023-11-08T05:18:48.662 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local local_cluster=cluster1 2023-11-08T05:18:48.663 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:18:48.663 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:18:48.663 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_33 2023-11-08T05:18:48.663 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ mkfname cluster2-mirror-image_33.status 2023-11-08T05:18:48.663 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ echo cluster2-mirror-image_33.status 2023-11-08T05:18:48.663 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -e 's|[/ ]|_|g' 2023-11-08T05:18:48.664 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local status_log=/tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_33.status 2023-11-08T05:18:48.664 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ mkfname cluster1-mirror-image_33.status 2023-11-08T05:18:48.664 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ echo cluster1-mirror-image_33.status 2023-11-08T05:18:48.664 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -e 's|[/ ]|_|g' 2023-11-08T05:18:48.665 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local local_status_log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_33.status 2023-11-08T05:18:48.665 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ mirror_image_snapshot cluster2 mirror image_33 2023-11-08T05:18:48.666 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:18:48.666 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:18:48.666 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_33 2023-11-08T05:18:48.666 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_33 2023-11-08T05:18:49.291 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:Snapshot ID: 183 2023-11-08T05:18:49.300 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_33 /tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_33.status 2023-11-08T05:18:49.301 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:18:49.301 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:18:49.301 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_33 2023-11-08T05:18:49.301 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_33.status 2023-11-08T05:18:49.301 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_33 --format xml 2023-11-08T05:18:49.302 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T05:18:49.342 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-11-08T05:18:49.344 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snapshot_id=183 2023-11-08T05:18:49.344 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ true 2023-11-08T05:18:49.344 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-11-08T05:18:49.344 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0.2 2023-11-08T05:18:49.545 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_33 /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_33.status 2023-11-08T05:18:49.546 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:18:49.546 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:18:49.547 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_33 2023-11-08T05:18:49.547 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_33.status 2023-11-08T05:18:49.547 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_33 --format xml 2023-11-08T05:18:49.548 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T05:18:49.586 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ true 2023-11-08T05:18:49.587 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-08T05:18:49.588 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:-:1.1: Document is empty 2023-11-08T05:18:49.589 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: 2023-11-08T05:18:49.589 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:^ 2023-11-08T05:18:49.589 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local primary_snapshot_id= 2023-11-08T05:18:49.590 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test 183 = '' 2023-11-08T05:18:49.590 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-11-08T05:18:49.590 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0.4 2023-11-08T05:18:49.990 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_33 /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_33.status 2023-11-08T05:18:49.990 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:18:49.991 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:18:49.991 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_33 2023-11-08T05:18:49.991 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_33.status 2023-11-08T05:18:49.991 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_33 --format xml 2023-11-08T05:18:49.992 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T05:18:50.030 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ true 2023-11-08T05:18:50.030 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-08T05:18:50.031 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:-:1.1: Document is empty 2023-11-08T05:18:50.031 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: 2023-11-08T05:18:50.031 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:^ 2023-11-08T05:18:50.032 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local primary_snapshot_id= 2023-11-08T05:18:50.033 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test 183 = '' 2023-11-08T05:18:50.033 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-11-08T05:18:50.033 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0.8 2023-11-08T05:18:50.833 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_33 /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_33.status 2023-11-08T05:18:50.833 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:18:50.833 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:18:50.834 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_33 2023-11-08T05:18:50.834 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_33.status 2023-11-08T05:18:50.834 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_33 --format xml 2023-11-08T05:18:50.834 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T05:18:50.873 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-08T05:18:50.875 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local primary_snapshot_id=183 2023-11-08T05:18:50.876 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test 183 = 183 2023-11-08T05:18:50.876 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:18:50.876 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_snap_present cluster1 mirror image_33 snap 2023-11-08T05:18:50.876 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:18:50.877 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:18:50.877 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_33 2023-11-08T05:18:50.877 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap_name=snap 2023-11-08T05:18:50.877 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local s 2023-11-08T05:18:50.877 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-11-08T05:18:50.878 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 1 2023-11-08T05:18:51.876 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 info mirror/image_33@snap 2023-11-08T05:18:51.914 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:rbd image 'image_33': 2023-11-08T05:18:51.914 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: size 128 MiB in 32 objects 2023-11-08T05:18:51.914 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: order 22 (4 MiB objects) 2023-11-08T05:18:51.915 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: snapshot_count: 2 2023-11-08T05:18:51.915 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: id: 10e7251a2cb 2023-11-08T05:18:51.915 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: block_name_prefix: rbd_data.10e7251a2cb 2023-11-08T05:18:51.915 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: format: 2 2023-11-08T05:18:51.915 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: features: layering, exclusive-lock, non-primary 2023-11-08T05:18:51.916 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: op_features: 2023-11-08T05:18:51.916 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: flags: 2023-11-08T05:18:51.916 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: create_timestamp: Wed Nov 8 05:10:11 2023 2023-11-08T05:18:51.916 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: access_timestamp: Wed Nov 8 05:10:11 2023 2023-11-08T05:18:51.916 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: modify_timestamp: Wed Nov 8 05:10:11 2023 2023-11-08T05:18:51.917 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: protected: False 2023-11-08T05:18:51.917 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring state: enabled 2023-11-08T05:18:51.917 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring mode: snapshot 2023-11-08T05:18:51.917 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring global id: 6f088a8f-d677-4bda-b50c-7fba63dd233e 2023-11-08T05:18:51.917 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring primary: false 2023-11-08T05:18:51.919 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:18:51.919 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ compare_image_snaps mirror image_33 snap 2023-11-08T05:18:51.919 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:18:51.919 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_33 2023-11-08T05:18:51.919 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap_name=snap 2023-11-08T05:18:51.920 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local ret=0 2023-11-08T05:18:51.920 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local rmt_export=/tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_33.export 2023-11-08T05:18:51.920 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local loc_export=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_33.export 2023-11-08T05:18:51.920 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rm -f /tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_33.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_33.export 2023-11-08T05:18:51.920 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 -p mirror export image_33@snap /tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_33.export 2023-11-08T05:18:52.094 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-11-08T05:18:52.097 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 -p mirror export image_33@snap /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_33.export 2023-11-08T05:18:52.322 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-11-08T05:18:52.325 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ cmp /tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_33.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_33.export 2023-11-08T05:18:52.381 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rm -f /tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_33.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_33.export 2023-11-08T05:18:52.417 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:18:52.418 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-11-08T05:18:52.418 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image=image_34 2023-11-08T05:18:52.418 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ create_snap cluster2 mirror image_34 snap 2023-11-08T05:18:52.418 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:18:52.419 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:18:52.419 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_34 2023-11-08T05:18:52.419 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap_name=snap 2023-11-08T05:18:52.419 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 -p mirror snap create image_34@snap --debug-rbd=20 --debug-journaler=20 2023-11-08T05:18:54.318 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_image_replay_started cluster1 mirror image_34 2023-11-08T05:18:54.319 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:18:54.319 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:18:54.319 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_34 2023-11-08T05:18:54.320 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_image_replay_state cluster1 mirror image_34 started 2023-11-08T05:18:54.320 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:18:54.320 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:18:54.320 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_34 2023-11-08T05:18:54.321 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local state=started 2023-11-08T05:18:54.321 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local s 2023-11-08T05:18:54.321 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-11-08T05:18:54.322 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 1 2023-11-08T05:18:55.320 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test_image_replay_state cluster1 mirror image_34 started 2023-11-08T05:18:55.320 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:18:55.321 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:18:55.321 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_34 2023-11-08T05:18:55.321 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local test_state=started 2023-11-08T05:18:55.321 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local status_result 2023-11-08T05:18:55.322 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local current_state=stopped 2023-11-08T05:18:55.324 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_34 2023-11-08T05:18:55.324 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_instance=cluster1 2023-11-08T05:18:55.325 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T05:18:55.325 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1 2023-11-08T05:18:55.325 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ grep -i state 2023-11-08T05:18:55.325 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance=cluster1 2023-11-08T05:18:55.325 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local loop_instance 2023-11-08T05:18:55.326 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-11-08T05:18:55.326 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sleep 0 2023-11-08T05:18:55.326 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-11-08T05:18:55.326 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ seq 0 3 2023-11-08T05:18:55.326 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T05:18:55.327 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_34 2023-11-08T05:18:55.327 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1:0 2023-11-08T05:18:55.327 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T05:18:55.327 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance 2023-11-08T05:18:55.327 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-11-08T05:18:55.327 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local val=cluster1:0 2023-11-08T05:18:55.328 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_var_name=cluster 2023-11-08T05:18:55.328 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance_var_name=instance 2023-11-08T05:18:55.328 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cluster=cluster1 2023-11-08T05:18:55.328 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ instance=0 2023-11-08T05:18:55.328 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' 0 = cluster1:0 ']' 2023-11-08T05:18:55.329 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval cluster=cluster1 2023-11-08T05:18:55.329 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:18:55.329 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval instance=0 2023-11-08T05:18:55.329 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=0 2023-11-08T05:18:55.329 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-11-08T05:18:55.330 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local local_cluster=cluster1:0 2023-11-08T05:18:55.330 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster=cluster1 2023-11-08T05:18:55.330 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance 2023-11-08T05:18:55.330 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-11-08T05:18:55.331 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local val=cluster1:0 2023-11-08T05:18:55.332 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster_var_name=local_cluster 2023-11-08T05:18:55.332 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance_var_name=instance 2023-11-08T05:18:55.332 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:18:55.332 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=0 2023-11-08T05:18:55.332 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ '[' 0 = cluster1:0 ']' 2023-11-08T05:18:55.333 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval local_cluster=cluster1 2023-11-08T05:18:55.333 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ local_cluster=cluster1 2023-11-08T05:18:55.333 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval instance=0 2023-11-08T05:18:55.333 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ instance=0 2023-11-08T05:18:55.333 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-11-08T05:18:55.342 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-08T05:18:55.342 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-08T05:18:55.342 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-08T05:18:55.343 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_34 2023-11-08T05:18:55.450 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:Invalid command: unused arguments: ['mirror/image_34'] 2023-11-08T05:18:55.451 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd mirror status : get status for rbd mirror 2023-11-08T05:18:55.451 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:admin_socket: invalid command 2023-11-08T05:18:55.454 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T05:18:55.454 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_34 2023-11-08T05:18:55.454 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1:1 2023-11-08T05:18:55.454 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T05:18:55.455 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance 2023-11-08T05:18:55.455 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-11-08T05:18:55.455 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local val=cluster1:1 2023-11-08T05:18:55.455 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_var_name=cluster 2023-11-08T05:18:55.455 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance_var_name=instance 2023-11-08T05:18:55.456 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cluster=cluster1 2023-11-08T05:18:55.456 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ instance=1 2023-11-08T05:18:55.456 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' 1 = cluster1:1 ']' 2023-11-08T05:18:55.456 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval cluster=cluster1 2023-11-08T05:18:55.456 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:18:55.457 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval instance=1 2023-11-08T05:18:55.457 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=1 2023-11-08T05:18:55.457 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-11-08T05:18:55.457 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local local_cluster=cluster1:1 2023-11-08T05:18:55.457 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster=cluster1 2023-11-08T05:18:55.458 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance 2023-11-08T05:18:55.458 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-11-08T05:18:55.458 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local val=cluster1:1 2023-11-08T05:18:55.458 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster_var_name=local_cluster 2023-11-08T05:18:55.458 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance_var_name=instance 2023-11-08T05:18:55.459 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:18:55.459 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=1 2023-11-08T05:18:55.459 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ '[' 1 = cluster1:1 ']' 2023-11-08T05:18:55.459 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval local_cluster=cluster1 2023-11-08T05:18:55.459 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ local_cluster=cluster1 2023-11-08T05:18:55.460 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval instance=1 2023-11-08T05:18:55.460 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ instance=1 2023-11-08T05:18:55.460 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-11-08T05:18:55.468 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-08T05:18:55.468 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-08T05:18:55.468 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-08T05:18:55.468 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_34 2023-11-08T05:18:55.569 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:Invalid command: unused arguments: ['mirror/image_34'] 2023-11-08T05:18:55.569 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd mirror status : get status for rbd mirror 2023-11-08T05:18:55.570 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:admin_socket: invalid command 2023-11-08T05:18:55.572 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T05:18:55.572 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_34 2023-11-08T05:18:55.573 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1:2 2023-11-08T05:18:55.573 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T05:18:55.573 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance 2023-11-08T05:18:55.573 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-11-08T05:18:55.573 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local val=cluster1:2 2023-11-08T05:18:55.574 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_var_name=cluster 2023-11-08T05:18:55.574 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance_var_name=instance 2023-11-08T05:18:55.574 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cluster=cluster1 2023-11-08T05:18:55.574 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ instance=2 2023-11-08T05:18:55.574 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' 2 = cluster1:2 ']' 2023-11-08T05:18:55.575 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval cluster=cluster1 2023-11-08T05:18:55.575 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:18:55.575 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval instance=2 2023-11-08T05:18:55.575 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=2 2023-11-08T05:18:55.576 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-11-08T05:18:55.576 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local local_cluster=cluster1:2 2023-11-08T05:18:55.576 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster=cluster1 2023-11-08T05:18:55.576 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance 2023-11-08T05:18:55.576 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-11-08T05:18:55.577 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local val=cluster1:2 2023-11-08T05:18:55.577 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster_var_name=local_cluster 2023-11-08T05:18:55.577 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance_var_name=instance 2023-11-08T05:18:55.577 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:18:55.577 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=2 2023-11-08T05:18:55.578 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ '[' 2 = cluster1:2 ']' 2023-11-08T05:18:55.578 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval local_cluster=cluster1 2023-11-08T05:18:55.578 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ local_cluster=cluster1 2023-11-08T05:18:55.578 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval instance=2 2023-11-08T05:18:55.578 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ instance=2 2023-11-08T05:18:55.579 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-11-08T05:18:55.587 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-08T05:18:55.587 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-08T05:18:55.587 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-08T05:18:55.587 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_34 2023-11-08T05:18:55.688 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:Invalid command: unused arguments: ['mirror/image_34'] 2023-11-08T05:18:55.689 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd mirror status : get status for rbd mirror 2023-11-08T05:18:55.689 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:admin_socket: invalid command 2023-11-08T05:18:55.692 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T05:18:55.692 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_34 2023-11-08T05:18:55.693 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1:3 2023-11-08T05:18:55.693 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T05:18:55.693 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance 2023-11-08T05:18:55.693 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ set_cluster_instance cluster1:3 cluster instance 2023-11-08T05:18:55.694 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local val=cluster1:3 2023-11-08T05:18:55.694 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_var_name=cluster 2023-11-08T05:18:55.694 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance_var_name=instance 2023-11-08T05:18:55.694 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cluster=cluster1 2023-11-08T05:18:55.695 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ instance=3 2023-11-08T05:18:55.695 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' 3 = cluster1:3 ']' 2023-11-08T05:18:55.695 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval cluster=cluster1 2023-11-08T05:18:55.695 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:18:55.695 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval instance=3 2023-11-08T05:18:55.696 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=3 2023-11-08T05:18:55.696 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ daemon_asok_file cluster1:3 cluster1 2023-11-08T05:18:55.696 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local local_cluster=cluster1:3 2023-11-08T05:18:55.697 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster=cluster1 2023-11-08T05:18:55.697 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance 2023-11-08T05:18:55.697 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2023-11-08T05:18:55.697 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local val=cluster1:3 2023-11-08T05:18:55.697 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster_var_name=local_cluster 2023-11-08T05:18:55.698 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance_var_name=instance 2023-11-08T05:18:55.698 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:18:55.698 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=3 2023-11-08T05:18:55.698 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ '[' 3 = cluster1:3 ']' 2023-11-08T05:18:55.699 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval local_cluster=cluster1 2023-11-08T05:18:55.699 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ local_cluster=cluster1 2023-11-08T05:18:55.699 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval instance=3 2023-11-08T05:18:55.699 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ instance=3 2023-11-08T05:18:55.700 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2023-11-08T05:18:55.706 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-11-08T05:18:55.706 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-11-08T05:18:55.706 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-11-08T05:18:55.707 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_34 2023-11-08T05:18:55.814 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ return 0 2023-11-08T05:18:55.814 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ status_result=' "state": "Replaying"' 2023-11-08T05:18:55.814 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ echo ' "state": "Replaying"' 2023-11-08T05:18:55.814 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ grep -i Replaying 2023-11-08T05:18:55.815 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: "state": "Replaying" 2023-11-08T05:18:55.817 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ current_state=started 2023-11-08T05:18:55.818 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test started = started 2023-11-08T05:18:55.818 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:18:55.818 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_34 2023-11-08T05:18:55.818 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local local_cluster=cluster1 2023-11-08T05:18:55.819 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:18:55.819 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:18:55.819 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_34 2023-11-08T05:18:55.819 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' snapshot = journal ']' 2023-11-08T05:18:55.819 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' snapshot = snapshot ']' 2023-11-08T05:18:55.820 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_34 2023-11-08T05:18:55.820 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local local_cluster=cluster1 2023-11-08T05:18:55.820 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:18:55.820 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:18:55.821 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_34 2023-11-08T05:18:55.821 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ mkfname cluster2-mirror-image_34.status 2023-11-08T05:18:55.821 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ echo cluster2-mirror-image_34.status 2023-11-08T05:18:55.821 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -e 's|[/ ]|_|g' 2023-11-08T05:18:55.821 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local status_log=/tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_34.status 2023-11-08T05:18:55.822 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ mkfname cluster1-mirror-image_34.status 2023-11-08T05:18:55.822 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ echo cluster1-mirror-image_34.status 2023-11-08T05:18:55.822 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -e 's|[/ ]|_|g' 2023-11-08T05:18:55.822 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local local_status_log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_34.status 2023-11-08T05:18:55.822 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ mirror_image_snapshot cluster2 mirror image_34 2023-11-08T05:18:55.823 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:18:55.823 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:18:55.823 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_34 2023-11-08T05:18:55.823 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_34 2023-11-08T05:18:56.317 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:Snapshot ID: 185 2023-11-08T05:18:56.326 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_34 /tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_34.status 2023-11-08T05:18:56.326 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:18:56.326 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:18:56.327 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_34 2023-11-08T05:18:56.327 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_34.status 2023-11-08T05:18:56.327 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_34 --format xml 2023-11-08T05:18:56.327 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T05:18:56.368 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-11-08T05:18:56.369 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snapshot_id=185 2023-11-08T05:18:56.370 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ true 2023-11-08T05:18:56.370 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-11-08T05:18:56.370 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0.2 2023-11-08T05:18:56.571 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_34 /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_34.status 2023-11-08T05:18:56.571 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:18:56.571 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:18:56.572 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_34 2023-11-08T05:18:56.572 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_34.status 2023-11-08T05:18:56.572 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_34 --format xml 2023-11-08T05:18:56.572 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T05:18:56.610 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ true 2023-11-08T05:18:56.610 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-08T05:18:56.611 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:-:1.1: Document is empty 2023-11-08T05:18:56.611 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: 2023-11-08T05:18:56.612 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:^ 2023-11-08T05:18:56.612 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local primary_snapshot_id= 2023-11-08T05:18:56.613 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test 185 = '' 2023-11-08T05:18:56.613 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-11-08T05:18:56.613 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0.4 2023-11-08T05:18:57.013 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_34 /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_34.status 2023-11-08T05:18:57.014 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:18:57.014 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:18:57.014 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_34 2023-11-08T05:18:57.014 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_34.status 2023-11-08T05:18:57.015 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_34 --format xml 2023-11-08T05:18:57.015 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T05:18:57.054 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ true 2023-11-08T05:18:57.054 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-08T05:18:57.055 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:-:1.1: Document is empty 2023-11-08T05:18:57.055 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: 2023-11-08T05:18:57.056 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:^ 2023-11-08T05:18:57.056 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local primary_snapshot_id= 2023-11-08T05:18:57.056 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test 185 = '' 2023-11-08T05:18:57.057 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-11-08T05:18:57.057 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0.8 2023-11-08T05:18:57.857 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_34 /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_34.status 2023-11-08T05:18:57.858 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:18:57.858 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:18:57.858 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_34 2023-11-08T05:18:57.858 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_34.status 2023-11-08T05:18:57.859 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_34 --format xml 2023-11-08T05:18:57.859 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T05:18:57.896 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-08T05:18:57.899 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local primary_snapshot_id=185 2023-11-08T05:18:57.899 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test 185 = 185 2023-11-08T05:18:57.899 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:18:57.899 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_snap_present cluster1 mirror image_34 snap 2023-11-08T05:18:57.900 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:18:57.900 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:18:57.900 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_34 2023-11-08T05:18:57.900 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap_name=snap 2023-11-08T05:18:57.900 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local s 2023-11-08T05:18:57.901 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-11-08T05:18:57.901 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 1 2023-11-08T05:18:58.899 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 info mirror/image_34@snap 2023-11-08T05:18:58.932 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:rbd image 'image_34': 2023-11-08T05:18:58.933 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: size 128 MiB in 32 objects 2023-11-08T05:18:58.933 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: order 22 (4 MiB objects) 2023-11-08T05:18:58.933 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: snapshot_count: 2 2023-11-08T05:18:58.934 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: id: 11d827d13d4e 2023-11-08T05:18:58.934 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: block_name_prefix: rbd_data.11d827d13d4e 2023-11-08T05:18:58.934 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: format: 2 2023-11-08T05:18:58.934 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: features: layering, exclusive-lock, non-primary 2023-11-08T05:18:58.934 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: op_features: 2023-11-08T05:18:58.935 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: flags: 2023-11-08T05:18:58.935 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: create_timestamp: Wed Nov 8 05:10:25 2023 2023-11-08T05:18:58.935 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: access_timestamp: Wed Nov 8 05:10:25 2023 2023-11-08T05:18:58.935 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: modify_timestamp: Wed Nov 8 05:10:25 2023 2023-11-08T05:18:58.935 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: protected: False 2023-11-08T05:18:58.936 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring state: enabled 2023-11-08T05:18:58.936 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring mode: snapshot 2023-11-08T05:18:58.936 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring global id: 9fa92707-a7d6-4c46-967b-87ad9015703b 2023-11-08T05:18:58.936 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring primary: false 2023-11-08T05:18:58.937 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:18:58.938 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ compare_image_snaps mirror image_34 snap 2023-11-08T05:18:58.938 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:18:58.938 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_34 2023-11-08T05:18:58.938 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap_name=snap 2023-11-08T05:18:58.939 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local ret=0 2023-11-08T05:18:58.939 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local rmt_export=/tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_34.export 2023-11-08T05:18:58.939 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local loc_export=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_34.export 2023-11-08T05:18:58.939 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rm -f /tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_34.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_34.export 2023-11-08T05:18:58.939 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 -p mirror export image_34@snap /tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_34.export 2023-11-08T05:18:59.110 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-11-08T05:18:59.113 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 -p mirror export image_34@snap /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_34.export 2023-11-08T05:18:59.367 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-11-08T05:18:59.370 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ cmp /tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_34.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_34.export 2023-11-08T05:18:59.425 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rm -f /tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_34.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_34.export 2023-11-08T05:18:59.461 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:18:59.462 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-11-08T05:18:59.462 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image=image_35 2023-11-08T05:18:59.462 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ create_snap cluster2 mirror image_35 snap 2023-11-08T05:18:59.462 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:18:59.463 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:18:59.463 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_35 2023-11-08T05:18:59.463 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap_name=snap 2023-11-08T05:18:59.463 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 -p mirror snap create image_35@snap --debug-rbd=20 --debug-journaler=20 2023-11-08T05:19:01.325 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_image_replay_started cluster1 mirror image_35 2023-11-08T05:19:01.326 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:19:01.326 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:19:01.326 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_35 2023-11-08T05:19:01.326 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_image_replay_state cluster1 mirror image_35 started 2023-11-08T05:19:01.327 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:19:01.327 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:19:01.327 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_35 2023-11-08T05:19:01.327 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local state=started 2023-11-08T05:19:01.327 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local s 2023-11-08T05:19:01.328 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-11-08T05:19:01.328 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 1 2023-11-08T05:19:02.326 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test_image_replay_state cluster1 mirror image_35 started 2023-11-08T05:19:02.327 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:19:02.327 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:19:02.327 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_35 2023-11-08T05:19:02.328 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local test_state=started 2023-11-08T05:19:02.328 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local status_result 2023-11-08T05:19:02.328 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local current_state=stopped 2023-11-08T05:19:02.331 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_35 2023-11-08T05:19:02.331 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_instance=cluster1 2023-11-08T05:19:02.331 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ grep -i state 2023-11-08T05:19:02.332 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T05:19:02.332 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1 2023-11-08T05:19:02.332 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance=cluster1 2023-11-08T05:19:02.332 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local loop_instance 2023-11-08T05:19:02.333 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-11-08T05:19:02.333 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sleep 0 2023-11-08T05:19:02.333 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-11-08T05:19:02.333 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ seq 0 3 2023-11-08T05:19:02.333 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T05:19:02.334 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_35 2023-11-08T05:19:02.334 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1:0 2023-11-08T05:19:02.334 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T05:19:02.334 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance 2023-11-08T05:19:02.334 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-11-08T05:19:02.335 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local val=cluster1:0 2023-11-08T05:19:02.335 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_var_name=cluster 2023-11-08T05:19:02.335 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance_var_name=instance 2023-11-08T05:19:02.335 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cluster=cluster1 2023-11-08T05:19:02.335 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ instance=0 2023-11-08T05:19:02.336 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' 0 = cluster1:0 ']' 2023-11-08T05:19:02.336 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval cluster=cluster1 2023-11-08T05:19:02.336 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:19:02.336 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval instance=0 2023-11-08T05:19:02.336 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=0 2023-11-08T05:19:02.337 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-11-08T05:19:02.337 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local local_cluster=cluster1:0 2023-11-08T05:19:02.337 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster=cluster1 2023-11-08T05:19:02.338 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance 2023-11-08T05:19:02.339 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-11-08T05:19:02.339 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local val=cluster1:0 2023-11-08T05:19:02.339 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster_var_name=local_cluster 2023-11-08T05:19:02.339 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance_var_name=instance 2023-11-08T05:19:02.340 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:19:02.340 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=0 2023-11-08T05:19:02.340 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ '[' 0 = cluster1:0 ']' 2023-11-08T05:19:02.340 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval local_cluster=cluster1 2023-11-08T05:19:02.340 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ local_cluster=cluster1 2023-11-08T05:19:02.341 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval instance=0 2023-11-08T05:19:02.341 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ instance=0 2023-11-08T05:19:02.341 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-11-08T05:19:02.348 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-08T05:19:02.349 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-08T05:19:02.350 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-08T05:19:02.350 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_35 2023-11-08T05:19:02.465 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ return 0 2023-11-08T05:19:02.465 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ status_result=' "state": "Replaying"' 2023-11-08T05:19:02.466 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ echo ' "state": "Replaying"' 2023-11-08T05:19:02.466 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ grep -i Replaying 2023-11-08T05:19:02.466 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: "state": "Replaying" 2023-11-08T05:19:02.468 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ current_state=started 2023-11-08T05:19:02.469 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test started = started 2023-11-08T05:19:02.469 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:19:02.469 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_35 2023-11-08T05:19:02.469 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local local_cluster=cluster1 2023-11-08T05:19:02.470 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:19:02.470 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:19:02.470 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_35 2023-11-08T05:19:02.470 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' snapshot = journal ']' 2023-11-08T05:19:02.471 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' snapshot = snapshot ']' 2023-11-08T05:19:02.471 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_35 2023-11-08T05:19:02.471 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local local_cluster=cluster1 2023-11-08T05:19:02.471 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:19:02.471 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:19:02.472 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_35 2023-11-08T05:19:02.472 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ mkfname cluster2-mirror-image_35.status 2023-11-08T05:19:02.472 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ echo cluster2-mirror-image_35.status 2023-11-08T05:19:02.472 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -e 's|[/ ]|_|g' 2023-11-08T05:19:02.472 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local status_log=/tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_35.status 2023-11-08T05:19:02.473 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ mkfname cluster1-mirror-image_35.status 2023-11-08T05:19:02.473 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ echo cluster1-mirror-image_35.status 2023-11-08T05:19:02.473 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -e 's|[/ ]|_|g' 2023-11-08T05:19:02.474 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local local_status_log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_35.status 2023-11-08T05:19:02.474 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ mirror_image_snapshot cluster2 mirror image_35 2023-11-08T05:19:02.474 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:19:02.475 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:19:02.475 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_35 2023-11-08T05:19:02.475 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_35 2023-11-08T05:19:03.334 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:Snapshot ID: 187 2023-11-08T05:19:03.344 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_35 /tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_35.status 2023-11-08T05:19:03.344 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:19:03.345 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:19:03.345 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_35 2023-11-08T05:19:03.345 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_35.status 2023-11-08T05:19:03.345 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_35 --format xml 2023-11-08T05:19:03.345 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T05:19:03.383 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-11-08T05:19:03.385 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snapshot_id=187 2023-11-08T05:19:03.385 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ true 2023-11-08T05:19:03.385 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-11-08T05:19:03.385 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0.2 2023-11-08T05:19:03.586 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_35 /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_35.status 2023-11-08T05:19:03.586 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:19:03.587 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:19:03.587 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_35 2023-11-08T05:19:03.587 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_35.status 2023-11-08T05:19:03.587 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_35 --format xml 2023-11-08T05:19:03.587 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T05:19:03.627 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ true 2023-11-08T05:19:03.628 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-08T05:19:03.628 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:-:1.1: Document is empty 2023-11-08T05:19:03.629 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: 2023-11-08T05:19:03.629 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:^ 2023-11-08T05:19:03.630 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local primary_snapshot_id= 2023-11-08T05:19:03.630 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test 187 = '' 2023-11-08T05:19:03.630 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-11-08T05:19:03.630 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0.4 2023-11-08T05:19:04.031 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_35 /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_35.status 2023-11-08T05:19:04.031 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:19:04.031 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:19:04.031 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_35 2023-11-08T05:19:04.032 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_35.status 2023-11-08T05:19:04.032 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_35 --format xml 2023-11-08T05:19:04.032 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T05:19:04.070 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ true 2023-11-08T05:19:04.070 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-08T05:19:04.071 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:-:1.1: Document is empty 2023-11-08T05:19:04.072 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: 2023-11-08T05:19:04.072 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:^ 2023-11-08T05:19:04.073 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local primary_snapshot_id= 2023-11-08T05:19:04.073 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test 187 = '' 2023-11-08T05:19:04.073 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-11-08T05:19:04.073 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0.8 2023-11-08T05:19:04.874 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_35 /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_35.status 2023-11-08T05:19:04.875 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:19:04.875 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:19:04.876 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_35 2023-11-08T05:19:04.876 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_35.status 2023-11-08T05:19:04.876 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_35 --format xml 2023-11-08T05:19:04.877 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T05:19:04.912 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-08T05:19:04.915 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local primary_snapshot_id=187 2023-11-08T05:19:04.916 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test 187 = 187 2023-11-08T05:19:04.916 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:19:04.916 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_snap_present cluster1 mirror image_35 snap 2023-11-08T05:19:04.917 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:19:04.917 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:19:04.917 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_35 2023-11-08T05:19:04.918 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap_name=snap 2023-11-08T05:19:04.918 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local s 2023-11-08T05:19:04.919 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-11-08T05:19:04.919 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 1 2023-11-08T05:19:05.915 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 info mirror/image_35@snap 2023-11-08T05:19:05.951 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:rbd image 'image_35': 2023-11-08T05:19:05.952 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: size 128 MiB in 32 objects 2023-11-08T05:19:05.952 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: order 22 (4 MiB objects) 2023-11-08T05:19:05.952 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: snapshot_count: 2 2023-11-08T05:19:05.952 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: id: 1287de82ebe2 2023-11-08T05:19:05.953 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: block_name_prefix: rbd_data.1287de82ebe2 2023-11-08T05:19:05.953 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: format: 2 2023-11-08T05:19:05.953 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: features: layering, exclusive-lock, non-primary 2023-11-08T05:19:05.953 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: op_features: 2023-11-08T05:19:05.953 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: flags: 2023-11-08T05:19:05.954 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: create_timestamp: Wed Nov 8 05:10:34 2023 2023-11-08T05:19:05.954 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: access_timestamp: Wed Nov 8 05:10:34 2023 2023-11-08T05:19:05.954 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: modify_timestamp: Wed Nov 8 05:10:34 2023 2023-11-08T05:19:05.954 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: protected: False 2023-11-08T05:19:05.954 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring state: enabled 2023-11-08T05:19:05.955 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring mode: snapshot 2023-11-08T05:19:05.955 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring global id: 4f580749-5b1c-4916-870b-0406e147094b 2023-11-08T05:19:05.955 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring primary: false 2023-11-08T05:19:05.956 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:19:05.956 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ compare_image_snaps mirror image_35 snap 2023-11-08T05:19:05.957 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:19:05.957 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_35 2023-11-08T05:19:05.957 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap_name=snap 2023-11-08T05:19:05.957 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local ret=0 2023-11-08T05:19:05.957 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local rmt_export=/tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_35.export 2023-11-08T05:19:05.958 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local loc_export=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_35.export 2023-11-08T05:19:05.958 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rm -f /tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_35.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_35.export 2023-11-08T05:19:05.958 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 -p mirror export image_35@snap /tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_35.export 2023-11-08T05:19:06.133 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-11-08T05:19:06.136 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 -p mirror export image_35@snap /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_35.export 2023-11-08T05:19:06.341 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-11-08T05:19:06.344 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ cmp /tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_35.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_35.export 2023-11-08T05:19:06.399 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rm -f /tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_35.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_35.export 2023-11-08T05:19:06.435 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:19:06.435 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-11-08T05:19:06.436 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image=image_36 2023-11-08T05:19:06.436 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ create_snap cluster2 mirror image_36 snap 2023-11-08T05:19:06.436 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:19:06.436 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:19:06.436 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_36 2023-11-08T05:19:06.437 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap_name=snap 2023-11-08T05:19:06.437 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 -p mirror snap create image_36@snap --debug-rbd=20 --debug-journaler=20 2023-11-08T05:19:08.349 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_image_replay_started cluster1 mirror image_36 2023-11-08T05:19:08.350 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:19:08.350 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:19:08.350 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_36 2023-11-08T05:19:08.351 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_image_replay_state cluster1 mirror image_36 started 2023-11-08T05:19:08.351 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:19:08.351 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:19:08.351 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_36 2023-11-08T05:19:08.351 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local state=started 2023-11-08T05:19:08.352 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local s 2023-11-08T05:19:08.352 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-11-08T05:19:08.352 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 1 2023-11-08T05:19:09.351 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test_image_replay_state cluster1 mirror image_36 started 2023-11-08T05:19:09.351 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:19:09.351 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:19:09.352 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_36 2023-11-08T05:19:09.352 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local test_state=started 2023-11-08T05:19:09.352 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local status_result 2023-11-08T05:19:09.352 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local current_state=stopped 2023-11-08T05:19:09.354 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_36 2023-11-08T05:19:09.354 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_instance=cluster1 2023-11-08T05:19:09.354 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ grep -i state 2023-11-08T05:19:09.354 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T05:19:09.354 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1 2023-11-08T05:19:09.355 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance=cluster1 2023-11-08T05:19:09.355 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local loop_instance 2023-11-08T05:19:09.355 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-11-08T05:19:09.355 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sleep 0 2023-11-08T05:19:09.355 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-11-08T05:19:09.356 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ seq 0 3 2023-11-08T05:19:09.357 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T05:19:09.358 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_36 2023-11-08T05:19:09.358 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1:0 2023-11-08T05:19:09.358 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T05:19:09.358 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance 2023-11-08T05:19:09.359 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-11-08T05:19:09.359 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local val=cluster1:0 2023-11-08T05:19:09.359 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_var_name=cluster 2023-11-08T05:19:09.359 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance_var_name=instance 2023-11-08T05:19:09.359 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cluster=cluster1 2023-11-08T05:19:09.360 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ instance=0 2023-11-08T05:19:09.360 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' 0 = cluster1:0 ']' 2023-11-08T05:19:09.360 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval cluster=cluster1 2023-11-08T05:19:09.360 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:19:09.360 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval instance=0 2023-11-08T05:19:09.362 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=0 2023-11-08T05:19:09.362 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-11-08T05:19:09.363 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local local_cluster=cluster1:0 2023-11-08T05:19:09.363 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster=cluster1 2023-11-08T05:19:09.363 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance 2023-11-08T05:19:09.363 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-11-08T05:19:09.363 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local val=cluster1:0 2023-11-08T05:19:09.364 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster_var_name=local_cluster 2023-11-08T05:19:09.364 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance_var_name=instance 2023-11-08T05:19:09.364 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:19:09.364 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=0 2023-11-08T05:19:09.364 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ '[' 0 = cluster1:0 ']' 2023-11-08T05:19:09.365 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval local_cluster=cluster1 2023-11-08T05:19:09.365 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ local_cluster=cluster1 2023-11-08T05:19:09.365 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval instance=0 2023-11-08T05:19:09.365 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ instance=0 2023-11-08T05:19:09.365 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-11-08T05:19:09.374 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-08T05:19:09.374 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-08T05:19:09.374 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-08T05:19:09.374 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_36 2023-11-08T05:19:09.480 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:Invalid command: unused arguments: ['mirror/image_36'] 2023-11-08T05:19:09.480 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd mirror status : get status for rbd mirror 2023-11-08T05:19:09.480 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:admin_socket: invalid command 2023-11-08T05:19:09.483 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T05:19:09.483 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_36 2023-11-08T05:19:09.484 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1:1 2023-11-08T05:19:09.484 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T05:19:09.484 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance 2023-11-08T05:19:09.484 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-11-08T05:19:09.484 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local val=cluster1:1 2023-11-08T05:19:09.485 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_var_name=cluster 2023-11-08T05:19:09.485 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance_var_name=instance 2023-11-08T05:19:09.485 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cluster=cluster1 2023-11-08T05:19:09.487 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ instance=1 2023-11-08T05:19:09.487 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' 1 = cluster1:1 ']' 2023-11-08T05:19:09.487 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval cluster=cluster1 2023-11-08T05:19:09.487 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:19:09.487 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval instance=1 2023-11-08T05:19:09.488 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=1 2023-11-08T05:19:09.488 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-11-08T05:19:09.488 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local local_cluster=cluster1:1 2023-11-08T05:19:09.488 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster=cluster1 2023-11-08T05:19:09.488 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance 2023-11-08T05:19:09.489 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-11-08T05:19:09.489 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local val=cluster1:1 2023-11-08T05:19:09.489 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster_var_name=local_cluster 2023-11-08T05:19:09.489 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance_var_name=instance 2023-11-08T05:19:09.489 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:19:09.490 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=1 2023-11-08T05:19:09.490 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ '[' 1 = cluster1:1 ']' 2023-11-08T05:19:09.490 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval local_cluster=cluster1 2023-11-08T05:19:09.491 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ local_cluster=cluster1 2023-11-08T05:19:09.491 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval instance=1 2023-11-08T05:19:09.491 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ instance=1 2023-11-08T05:19:09.491 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-11-08T05:19:09.497 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-08T05:19:09.498 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-08T05:19:09.498 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-08T05:19:09.498 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_36 2023-11-08T05:19:09.606 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ return 0 2023-11-08T05:19:09.607 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ status_result=' "state": "Replaying"' 2023-11-08T05:19:09.608 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ echo ' "state": "Replaying"' 2023-11-08T05:19:09.608 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ grep -i Replaying 2023-11-08T05:19:09.608 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: "state": "Replaying" 2023-11-08T05:19:09.611 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ current_state=started 2023-11-08T05:19:09.611 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test started = started 2023-11-08T05:19:09.611 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:19:09.611 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_36 2023-11-08T05:19:09.612 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local local_cluster=cluster1 2023-11-08T05:19:09.612 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:19:09.612 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:19:09.612 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_36 2023-11-08T05:19:09.613 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' snapshot = journal ']' 2023-11-08T05:19:09.613 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' snapshot = snapshot ']' 2023-11-08T05:19:09.613 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_36 2023-11-08T05:19:09.613 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local local_cluster=cluster1 2023-11-08T05:19:09.613 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:19:09.613 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:19:09.614 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_36 2023-11-08T05:19:09.614 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ mkfname cluster2-mirror-image_36.status 2023-11-08T05:19:09.614 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ echo cluster2-mirror-image_36.status 2023-11-08T05:19:09.614 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -e 's|[/ ]|_|g' 2023-11-08T05:19:09.614 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local status_log=/tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_36.status 2023-11-08T05:19:09.615 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ mkfname cluster1-mirror-image_36.status 2023-11-08T05:19:09.615 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ echo cluster1-mirror-image_36.status 2023-11-08T05:19:09.615 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -e 's|[/ ]|_|g' 2023-11-08T05:19:09.616 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local local_status_log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_36.status 2023-11-08T05:19:09.616 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ mirror_image_snapshot cluster2 mirror image_36 2023-11-08T05:19:09.616 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:19:09.617 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:19:09.617 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_36 2023-11-08T05:19:09.617 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_36 2023-11-08T05:19:10.353 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:Snapshot ID: 189 2023-11-08T05:19:10.360 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_36 /tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_36.status 2023-11-08T05:19:10.360 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:19:10.361 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:19:10.361 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_36 2023-11-08T05:19:10.361 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_36.status 2023-11-08T05:19:10.361 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_36 --format xml 2023-11-08T05:19:10.361 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T05:19:10.398 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-11-08T05:19:10.400 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snapshot_id=189 2023-11-08T05:19:10.400 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ true 2023-11-08T05:19:10.400 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-11-08T05:19:10.400 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0.2 2023-11-08T05:19:10.601 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_36 /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_36.status 2023-11-08T05:19:10.601 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:19:10.602 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:19:10.602 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_36 2023-11-08T05:19:10.602 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_36.status 2023-11-08T05:19:10.602 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_36 --format xml 2023-11-08T05:19:10.602 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T05:19:10.641 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ true 2023-11-08T05:19:10.642 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-08T05:19:10.643 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:-:1.1: Document is empty 2023-11-08T05:19:10.643 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: 2023-11-08T05:19:10.643 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:^ 2023-11-08T05:19:10.644 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local primary_snapshot_id= 2023-11-08T05:19:10.644 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test 189 = '' 2023-11-08T05:19:10.644 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-11-08T05:19:10.644 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0.4 2023-11-08T05:19:11.044 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_36 /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_36.status 2023-11-08T05:19:11.045 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:19:11.045 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:19:11.045 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_36 2023-11-08T05:19:11.045 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_36.status 2023-11-08T05:19:11.046 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_36 --format xml 2023-11-08T05:19:11.046 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T05:19:11.089 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ true 2023-11-08T05:19:11.090 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-08T05:19:11.091 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:-:1.1: Document is empty 2023-11-08T05:19:11.091 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: 2023-11-08T05:19:11.091 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:^ 2023-11-08T05:19:11.092 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local primary_snapshot_id= 2023-11-08T05:19:11.092 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test 189 = '' 2023-11-08T05:19:11.092 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-11-08T05:19:11.093 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0.8 2023-11-08T05:19:11.893 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_36 /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_36.status 2023-11-08T05:19:11.893 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:19:11.894 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:19:11.894 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_36 2023-11-08T05:19:11.894 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_36.status 2023-11-08T05:19:11.894 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_36 --format xml 2023-11-08T05:19:11.895 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T05:19:11.942 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ true 2023-11-08T05:19:11.943 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-08T05:19:11.944 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:-:1.1: Document is empty 2023-11-08T05:19:11.944 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: 2023-11-08T05:19:11.944 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:^ 2023-11-08T05:19:11.945 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local primary_snapshot_id= 2023-11-08T05:19:11.945 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test 189 = '' 2023-11-08T05:19:11.945 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-11-08T05:19:11.946 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 1.6 2023-11-08T05:19:13.546 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_36 /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_36.status 2023-11-08T05:19:13.546 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:19:13.546 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:19:13.547 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_36 2023-11-08T05:19:13.547 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_36.status 2023-11-08T05:19:13.547 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_36 --format xml 2023-11-08T05:19:13.547 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T05:19:13.590 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-08T05:19:13.592 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local primary_snapshot_id=189 2023-11-08T05:19:13.592 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test 189 = 189 2023-11-08T05:19:13.592 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:19:13.593 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_snap_present cluster1 mirror image_36 snap 2023-11-08T05:19:13.593 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:19:13.593 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:19:13.593 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_36 2023-11-08T05:19:13.593 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap_name=snap 2023-11-08T05:19:13.594 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local s 2023-11-08T05:19:13.594 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-11-08T05:19:13.594 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 1 2023-11-08T05:19:14.593 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 info mirror/image_36@snap 2023-11-08T05:19:14.632 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:rbd image 'image_36': 2023-11-08T05:19:14.632 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: size 128 MiB in 32 objects 2023-11-08T05:19:14.632 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: order 22 (4 MiB objects) 2023-11-08T05:19:14.633 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: snapshot_count: 2 2023-11-08T05:19:14.633 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: id: 12b3cd21add0 2023-11-08T05:19:14.633 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: block_name_prefix: rbd_data.12b3cd21add0 2023-11-08T05:19:14.633 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: format: 2 2023-11-08T05:19:14.633 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: features: layering, exclusive-lock, non-primary 2023-11-08T05:19:14.634 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: op_features: 2023-11-08T05:19:14.634 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: flags: 2023-11-08T05:19:14.634 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: create_timestamp: Wed Nov 8 05:10:41 2023 2023-11-08T05:19:14.634 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: access_timestamp: Wed Nov 8 05:10:41 2023 2023-11-08T05:19:14.634 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: modify_timestamp: Wed Nov 8 05:10:41 2023 2023-11-08T05:19:14.635 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: protected: False 2023-11-08T05:19:14.635 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring state: enabled 2023-11-08T05:19:14.635 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring mode: snapshot 2023-11-08T05:19:14.635 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring global id: a868743f-f258-4032-ae79-7cb04d60e193 2023-11-08T05:19:14.635 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring primary: false 2023-11-08T05:19:14.641 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:19:14.641 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ compare_image_snaps mirror image_36 snap 2023-11-08T05:19:14.641 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:19:14.642 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_36 2023-11-08T05:19:14.642 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap_name=snap 2023-11-08T05:19:14.642 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local ret=0 2023-11-08T05:19:14.642 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local rmt_export=/tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_36.export 2023-11-08T05:19:14.642 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local loc_export=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_36.export 2023-11-08T05:19:14.643 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rm -f /tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_36.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_36.export 2023-11-08T05:19:14.643 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 -p mirror export image_36@snap /tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_36.export 2023-11-08T05:19:14.818 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-11-08T05:19:14.820 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 -p mirror export image_36@snap /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_36.export 2023-11-08T05:19:15.047 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-11-08T05:19:15.050 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ cmp /tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_36.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_36.export 2023-11-08T05:19:15.105 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rm -f /tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_36.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_36.export 2023-11-08T05:19:15.143 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:19:15.143 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-11-08T05:19:15.144 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image=image_37 2023-11-08T05:19:15.144 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ create_snap cluster2 mirror image_37 snap 2023-11-08T05:19:15.144 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:19:15.144 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:19:15.145 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_37 2023-11-08T05:19:15.145 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap_name=snap 2023-11-08T05:19:15.145 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 -p mirror snap create image_37@snap --debug-rbd=20 --debug-journaler=20 2023-11-08T05:19:16.364 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_image_replay_started cluster1 mirror image_37 2023-11-08T05:19:16.364 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:19:16.365 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:19:16.365 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_37 2023-11-08T05:19:16.365 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_image_replay_state cluster1 mirror image_37 started 2023-11-08T05:19:16.365 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:19:16.366 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:19:16.366 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_37 2023-11-08T05:19:16.366 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local state=started 2023-11-08T05:19:16.366 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local s 2023-11-08T05:19:16.366 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-11-08T05:19:16.367 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 1 2023-11-08T05:19:17.108 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.7974247163145597) > weight (0.25) 2023-11-08T05:19:17.109 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.3670482871841717) > weight (0.25) 2023-11-08T05:19:17.109 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.6883190655591905) > weight (0.25) 2023-11-08T05:19:17.109 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.3 with skip (0.9181274028845989) > weight (0.25) 2023-11-08T05:19:17.109 INFO:tasks.rbd_mirror_thrash:waiting for 72 secs before thrashing 2023-11-08T05:19:17.366 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test_image_replay_state cluster1 mirror image_37 started 2023-11-08T05:19:17.366 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:19:17.366 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:19:17.366 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_37 2023-11-08T05:19:17.367 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local test_state=started 2023-11-08T05:19:17.367 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local status_result 2023-11-08T05:19:17.367 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local current_state=stopped 2023-11-08T05:19:17.367 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_37 2023-11-08T05:19:17.367 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_instance=cluster1 2023-11-08T05:19:17.367 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T05:19:17.368 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ grep -i state 2023-11-08T05:19:17.368 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1 2023-11-08T05:19:17.368 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance=cluster1 2023-11-08T05:19:17.368 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local loop_instance 2023-11-08T05:19:17.368 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-11-08T05:19:17.369 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sleep 0 2023-11-08T05:19:17.371 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-11-08T05:19:17.371 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ seq 0 3 2023-11-08T05:19:17.372 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T05:19:17.372 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_37 2023-11-08T05:19:17.372 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1:0 2023-11-08T05:19:17.372 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T05:19:17.373 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance 2023-11-08T05:19:17.373 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-11-08T05:19:17.373 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local val=cluster1:0 2023-11-08T05:19:17.373 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_var_name=cluster 2023-11-08T05:19:17.373 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance_var_name=instance 2023-11-08T05:19:17.374 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cluster=cluster1 2023-11-08T05:19:17.374 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ instance=0 2023-11-08T05:19:17.374 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' 0 = cluster1:0 ']' 2023-11-08T05:19:17.374 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval cluster=cluster1 2023-11-08T05:19:17.374 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:19:17.375 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval instance=0 2023-11-08T05:19:17.375 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=0 2023-11-08T05:19:17.375 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-11-08T05:19:17.375 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local local_cluster=cluster1:0 2023-11-08T05:19:17.377 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster=cluster1 2023-11-08T05:19:17.377 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance 2023-11-08T05:19:17.377 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-11-08T05:19:17.377 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local val=cluster1:0 2023-11-08T05:19:17.377 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster_var_name=local_cluster 2023-11-08T05:19:17.378 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance_var_name=instance 2023-11-08T05:19:17.378 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:19:17.378 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=0 2023-11-08T05:19:17.378 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ '[' 0 = cluster1:0 ']' 2023-11-08T05:19:17.378 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval local_cluster=cluster1 2023-11-08T05:19:17.379 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ local_cluster=cluster1 2023-11-08T05:19:17.379 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval instance=0 2023-11-08T05:19:17.379 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ instance=0 2023-11-08T05:19:17.379 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-11-08T05:19:17.387 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-08T05:19:17.387 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-08T05:19:17.387 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-08T05:19:17.387 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_37 2023-11-08T05:19:17.494 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:Invalid command: unused arguments: ['mirror/image_37'] 2023-11-08T05:19:17.495 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd mirror status : get status for rbd mirror 2023-11-08T05:19:17.495 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:admin_socket: invalid command 2023-11-08T05:19:17.498 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T05:19:17.498 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_37 2023-11-08T05:19:17.499 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1:1 2023-11-08T05:19:17.499 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T05:19:17.499 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance 2023-11-08T05:19:17.499 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-11-08T05:19:17.500 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local val=cluster1:1 2023-11-08T05:19:17.500 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_var_name=cluster 2023-11-08T05:19:17.500 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance_var_name=instance 2023-11-08T05:19:17.500 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cluster=cluster1 2023-11-08T05:19:17.500 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ instance=1 2023-11-08T05:19:17.501 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' 1 = cluster1:1 ']' 2023-11-08T05:19:17.501 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval cluster=cluster1 2023-11-08T05:19:17.501 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:19:17.501 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval instance=1 2023-11-08T05:19:17.502 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=1 2023-11-08T05:19:17.502 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-11-08T05:19:17.502 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local local_cluster=cluster1:1 2023-11-08T05:19:17.502 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster=cluster1 2023-11-08T05:19:17.502 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance 2023-11-08T05:19:17.503 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-11-08T05:19:17.503 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local val=cluster1:1 2023-11-08T05:19:17.503 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster_var_name=local_cluster 2023-11-08T05:19:17.503 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance_var_name=instance 2023-11-08T05:19:17.503 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:19:17.504 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=1 2023-11-08T05:19:17.504 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ '[' 1 = cluster1:1 ']' 2023-11-08T05:19:17.504 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval local_cluster=cluster1 2023-11-08T05:19:17.504 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ local_cluster=cluster1 2023-11-08T05:19:17.504 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval instance=1 2023-11-08T05:19:17.505 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ instance=1 2023-11-08T05:19:17.505 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-11-08T05:19:17.512 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-08T05:19:17.512 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-08T05:19:17.513 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-08T05:19:17.513 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_37 2023-11-08T05:19:17.613 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:Invalid command: unused arguments: ['mirror/image_37'] 2023-11-08T05:19:17.614 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd mirror status : get status for rbd mirror 2023-11-08T05:19:17.614 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:admin_socket: invalid command 2023-11-08T05:19:17.617 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T05:19:17.618 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_37 2023-11-08T05:19:17.618 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1:2 2023-11-08T05:19:17.618 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T05:19:17.618 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance 2023-11-08T05:19:17.619 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-11-08T05:19:17.619 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local val=cluster1:2 2023-11-08T05:19:17.619 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_var_name=cluster 2023-11-08T05:19:17.619 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance_var_name=instance 2023-11-08T05:19:17.620 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cluster=cluster1 2023-11-08T05:19:17.620 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ instance=2 2023-11-08T05:19:17.620 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' 2 = cluster1:2 ']' 2023-11-08T05:19:17.620 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval cluster=cluster1 2023-11-08T05:19:17.621 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:19:17.621 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval instance=2 2023-11-08T05:19:17.621 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=2 2023-11-08T05:19:17.621 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-11-08T05:19:17.621 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local local_cluster=cluster1:2 2023-11-08T05:19:17.622 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster=cluster1 2023-11-08T05:19:17.622 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance 2023-11-08T05:19:17.622 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-11-08T05:19:17.622 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local val=cluster1:2 2023-11-08T05:19:17.623 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster_var_name=local_cluster 2023-11-08T05:19:17.623 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance_var_name=instance 2023-11-08T05:19:17.623 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:19:17.623 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=2 2023-11-08T05:19:17.624 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ '[' 2 = cluster1:2 ']' 2023-11-08T05:19:17.624 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval local_cluster=cluster1 2023-11-08T05:19:17.624 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ local_cluster=cluster1 2023-11-08T05:19:17.624 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval instance=2 2023-11-08T05:19:17.624 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ instance=2 2023-11-08T05:19:17.625 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-11-08T05:19:17.632 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-08T05:19:17.632 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-08T05:19:17.632 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-08T05:19:17.633 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_37 2023-11-08T05:19:17.739 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ return 0 2023-11-08T05:19:17.739 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ status_result=' "state": "Replaying"' 2023-11-08T05:19:17.740 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ echo ' "state": "Replaying"' 2023-11-08T05:19:17.740 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ grep -i Replaying 2023-11-08T05:19:17.740 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: "state": "Replaying" 2023-11-08T05:19:17.742 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ current_state=started 2023-11-08T05:19:17.743 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test started = started 2023-11-08T05:19:17.743 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:19:17.743 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_37 2023-11-08T05:19:17.743 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local local_cluster=cluster1 2023-11-08T05:19:17.743 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:19:17.744 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:19:17.744 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_37 2023-11-08T05:19:17.744 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' snapshot = journal ']' 2023-11-08T05:19:17.744 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' snapshot = snapshot ']' 2023-11-08T05:19:17.744 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_37 2023-11-08T05:19:17.745 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local local_cluster=cluster1 2023-11-08T05:19:17.745 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:19:17.745 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:19:17.745 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_37 2023-11-08T05:19:17.745 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ mkfname cluster2-mirror-image_37.status 2023-11-08T05:19:17.746 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ echo cluster2-mirror-image_37.status 2023-11-08T05:19:17.746 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -e 's|[/ ]|_|g' 2023-11-08T05:19:17.746 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local status_log=/tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_37.status 2023-11-08T05:19:17.746 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ mkfname cluster1-mirror-image_37.status 2023-11-08T05:19:17.746 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ echo cluster1-mirror-image_37.status 2023-11-08T05:19:17.746 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -e 's|[/ ]|_|g' 2023-11-08T05:19:17.748 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local local_status_log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_37.status 2023-11-08T05:19:17.748 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ mirror_image_snapshot cluster2 mirror image_37 2023-11-08T05:19:17.748 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:19:17.748 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:19:17.749 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_37 2023-11-08T05:19:17.749 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_37 2023-11-08T05:19:18.374 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:Snapshot ID: 191 2023-11-08T05:19:18.383 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_37 /tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_37.status 2023-11-08T05:19:18.383 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:19:18.384 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:19:18.384 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_37 2023-11-08T05:19:18.384 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_37.status 2023-11-08T05:19:18.384 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_37 --format xml 2023-11-08T05:19:18.384 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T05:19:18.420 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-11-08T05:19:18.430 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snapshot_id=191 2023-11-08T05:19:18.431 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ true 2023-11-08T05:19:18.431 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-11-08T05:19:18.431 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0.2 2023-11-08T05:19:18.623 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_37 /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_37.status 2023-11-08T05:19:18.624 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:19:18.624 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:19:18.624 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_37 2023-11-08T05:19:18.624 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_37.status 2023-11-08T05:19:18.625 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_37 --format xml 2023-11-08T05:19:18.625 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T05:19:18.664 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ true 2023-11-08T05:19:18.664 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-08T05:19:18.665 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:-:1.1: Document is empty 2023-11-08T05:19:18.665 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: 2023-11-08T05:19:18.666 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:^ 2023-11-08T05:19:18.666 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local primary_snapshot_id= 2023-11-08T05:19:18.666 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test 191 = '' 2023-11-08T05:19:18.666 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-11-08T05:19:18.666 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0.4 2023-11-08T05:19:19.066 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_37 /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_37.status 2023-11-08T05:19:19.067 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:19:19.067 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:19:19.067 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_37 2023-11-08T05:19:19.067 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_37.status 2023-11-08T05:19:19.067 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_37 --format xml 2023-11-08T05:19:19.068 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T05:19:19.107 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ true 2023-11-08T05:19:19.108 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-08T05:19:19.109 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:-:1.1: Document is empty 2023-11-08T05:19:19.109 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: 2023-11-08T05:19:19.109 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:^ 2023-11-08T05:19:19.110 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local primary_snapshot_id= 2023-11-08T05:19:19.110 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test 191 = '' 2023-11-08T05:19:19.110 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-11-08T05:19:19.110 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0.8 2023-11-08T05:19:19.911 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_37 /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_37.status 2023-11-08T05:19:19.911 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:19:19.912 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:19:19.912 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_37 2023-11-08T05:19:19.912 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_37.status 2023-11-08T05:19:19.912 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_37 --format xml 2023-11-08T05:19:19.912 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T05:19:19.951 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-08T05:19:19.953 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local primary_snapshot_id=191 2023-11-08T05:19:19.953 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test 191 = 191 2023-11-08T05:19:19.953 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:19:19.953 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_snap_present cluster1 mirror image_37 snap 2023-11-08T05:19:19.954 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:19:19.954 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:19:19.954 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_37 2023-11-08T05:19:19.954 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap_name=snap 2023-11-08T05:19:19.954 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local s 2023-11-08T05:19:19.955 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-11-08T05:19:19.955 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 1 2023-11-08T05:19:20.954 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 info mirror/image_37@snap 2023-11-08T05:19:20.993 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:rbd image 'image_37': 2023-11-08T05:19:20.994 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: size 128 MiB in 32 objects 2023-11-08T05:19:20.994 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: order 22 (4 MiB objects) 2023-11-08T05:19:20.994 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: snapshot_count: 2 2023-11-08T05:19:20.994 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: id: 12b3726cfd43 2023-11-08T05:19:20.994 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: block_name_prefix: rbd_data.12b3726cfd43 2023-11-08T05:19:20.995 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: format: 2 2023-11-08T05:19:20.995 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: features: layering, exclusive-lock, non-primary 2023-11-08T05:19:20.995 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: op_features: 2023-11-08T05:19:20.995 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: flags: 2023-11-08T05:19:20.995 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: create_timestamp: Wed Nov 8 05:10:54 2023 2023-11-08T05:19:20.996 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: access_timestamp: Wed Nov 8 05:10:54 2023 2023-11-08T05:19:20.996 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: modify_timestamp: Wed Nov 8 05:10:54 2023 2023-11-08T05:19:20.996 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: protected: False 2023-11-08T05:19:20.996 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring state: enabled 2023-11-08T05:19:20.996 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring mode: snapshot 2023-11-08T05:19:20.997 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring global id: 7c36d62d-ec90-4bfc-b81d-9054fbd92513 2023-11-08T05:19:20.997 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring primary: false 2023-11-08T05:19:20.998 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:19:20.998 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ compare_image_snaps mirror image_37 snap 2023-11-08T05:19:20.998 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:19:20.999 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_37 2023-11-08T05:19:20.999 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap_name=snap 2023-11-08T05:19:20.999 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local ret=0 2023-11-08T05:19:20.999 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local rmt_export=/tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_37.export 2023-11-08T05:19:20.999 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local loc_export=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_37.export 2023-11-08T05:19:21.000 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rm -f /tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_37.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_37.export 2023-11-08T05:19:21.000 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 -p mirror export image_37@snap /tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_37.export 2023-11-08T05:19:21.175 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-11-08T05:19:21.178 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 -p mirror export image_37@snap /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_37.export 2023-11-08T05:19:21.364 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-11-08T05:19:21.367 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ cmp /tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_37.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_37.export 2023-11-08T05:19:21.422 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rm -f /tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_37.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_37.export 2023-11-08T05:19:21.458 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:19:21.459 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-11-08T05:19:21.459 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image=image_38 2023-11-08T05:19:21.459 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ create_snap cluster2 mirror image_38 snap 2023-11-08T05:19:21.459 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:19:21.460 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:19:21.460 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_38 2023-11-08T05:19:21.460 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap_name=snap 2023-11-08T05:19:21.460 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 -p mirror snap create image_38@snap --debug-rbd=20 --debug-journaler=20 2023-11-08T05:19:23.376 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_image_replay_started cluster1 mirror image_38 2023-11-08T05:19:23.376 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:19:23.376 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:19:23.377 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_38 2023-11-08T05:19:23.377 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_image_replay_state cluster1 mirror image_38 started 2023-11-08T05:19:23.377 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:19:23.377 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:19:23.377 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_38 2023-11-08T05:19:23.378 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local state=started 2023-11-08T05:19:23.378 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local s 2023-11-08T05:19:23.378 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-11-08T05:19:23.378 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 1 2023-11-08T05:19:24.378 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test_image_replay_state cluster1 mirror image_38 started 2023-11-08T05:19:24.378 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:19:24.378 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:19:24.379 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_38 2023-11-08T05:19:24.379 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local test_state=started 2023-11-08T05:19:24.379 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local status_result 2023-11-08T05:19:24.379 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local current_state=stopped 2023-11-08T05:19:24.379 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_38 2023-11-08T05:19:24.380 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_instance=cluster1 2023-11-08T05:19:24.380 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T05:19:24.380 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1 2023-11-08T05:19:24.380 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance=cluster1 2023-11-08T05:19:24.380 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ grep -i state 2023-11-08T05:19:24.380 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local loop_instance 2023-11-08T05:19:24.381 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-11-08T05:19:24.381 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sleep 0 2023-11-08T05:19:24.383 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-11-08T05:19:24.383 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ seq 0 3 2023-11-08T05:19:24.384 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T05:19:24.384 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_38 2023-11-08T05:19:24.384 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1:0 2023-11-08T05:19:24.384 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T05:19:24.384 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance 2023-11-08T05:19:24.385 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-11-08T05:19:24.385 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local val=cluster1:0 2023-11-08T05:19:24.385 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_var_name=cluster 2023-11-08T05:19:24.385 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance_var_name=instance 2023-11-08T05:19:24.385 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cluster=cluster1 2023-11-08T05:19:24.386 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ instance=0 2023-11-08T05:19:24.386 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' 0 = cluster1:0 ']' 2023-11-08T05:19:24.386 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval cluster=cluster1 2023-11-08T05:19:24.386 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:19:24.386 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval instance=0 2023-11-08T05:19:24.387 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=0 2023-11-08T05:19:24.387 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-11-08T05:19:24.388 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local local_cluster=cluster1:0 2023-11-08T05:19:24.388 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster=cluster1 2023-11-08T05:19:24.389 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance 2023-11-08T05:19:24.389 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-11-08T05:19:24.389 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local val=cluster1:0 2023-11-08T05:19:24.389 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster_var_name=local_cluster 2023-11-08T05:19:24.389 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance_var_name=instance 2023-11-08T05:19:24.390 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:19:24.390 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=0 2023-11-08T05:19:24.390 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ '[' 0 = cluster1:0 ']' 2023-11-08T05:19:24.390 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval local_cluster=cluster1 2023-11-08T05:19:24.390 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ local_cluster=cluster1 2023-11-08T05:19:24.391 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval instance=0 2023-11-08T05:19:24.391 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ instance=0 2023-11-08T05:19:24.391 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-11-08T05:19:24.399 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-08T05:19:24.399 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-08T05:19:24.400 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-08T05:19:24.400 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_38 2023-11-08T05:19:24.505 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:Invalid command: unused arguments: ['mirror/image_38'] 2023-11-08T05:19:24.506 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd mirror status : get status for rbd mirror 2023-11-08T05:19:24.506 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:admin_socket: invalid command 2023-11-08T05:19:24.508 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T05:19:24.509 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_38 2023-11-08T05:19:24.509 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1:1 2023-11-08T05:19:24.509 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T05:19:24.509 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance 2023-11-08T05:19:24.509 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-11-08T05:19:24.510 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local val=cluster1:1 2023-11-08T05:19:24.510 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_var_name=cluster 2023-11-08T05:19:24.510 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance_var_name=instance 2023-11-08T05:19:24.510 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cluster=cluster1 2023-11-08T05:19:24.510 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ instance=1 2023-11-08T05:19:24.511 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' 1 = cluster1:1 ']' 2023-11-08T05:19:24.511 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval cluster=cluster1 2023-11-08T05:19:24.511 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:19:24.511 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval instance=1 2023-11-08T05:19:24.511 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=1 2023-11-08T05:19:24.512 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-11-08T05:19:24.512 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local local_cluster=cluster1:1 2023-11-08T05:19:24.512 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster=cluster1 2023-11-08T05:19:24.512 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance 2023-11-08T05:19:24.512 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-11-08T05:19:24.513 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local val=cluster1:1 2023-11-08T05:19:24.513 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster_var_name=local_cluster 2023-11-08T05:19:24.513 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance_var_name=instance 2023-11-08T05:19:24.513 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:19:24.513 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=1 2023-11-08T05:19:24.514 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ '[' 1 = cluster1:1 ']' 2023-11-08T05:19:24.514 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval local_cluster=cluster1 2023-11-08T05:19:24.514 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ local_cluster=cluster1 2023-11-08T05:19:24.514 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval instance=1 2023-11-08T05:19:24.514 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ instance=1 2023-11-08T05:19:24.515 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-11-08T05:19:24.523 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-08T05:19:24.523 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-08T05:19:24.524 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-08T05:19:24.524 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_38 2023-11-08T05:19:24.631 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ return 0 2023-11-08T05:19:24.632 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ status_result=' "state": "Replaying"' 2023-11-08T05:19:24.633 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ echo ' "state": "Replaying"' 2023-11-08T05:19:24.633 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ grep -i Replaying 2023-11-08T05:19:24.633 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: "state": "Replaying" 2023-11-08T05:19:24.639 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ current_state=started 2023-11-08T05:19:24.640 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test started = started 2023-11-08T05:19:24.640 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:19:24.641 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_38 2023-11-08T05:19:24.641 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local local_cluster=cluster1 2023-11-08T05:19:24.641 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:19:24.642 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:19:24.642 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_38 2023-11-08T05:19:24.642 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' snapshot = journal ']' 2023-11-08T05:19:24.643 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' snapshot = snapshot ']' 2023-11-08T05:19:24.643 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_38 2023-11-08T05:19:24.643 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local local_cluster=cluster1 2023-11-08T05:19:24.643 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:19:24.644 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:19:24.644 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_38 2023-11-08T05:19:24.644 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ mkfname cluster2-mirror-image_38.status 2023-11-08T05:19:24.644 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ echo cluster2-mirror-image_38.status 2023-11-08T05:19:24.645 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -e 's|[/ ]|_|g' 2023-11-08T05:19:24.645 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local status_log=/tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_38.status 2023-11-08T05:19:24.645 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ mkfname cluster1-mirror-image_38.status 2023-11-08T05:19:24.646 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ echo cluster1-mirror-image_38.status 2023-11-08T05:19:24.646 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -e 's|[/ ]|_|g' 2023-11-08T05:19:24.646 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local local_status_log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_38.status 2023-11-08T05:19:24.646 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ mirror_image_snapshot cluster2 mirror image_38 2023-11-08T05:19:24.647 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:19:24.647 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:19:24.647 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_38 2023-11-08T05:19:24.648 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_38 2023-11-08T05:19:25.382 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:Snapshot ID: 193 2023-11-08T05:19:25.392 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_38 /tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_38.status 2023-11-08T05:19:25.392 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:19:25.392 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:19:25.393 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_38 2023-11-08T05:19:25.393 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_38.status 2023-11-08T05:19:25.393 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_38 --format xml 2023-11-08T05:19:25.393 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T05:19:25.430 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-11-08T05:19:25.432 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snapshot_id=193 2023-11-08T05:19:25.432 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ true 2023-11-08T05:19:25.432 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-11-08T05:19:25.432 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0.2 2023-11-08T05:19:25.633 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_38 /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_38.status 2023-11-08T05:19:25.634 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:19:25.634 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:19:25.634 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_38 2023-11-08T05:19:25.634 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_38.status 2023-11-08T05:19:25.635 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_38 --format xml 2023-11-08T05:19:25.635 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T05:19:25.673 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ true 2023-11-08T05:19:25.674 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-08T05:19:25.675 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:-:1.1: Document is empty 2023-11-08T05:19:25.675 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: 2023-11-08T05:19:25.675 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:^ 2023-11-08T05:19:25.676 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local primary_snapshot_id= 2023-11-08T05:19:25.676 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test 193 = '' 2023-11-08T05:19:25.676 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-11-08T05:19:25.676 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0.4 2023-11-08T05:19:26.076 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_38 /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_38.status 2023-11-08T05:19:26.077 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:19:26.077 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:19:26.077 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_38 2023-11-08T05:19:26.077 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_38.status 2023-11-08T05:19:26.078 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_38 --format xml 2023-11-08T05:19:26.078 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T05:19:26.117 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ true 2023-11-08T05:19:26.117 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-08T05:19:26.118 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:-:1.1: Document is empty 2023-11-08T05:19:26.118 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: 2023-11-08T05:19:26.119 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:^ 2023-11-08T05:19:26.120 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local primary_snapshot_id= 2023-11-08T05:19:26.120 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test 193 = '' 2023-11-08T05:19:26.120 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-11-08T05:19:26.120 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0.8 2023-11-08T05:19:26.920 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_38 /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_38.status 2023-11-08T05:19:26.920 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:19:26.921 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:19:26.921 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_38 2023-11-08T05:19:26.921 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_38.status 2023-11-08T05:19:26.921 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_38 --format xml 2023-11-08T05:19:26.921 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T05:19:26.979 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ true 2023-11-08T05:19:26.979 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-08T05:19:26.980 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:-:1.1: Document is empty 2023-11-08T05:19:26.981 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: 2023-11-08T05:19:26.981 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:^ 2023-11-08T05:19:26.982 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local primary_snapshot_id= 2023-11-08T05:19:26.982 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test 193 = '' 2023-11-08T05:19:26.982 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-11-08T05:19:26.982 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 1.6 2023-11-08T05:19:27.216 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.42033851032001635) > weight (0.25) 2023-11-08T05:19:27.216 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.6557157650216187) > weight (0.25) 2023-11-08T05:19:27.217 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.901255139830699) > weight (0.25) 2023-11-08T05:19:27.217 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.3 2023-11-08T05:19:27.217 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Sent signal 15 2023-11-08T05:19:27.218 INFO:tasks.rbd_mirror_thrash:waiting for 1 secs before reviving daemons 2023-11-08T05:19:27.218 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi087.stderr:2023-11-08T05:19:27.217+0000 7fe44d444700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.3 (PID: 143267) UID: 1000 2023-11-08T05:19:27.419 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi087.stderr:daemon-helper: command failed with exit status 1 2023-11-08T05:19:28.219 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.3 2023-11-08T05:19:28.219 INFO:teuthology.orchestra.run:waiting for 600 2023-11-08T05:19:28.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-08T05:19:28.220 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.3 2023-11-08T05:19:28.220 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Restarting daemon 2023-11-08T05:19:28.221 DEBUG:teuthology.orchestra.run.smithi087:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.3 2023-11-08T05:19:28.223 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Started 2023-11-08T05:19:28.224 INFO:tasks.rbd_mirror_thrash:waiting for 108 secs before thrashing 2023-11-08T05:19:28.582 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_38 /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_38.status 2023-11-08T05:19:28.583 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:19:28.583 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:19:28.583 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_38 2023-11-08T05:19:28.583 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_38.status 2023-11-08T05:19:28.584 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_38 --format xml 2023-11-08T05:19:28.584 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T05:19:28.622 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-08T05:19:28.624 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local primary_snapshot_id=193 2023-11-08T05:19:28.624 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test 193 = 193 2023-11-08T05:19:28.624 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:19:28.624 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_snap_present cluster1 mirror image_38 snap 2023-11-08T05:19:28.624 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:19:28.625 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:19:28.625 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_38 2023-11-08T05:19:28.625 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap_name=snap 2023-11-08T05:19:28.625 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local s 2023-11-08T05:19:28.625 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-11-08T05:19:28.626 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 1 2023-11-08T05:19:29.625 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 info mirror/image_38@snap 2023-11-08T05:19:29.663 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:rbd image 'image_38': 2023-11-08T05:19:29.664 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: size 128 MiB in 32 objects 2023-11-08T05:19:29.664 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: order 22 (4 MiB objects) 2023-11-08T05:19:29.664 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: snapshot_count: 2 2023-11-08T05:19:29.664 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: id: 1287f2b4c505 2023-11-08T05:19:29.665 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: block_name_prefix: rbd_data.1287f2b4c505 2023-11-08T05:19:29.665 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: format: 2 2023-11-08T05:19:29.665 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: features: layering, exclusive-lock, non-primary 2023-11-08T05:19:29.665 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: op_features: 2023-11-08T05:19:29.665 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: flags: 2023-11-08T05:19:29.666 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: create_timestamp: Wed Nov 8 05:11:03 2023 2023-11-08T05:19:29.666 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: access_timestamp: Wed Nov 8 05:11:03 2023 2023-11-08T05:19:29.666 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: modify_timestamp: Wed Nov 8 05:11:03 2023 2023-11-08T05:19:29.666 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: protected: False 2023-11-08T05:19:29.666 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring state: enabled 2023-11-08T05:19:29.667 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring mode: snapshot 2023-11-08T05:19:29.667 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring global id: c3a4ad2f-2430-4db4-a78e-1b1f4255422f 2023-11-08T05:19:29.667 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring primary: false 2023-11-08T05:19:29.668 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:19:29.668 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ compare_image_snaps mirror image_38 snap 2023-11-08T05:19:29.669 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:19:29.669 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_38 2023-11-08T05:19:29.669 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap_name=snap 2023-11-08T05:19:29.669 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local ret=0 2023-11-08T05:19:29.669 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local rmt_export=/tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_38.export 2023-11-08T05:19:29.670 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local loc_export=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_38.export 2023-11-08T05:19:29.670 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rm -f /tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_38.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_38.export 2023-11-08T05:19:29.670 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 -p mirror export image_38@snap /tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_38.export 2023-11-08T05:19:29.851 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-11-08T05:19:29.854 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 -p mirror export image_38@snap /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_38.export 2023-11-08T05:19:30.051 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-11-08T05:19:30.054 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ cmp /tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_38.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_38.export 2023-11-08T05:19:30.109 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rm -f /tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_38.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_38.export 2023-11-08T05:19:30.146 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:19:30.146 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-11-08T05:19:30.146 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image=image_39 2023-11-08T05:19:30.147 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ create_snap cluster2 mirror image_39 snap 2023-11-08T05:19:30.147 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:19:30.147 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:19:30.147 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_39 2023-11-08T05:19:30.147 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap_name=snap 2023-11-08T05:19:30.148 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 -p mirror snap create image_39@snap --debug-rbd=20 --debug-journaler=20 2023-11-08T05:19:31.396 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_image_replay_started cluster1 mirror image_39 2023-11-08T05:19:31.397 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:19:31.397 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:19:31.397 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_39 2023-11-08T05:19:31.397 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_image_replay_state cluster1 mirror image_39 started 2023-11-08T05:19:31.398 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:19:31.398 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:19:31.398 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_39 2023-11-08T05:19:31.398 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local state=started 2023-11-08T05:19:31.399 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local s 2023-11-08T05:19:31.399 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-11-08T05:19:31.399 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 1 2023-11-08T05:19:32.398 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test_image_replay_state cluster1 mirror image_39 started 2023-11-08T05:19:32.398 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:19:32.398 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:19:32.398 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_39 2023-11-08T05:19:32.399 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local test_state=started 2023-11-08T05:19:32.399 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local status_result 2023-11-08T05:19:32.399 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local current_state=stopped 2023-11-08T05:19:32.400 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_39 2023-11-08T05:19:32.400 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_instance=cluster1 2023-11-08T05:19:32.400 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T05:19:32.401 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1 2023-11-08T05:19:32.401 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ grep -i state 2023-11-08T05:19:32.401 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance=cluster1 2023-11-08T05:19:32.401 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local loop_instance 2023-11-08T05:19:32.401 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-11-08T05:19:32.402 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sleep 0 2023-11-08T05:19:32.402 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-11-08T05:19:32.402 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ seq 0 3 2023-11-08T05:19:32.404 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T05:19:32.404 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_39 2023-11-08T05:19:32.404 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1:0 2023-11-08T05:19:32.405 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T05:19:32.405 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance 2023-11-08T05:19:32.405 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-11-08T05:19:32.405 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local val=cluster1:0 2023-11-08T05:19:32.405 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_var_name=cluster 2023-11-08T05:19:32.406 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance_var_name=instance 2023-11-08T05:19:32.406 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cluster=cluster1 2023-11-08T05:19:32.406 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ instance=0 2023-11-08T05:19:32.406 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' 0 = cluster1:0 ']' 2023-11-08T05:19:32.406 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval cluster=cluster1 2023-11-08T05:19:32.407 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:19:32.407 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval instance=0 2023-11-08T05:19:32.407 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=0 2023-11-08T05:19:32.407 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-11-08T05:19:32.407 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local local_cluster=cluster1:0 2023-11-08T05:19:32.408 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster=cluster1 2023-11-08T05:19:32.408 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance 2023-11-08T05:19:32.408 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-11-08T05:19:32.408 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local val=cluster1:0 2023-11-08T05:19:32.408 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster_var_name=local_cluster 2023-11-08T05:19:32.409 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance_var_name=instance 2023-11-08T05:19:32.410 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:19:32.410 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=0 2023-11-08T05:19:32.410 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ '[' 0 = cluster1:0 ']' 2023-11-08T05:19:32.410 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval local_cluster=cluster1 2023-11-08T05:19:32.410 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ local_cluster=cluster1 2023-11-08T05:19:32.413 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval instance=0 2023-11-08T05:19:32.413 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ instance=0 2023-11-08T05:19:32.413 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-11-08T05:19:32.420 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-08T05:19:32.420 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-08T05:19:32.420 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-08T05:19:32.421 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_39 2023-11-08T05:19:32.537 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ return 0 2023-11-08T05:19:32.538 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ status_result=' "state": "Replaying"' 2023-11-08T05:19:32.538 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ echo ' "state": "Replaying"' 2023-11-08T05:19:32.538 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ grep -i Replaying 2023-11-08T05:19:32.539 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: "state": "Replaying" 2023-11-08T05:19:32.541 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ current_state=started 2023-11-08T05:19:32.541 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test started = started 2023-11-08T05:19:32.541 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:19:32.541 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_39 2023-11-08T05:19:32.542 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local local_cluster=cluster1 2023-11-08T05:19:32.542 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:19:32.542 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:19:32.542 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_39 2023-11-08T05:19:32.542 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' snapshot = journal ']' 2023-11-08T05:19:32.543 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' snapshot = snapshot ']' 2023-11-08T05:19:32.543 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_39 2023-11-08T05:19:32.543 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local local_cluster=cluster1 2023-11-08T05:19:32.543 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:19:32.544 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:19:32.544 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_39 2023-11-08T05:19:32.544 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ mkfname cluster2-mirror-image_39.status 2023-11-08T05:19:32.544 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ echo cluster2-mirror-image_39.status 2023-11-08T05:19:32.545 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -e 's|[/ ]|_|g' 2023-11-08T05:19:32.545 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local status_log=/tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_39.status 2023-11-08T05:19:32.545 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ mkfname cluster1-mirror-image_39.status 2023-11-08T05:19:32.545 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ echo cluster1-mirror-image_39.status 2023-11-08T05:19:32.545 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -e 's|[/ ]|_|g' 2023-11-08T05:19:32.547 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local local_status_log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_39.status 2023-11-08T05:19:32.547 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ mirror_image_snapshot cluster2 mirror image_39 2023-11-08T05:19:32.547 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:19:32.548 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:19:32.548 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_39 2023-11-08T05:19:32.548 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_39 2023-11-08T05:19:33.409 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:Snapshot ID: 195 2023-11-08T05:19:33.419 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_39 /tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_39.status 2023-11-08T05:19:33.420 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:19:33.420 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:19:33.420 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_39 2023-11-08T05:19:33.420 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_39.status 2023-11-08T05:19:33.420 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_39 --format xml 2023-11-08T05:19:33.421 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T05:19:33.458 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-11-08T05:19:33.460 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snapshot_id=195 2023-11-08T05:19:33.461 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ true 2023-11-08T05:19:33.461 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-11-08T05:19:33.461 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0.2 2023-11-08T05:19:33.662 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_39 /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_39.status 2023-11-08T05:19:33.662 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:19:33.662 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:19:33.662 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_39 2023-11-08T05:19:33.663 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_39.status 2023-11-08T05:19:33.663 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_39 --format xml 2023-11-08T05:19:33.663 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T05:19:33.703 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ true 2023-11-08T05:19:33.703 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-08T05:19:33.704 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:-:1.1: Document is empty 2023-11-08T05:19:33.704 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: 2023-11-08T05:19:33.704 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:^ 2023-11-08T05:19:33.705 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local primary_snapshot_id= 2023-11-08T05:19:33.705 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test 195 = '' 2023-11-08T05:19:33.705 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-11-08T05:19:33.706 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0.4 2023-11-08T05:19:34.106 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_39 /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_39.status 2023-11-08T05:19:34.106 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:19:34.107 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:19:34.107 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_39 2023-11-08T05:19:34.107 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_39.status 2023-11-08T05:19:34.108 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_39 --format xml 2023-11-08T05:19:34.108 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T05:19:34.147 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ true 2023-11-08T05:19:34.148 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-08T05:19:34.149 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:-:1.1: Document is empty 2023-11-08T05:19:34.149 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: 2023-11-08T05:19:34.149 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:^ 2023-11-08T05:19:34.150 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local primary_snapshot_id= 2023-11-08T05:19:34.150 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test 195 = '' 2023-11-08T05:19:34.150 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-11-08T05:19:34.150 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0.8 2023-11-08T05:19:34.950 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_39 /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_39.status 2023-11-08T05:19:34.950 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:19:34.951 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:19:34.951 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_39 2023-11-08T05:19:34.951 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_39.status 2023-11-08T05:19:34.951 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_39 --format xml 2023-11-08T05:19:34.951 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T05:19:34.991 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-08T05:19:34.993 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local primary_snapshot_id=195 2023-11-08T05:19:34.994 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test 195 = 195 2023-11-08T05:19:34.994 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:19:34.994 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_snap_present cluster1 mirror image_39 snap 2023-11-08T05:19:34.994 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:19:34.994 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:19:34.995 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_39 2023-11-08T05:19:34.995 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap_name=snap 2023-11-08T05:19:34.995 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local s 2023-11-08T05:19:34.995 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-11-08T05:19:34.995 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 1 2023-11-08T05:19:35.994 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 info mirror/image_39@snap 2023-11-08T05:19:36.031 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:rbd image 'image_39': 2023-11-08T05:19:36.032 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: size 128 MiB in 32 objects 2023-11-08T05:19:36.032 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: order 22 (4 MiB objects) 2023-11-08T05:19:36.032 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: snapshot_count: 2 2023-11-08T05:19:36.032 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: id: 12a956d88f8d 2023-11-08T05:19:36.032 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: block_name_prefix: rbd_data.12a956d88f8d 2023-11-08T05:19:36.033 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: format: 2 2023-11-08T05:19:36.033 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: features: layering, exclusive-lock, non-primary 2023-11-08T05:19:36.034 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: op_features: 2023-11-08T05:19:36.034 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: flags: 2023-11-08T05:19:36.034 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: create_timestamp: Wed Nov 8 05:11:13 2023 2023-11-08T05:19:36.034 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: access_timestamp: Wed Nov 8 05:11:13 2023 2023-11-08T05:19:36.035 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: modify_timestamp: Wed Nov 8 05:11:13 2023 2023-11-08T05:19:36.035 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: protected: False 2023-11-08T05:19:36.035 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring state: enabled 2023-11-08T05:19:36.035 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring mode: snapshot 2023-11-08T05:19:36.035 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring global id: a4b03d0b-f737-4c8f-8a6c-6971fcf77e8a 2023-11-08T05:19:36.036 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring primary: false 2023-11-08T05:19:36.037 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:19:36.037 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ compare_image_snaps mirror image_39 snap 2023-11-08T05:19:36.037 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:19:36.038 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_39 2023-11-08T05:19:36.038 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap_name=snap 2023-11-08T05:19:36.038 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local ret=0 2023-11-08T05:19:36.038 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local rmt_export=/tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_39.export 2023-11-08T05:19:36.038 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local loc_export=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_39.export 2023-11-08T05:19:36.039 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rm -f /tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_39.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_39.export 2023-11-08T05:19:36.039 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 -p mirror export image_39@snap /tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_39.export 2023-11-08T05:19:36.213 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-11-08T05:19:36.217 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 -p mirror export image_39@snap /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_39.export 2023-11-08T05:19:36.431 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-11-08T05:19:36.434 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ cmp /tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_39.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_39.export 2023-11-08T05:19:36.490 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rm -f /tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_39.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_39.export 2023-11-08T05:19:36.527 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:19:36.527 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-11-08T05:19:36.528 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image=image_40 2023-11-08T05:19:36.528 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ create_snap cluster2 mirror image_40 snap 2023-11-08T05:19:36.528 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:19:36.528 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:19:36.528 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_40 2023-11-08T05:19:36.529 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap_name=snap 2023-11-08T05:19:36.529 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 -p mirror snap create image_40@snap --debug-rbd=20 --debug-journaler=20 2023-11-08T05:19:38.421 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_image_replay_started cluster1 mirror image_40 2023-11-08T05:19:38.422 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:19:38.422 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:19:38.422 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_40 2023-11-08T05:19:38.422 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_image_replay_state cluster1 mirror image_40 started 2023-11-08T05:19:38.422 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:19:38.423 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:19:38.423 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_40 2023-11-08T05:19:38.423 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local state=started 2023-11-08T05:19:38.423 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local s 2023-11-08T05:19:38.423 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-11-08T05:19:38.424 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 1 2023-11-08T05:19:39.423 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test_image_replay_state cluster1 mirror image_40 started 2023-11-08T05:19:39.423 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:19:39.423 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:19:39.423 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_40 2023-11-08T05:19:39.424 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local test_state=started 2023-11-08T05:19:39.424 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local status_result 2023-11-08T05:19:39.424 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local current_state=stopped 2023-11-08T05:19:39.425 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_40 2023-11-08T05:19:39.425 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_instance=cluster1 2023-11-08T05:19:39.426 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T05:19:39.426 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ grep -i state 2023-11-08T05:19:39.426 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1 2023-11-08T05:19:39.426 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance=cluster1 2023-11-08T05:19:39.426 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local loop_instance 2023-11-08T05:19:39.427 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-11-08T05:19:39.427 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sleep 0 2023-11-08T05:19:39.427 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-11-08T05:19:39.427 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ seq 0 3 2023-11-08T05:19:39.430 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T05:19:39.431 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_40 2023-11-08T05:19:39.431 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1:0 2023-11-08T05:19:39.431 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T05:19:39.431 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance 2023-11-08T05:19:39.431 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-11-08T05:19:39.432 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local val=cluster1:0 2023-11-08T05:19:39.432 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_var_name=cluster 2023-11-08T05:19:39.432 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance_var_name=instance 2023-11-08T05:19:39.432 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cluster=cluster1 2023-11-08T05:19:39.432 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ instance=0 2023-11-08T05:19:39.433 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' 0 = cluster1:0 ']' 2023-11-08T05:19:39.433 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval cluster=cluster1 2023-11-08T05:19:39.433 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:19:39.433 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval instance=0 2023-11-08T05:19:39.433 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=0 2023-11-08T05:19:39.434 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-11-08T05:19:39.434 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local local_cluster=cluster1:0 2023-11-08T05:19:39.434 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster=cluster1 2023-11-08T05:19:39.434 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance 2023-11-08T05:19:39.434 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-11-08T05:19:39.435 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local val=cluster1:0 2023-11-08T05:19:39.435 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster_var_name=local_cluster 2023-11-08T05:19:39.435 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance_var_name=instance 2023-11-08T05:19:39.436 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:19:39.436 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=0 2023-11-08T05:19:39.436 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ '[' 0 = cluster1:0 ']' 2023-11-08T05:19:39.436 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval local_cluster=cluster1 2023-11-08T05:19:39.436 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ local_cluster=cluster1 2023-11-08T05:19:39.437 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval instance=0 2023-11-08T05:19:39.437 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ instance=0 2023-11-08T05:19:39.437 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-11-08T05:19:39.444 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-08T05:19:39.445 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-08T05:19:39.445 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-08T05:19:39.445 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_40 2023-11-08T05:19:39.552 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:Invalid command: unused arguments: ['mirror/image_40'] 2023-11-08T05:19:39.552 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd mirror status : get status for rbd mirror 2023-11-08T05:19:39.553 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:admin_socket: invalid command 2023-11-08T05:19:39.555 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T05:19:39.556 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_40 2023-11-08T05:19:39.556 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1:1 2023-11-08T05:19:39.556 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T05:19:39.556 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance 2023-11-08T05:19:39.556 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-11-08T05:19:39.557 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local val=cluster1:1 2023-11-08T05:19:39.557 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_var_name=cluster 2023-11-08T05:19:39.557 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance_var_name=instance 2023-11-08T05:19:39.557 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cluster=cluster1 2023-11-08T05:19:39.557 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ instance=1 2023-11-08T05:19:39.558 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' 1 = cluster1:1 ']' 2023-11-08T05:19:39.558 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval cluster=cluster1 2023-11-08T05:19:39.558 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:19:39.558 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval instance=1 2023-11-08T05:19:39.559 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=1 2023-11-08T05:19:39.559 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-11-08T05:19:39.559 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local local_cluster=cluster1:1 2023-11-08T05:19:39.559 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster=cluster1 2023-11-08T05:19:39.560 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance 2023-11-08T05:19:39.560 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-11-08T05:19:39.560 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local val=cluster1:1 2023-11-08T05:19:39.560 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster_var_name=local_cluster 2023-11-08T05:19:39.561 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance_var_name=instance 2023-11-08T05:19:39.561 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:19:39.561 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=1 2023-11-08T05:19:39.561 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ '[' 1 = cluster1:1 ']' 2023-11-08T05:19:39.561 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval local_cluster=cluster1 2023-11-08T05:19:39.561 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ local_cluster=cluster1 2023-11-08T05:19:39.562 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval instance=1 2023-11-08T05:19:39.562 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ instance=1 2023-11-08T05:19:39.562 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-11-08T05:19:39.570 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-08T05:19:39.570 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-08T05:19:39.571 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-08T05:19:39.571 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_40 2023-11-08T05:19:39.672 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:Invalid command: unused arguments: ['mirror/image_40'] 2023-11-08T05:19:39.672 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd mirror status : get status for rbd mirror 2023-11-08T05:19:39.672 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:admin_socket: invalid command 2023-11-08T05:19:39.673 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T05:19:39.673 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_40 2023-11-08T05:19:39.673 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1:2 2023-11-08T05:19:39.673 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T05:19:39.673 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance 2023-11-08T05:19:39.674 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-11-08T05:19:39.674 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local val=cluster1:2 2023-11-08T05:19:39.674 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_var_name=cluster 2023-11-08T05:19:39.674 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance_var_name=instance 2023-11-08T05:19:39.674 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cluster=cluster1 2023-11-08T05:19:39.675 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ instance=2 2023-11-08T05:19:39.675 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' 2 = cluster1:2 ']' 2023-11-08T05:19:39.675 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval cluster=cluster1 2023-11-08T05:19:39.675 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:19:39.675 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval instance=2 2023-11-08T05:19:39.676 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=2 2023-11-08T05:19:39.676 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-11-08T05:19:39.676 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local local_cluster=cluster1:2 2023-11-08T05:19:39.676 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster=cluster1 2023-11-08T05:19:39.676 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance 2023-11-08T05:19:39.677 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-11-08T05:19:39.677 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local val=cluster1:2 2023-11-08T05:19:39.677 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster_var_name=local_cluster 2023-11-08T05:19:39.677 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance_var_name=instance 2023-11-08T05:19:39.677 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:19:39.678 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=2 2023-11-08T05:19:39.678 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ '[' 2 = cluster1:2 ']' 2023-11-08T05:19:39.678 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval local_cluster=cluster1 2023-11-08T05:19:39.678 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ local_cluster=cluster1 2023-11-08T05:19:39.678 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval instance=2 2023-11-08T05:19:39.679 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ instance=2 2023-11-08T05:19:39.679 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-11-08T05:19:39.684 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-08T05:19:39.684 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-08T05:19:39.685 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-08T05:19:39.685 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_40 2023-11-08T05:19:39.793 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ return 0 2023-11-08T05:19:39.794 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ status_result=' "state": "Replaying"' 2023-11-08T05:19:39.794 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ echo ' "state": "Replaying"' 2023-11-08T05:19:39.794 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ grep -i Replaying 2023-11-08T05:19:39.795 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: "state": "Replaying" 2023-11-08T05:19:39.797 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ current_state=started 2023-11-08T05:19:39.797 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test started = started 2023-11-08T05:19:39.797 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:19:39.797 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_40 2023-11-08T05:19:39.798 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local local_cluster=cluster1 2023-11-08T05:19:39.798 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:19:39.798 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:19:39.798 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_40 2023-11-08T05:19:39.798 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' snapshot = journal ']' 2023-11-08T05:19:39.799 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' snapshot = snapshot ']' 2023-11-08T05:19:39.799 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_40 2023-11-08T05:19:39.799 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local local_cluster=cluster1 2023-11-08T05:19:39.799 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:19:39.799 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:19:39.800 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_40 2023-11-08T05:19:39.800 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ mkfname cluster2-mirror-image_40.status 2023-11-08T05:19:39.800 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ echo cluster2-mirror-image_40.status 2023-11-08T05:19:39.800 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -e 's|[/ ]|_|g' 2023-11-08T05:19:39.800 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local status_log=/tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_40.status 2023-11-08T05:19:39.801 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ mkfname cluster1-mirror-image_40.status 2023-11-08T05:19:39.801 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ echo cluster1-mirror-image_40.status 2023-11-08T05:19:39.801 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -e 's|[/ ]|_|g' 2023-11-08T05:19:39.802 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local local_status_log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_40.status 2023-11-08T05:19:39.802 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ mirror_image_snapshot cluster2 mirror image_40 2023-11-08T05:19:39.802 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:19:39.803 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:19:39.803 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_40 2023-11-08T05:19:39.803 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_40 2023-11-08T05:19:40.426 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:Snapshot ID: 197 2023-11-08T05:19:40.436 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_40 /tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_40.status 2023-11-08T05:19:40.436 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:19:40.436 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:19:40.437 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_40 2023-11-08T05:19:40.437 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_40.status 2023-11-08T05:19:40.437 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_40 --format xml 2023-11-08T05:19:40.437 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T05:19:40.474 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-11-08T05:19:40.476 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snapshot_id=197 2023-11-08T05:19:40.476 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ true 2023-11-08T05:19:40.477 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-11-08T05:19:40.477 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0.2 2023-11-08T05:19:40.677 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_40 /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_40.status 2023-11-08T05:19:40.678 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:19:40.678 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:19:40.678 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_40 2023-11-08T05:19:40.678 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_40.status 2023-11-08T05:19:40.679 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_40 --format xml 2023-11-08T05:19:40.679 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T05:19:40.719 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ true 2023-11-08T05:19:40.719 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-08T05:19:40.720 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:-:1.1: Document is empty 2023-11-08T05:19:40.720 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: 2023-11-08T05:19:40.720 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:^ 2023-11-08T05:19:40.721 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local primary_snapshot_id= 2023-11-08T05:19:40.721 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test 197 = '' 2023-11-08T05:19:40.721 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-11-08T05:19:40.722 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0.4 2023-11-08T05:19:41.122 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_40 /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_40.status 2023-11-08T05:19:41.122 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:19:41.123 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:19:41.123 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_40 2023-11-08T05:19:41.123 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_40.status 2023-11-08T05:19:41.123 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_40 --format xml 2023-11-08T05:19:41.123 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T05:19:41.165 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ true 2023-11-08T05:19:41.165 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-08T05:19:41.166 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:-:1.1: Document is empty 2023-11-08T05:19:41.167 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: 2023-11-08T05:19:41.167 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:^ 2023-11-08T05:19:41.168 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local primary_snapshot_id= 2023-11-08T05:19:41.168 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test 197 = '' 2023-11-08T05:19:41.168 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-11-08T05:19:41.168 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0.8 2023-11-08T05:19:41.969 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_40 /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_40.status 2023-11-08T05:19:41.969 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:19:41.969 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:19:41.970 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_40 2023-11-08T05:19:41.970 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_40.status 2023-11-08T05:19:41.970 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_40 --format xml 2023-11-08T05:19:41.970 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T05:19:42.020 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ true 2023-11-08T05:19:42.020 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-08T05:19:42.022 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:-:1.1: Document is empty 2023-11-08T05:19:42.022 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: 2023-11-08T05:19:42.022 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:^ 2023-11-08T05:19:42.022 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local primary_snapshot_id= 2023-11-08T05:19:42.023 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test 197 = '' 2023-11-08T05:19:42.023 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-11-08T05:19:42.023 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 1.6 2023-11-08T05:19:43.623 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_40 /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_40.status 2023-11-08T05:19:43.623 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:19:43.624 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:19:43.624 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_40 2023-11-08T05:19:43.624 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_40.status 2023-11-08T05:19:43.624 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_40 --format xml 2023-11-08T05:19:43.625 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T05:19:43.666 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-08T05:19:43.669 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local primary_snapshot_id=197 2023-11-08T05:19:43.669 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test 197 = 197 2023-11-08T05:19:43.670 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:19:43.670 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_snap_present cluster1 mirror image_40 snap 2023-11-08T05:19:43.670 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:19:43.670 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:19:43.670 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_40 2023-11-08T05:19:43.671 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap_name=snap 2023-11-08T05:19:43.671 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local s 2023-11-08T05:19:43.671 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-11-08T05:19:43.671 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 1 2023-11-08T05:19:44.670 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 info mirror/image_40@snap 2023-11-08T05:19:44.710 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:rbd image 'image_40': 2023-11-08T05:19:44.710 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: size 128 MiB in 32 objects 2023-11-08T05:19:44.710 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: order 22 (4 MiB objects) 2023-11-08T05:19:44.710 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: snapshot_count: 2 2023-11-08T05:19:44.711 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: id: 12b3bdcb5378 2023-11-08T05:19:44.711 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: block_name_prefix: rbd_data.12b3bdcb5378 2023-11-08T05:19:44.711 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: format: 2 2023-11-08T05:19:44.711 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: features: layering, exclusive-lock, non-primary 2023-11-08T05:19:44.711 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: op_features: 2023-11-08T05:19:44.712 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: flags: 2023-11-08T05:19:44.712 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: create_timestamp: Wed Nov 8 05:11:26 2023 2023-11-08T05:19:44.712 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: access_timestamp: Wed Nov 8 05:11:26 2023 2023-11-08T05:19:44.712 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: modify_timestamp: Wed Nov 8 05:11:26 2023 2023-11-08T05:19:44.712 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: protected: False 2023-11-08T05:19:44.713 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring state: enabled 2023-11-08T05:19:44.713 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring mode: snapshot 2023-11-08T05:19:44.713 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring global id: 17ed47a8-b4d3-4db4-8592-15e686f59163 2023-11-08T05:19:44.713 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring primary: false 2023-11-08T05:19:44.714 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:19:44.715 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ compare_image_snaps mirror image_40 snap 2023-11-08T05:19:44.715 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:19:44.715 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_40 2023-11-08T05:19:44.715 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap_name=snap 2023-11-08T05:19:44.715 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local ret=0 2023-11-08T05:19:44.716 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local rmt_export=/tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_40.export 2023-11-08T05:19:44.716 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local loc_export=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_40.export 2023-11-08T05:19:44.716 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rm -f /tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_40.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_40.export 2023-11-08T05:19:44.716 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 -p mirror export image_40@snap /tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_40.export 2023-11-08T05:19:44.896 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-11-08T05:19:44.898 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 -p mirror export image_40@snap /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_40.export 2023-11-08T05:19:45.090 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-11-08T05:19:45.093 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ cmp /tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_40.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_40.export 2023-11-08T05:19:45.148 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rm -f /tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_40.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_40.export 2023-11-08T05:19:45.186 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:19:45.186 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-11-08T05:19:45.186 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image=image_41 2023-11-08T05:19:45.187 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ create_snap cluster2 mirror image_41 snap 2023-11-08T05:19:45.187 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:19:45.187 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:19:45.187 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_41 2023-11-08T05:19:45.187 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap_name=snap 2023-11-08T05:19:45.188 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 -p mirror snap create image_41@snap --debug-rbd=20 --debug-journaler=20 2023-11-08T05:19:46.434 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_image_replay_started cluster1 mirror image_41 2023-11-08T05:19:46.434 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:19:46.435 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:19:46.435 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_41 2023-11-08T05:19:46.435 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_image_replay_state cluster1 mirror image_41 started 2023-11-08T05:19:46.435 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:19:46.435 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:19:46.436 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_41 2023-11-08T05:19:46.436 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local state=started 2023-11-08T05:19:46.436 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local s 2023-11-08T05:19:46.436 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-11-08T05:19:46.436 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 1 2023-11-08T05:19:47.435 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test_image_replay_state cluster1 mirror image_41 started 2023-11-08T05:19:47.435 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:19:47.436 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:19:47.436 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_41 2023-11-08T05:19:47.436 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local test_state=started 2023-11-08T05:19:47.436 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local status_result 2023-11-08T05:19:47.436 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local current_state=stopped 2023-11-08T05:19:47.440 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_41 2023-11-08T05:19:47.440 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_instance=cluster1 2023-11-08T05:19:47.441 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T05:19:47.441 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1 2023-11-08T05:19:47.441 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance=cluster1 2023-11-08T05:19:47.441 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ grep -i state 2023-11-08T05:19:47.441 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local loop_instance 2023-11-08T05:19:47.442 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-11-08T05:19:47.442 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sleep 0 2023-11-08T05:19:47.442 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-11-08T05:19:47.442 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ seq 0 3 2023-11-08T05:19:47.443 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T05:19:47.443 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_41 2023-11-08T05:19:47.443 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1:0 2023-11-08T05:19:47.443 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T05:19:47.444 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance 2023-11-08T05:19:47.444 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-11-08T05:19:47.444 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local val=cluster1:0 2023-11-08T05:19:47.444 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_var_name=cluster 2023-11-08T05:19:47.444 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance_var_name=instance 2023-11-08T05:19:47.445 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cluster=cluster1 2023-11-08T05:19:47.445 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ instance=0 2023-11-08T05:19:47.445 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' 0 = cluster1:0 ']' 2023-11-08T05:19:47.445 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval cluster=cluster1 2023-11-08T05:19:47.445 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:19:47.446 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval instance=0 2023-11-08T05:19:47.446 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=0 2023-11-08T05:19:47.446 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-11-08T05:19:47.446 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local local_cluster=cluster1:0 2023-11-08T05:19:47.446 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster=cluster1 2023-11-08T05:19:47.447 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance 2023-11-08T05:19:47.447 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-11-08T05:19:47.447 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local val=cluster1:0 2023-11-08T05:19:47.447 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster_var_name=local_cluster 2023-11-08T05:19:47.447 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance_var_name=instance 2023-11-08T05:19:47.448 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:19:47.448 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=0 2023-11-08T05:19:47.448 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ '[' 0 = cluster1:0 ']' 2023-11-08T05:19:47.448 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval local_cluster=cluster1 2023-11-08T05:19:47.448 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ local_cluster=cluster1 2023-11-08T05:19:47.449 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval instance=0 2023-11-08T05:19:47.449 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ instance=0 2023-11-08T05:19:47.449 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-11-08T05:19:47.459 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-08T05:19:47.459 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-08T05:19:47.459 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-08T05:19:47.460 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_41 2023-11-08T05:19:47.565 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:Invalid command: unused arguments: ['mirror/image_41'] 2023-11-08T05:19:47.565 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd mirror status : get status for rbd mirror 2023-11-08T05:19:47.565 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:admin_socket: invalid command 2023-11-08T05:19:47.568 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T05:19:47.568 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_41 2023-11-08T05:19:47.569 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1:1 2023-11-08T05:19:47.569 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T05:19:47.569 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance 2023-11-08T05:19:47.569 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-11-08T05:19:47.569 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local val=cluster1:1 2023-11-08T05:19:47.570 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_var_name=cluster 2023-11-08T05:19:47.570 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance_var_name=instance 2023-11-08T05:19:47.570 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cluster=cluster1 2023-11-08T05:19:47.570 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ instance=1 2023-11-08T05:19:47.571 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' 1 = cluster1:1 ']' 2023-11-08T05:19:47.571 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval cluster=cluster1 2023-11-08T05:19:47.571 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:19:47.571 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval instance=1 2023-11-08T05:19:47.571 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=1 2023-11-08T05:19:47.572 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-11-08T05:19:47.572 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local local_cluster=cluster1:1 2023-11-08T05:19:47.572 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster=cluster1 2023-11-08T05:19:47.572 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance 2023-11-08T05:19:47.572 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-11-08T05:19:47.573 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local val=cluster1:1 2023-11-08T05:19:47.573 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster_var_name=local_cluster 2023-11-08T05:19:47.573 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance_var_name=instance 2023-11-08T05:19:47.573 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:19:47.573 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=1 2023-11-08T05:19:47.574 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ '[' 1 = cluster1:1 ']' 2023-11-08T05:19:47.574 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval local_cluster=cluster1 2023-11-08T05:19:47.574 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ local_cluster=cluster1 2023-11-08T05:19:47.574 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval instance=1 2023-11-08T05:19:47.575 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ instance=1 2023-11-08T05:19:47.575 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-11-08T05:19:47.582 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-08T05:19:47.583 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-08T05:19:47.583 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-08T05:19:47.583 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_41 2023-11-08T05:19:47.680 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:Invalid command: unused arguments: ['mirror/image_41'] 2023-11-08T05:19:47.680 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd mirror status : get status for rbd mirror 2023-11-08T05:19:47.681 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:admin_socket: invalid command 2023-11-08T05:19:47.683 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T05:19:47.684 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_41 2023-11-08T05:19:47.684 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1:2 2023-11-08T05:19:47.684 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T05:19:47.684 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance 2023-11-08T05:19:47.684 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-11-08T05:19:47.685 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local val=cluster1:2 2023-11-08T05:19:47.685 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_var_name=cluster 2023-11-08T05:19:47.685 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance_var_name=instance 2023-11-08T05:19:47.685 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cluster=cluster1 2023-11-08T05:19:47.685 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ instance=2 2023-11-08T05:19:47.686 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' 2 = cluster1:2 ']' 2023-11-08T05:19:47.686 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval cluster=cluster1 2023-11-08T05:19:47.686 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:19:47.686 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval instance=2 2023-11-08T05:19:47.686 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=2 2023-11-08T05:19:47.687 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-11-08T05:19:47.687 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local local_cluster=cluster1:2 2023-11-08T05:19:47.687 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster=cluster1 2023-11-08T05:19:47.687 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance 2023-11-08T05:19:47.687 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-11-08T05:19:47.688 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local val=cluster1:2 2023-11-08T05:19:47.688 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster_var_name=local_cluster 2023-11-08T05:19:47.688 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance_var_name=instance 2023-11-08T05:19:47.688 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:19:47.688 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=2 2023-11-08T05:19:47.689 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ '[' 2 = cluster1:2 ']' 2023-11-08T05:19:47.689 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval local_cluster=cluster1 2023-11-08T05:19:47.689 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ local_cluster=cluster1 2023-11-08T05:19:47.689 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval instance=2 2023-11-08T05:19:47.689 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ instance=2 2023-11-08T05:19:47.690 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-11-08T05:19:47.697 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-08T05:19:47.697 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-08T05:19:47.697 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-08T05:19:47.698 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_41 2023-11-08T05:19:47.796 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:Invalid command: unused arguments: ['mirror/image_41'] 2023-11-08T05:19:47.796 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd mirror status : get status for rbd mirror 2023-11-08T05:19:47.796 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:admin_socket: invalid command 2023-11-08T05:19:47.799 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T05:19:47.800 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_41 2023-11-08T05:19:47.800 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1:3 2023-11-08T05:19:47.800 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T05:19:47.800 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance 2023-11-08T05:19:47.800 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ set_cluster_instance cluster1:3 cluster instance 2023-11-08T05:19:47.801 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local val=cluster1:3 2023-11-08T05:19:47.801 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_var_name=cluster 2023-11-08T05:19:47.801 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance_var_name=instance 2023-11-08T05:19:47.801 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cluster=cluster1 2023-11-08T05:19:47.801 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ instance=3 2023-11-08T05:19:47.802 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' 3 = cluster1:3 ']' 2023-11-08T05:19:47.802 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval cluster=cluster1 2023-11-08T05:19:47.802 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:19:47.802 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval instance=3 2023-11-08T05:19:47.802 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=3 2023-11-08T05:19:47.803 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ daemon_asok_file cluster1:3 cluster1 2023-11-08T05:19:47.803 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local local_cluster=cluster1:3 2023-11-08T05:19:47.803 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster=cluster1 2023-11-08T05:19:47.803 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance 2023-11-08T05:19:47.803 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2023-11-08T05:19:47.804 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local val=cluster1:3 2023-11-08T05:19:47.804 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster_var_name=local_cluster 2023-11-08T05:19:47.804 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance_var_name=instance 2023-11-08T05:19:47.804 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:19:47.804 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=3 2023-11-08T05:19:47.805 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ '[' 3 = cluster1:3 ']' 2023-11-08T05:19:47.805 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval local_cluster=cluster1 2023-11-08T05:19:47.805 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ local_cluster=cluster1 2023-11-08T05:19:47.805 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval instance=3 2023-11-08T05:19:47.805 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ instance=3 2023-11-08T05:19:47.806 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2023-11-08T05:19:47.814 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-11-08T05:19:47.814 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-11-08T05:19:47.815 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-11-08T05:19:47.815 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_41 2023-11-08T05:19:47.914 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ return 0 2023-11-08T05:19:47.915 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ status_result=' "state": "Replaying"' 2023-11-08T05:19:47.915 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ echo ' "state": "Replaying"' 2023-11-08T05:19:47.916 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ grep -i Replaying 2023-11-08T05:19:47.916 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: "state": "Replaying" 2023-11-08T05:19:47.918 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ current_state=started 2023-11-08T05:19:47.918 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test started = started 2023-11-08T05:19:47.919 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:19:47.919 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_41 2023-11-08T05:19:47.919 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local local_cluster=cluster1 2023-11-08T05:19:47.919 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:19:47.919 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:19:47.920 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_41 2023-11-08T05:19:47.920 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' snapshot = journal ']' 2023-11-08T05:19:47.920 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' snapshot = snapshot ']' 2023-11-08T05:19:47.920 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_41 2023-11-08T05:19:47.920 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local local_cluster=cluster1 2023-11-08T05:19:47.921 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:19:47.921 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:19:47.921 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_41 2023-11-08T05:19:47.921 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ mkfname cluster2-mirror-image_41.status 2023-11-08T05:19:47.922 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ echo cluster2-mirror-image_41.status 2023-11-08T05:19:47.922 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -e 's|[/ ]|_|g' 2023-11-08T05:19:47.922 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local status_log=/tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_41.status 2023-11-08T05:19:47.922 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ mkfname cluster1-mirror-image_41.status 2023-11-08T05:19:47.922 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ echo cluster1-mirror-image_41.status 2023-11-08T05:19:47.923 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -e 's|[/ ]|_|g' 2023-11-08T05:19:47.923 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local local_status_log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_41.status 2023-11-08T05:19:47.924 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ mirror_image_snapshot cluster2 mirror image_41 2023-11-08T05:19:47.924 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:19:47.924 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:19:47.924 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_41 2023-11-08T05:19:47.925 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_41 2023-11-08T05:19:48.438 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:Snapshot ID: 199 2023-11-08T05:19:48.446 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_41 /tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_41.status 2023-11-08T05:19:48.446 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:19:48.446 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:19:48.447 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_41 2023-11-08T05:19:48.447 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_41.status 2023-11-08T05:19:48.447 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_41 --format xml 2023-11-08T05:19:48.447 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T05:19:48.482 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-11-08T05:19:48.484 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snapshot_id=199 2023-11-08T05:19:48.485 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ true 2023-11-08T05:19:48.485 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-11-08T05:19:48.485 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0.2 2023-11-08T05:19:48.685 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_41 /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_41.status 2023-11-08T05:19:48.686 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:19:48.686 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:19:48.686 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_41 2023-11-08T05:19:48.687 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_41.status 2023-11-08T05:19:48.687 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_41 --format xml 2023-11-08T05:19:48.687 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T05:19:48.725 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ true 2023-11-08T05:19:48.725 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-08T05:19:48.726 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:-:1.1: Document is empty 2023-11-08T05:19:48.726 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: 2023-11-08T05:19:48.726 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:^ 2023-11-08T05:19:48.727 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local primary_snapshot_id= 2023-11-08T05:19:48.728 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test 199 = '' 2023-11-08T05:19:48.728 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-11-08T05:19:48.728 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0.4 2023-11-08T05:19:49.128 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_41 /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_41.status 2023-11-08T05:19:49.128 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:19:49.129 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:19:49.129 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_41 2023-11-08T05:19:49.129 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_41.status 2023-11-08T05:19:49.129 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_41 --format xml 2023-11-08T05:19:49.130 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T05:19:49.167 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ true 2023-11-08T05:19:49.168 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-08T05:19:49.169 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:-:1.1: Document is empty 2023-11-08T05:19:49.169 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: 2023-11-08T05:19:49.169 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:^ 2023-11-08T05:19:49.170 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local primary_snapshot_id= 2023-11-08T05:19:49.170 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test 199 = '' 2023-11-08T05:19:49.170 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-11-08T05:19:49.171 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0.8 2023-11-08T05:19:49.971 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_41 /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_41.status 2023-11-08T05:19:49.971 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:19:49.971 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:19:49.972 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_41 2023-11-08T05:19:49.972 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_41.status 2023-11-08T05:19:49.972 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_41 --format xml 2023-11-08T05:19:49.972 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T05:19:50.012 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-08T05:19:50.014 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local primary_snapshot_id=199 2023-11-08T05:19:50.014 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test 199 = 199 2023-11-08T05:19:50.014 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:19:50.015 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_snap_present cluster1 mirror image_41 snap 2023-11-08T05:19:50.015 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:19:50.015 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:19:50.015 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_41 2023-11-08T05:19:50.015 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap_name=snap 2023-11-08T05:19:50.016 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local s 2023-11-08T05:19:50.016 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-11-08T05:19:50.016 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 1 2023-11-08T05:19:51.015 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 info mirror/image_41@snap 2023-11-08T05:19:51.054 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:rbd image 'image_41': 2023-11-08T05:19:51.054 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: size 128 MiB in 32 objects 2023-11-08T05:19:51.054 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: order 22 (4 MiB objects) 2023-11-08T05:19:51.055 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: snapshot_count: 2 2023-11-08T05:19:51.055 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: id: 11d8b03471f9 2023-11-08T05:19:51.055 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: block_name_prefix: rbd_data.11d8b03471f9 2023-11-08T05:19:51.055 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: format: 2 2023-11-08T05:19:51.055 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: features: layering, exclusive-lock, non-primary 2023-11-08T05:19:51.056 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: op_features: 2023-11-08T05:19:51.056 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: flags: 2023-11-08T05:19:51.056 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: create_timestamp: Wed Nov 8 05:11:40 2023 2023-11-08T05:19:51.056 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: access_timestamp: Wed Nov 8 05:11:40 2023 2023-11-08T05:19:51.056 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: modify_timestamp: Wed Nov 8 05:11:40 2023 2023-11-08T05:19:51.057 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: protected: False 2023-11-08T05:19:51.057 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring state: enabled 2023-11-08T05:19:51.057 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring mode: snapshot 2023-11-08T05:19:51.057 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring global id: de215ca4-d87f-4583-9648-dab2d031119f 2023-11-08T05:19:51.057 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring primary: false 2023-11-08T05:19:51.059 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:19:51.059 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ compare_image_snaps mirror image_41 snap 2023-11-08T05:19:51.059 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:19:51.059 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_41 2023-11-08T05:19:51.060 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap_name=snap 2023-11-08T05:19:51.060 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local ret=0 2023-11-08T05:19:51.060 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local rmt_export=/tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_41.export 2023-11-08T05:19:51.060 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local loc_export=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_41.export 2023-11-08T05:19:51.060 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rm -f /tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_41.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_41.export 2023-11-08T05:19:51.061 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 -p mirror export image_41@snap /tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_41.export 2023-11-08T05:19:51.441 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-11-08T05:19:51.444 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 -p mirror export image_41@snap /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_41.export 2023-11-08T05:19:51.684 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-11-08T05:19:51.686 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ cmp /tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_41.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_41.export 2023-11-08T05:19:51.742 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rm -f /tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_41.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_41.export 2023-11-08T05:19:51.780 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:19:51.781 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-11-08T05:19:51.781 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image=image_42 2023-11-08T05:19:51.781 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ create_snap cluster2 mirror image_42 snap 2023-11-08T05:19:51.781 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:19:51.781 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:19:51.782 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_42 2023-11-08T05:19:51.782 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap_name=snap 2023-11-08T05:19:51.782 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 -p mirror snap create image_42@snap --debug-rbd=20 --debug-journaler=20 2023-11-08T05:19:53.242 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_image_replay_started cluster1 mirror image_42 2023-11-08T05:19:53.242 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:19:53.242 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:19:53.242 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_42 2023-11-08T05:19:53.243 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_image_replay_state cluster1 mirror image_42 started 2023-11-08T05:19:53.243 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:19:53.243 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:19:53.243 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_42 2023-11-08T05:19:53.243 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local state=started 2023-11-08T05:19:53.244 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local s 2023-11-08T05:19:53.244 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-11-08T05:19:53.244 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 1 2023-11-08T05:19:54.243 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test_image_replay_state cluster1 mirror image_42 started 2023-11-08T05:19:54.244 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:19:54.244 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:19:54.244 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_42 2023-11-08T05:19:54.244 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local test_state=started 2023-11-08T05:19:54.244 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local status_result 2023-11-08T05:19:54.245 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local current_state=stopped 2023-11-08T05:19:54.245 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_42 2023-11-08T05:19:54.245 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_instance=cluster1 2023-11-08T05:19:54.245 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T05:19:54.245 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ grep -i state 2023-11-08T05:19:54.246 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1 2023-11-08T05:19:54.246 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance=cluster1 2023-11-08T05:19:54.246 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local loop_instance 2023-11-08T05:19:54.246 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-11-08T05:19:54.246 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sleep 0 2023-11-08T05:19:54.249 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-11-08T05:19:54.250 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ seq 0 3 2023-11-08T05:19:54.250 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T05:19:54.250 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_42 2023-11-08T05:19:54.250 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1:0 2023-11-08T05:19:54.251 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T05:19:54.251 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance 2023-11-08T05:19:54.251 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-11-08T05:19:54.251 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local val=cluster1:0 2023-11-08T05:19:54.251 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_var_name=cluster 2023-11-08T05:19:54.252 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance_var_name=instance 2023-11-08T05:19:54.252 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cluster=cluster1 2023-11-08T05:19:54.252 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ instance=0 2023-11-08T05:19:54.252 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' 0 = cluster1:0 ']' 2023-11-08T05:19:54.252 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval cluster=cluster1 2023-11-08T05:19:54.253 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:19:54.253 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval instance=0 2023-11-08T05:19:54.253 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=0 2023-11-08T05:19:54.253 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-11-08T05:19:54.253 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local local_cluster=cluster1:0 2023-11-08T05:19:54.254 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster=cluster1 2023-11-08T05:19:54.254 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance 2023-11-08T05:19:54.255 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-11-08T05:19:54.255 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local val=cluster1:0 2023-11-08T05:19:54.256 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster_var_name=local_cluster 2023-11-08T05:19:54.256 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance_var_name=instance 2023-11-08T05:19:54.256 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:19:54.257 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=0 2023-11-08T05:19:54.257 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ '[' 0 = cluster1:0 ']' 2023-11-08T05:19:54.257 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval local_cluster=cluster1 2023-11-08T05:19:54.257 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ local_cluster=cluster1 2023-11-08T05:19:54.257 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval instance=0 2023-11-08T05:19:54.257 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ instance=0 2023-11-08T05:19:54.258 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-11-08T05:19:54.266 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-08T05:19:54.266 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-08T05:19:54.267 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-08T05:19:54.267 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_42 2023-11-08T05:19:54.381 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ return 0 2023-11-08T05:19:54.382 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ status_result=' "state": "Replaying"' 2023-11-08T05:19:54.382 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ echo ' "state": "Replaying"' 2023-11-08T05:19:54.382 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ grep -i Replaying 2023-11-08T05:19:54.383 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: "state": "Replaying" 2023-11-08T05:19:54.385 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ current_state=started 2023-11-08T05:19:54.385 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test started = started 2023-11-08T05:19:54.386 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:19:54.386 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_42 2023-11-08T05:19:54.386 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local local_cluster=cluster1 2023-11-08T05:19:54.386 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:19:54.386 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:19:54.387 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_42 2023-11-08T05:19:54.387 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' snapshot = journal ']' 2023-11-08T05:19:54.387 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' snapshot = snapshot ']' 2023-11-08T05:19:54.387 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_42 2023-11-08T05:19:54.387 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local local_cluster=cluster1 2023-11-08T05:19:54.388 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:19:54.388 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:19:54.388 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_42 2023-11-08T05:19:54.388 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ mkfname cluster2-mirror-image_42.status 2023-11-08T05:19:54.388 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ echo cluster2-mirror-image_42.status 2023-11-08T05:19:54.389 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -e 's|[/ ]|_|g' 2023-11-08T05:19:54.389 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local status_log=/tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_42.status 2023-11-08T05:19:54.389 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ mkfname cluster1-mirror-image_42.status 2023-11-08T05:19:54.390 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ echo cluster1-mirror-image_42.status 2023-11-08T05:19:54.390 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -e 's|[/ ]|_|g' 2023-11-08T05:19:54.391 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local local_status_log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_42.status 2023-11-08T05:19:54.391 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ mirror_image_snapshot cluster2 mirror image_42 2023-11-08T05:19:54.391 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:19:54.391 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:19:54.392 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_42 2023-11-08T05:19:54.392 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_42 2023-11-08T05:19:55.250 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:Snapshot ID: 201 2023-11-08T05:19:55.260 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_42 /tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_42.status 2023-11-08T05:19:55.260 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:19:55.261 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:19:55.261 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_42 2023-11-08T05:19:55.261 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_42.status 2023-11-08T05:19:55.261 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_42 --format xml 2023-11-08T05:19:55.262 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T05:19:55.300 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-11-08T05:19:55.302 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snapshot_id=201 2023-11-08T05:19:55.302 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ true 2023-11-08T05:19:55.302 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-11-08T05:19:55.303 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0.2 2023-11-08T05:19:55.503 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_42 /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_42.status 2023-11-08T05:19:55.504 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:19:55.504 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:19:55.504 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_42 2023-11-08T05:19:55.504 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_42.status 2023-11-08T05:19:55.504 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_42 --format xml 2023-11-08T05:19:55.505 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T05:19:55.545 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-08T05:19:55.547 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local primary_snapshot_id=109 2023-11-08T05:19:55.547 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test 201 = 109 2023-11-08T05:19:55.547 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-11-08T05:19:55.547 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0.4 2023-11-08T05:19:55.948 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_42 /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_42.status 2023-11-08T05:19:55.949 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:19:55.949 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:19:55.949 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_42 2023-11-08T05:19:55.950 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_42.status 2023-11-08T05:19:55.950 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_42 --format xml 2023-11-08T05:19:55.950 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T05:19:55.994 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ true 2023-11-08T05:19:55.994 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-08T05:19:55.995 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:-:1.1: Document is empty 2023-11-08T05:19:55.995 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: 2023-11-08T05:19:55.996 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:^ 2023-11-08T05:19:55.996 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local primary_snapshot_id= 2023-11-08T05:19:55.996 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test 201 = '' 2023-11-08T05:19:55.997 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-11-08T05:19:55.997 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0.8 2023-11-08T05:19:56.798 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_42 /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_42.status 2023-11-08T05:19:56.798 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:19:56.798 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:19:56.799 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_42 2023-11-08T05:19:56.799 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_42.status 2023-11-08T05:19:56.799 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_42 --format xml 2023-11-08T05:19:56.799 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T05:19:56.836 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ true 2023-11-08T05:19:56.836 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-08T05:19:56.837 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:-:1.1: Document is empty 2023-11-08T05:19:56.837 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: 2023-11-08T05:19:56.837 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:^ 2023-11-08T05:19:56.838 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local primary_snapshot_id= 2023-11-08T05:19:56.838 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test 201 = '' 2023-11-08T05:19:56.838 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-11-08T05:19:56.838 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 1.6 2023-11-08T05:19:58.439 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_42 /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_42.status 2023-11-08T05:19:58.439 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:19:58.439 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:19:58.440 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_42 2023-11-08T05:19:58.440 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_42.status 2023-11-08T05:19:58.440 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_42 --format xml 2023-11-08T05:19:58.440 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T05:19:58.483 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-08T05:19:58.486 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local primary_snapshot_id=201 2023-11-08T05:19:58.486 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test 201 = 201 2023-11-08T05:19:58.486 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:19:58.487 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_snap_present cluster1 mirror image_42 snap 2023-11-08T05:19:58.487 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:19:58.487 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:19:58.487 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_42 2023-11-08T05:19:58.487 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap_name=snap 2023-11-08T05:19:58.488 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local s 2023-11-08T05:19:58.488 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-11-08T05:19:58.488 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 1 2023-11-08T05:19:59.486 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 info mirror/image_42@snap 2023-11-08T05:19:59.525 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:rbd image 'image_42': 2023-11-08T05:19:59.525 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: size 128 MiB in 32 objects 2023-11-08T05:19:59.526 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: order 22 (4 MiB objects) 2023-11-08T05:19:59.526 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: snapshot_count: 2 2023-11-08T05:19:59.526 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: id: 128762b0f47 2023-11-08T05:19:59.526 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: block_name_prefix: rbd_data.128762b0f47 2023-11-08T05:19:59.527 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: format: 2 2023-11-08T05:19:59.527 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: features: layering, exclusive-lock, non-primary 2023-11-08T05:19:59.527 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: op_features: 2023-11-08T05:19:59.527 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: flags: 2023-11-08T05:19:59.527 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: create_timestamp: Wed Nov 8 05:11:46 2023 2023-11-08T05:19:59.527 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: access_timestamp: Wed Nov 8 05:11:46 2023 2023-11-08T05:19:59.528 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: modify_timestamp: Wed Nov 8 05:11:46 2023 2023-11-08T05:19:59.528 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: protected: False 2023-11-08T05:19:59.528 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring state: enabled 2023-11-08T05:19:59.528 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring mode: snapshot 2023-11-08T05:19:59.528 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring global id: 2409fb55-b412-43a5-9d65-43c4ee962379 2023-11-08T05:19:59.529 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring primary: false 2023-11-08T05:19:59.530 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:19:59.530 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ compare_image_snaps mirror image_42 snap 2023-11-08T05:19:59.530 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:19:59.531 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_42 2023-11-08T05:19:59.531 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap_name=snap 2023-11-08T05:19:59.531 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local ret=0 2023-11-08T05:19:59.531 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local rmt_export=/tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_42.export 2023-11-08T05:19:59.531 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local loc_export=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_42.export 2023-11-08T05:19:59.532 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rm -f /tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_42.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_42.export 2023-11-08T05:19:59.532 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 -p mirror export image_42@snap /tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_42.export 2023-11-08T05:19:59.705 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-11-08T05:19:59.710 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 -p mirror export image_42@snap /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_42.export 2023-11-08T05:19:59.902 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-11-08T05:19:59.904 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ cmp /tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_42.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_42.export 2023-11-08T05:19:59.959 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rm -f /tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_42.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_42.export 2023-11-08T05:19:59.995 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:19:59.996 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-11-08T05:19:59.996 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image=image_43 2023-11-08T05:19:59.996 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ create_snap cluster2 mirror image_43 snap 2023-11-08T05:19:59.996 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:19:59.997 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:19:59.997 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_43 2023-11-08T05:19:59.997 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap_name=snap 2023-11-08T05:19:59.997 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 -p mirror snap create image_43@snap --debug-rbd=20 --debug-journaler=20 2023-11-08T05:20:01.267 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_image_replay_started cluster1 mirror image_43 2023-11-08T05:20:01.267 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:20:01.267 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:20:01.267 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_43 2023-11-08T05:20:01.268 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_image_replay_state cluster1 mirror image_43 started 2023-11-08T05:20:01.268 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:20:01.268 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:20:01.268 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_43 2023-11-08T05:20:01.268 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local state=started 2023-11-08T05:20:01.269 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local s 2023-11-08T05:20:01.269 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-11-08T05:20:01.269 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 1 2023-11-08T05:20:02.269 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test_image_replay_state cluster1 mirror image_43 started 2023-11-08T05:20:02.269 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:20:02.269 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:20:02.269 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_43 2023-11-08T05:20:02.270 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local test_state=started 2023-11-08T05:20:02.270 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local status_result 2023-11-08T05:20:02.270 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local current_state=stopped 2023-11-08T05:20:02.270 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_43 2023-11-08T05:20:02.271 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_instance=cluster1 2023-11-08T05:20:02.271 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T05:20:02.271 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ grep -i state 2023-11-08T05:20:02.271 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1 2023-11-08T05:20:02.272 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance=cluster1 2023-11-08T05:20:02.272 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local loop_instance 2023-11-08T05:20:02.272 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-11-08T05:20:02.272 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sleep 0 2023-11-08T05:20:02.274 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-11-08T05:20:02.275 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ seq 0 3 2023-11-08T05:20:02.275 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T05:20:02.275 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_43 2023-11-08T05:20:02.275 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1:0 2023-11-08T05:20:02.276 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T05:20:02.276 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance 2023-11-08T05:20:02.276 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-11-08T05:20:02.276 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local val=cluster1:0 2023-11-08T05:20:02.276 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_var_name=cluster 2023-11-08T05:20:02.277 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance_var_name=instance 2023-11-08T05:20:02.277 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cluster=cluster1 2023-11-08T05:20:02.277 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ instance=0 2023-11-08T05:20:02.277 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' 0 = cluster1:0 ']' 2023-11-08T05:20:02.277 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval cluster=cluster1 2023-11-08T05:20:02.278 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:20:02.278 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval instance=0 2023-11-08T05:20:02.278 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=0 2023-11-08T05:20:02.278 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-11-08T05:20:02.279 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local local_cluster=cluster1:0 2023-11-08T05:20:02.279 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster=cluster1 2023-11-08T05:20:02.279 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance 2023-11-08T05:20:02.279 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-11-08T05:20:02.280 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local val=cluster1:0 2023-11-08T05:20:02.281 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster_var_name=local_cluster 2023-11-08T05:20:02.281 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance_var_name=instance 2023-11-08T05:20:02.281 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:20:02.282 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=0 2023-11-08T05:20:02.282 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ '[' 0 = cluster1:0 ']' 2023-11-08T05:20:02.282 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval local_cluster=cluster1 2023-11-08T05:20:02.282 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ local_cluster=cluster1 2023-11-08T05:20:02.283 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval instance=0 2023-11-08T05:20:02.283 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ instance=0 2023-11-08T05:20:02.283 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-11-08T05:20:02.289 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-08T05:20:02.290 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-08T05:20:02.290 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-08T05:20:02.290 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_43 2023-11-08T05:20:02.399 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:Invalid command: unused arguments: ['mirror/image_43'] 2023-11-08T05:20:02.400 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd mirror status : get status for rbd mirror 2023-11-08T05:20:02.400 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:admin_socket: invalid command 2023-11-08T05:20:02.403 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T05:20:02.403 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_43 2023-11-08T05:20:02.403 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1:1 2023-11-08T05:20:02.403 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T05:20:02.404 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance 2023-11-08T05:20:02.404 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-11-08T05:20:02.404 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local val=cluster1:1 2023-11-08T05:20:02.404 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_var_name=cluster 2023-11-08T05:20:02.405 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance_var_name=instance 2023-11-08T05:20:02.405 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cluster=cluster1 2023-11-08T05:20:02.405 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ instance=1 2023-11-08T05:20:02.405 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' 1 = cluster1:1 ']' 2023-11-08T05:20:02.405 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval cluster=cluster1 2023-11-08T05:20:02.406 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:20:02.406 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval instance=1 2023-11-08T05:20:02.406 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=1 2023-11-08T05:20:02.406 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-11-08T05:20:02.406 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local local_cluster=cluster1:1 2023-11-08T05:20:02.407 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster=cluster1 2023-11-08T05:20:02.407 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance 2023-11-08T05:20:02.407 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-11-08T05:20:02.407 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local val=cluster1:1 2023-11-08T05:20:02.407 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster_var_name=local_cluster 2023-11-08T05:20:02.408 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance_var_name=instance 2023-11-08T05:20:02.408 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:20:02.408 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=1 2023-11-08T05:20:02.408 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ '[' 1 = cluster1:1 ']' 2023-11-08T05:20:02.409 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval local_cluster=cluster1 2023-11-08T05:20:02.409 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ local_cluster=cluster1 2023-11-08T05:20:02.409 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval instance=1 2023-11-08T05:20:02.409 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ instance=1 2023-11-08T05:20:02.409 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-11-08T05:20:02.417 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-08T05:20:02.417 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-08T05:20:02.417 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-08T05:20:02.417 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_43 2023-11-08T05:20:02.522 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:Invalid command: unused arguments: ['mirror/image_43'] 2023-11-08T05:20:02.522 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd mirror status : get status for rbd mirror 2023-11-08T05:20:02.523 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:admin_socket: invalid command 2023-11-08T05:20:02.526 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T05:20:02.526 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_43 2023-11-08T05:20:02.526 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1:2 2023-11-08T05:20:02.526 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T05:20:02.527 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance 2023-11-08T05:20:02.527 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-11-08T05:20:02.527 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local val=cluster1:2 2023-11-08T05:20:02.527 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_var_name=cluster 2023-11-08T05:20:02.528 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance_var_name=instance 2023-11-08T05:20:02.528 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cluster=cluster1 2023-11-08T05:20:02.528 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ instance=2 2023-11-08T05:20:02.528 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' 2 = cluster1:2 ']' 2023-11-08T05:20:02.528 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval cluster=cluster1 2023-11-08T05:20:02.529 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:20:02.529 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval instance=2 2023-11-08T05:20:02.529 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=2 2023-11-08T05:20:02.529 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-11-08T05:20:02.530 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local local_cluster=cluster1:2 2023-11-08T05:20:02.530 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster=cluster1 2023-11-08T05:20:02.530 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance 2023-11-08T05:20:02.530 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-11-08T05:20:02.530 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local val=cluster1:2 2023-11-08T05:20:02.531 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster_var_name=local_cluster 2023-11-08T05:20:02.531 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance_var_name=instance 2023-11-08T05:20:02.531 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:20:02.531 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=2 2023-11-08T05:20:02.531 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ '[' 2 = cluster1:2 ']' 2023-11-08T05:20:02.532 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval local_cluster=cluster1 2023-11-08T05:20:02.532 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ local_cluster=cluster1 2023-11-08T05:20:02.532 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval instance=2 2023-11-08T05:20:02.532 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ instance=2 2023-11-08T05:20:02.532 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-11-08T05:20:02.539 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-08T05:20:02.540 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-08T05:20:02.540 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-08T05:20:02.540 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_43 2023-11-08T05:20:02.640 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:Invalid command: unused arguments: ['mirror/image_43'] 2023-11-08T05:20:02.640 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd mirror status : get status for rbd mirror 2023-11-08T05:20:02.640 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:admin_socket: invalid command 2023-11-08T05:20:02.643 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T05:20:02.643 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_43 2023-11-08T05:20:02.644 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1:3 2023-11-08T05:20:02.644 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T05:20:02.644 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance 2023-11-08T05:20:02.644 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ set_cluster_instance cluster1:3 cluster instance 2023-11-08T05:20:02.644 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local val=cluster1:3 2023-11-08T05:20:02.645 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_var_name=cluster 2023-11-08T05:20:02.645 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance_var_name=instance 2023-11-08T05:20:02.645 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cluster=cluster1 2023-11-08T05:20:02.645 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ instance=3 2023-11-08T05:20:02.645 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' 3 = cluster1:3 ']' 2023-11-08T05:20:02.646 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval cluster=cluster1 2023-11-08T05:20:02.646 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:20:02.646 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval instance=3 2023-11-08T05:20:02.646 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=3 2023-11-08T05:20:02.647 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ daemon_asok_file cluster1:3 cluster1 2023-11-08T05:20:02.647 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local local_cluster=cluster1:3 2023-11-08T05:20:02.649 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster=cluster1 2023-11-08T05:20:02.649 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance 2023-11-08T05:20:02.650 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2023-11-08T05:20:02.650 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local val=cluster1:3 2023-11-08T05:20:02.650 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster_var_name=local_cluster 2023-11-08T05:20:02.650 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance_var_name=instance 2023-11-08T05:20:02.650 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:20:02.651 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=3 2023-11-08T05:20:02.651 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ '[' 3 = cluster1:3 ']' 2023-11-08T05:20:02.651 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval local_cluster=cluster1 2023-11-08T05:20:02.651 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ local_cluster=cluster1 2023-11-08T05:20:02.651 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval instance=3 2023-11-08T05:20:02.652 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ instance=3 2023-11-08T05:20:02.652 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2023-11-08T05:20:02.656 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-11-08T05:20:02.657 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-11-08T05:20:02.657 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-11-08T05:20:02.657 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_43 2023-11-08T05:20:02.763 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ return 0 2023-11-08T05:20:02.765 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ status_result=' "state": "Replaying"' 2023-11-08T05:20:02.765 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ echo ' "state": "Replaying"' 2023-11-08T05:20:02.765 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ grep -i Replaying 2023-11-08T05:20:02.766 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: "state": "Replaying" 2023-11-08T05:20:02.768 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ current_state=started 2023-11-08T05:20:02.768 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test started = started 2023-11-08T05:20:02.768 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:20:02.768 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_43 2023-11-08T05:20:02.769 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local local_cluster=cluster1 2023-11-08T05:20:02.769 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:20:02.769 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:20:02.769 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_43 2023-11-08T05:20:02.769 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' snapshot = journal ']' 2023-11-08T05:20:02.770 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' snapshot = snapshot ']' 2023-11-08T05:20:02.770 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_43 2023-11-08T05:20:02.770 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local local_cluster=cluster1 2023-11-08T05:20:02.771 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:20:02.771 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:20:02.771 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_43 2023-11-08T05:20:02.771 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ mkfname cluster2-mirror-image_43.status 2023-11-08T05:20:02.771 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ echo cluster2-mirror-image_43.status 2023-11-08T05:20:02.772 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -e 's|[/ ]|_|g' 2023-11-08T05:20:02.772 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local status_log=/tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_43.status 2023-11-08T05:20:02.772 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ mkfname cluster1-mirror-image_43.status 2023-11-08T05:20:02.772 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ echo cluster1-mirror-image_43.status 2023-11-08T05:20:02.772 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -e 's|[/ ]|_|g' 2023-11-08T05:20:02.773 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local local_status_log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_43.status 2023-11-08T05:20:02.774 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ mirror_image_snapshot cluster2 mirror image_43 2023-11-08T05:20:02.774 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:20:02.774 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:20:02.774 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_43 2023-11-08T05:20:02.774 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_43 2023-11-08T05:20:03.273 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:Snapshot ID: 203 2023-11-08T05:20:03.282 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_43 /tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_43.status 2023-11-08T05:20:03.282 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:20:03.282 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:20:03.282 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_43 2023-11-08T05:20:03.283 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_43.status 2023-11-08T05:20:03.283 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_43 --format xml 2023-11-08T05:20:03.283 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T05:20:03.325 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-11-08T05:20:03.327 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snapshot_id=203 2023-11-08T05:20:03.327 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ true 2023-11-08T05:20:03.328 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-11-08T05:20:03.328 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0.2 2023-11-08T05:20:03.529 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_43 /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_43.status 2023-11-08T05:20:03.529 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:20:03.529 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:20:03.529 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_43 2023-11-08T05:20:03.530 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_43.status 2023-11-08T05:20:03.530 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_43 --format xml 2023-11-08T05:20:03.530 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T05:20:03.578 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-08T05:20:03.580 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local primary_snapshot_id=110 2023-11-08T05:20:03.580 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test 203 = 110 2023-11-08T05:20:03.580 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-11-08T05:20:03.581 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0.4 2023-11-08T05:20:03.981 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_43 /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_43.status 2023-11-08T05:20:03.982 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:20:03.982 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:20:03.982 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_43 2023-11-08T05:20:03.982 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_43.status 2023-11-08T05:20:03.983 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_43 --format xml 2023-11-08T05:20:03.983 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T05:20:04.022 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ true 2023-11-08T05:20:04.023 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-08T05:20:04.024 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:-:1.1: Document is empty 2023-11-08T05:20:04.024 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: 2023-11-08T05:20:04.024 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:^ 2023-11-08T05:20:04.025 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local primary_snapshot_id= 2023-11-08T05:20:04.025 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test 203 = '' 2023-11-08T05:20:04.025 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-11-08T05:20:04.025 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0.8 2023-11-08T05:20:04.826 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_43 /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_43.status 2023-11-08T05:20:04.827 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:20:04.827 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:20:04.827 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_43 2023-11-08T05:20:04.828 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_43.status 2023-11-08T05:20:04.828 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_43 --format xml 2023-11-08T05:20:04.828 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T05:20:04.882 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-08T05:20:04.884 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local primary_snapshot_id=203 2023-11-08T05:20:04.884 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test 203 = 203 2023-11-08T05:20:04.884 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:20:04.885 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_snap_present cluster1 mirror image_43 snap 2023-11-08T05:20:04.885 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:20:04.885 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:20:04.885 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_43 2023-11-08T05:20:04.885 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap_name=snap 2023-11-08T05:20:04.886 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local s 2023-11-08T05:20:04.886 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-11-08T05:20:04.886 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 1 2023-11-08T05:20:05.885 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 info mirror/image_43@snap 2023-11-08T05:20:05.923 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:rbd image 'image_43': 2023-11-08T05:20:05.923 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: size 128 MiB in 32 objects 2023-11-08T05:20:05.923 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: order 22 (4 MiB objects) 2023-11-08T05:20:05.924 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: snapshot_count: 2 2023-11-08T05:20:05.924 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: id: 12a9dbd72094 2023-11-08T05:20:05.924 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: block_name_prefix: rbd_data.12a9dbd72094 2023-11-08T05:20:05.924 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: format: 2 2023-11-08T05:20:05.924 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: features: layering, exclusive-lock, non-primary 2023-11-08T05:20:05.925 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: op_features: 2023-11-08T05:20:05.925 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: flags: 2023-11-08T05:20:05.925 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: create_timestamp: Wed Nov 8 05:11:53 2023 2023-11-08T05:20:05.925 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: access_timestamp: Wed Nov 8 05:11:53 2023 2023-11-08T05:20:05.926 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: modify_timestamp: Wed Nov 8 05:11:53 2023 2023-11-08T05:20:05.926 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: protected: False 2023-11-08T05:20:05.926 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring state: enabled 2023-11-08T05:20:05.926 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring mode: snapshot 2023-11-08T05:20:05.927 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring global id: 0cc6b9f9-7eeb-414c-bd5f-685fa6d23f33 2023-11-08T05:20:05.927 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring primary: false 2023-11-08T05:20:05.931 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:20:05.931 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ compare_image_snaps mirror image_43 snap 2023-11-08T05:20:05.931 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:20:05.932 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_43 2023-11-08T05:20:05.932 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap_name=snap 2023-11-08T05:20:05.932 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local ret=0 2023-11-08T05:20:05.933 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local rmt_export=/tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_43.export 2023-11-08T05:20:05.933 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local loc_export=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_43.export 2023-11-08T05:20:05.933 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rm -f /tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_43.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_43.export 2023-11-08T05:20:05.934 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 -p mirror export image_43@snap /tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_43.export 2023-11-08T05:20:06.093 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-11-08T05:20:06.096 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 -p mirror export image_43@snap /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_43.export 2023-11-08T05:20:06.330 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-11-08T05:20:06.333 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ cmp /tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_43.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_43.export 2023-11-08T05:20:06.389 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rm -f /tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_43.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_43.export 2023-11-08T05:20:06.426 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:20:06.426 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-11-08T05:20:06.426 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image=image_44 2023-11-08T05:20:06.427 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ create_snap cluster2 mirror image_44 snap 2023-11-08T05:20:06.427 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:20:06.427 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:20:06.427 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_44 2023-11-08T05:20:06.427 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap_name=snap 2023-11-08T05:20:06.428 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 -p mirror snap create image_44@snap --debug-rbd=20 --debug-journaler=20 2023-11-08T05:20:08.297 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_image_replay_started cluster1 mirror image_44 2023-11-08T05:20:08.297 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:20:08.297 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:20:08.298 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_44 2023-11-08T05:20:08.298 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_image_replay_state cluster1 mirror image_44 started 2023-11-08T05:20:08.299 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:20:08.299 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:20:08.299 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_44 2023-11-08T05:20:08.300 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local state=started 2023-11-08T05:20:08.300 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local s 2023-11-08T05:20:08.300 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-11-08T05:20:08.301 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 1 2023-11-08T05:20:09.299 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test_image_replay_state cluster1 mirror image_44 started 2023-11-08T05:20:09.299 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:20:09.299 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:20:09.300 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_44 2023-11-08T05:20:09.301 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local test_state=started 2023-11-08T05:20:09.301 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local status_result 2023-11-08T05:20:09.301 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local current_state=stopped 2023-11-08T05:20:09.301 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_44 2023-11-08T05:20:09.302 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_instance=cluster1 2023-11-08T05:20:09.302 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T05:20:09.302 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ grep -i state 2023-11-08T05:20:09.302 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1 2023-11-08T05:20:09.302 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance=cluster1 2023-11-08T05:20:09.303 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local loop_instance 2023-11-08T05:20:09.303 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-11-08T05:20:09.303 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sleep 0 2023-11-08T05:20:09.306 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-11-08T05:20:09.306 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ seq 0 3 2023-11-08T05:20:09.307 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T05:20:09.307 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_44 2023-11-08T05:20:09.307 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1:0 2023-11-08T05:20:09.307 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T05:20:09.308 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance 2023-11-08T05:20:09.308 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-11-08T05:20:09.308 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local val=cluster1:0 2023-11-08T05:20:09.308 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_var_name=cluster 2023-11-08T05:20:09.309 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance_var_name=instance 2023-11-08T05:20:09.309 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cluster=cluster1 2023-11-08T05:20:09.309 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ instance=0 2023-11-08T05:20:09.309 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' 0 = cluster1:0 ']' 2023-11-08T05:20:09.310 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval cluster=cluster1 2023-11-08T05:20:09.310 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:20:09.310 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval instance=0 2023-11-08T05:20:09.310 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=0 2023-11-08T05:20:09.310 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-11-08T05:20:09.311 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local local_cluster=cluster1:0 2023-11-08T05:20:09.311 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster=cluster1 2023-11-08T05:20:09.311 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance 2023-11-08T05:20:09.311 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-11-08T05:20:09.311 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local val=cluster1:0 2023-11-08T05:20:09.312 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster_var_name=local_cluster 2023-11-08T05:20:09.312 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance_var_name=instance 2023-11-08T05:20:09.312 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:20:09.312 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=0 2023-11-08T05:20:09.312 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ '[' 0 = cluster1:0 ']' 2023-11-08T05:20:09.313 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval local_cluster=cluster1 2023-11-08T05:20:09.313 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ local_cluster=cluster1 2023-11-08T05:20:09.313 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval instance=0 2023-11-08T05:20:09.313 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ instance=0 2023-11-08T05:20:09.314 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-11-08T05:20:09.320 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-08T05:20:09.320 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-08T05:20:09.321 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-08T05:20:09.321 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_44 2023-11-08T05:20:09.428 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:Invalid command: unused arguments: ['mirror/image_44'] 2023-11-08T05:20:09.428 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd mirror status : get status for rbd mirror 2023-11-08T05:20:09.429 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:admin_socket: invalid command 2023-11-08T05:20:09.432 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T05:20:09.432 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_44 2023-11-08T05:20:09.432 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1:1 2023-11-08T05:20:09.432 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T05:20:09.433 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance 2023-11-08T05:20:09.433 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-11-08T05:20:09.433 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local val=cluster1:1 2023-11-08T05:20:09.433 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_var_name=cluster 2023-11-08T05:20:09.433 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance_var_name=instance 2023-11-08T05:20:09.434 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cluster=cluster1 2023-11-08T05:20:09.434 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ instance=1 2023-11-08T05:20:09.435 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' 1 = cluster1:1 ']' 2023-11-08T05:20:09.435 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval cluster=cluster1 2023-11-08T05:20:09.435 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:20:09.435 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval instance=1 2023-11-08T05:20:09.435 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=1 2023-11-08T05:20:09.436 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-11-08T05:20:09.436 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local local_cluster=cluster1:1 2023-11-08T05:20:09.436 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster=cluster1 2023-11-08T05:20:09.436 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance 2023-11-08T05:20:09.436 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-11-08T05:20:09.437 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local val=cluster1:1 2023-11-08T05:20:09.437 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster_var_name=local_cluster 2023-11-08T05:20:09.437 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance_var_name=instance 2023-11-08T05:20:09.437 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:20:09.438 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=1 2023-11-08T05:20:09.438 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ '[' 1 = cluster1:1 ']' 2023-11-08T05:20:09.439 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval local_cluster=cluster1 2023-11-08T05:20:09.439 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ local_cluster=cluster1 2023-11-08T05:20:09.439 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval instance=1 2023-11-08T05:20:09.439 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ instance=1 2023-11-08T05:20:09.439 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-11-08T05:20:09.446 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-08T05:20:09.446 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-08T05:20:09.446 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-08T05:20:09.447 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_44 2023-11-08T05:20:09.552 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:Invalid command: unused arguments: ['mirror/image_44'] 2023-11-08T05:20:09.552 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd mirror status : get status for rbd mirror 2023-11-08T05:20:09.553 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:admin_socket: invalid command 2023-11-08T05:20:09.558 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T05:20:09.558 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_44 2023-11-08T05:20:09.558 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1:2 2023-11-08T05:20:09.559 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T05:20:09.559 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance 2023-11-08T05:20:09.559 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-11-08T05:20:09.559 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local val=cluster1:2 2023-11-08T05:20:09.559 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_var_name=cluster 2023-11-08T05:20:09.559 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance_var_name=instance 2023-11-08T05:20:09.560 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cluster=cluster1 2023-11-08T05:20:09.560 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ instance=2 2023-11-08T05:20:09.560 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' 2 = cluster1:2 ']' 2023-11-08T05:20:09.560 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval cluster=cluster1 2023-11-08T05:20:09.560 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:20:09.561 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval instance=2 2023-11-08T05:20:09.561 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=2 2023-11-08T05:20:09.561 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-11-08T05:20:09.562 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local local_cluster=cluster1:2 2023-11-08T05:20:09.562 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster=cluster1 2023-11-08T05:20:09.562 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance 2023-11-08T05:20:09.562 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-11-08T05:20:09.563 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local val=cluster1:2 2023-11-08T05:20:09.563 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster_var_name=local_cluster 2023-11-08T05:20:09.563 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance_var_name=instance 2023-11-08T05:20:09.564 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:20:09.564 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=2 2023-11-08T05:20:09.564 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ '[' 2 = cluster1:2 ']' 2023-11-08T05:20:09.565 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval local_cluster=cluster1 2023-11-08T05:20:09.565 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ local_cluster=cluster1 2023-11-08T05:20:09.565 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval instance=2 2023-11-08T05:20:09.566 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ instance=2 2023-11-08T05:20:09.566 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-11-08T05:20:09.569 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-08T05:20:09.569 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-08T05:20:09.570 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-08T05:20:09.570 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_44 2023-11-08T05:20:09.677 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ return 0 2023-11-08T05:20:09.678 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ status_result=' "state": "Replaying"' 2023-11-08T05:20:09.678 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ echo ' "state": "Replaying"' 2023-11-08T05:20:09.678 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ grep -i Replaying 2023-11-08T05:20:09.679 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: "state": "Replaying" 2023-11-08T05:20:09.681 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ current_state=started 2023-11-08T05:20:09.681 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test started = started 2023-11-08T05:20:09.681 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:20:09.681 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_44 2023-11-08T05:20:09.681 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local local_cluster=cluster1 2023-11-08T05:20:09.682 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:20:09.682 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:20:09.682 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_44 2023-11-08T05:20:09.682 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' snapshot = journal ']' 2023-11-08T05:20:09.682 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' snapshot = snapshot ']' 2023-11-08T05:20:09.683 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_44 2023-11-08T05:20:09.683 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local local_cluster=cluster1 2023-11-08T05:20:09.683 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:20:09.683 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:20:09.683 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_44 2023-11-08T05:20:09.684 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ mkfname cluster2-mirror-image_44.status 2023-11-08T05:20:09.684 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ echo cluster2-mirror-image_44.status 2023-11-08T05:20:09.684 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -e 's|[/ ]|_|g' 2023-11-08T05:20:09.684 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local status_log=/tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_44.status 2023-11-08T05:20:09.684 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ mkfname cluster1-mirror-image_44.status 2023-11-08T05:20:09.685 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ echo cluster1-mirror-image_44.status 2023-11-08T05:20:09.686 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -e 's|[/ ]|_|g' 2023-11-08T05:20:09.686 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local local_status_log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_44.status 2023-11-08T05:20:09.687 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ mirror_image_snapshot cluster2 mirror image_44 2023-11-08T05:20:09.687 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:20:09.687 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:20:09.687 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_44 2023-11-08T05:20:09.687 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_44 2023-11-08T05:20:10.297 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:Snapshot ID: 205 2023-11-08T05:20:10.307 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_44 /tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_44.status 2023-11-08T05:20:10.307 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:20:10.308 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:20:10.308 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_44 2023-11-08T05:20:10.308 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_44.status 2023-11-08T05:20:10.309 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_44 --format xml 2023-11-08T05:20:10.309 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T05:20:10.344 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-11-08T05:20:10.346 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snapshot_id=205 2023-11-08T05:20:10.346 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ true 2023-11-08T05:20:10.346 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-11-08T05:20:10.346 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0.2 2023-11-08T05:20:10.547 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_44 /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_44.status 2023-11-08T05:20:10.547 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:20:10.548 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:20:10.548 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_44 2023-11-08T05:20:10.548 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_44.status 2023-11-08T05:20:10.548 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_44 --format xml 2023-11-08T05:20:10.548 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T05:20:10.593 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ true 2023-11-08T05:20:10.593 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-08T05:20:10.594 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:-:1.1: Document is empty 2023-11-08T05:20:10.595 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: 2023-11-08T05:20:10.595 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:^ 2023-11-08T05:20:10.595 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local primary_snapshot_id= 2023-11-08T05:20:10.595 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test 205 = '' 2023-11-08T05:20:10.596 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-11-08T05:20:10.596 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0.4 2023-11-08T05:20:10.996 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_44 /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_44.status 2023-11-08T05:20:10.996 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:20:10.997 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:20:10.997 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_44 2023-11-08T05:20:10.997 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_44.status 2023-11-08T05:20:10.997 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_44 --format xml 2023-11-08T05:20:10.997 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T05:20:11.035 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ true 2023-11-08T05:20:11.035 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-08T05:20:11.036 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:-:1.1: Document is empty 2023-11-08T05:20:11.037 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: 2023-11-08T05:20:11.037 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:^ 2023-11-08T05:20:11.038 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local primary_snapshot_id= 2023-11-08T05:20:11.038 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test 205 = '' 2023-11-08T05:20:11.038 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-11-08T05:20:11.038 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0.8 2023-11-08T05:20:11.838 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_44 /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_44.status 2023-11-08T05:20:11.839 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:20:11.839 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:20:11.839 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_44 2023-11-08T05:20:11.839 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_44.status 2023-11-08T05:20:11.840 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_44 --format xml 2023-11-08T05:20:11.840 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T05:20:11.880 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ true 2023-11-08T05:20:11.881 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-08T05:20:11.882 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:-:1.1: Document is empty 2023-11-08T05:20:11.882 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: 2023-11-08T05:20:11.882 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:^ 2023-11-08T05:20:11.883 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local primary_snapshot_id= 2023-11-08T05:20:11.883 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test 205 = '' 2023-11-08T05:20:11.883 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-11-08T05:20:11.883 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 1.6 2023-11-08T05:20:13.484 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_44 /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_44.status 2023-11-08T05:20:13.484 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:20:13.485 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:20:13.485 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_44 2023-11-08T05:20:13.485 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_44.status 2023-11-08T05:20:13.485 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_44 --format xml 2023-11-08T05:20:13.485 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T05:20:13.528 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-08T05:20:13.530 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local primary_snapshot_id=205 2023-11-08T05:20:13.531 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test 205 = 205 2023-11-08T05:20:13.531 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:20:13.531 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_snap_present cluster1 mirror image_44 snap 2023-11-08T05:20:13.531 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:20:13.531 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:20:13.532 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_44 2023-11-08T05:20:13.532 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap_name=snap 2023-11-08T05:20:13.532 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local s 2023-11-08T05:20:13.532 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-11-08T05:20:13.532 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 1 2023-11-08T05:20:14.532 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 info mirror/image_44@snap 2023-11-08T05:20:14.571 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:rbd image 'image_44': 2023-11-08T05:20:14.572 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: size 128 MiB in 32 objects 2023-11-08T05:20:14.572 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: order 22 (4 MiB objects) 2023-11-08T05:20:14.572 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: snapshot_count: 2 2023-11-08T05:20:14.572 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: id: 12b350d23e7 2023-11-08T05:20:14.573 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: block_name_prefix: rbd_data.12b350d23e7 2023-11-08T05:20:14.573 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: format: 2 2023-11-08T05:20:14.573 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: features: layering, exclusive-lock, non-primary 2023-11-08T05:20:14.573 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: op_features: 2023-11-08T05:20:14.573 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: flags: 2023-11-08T05:20:14.574 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: create_timestamp: Wed Nov 8 05:12:06 2023 2023-11-08T05:20:14.574 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: access_timestamp: Wed Nov 8 05:12:06 2023 2023-11-08T05:20:14.574 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: modify_timestamp: Wed Nov 8 05:12:06 2023 2023-11-08T05:20:14.575 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: protected: False 2023-11-08T05:20:14.575 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring state: enabled 2023-11-08T05:20:14.575 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring mode: snapshot 2023-11-08T05:20:14.575 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring global id: 211c5392-1879-4787-844d-06dee2447ed6 2023-11-08T05:20:14.576 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring primary: false 2023-11-08T05:20:14.578 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:20:14.579 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ compare_image_snaps mirror image_44 snap 2023-11-08T05:20:14.579 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:20:14.580 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_44 2023-11-08T05:20:14.580 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap_name=snap 2023-11-08T05:20:14.580 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local ret=0 2023-11-08T05:20:14.581 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local rmt_export=/tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_44.export 2023-11-08T05:20:14.581 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local loc_export=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_44.export 2023-11-08T05:20:14.581 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rm -f /tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_44.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_44.export 2023-11-08T05:20:14.581 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 -p mirror export image_44@snap /tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_44.export 2023-11-08T05:20:14.752 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-11-08T05:20:14.755 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 -p mirror export image_44@snap /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_44.export 2023-11-08T05:20:14.990 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-11-08T05:20:14.992 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ cmp /tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_44.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_44.export 2023-11-08T05:20:15.047 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rm -f /tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_44.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_44.export 2023-11-08T05:20:15.085 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:20:15.085 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-11-08T05:20:15.086 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image=image_45 2023-11-08T05:20:15.086 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ create_snap cluster2 mirror image_45 snap 2023-11-08T05:20:15.086 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:20:15.086 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:20:15.086 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_45 2023-11-08T05:20:15.087 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap_name=snap 2023-11-08T05:20:15.087 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 -p mirror snap create image_45@snap --debug-rbd=20 --debug-journaler=20 2023-11-08T05:20:17.314 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_image_replay_started cluster1 mirror image_45 2023-11-08T05:20:17.314 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:20:17.314 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:20:17.315 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_45 2023-11-08T05:20:17.315 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_image_replay_state cluster1 mirror image_45 started 2023-11-08T05:20:17.315 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:20:17.315 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:20:17.315 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_45 2023-11-08T05:20:17.316 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local state=started 2023-11-08T05:20:17.316 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local s 2023-11-08T05:20:17.316 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-11-08T05:20:17.316 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 1 2023-11-08T05:20:18.315 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test_image_replay_state cluster1 mirror image_45 started 2023-11-08T05:20:18.315 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:20:18.315 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:20:18.316 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_45 2023-11-08T05:20:18.316 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local test_state=started 2023-11-08T05:20:18.316 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local status_result 2023-11-08T05:20:18.316 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local current_state=stopped 2023-11-08T05:20:18.317 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_45 2023-11-08T05:20:18.318 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ grep -i state 2023-11-08T05:20:18.318 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_instance=cluster1 2023-11-08T05:20:18.318 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T05:20:18.318 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1 2023-11-08T05:20:18.318 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance=cluster1 2023-11-08T05:20:18.319 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local loop_instance 2023-11-08T05:20:18.319 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-11-08T05:20:18.319 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sleep 0 2023-11-08T05:20:18.319 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-11-08T05:20:18.319 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ seq 0 3 2023-11-08T05:20:18.322 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T05:20:18.322 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_45 2023-11-08T05:20:18.323 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1:0 2023-11-08T05:20:18.323 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T05:20:18.323 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance 2023-11-08T05:20:18.323 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-11-08T05:20:18.324 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local val=cluster1:0 2023-11-08T05:20:18.324 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_var_name=cluster 2023-11-08T05:20:18.324 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance_var_name=instance 2023-11-08T05:20:18.324 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cluster=cluster1 2023-11-08T05:20:18.324 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ instance=0 2023-11-08T05:20:18.325 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' 0 = cluster1:0 ']' 2023-11-08T05:20:18.325 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval cluster=cluster1 2023-11-08T05:20:18.325 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:20:18.325 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval instance=0 2023-11-08T05:20:18.325 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=0 2023-11-08T05:20:18.326 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-11-08T05:20:18.326 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local local_cluster=cluster1:0 2023-11-08T05:20:18.326 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster=cluster1 2023-11-08T05:20:18.326 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance 2023-11-08T05:20:18.326 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-11-08T05:20:18.327 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local val=cluster1:0 2023-11-08T05:20:18.327 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster_var_name=local_cluster 2023-11-08T05:20:18.327 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance_var_name=instance 2023-11-08T05:20:18.327 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:20:18.327 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=0 2023-11-08T05:20:18.328 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ '[' 0 = cluster1:0 ']' 2023-11-08T05:20:18.328 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval local_cluster=cluster1 2023-11-08T05:20:18.328 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ local_cluster=cluster1 2023-11-08T05:20:18.328 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval instance=0 2023-11-08T05:20:18.328 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ instance=0 2023-11-08T05:20:18.329 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-11-08T05:20:18.336 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-08T05:20:18.337 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-08T05:20:18.337 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-08T05:20:18.337 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_45 2023-11-08T05:20:18.443 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:Invalid command: unused arguments: ['mirror/image_45'] 2023-11-08T05:20:18.443 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd mirror status : get status for rbd mirror 2023-11-08T05:20:18.443 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:admin_socket: invalid command 2023-11-08T05:20:18.446 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T05:20:18.446 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_45 2023-11-08T05:20:18.447 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1:1 2023-11-08T05:20:18.447 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T05:20:18.447 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance 2023-11-08T05:20:18.447 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-11-08T05:20:18.447 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local val=cluster1:1 2023-11-08T05:20:18.448 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_var_name=cluster 2023-11-08T05:20:18.448 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance_var_name=instance 2023-11-08T05:20:18.448 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cluster=cluster1 2023-11-08T05:20:18.448 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ instance=1 2023-11-08T05:20:18.448 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' 1 = cluster1:1 ']' 2023-11-08T05:20:18.449 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval cluster=cluster1 2023-11-08T05:20:18.449 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:20:18.449 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval instance=1 2023-11-08T05:20:18.449 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=1 2023-11-08T05:20:18.449 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-11-08T05:20:18.450 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local local_cluster=cluster1:1 2023-11-08T05:20:18.450 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster=cluster1 2023-11-08T05:20:18.450 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance 2023-11-08T05:20:18.450 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-11-08T05:20:18.450 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local val=cluster1:1 2023-11-08T05:20:18.451 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster_var_name=local_cluster 2023-11-08T05:20:18.451 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance_var_name=instance 2023-11-08T05:20:18.451 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:20:18.451 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=1 2023-11-08T05:20:18.451 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ '[' 1 = cluster1:1 ']' 2023-11-08T05:20:18.451 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval local_cluster=cluster1 2023-11-08T05:20:18.452 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ local_cluster=cluster1 2023-11-08T05:20:18.452 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval instance=1 2023-11-08T05:20:18.452 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ instance=1 2023-11-08T05:20:18.452 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-11-08T05:20:18.460 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-08T05:20:18.460 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-08T05:20:18.461 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-08T05:20:18.461 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_45 2023-11-08T05:20:18.564 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:Invalid command: unused arguments: ['mirror/image_45'] 2023-11-08T05:20:18.564 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd mirror status : get status for rbd mirror 2023-11-08T05:20:18.564 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:admin_socket: invalid command 2023-11-08T05:20:18.567 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T05:20:18.567 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_45 2023-11-08T05:20:18.567 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1:2 2023-11-08T05:20:18.567 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T05:20:18.568 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance 2023-11-08T05:20:18.568 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-11-08T05:20:18.568 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local val=cluster1:2 2023-11-08T05:20:18.568 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_var_name=cluster 2023-11-08T05:20:18.568 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance_var_name=instance 2023-11-08T05:20:18.569 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cluster=cluster1 2023-11-08T05:20:18.569 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ instance=2 2023-11-08T05:20:18.569 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' 2 = cluster1:2 ']' 2023-11-08T05:20:18.569 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval cluster=cluster1 2023-11-08T05:20:18.569 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:20:18.570 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval instance=2 2023-11-08T05:20:18.570 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=2 2023-11-08T05:20:18.570 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-11-08T05:20:18.570 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local local_cluster=cluster1:2 2023-11-08T05:20:18.570 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster=cluster1 2023-11-08T05:20:18.571 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance 2023-11-08T05:20:18.571 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-11-08T05:20:18.571 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local val=cluster1:2 2023-11-08T05:20:18.571 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster_var_name=local_cluster 2023-11-08T05:20:18.572 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance_var_name=instance 2023-11-08T05:20:18.572 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:20:18.572 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=2 2023-11-08T05:20:18.572 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ '[' 2 = cluster1:2 ']' 2023-11-08T05:20:18.572 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval local_cluster=cluster1 2023-11-08T05:20:18.573 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ local_cluster=cluster1 2023-11-08T05:20:18.573 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval instance=2 2023-11-08T05:20:18.573 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ instance=2 2023-11-08T05:20:18.573 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-11-08T05:20:18.581 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-08T05:20:18.581 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-08T05:20:18.582 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-08T05:20:18.582 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_45 2023-11-08T05:20:18.681 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:Invalid command: unused arguments: ['mirror/image_45'] 2023-11-08T05:20:18.682 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd mirror status : get status for rbd mirror 2023-11-08T05:20:18.682 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:admin_socket: invalid command 2023-11-08T05:20:18.685 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T05:20:18.685 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_45 2023-11-08T05:20:18.685 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1:3 2023-11-08T05:20:18.685 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T05:20:18.686 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance 2023-11-08T05:20:18.686 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ set_cluster_instance cluster1:3 cluster instance 2023-11-08T05:20:18.686 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local val=cluster1:3 2023-11-08T05:20:18.686 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_var_name=cluster 2023-11-08T05:20:18.687 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance_var_name=instance 2023-11-08T05:20:18.687 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cluster=cluster1 2023-11-08T05:20:18.687 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ instance=3 2023-11-08T05:20:18.687 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' 3 = cluster1:3 ']' 2023-11-08T05:20:18.687 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval cluster=cluster1 2023-11-08T05:20:18.688 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:20:18.688 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval instance=3 2023-11-08T05:20:18.688 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=3 2023-11-08T05:20:18.688 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ daemon_asok_file cluster1:3 cluster1 2023-11-08T05:20:18.688 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local local_cluster=cluster1:3 2023-11-08T05:20:18.688 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster=cluster1 2023-11-08T05:20:18.689 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance 2023-11-08T05:20:18.689 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2023-11-08T05:20:18.689 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local val=cluster1:3 2023-11-08T05:20:18.689 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster_var_name=local_cluster 2023-11-08T05:20:18.689 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance_var_name=instance 2023-11-08T05:20:18.690 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:20:18.690 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=3 2023-11-08T05:20:18.690 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ '[' 3 = cluster1:3 ']' 2023-11-08T05:20:18.690 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval local_cluster=cluster1 2023-11-08T05:20:18.690 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ local_cluster=cluster1 2023-11-08T05:20:18.691 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval instance=3 2023-11-08T05:20:18.691 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ instance=3 2023-11-08T05:20:18.691 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2023-11-08T05:20:18.698 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-11-08T05:20:18.699 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-11-08T05:20:18.699 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-11-08T05:20:18.699 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_45 2023-11-08T05:20:18.806 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ return 0 2023-11-08T05:20:18.807 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ status_result=' "state": "Replaying"' 2023-11-08T05:20:18.807 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ echo ' "state": "Replaying"' 2023-11-08T05:20:18.807 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ grep -i Replaying 2023-11-08T05:20:18.808 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: "state": "Replaying" 2023-11-08T05:20:18.810 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ current_state=started 2023-11-08T05:20:18.810 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test started = started 2023-11-08T05:20:18.810 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:20:18.810 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_45 2023-11-08T05:20:18.811 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local local_cluster=cluster1 2023-11-08T05:20:18.811 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:20:18.811 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:20:18.811 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_45 2023-11-08T05:20:18.811 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' snapshot = journal ']' 2023-11-08T05:20:18.812 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' snapshot = snapshot ']' 2023-11-08T05:20:18.812 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_45 2023-11-08T05:20:18.812 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local local_cluster=cluster1 2023-11-08T05:20:18.812 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:20:18.812 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:20:18.813 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_45 2023-11-08T05:20:18.813 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ mkfname cluster2-mirror-image_45.status 2023-11-08T05:20:18.813 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ echo cluster2-mirror-image_45.status 2023-11-08T05:20:18.813 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -e 's|[/ ]|_|g' 2023-11-08T05:20:18.813 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local status_log=/tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_45.status 2023-11-08T05:20:18.813 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ mkfname cluster1-mirror-image_45.status 2023-11-08T05:20:18.814 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ echo cluster1-mirror-image_45.status 2023-11-08T05:20:18.814 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -e 's|[/ ]|_|g' 2023-11-08T05:20:18.815 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local local_status_log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_45.status 2023-11-08T05:20:18.815 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ mirror_image_snapshot cluster2 mirror image_45 2023-11-08T05:20:18.815 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:20:18.815 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:20:18.816 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_45 2023-11-08T05:20:18.816 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_45 2023-11-08T05:20:19.324 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:Snapshot ID: 207 2023-11-08T05:20:19.333 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_45 /tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_45.status 2023-11-08T05:20:19.333 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:20:19.333 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:20:19.333 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_45 2023-11-08T05:20:19.333 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_45.status 2023-11-08T05:20:19.334 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_45 --format xml 2023-11-08T05:20:19.334 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T05:20:19.371 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-11-08T05:20:19.372 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snapshot_id=207 2023-11-08T05:20:19.373 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ true 2023-11-08T05:20:19.373 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-11-08T05:20:19.373 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0.2 2023-11-08T05:20:19.574 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_45 /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_45.status 2023-11-08T05:20:19.574 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:20:19.574 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:20:19.575 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_45 2023-11-08T05:20:19.575 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_45.status 2023-11-08T05:20:19.575 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_45 --format xml 2023-11-08T05:20:19.575 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T05:20:19.613 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ true 2023-11-08T05:20:19.613 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-08T05:20:19.614 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:-:1.1: Document is empty 2023-11-08T05:20:19.614 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: 2023-11-08T05:20:19.614 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:^ 2023-11-08T05:20:19.615 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local primary_snapshot_id= 2023-11-08T05:20:19.615 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test 207 = '' 2023-11-08T05:20:19.615 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-11-08T05:20:19.615 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0.4 2023-11-08T05:20:20.016 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_45 /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_45.status 2023-11-08T05:20:20.016 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:20:20.016 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:20:20.017 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_45 2023-11-08T05:20:20.017 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_45.status 2023-11-08T05:20:20.017 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_45 --format xml 2023-11-08T05:20:20.017 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T05:20:20.055 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ true 2023-11-08T05:20:20.055 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-08T05:20:20.056 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:-:1.1: Document is empty 2023-11-08T05:20:20.057 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: 2023-11-08T05:20:20.057 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:^ 2023-11-08T05:20:20.058 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local primary_snapshot_id= 2023-11-08T05:20:20.058 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test 207 = '' 2023-11-08T05:20:20.058 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-11-08T05:20:20.058 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0.8 2023-11-08T05:20:20.859 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_45 /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_45.status 2023-11-08T05:20:20.859 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:20:20.859 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:20:20.859 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_45 2023-11-08T05:20:20.860 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_45.status 2023-11-08T05:20:20.860 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_45 --format xml 2023-11-08T05:20:20.860 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T05:20:20.916 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-08T05:20:20.918 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local primary_snapshot_id=207 2023-11-08T05:20:20.918 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test 207 = 207 2023-11-08T05:20:20.918 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:20:20.919 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_snap_present cluster1 mirror image_45 snap 2023-11-08T05:20:20.919 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:20:20.919 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:20:20.919 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_45 2023-11-08T05:20:20.919 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap_name=snap 2023-11-08T05:20:20.920 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local s 2023-11-08T05:20:20.920 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-11-08T05:20:20.920 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 1 2023-11-08T05:20:21.919 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 info mirror/image_45@snap 2023-11-08T05:20:21.957 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:rbd image 'image_45': 2023-11-08T05:20:21.957 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: size 128 MiB in 32 objects 2023-11-08T05:20:21.957 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: order 22 (4 MiB objects) 2023-11-08T05:20:21.957 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: snapshot_count: 2 2023-11-08T05:20:21.958 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: id: 11d8b60b8243 2023-11-08T05:20:21.958 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: block_name_prefix: rbd_data.11d8b60b8243 2023-11-08T05:20:21.958 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: format: 2 2023-11-08T05:20:21.958 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: features: layering, exclusive-lock, non-primary 2023-11-08T05:20:21.958 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: op_features: 2023-11-08T05:20:21.959 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: flags: 2023-11-08T05:20:21.959 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: create_timestamp: Wed Nov 8 05:12:20 2023 2023-11-08T05:20:21.959 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: access_timestamp: Wed Nov 8 05:12:20 2023 2023-11-08T05:20:21.959 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: modify_timestamp: Wed Nov 8 05:12:20 2023 2023-11-08T05:20:21.959 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: protected: False 2023-11-08T05:20:21.960 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring state: enabled 2023-11-08T05:20:21.960 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring mode: snapshot 2023-11-08T05:20:21.960 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring global id: 20b7d4e5-6730-4919-b471-a15668fab78b 2023-11-08T05:20:21.960 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring primary: false 2023-11-08T05:20:21.961 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:20:21.962 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ compare_image_snaps mirror image_45 snap 2023-11-08T05:20:21.962 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:20:21.962 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_45 2023-11-08T05:20:21.962 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap_name=snap 2023-11-08T05:20:21.962 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local ret=0 2023-11-08T05:20:21.963 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local rmt_export=/tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_45.export 2023-11-08T05:20:21.963 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local loc_export=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_45.export 2023-11-08T05:20:21.963 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rm -f /tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_45.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_45.export 2023-11-08T05:20:21.963 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 -p mirror export image_45@snap /tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_45.export 2023-11-08T05:20:22.144 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-11-08T05:20:22.147 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 -p mirror export image_45@snap /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_45.export 2023-11-08T05:20:22.377 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-11-08T05:20:22.380 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ cmp /tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_45.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_45.export 2023-11-08T05:20:22.435 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rm -f /tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_45.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_45.export 2023-11-08T05:20:22.472 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:20:22.472 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-11-08T05:20:22.472 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image=image_46 2023-11-08T05:20:22.473 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ create_snap cluster2 mirror image_46 snap 2023-11-08T05:20:22.473 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:20:22.473 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:20:22.473 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_46 2023-11-08T05:20:22.473 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap_name=snap 2023-11-08T05:20:22.474 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 -p mirror snap create image_46@snap --debug-rbd=20 --debug-journaler=20 2023-11-08T05:20:24.341 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_image_replay_started cluster1 mirror image_46 2023-11-08T05:20:24.342 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:20:24.342 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:20:24.342 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_46 2023-11-08T05:20:24.342 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_image_replay_state cluster1 mirror image_46 started 2023-11-08T05:20:24.342 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:20:24.343 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:20:24.343 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_46 2023-11-08T05:20:24.343 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local state=started 2023-11-08T05:20:24.343 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local s 2023-11-08T05:20:24.343 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-11-08T05:20:24.344 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 1 2023-11-08T05:20:25.343 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test_image_replay_state cluster1 mirror image_46 started 2023-11-08T05:20:25.343 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:20:25.344 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:20:25.344 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_46 2023-11-08T05:20:25.344 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local test_state=started 2023-11-08T05:20:25.344 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local status_result 2023-11-08T05:20:25.344 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local current_state=stopped 2023-11-08T05:20:25.345 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_46 2023-11-08T05:20:25.345 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_instance=cluster1 2023-11-08T05:20:25.345 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ grep -i state 2023-11-08T05:20:25.345 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T05:20:25.345 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1 2023-11-08T05:20:25.346 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance=cluster1 2023-11-08T05:20:25.346 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local loop_instance 2023-11-08T05:20:25.346 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-11-08T05:20:25.346 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sleep 0 2023-11-08T05:20:25.348 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-11-08T05:20:25.348 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ seq 0 3 2023-11-08T05:20:25.349 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T05:20:25.349 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_46 2023-11-08T05:20:25.349 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1:0 2023-11-08T05:20:25.349 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T05:20:25.349 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance 2023-11-08T05:20:25.350 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-11-08T05:20:25.350 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local val=cluster1:0 2023-11-08T05:20:25.350 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_var_name=cluster 2023-11-08T05:20:25.350 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance_var_name=instance 2023-11-08T05:20:25.350 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cluster=cluster1 2023-11-08T05:20:25.351 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ instance=0 2023-11-08T05:20:25.351 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' 0 = cluster1:0 ']' 2023-11-08T05:20:25.351 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval cluster=cluster1 2023-11-08T05:20:25.351 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:20:25.351 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval instance=0 2023-11-08T05:20:25.351 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=0 2023-11-08T05:20:25.352 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-11-08T05:20:25.353 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local local_cluster=cluster1:0 2023-11-08T05:20:25.353 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster=cluster1 2023-11-08T05:20:25.354 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance 2023-11-08T05:20:25.354 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-11-08T05:20:25.354 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local val=cluster1:0 2023-11-08T05:20:25.354 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster_var_name=local_cluster 2023-11-08T05:20:25.354 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance_var_name=instance 2023-11-08T05:20:25.355 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:20:25.355 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=0 2023-11-08T05:20:25.355 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ '[' 0 = cluster1:0 ']' 2023-11-08T05:20:25.355 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval local_cluster=cluster1 2023-11-08T05:20:25.355 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ local_cluster=cluster1 2023-11-08T05:20:25.356 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval instance=0 2023-11-08T05:20:25.356 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ instance=0 2023-11-08T05:20:25.356 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-11-08T05:20:25.364 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-08T05:20:25.365 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-08T05:20:25.365 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-08T05:20:25.365 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_46 2023-11-08T05:20:25.482 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ return 0 2023-11-08T05:20:25.483 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ status_result=' "state": "Replaying"' 2023-11-08T05:20:25.483 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ echo ' "state": "Replaying"' 2023-11-08T05:20:25.483 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ grep -i Replaying 2023-11-08T05:20:25.484 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: "state": "Replaying" 2023-11-08T05:20:25.486 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ current_state=started 2023-11-08T05:20:25.486 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test started = started 2023-11-08T05:20:25.486 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:20:25.486 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_46 2023-11-08T05:20:25.487 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local local_cluster=cluster1 2023-11-08T05:20:25.487 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:20:25.487 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:20:25.487 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_46 2023-11-08T05:20:25.487 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' snapshot = journal ']' 2023-11-08T05:20:25.488 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' snapshot = snapshot ']' 2023-11-08T05:20:25.488 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_46 2023-11-08T05:20:25.488 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local local_cluster=cluster1 2023-11-08T05:20:25.488 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:20:25.488 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:20:25.489 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_46 2023-11-08T05:20:25.489 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ mkfname cluster2-mirror-image_46.status 2023-11-08T05:20:25.489 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ echo cluster2-mirror-image_46.status 2023-11-08T05:20:25.489 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -e 's|[/ ]|_|g' 2023-11-08T05:20:25.489 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local status_log=/tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_46.status 2023-11-08T05:20:25.490 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ mkfname cluster1-mirror-image_46.status 2023-11-08T05:20:25.490 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ echo cluster1-mirror-image_46.status 2023-11-08T05:20:25.490 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -e 's|[/ ]|_|g' 2023-11-08T05:20:25.491 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local local_status_log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_46.status 2023-11-08T05:20:25.491 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ mirror_image_snapshot cluster2 mirror image_46 2023-11-08T05:20:25.491 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:20:25.491 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:20:25.492 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_46 2023-11-08T05:20:25.492 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_46 2023-11-08T05:20:26.347 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:Snapshot ID: 209 2023-11-08T05:20:26.356 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_46 /tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_46.status 2023-11-08T05:20:26.357 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:20:26.357 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:20:26.357 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_46 2023-11-08T05:20:26.357 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_46.status 2023-11-08T05:20:26.357 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_46 --format xml 2023-11-08T05:20:26.358 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T05:20:26.397 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-11-08T05:20:26.399 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snapshot_id=209 2023-11-08T05:20:26.399 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ true 2023-11-08T05:20:26.399 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-11-08T05:20:26.400 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0.2 2023-11-08T05:20:26.600 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_46 /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_46.status 2023-11-08T05:20:26.601 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:20:26.601 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:20:26.601 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_46 2023-11-08T05:20:26.601 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_46.status 2023-11-08T05:20:26.602 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_46 --format xml 2023-11-08T05:20:26.602 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T05:20:26.640 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ true 2023-11-08T05:20:26.640 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-08T05:20:26.641 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:-:1.1: Document is empty 2023-11-08T05:20:26.641 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: 2023-11-08T05:20:26.641 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:^ 2023-11-08T05:20:26.642 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local primary_snapshot_id= 2023-11-08T05:20:26.642 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test 209 = '' 2023-11-08T05:20:26.643 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-11-08T05:20:26.643 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0.4 2023-11-08T05:20:27.043 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_46 /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_46.status 2023-11-08T05:20:27.044 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:20:27.044 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:20:27.044 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_46 2023-11-08T05:20:27.044 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_46.status 2023-11-08T05:20:27.044 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_46 --format xml 2023-11-08T05:20:27.045 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T05:20:27.084 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ true 2023-11-08T05:20:27.084 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-08T05:20:27.085 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:-:1.1: Document is empty 2023-11-08T05:20:27.086 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: 2023-11-08T05:20:27.086 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:^ 2023-11-08T05:20:27.087 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local primary_snapshot_id= 2023-11-08T05:20:27.087 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test 209 = '' 2023-11-08T05:20:27.087 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-11-08T05:20:27.087 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0.8 2023-11-08T05:20:27.888 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_46 /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_46.status 2023-11-08T05:20:27.888 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:20:27.888 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:20:27.889 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_46 2023-11-08T05:20:27.889 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_46.status 2023-11-08T05:20:27.889 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_46 --format xml 2023-11-08T05:20:27.889 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T05:20:27.926 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-08T05:20:27.929 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local primary_snapshot_id=209 2023-11-08T05:20:27.929 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test 209 = 209 2023-11-08T05:20:27.929 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:20:27.929 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_snap_present cluster1 mirror image_46 snap 2023-11-08T05:20:27.929 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:20:27.930 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:20:27.930 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_46 2023-11-08T05:20:27.930 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap_name=snap 2023-11-08T05:20:27.930 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local s 2023-11-08T05:20:27.930 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-11-08T05:20:27.931 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 1 2023-11-08T05:20:28.659 INFO:tasks.ceph.cluster1.osd.2.smithi077.stderr:2023-11-08T05:20:28.658+0000 7f9900abf700 -1 reset not still connected to 0x555e8225a8f0 2023-11-08T05:20:28.660 INFO:tasks.ceph.cluster1.osd.2.smithi077.stderr:2023-11-08T05:20:28.658+0000 7f9900abf700 -1 reset not still connected to 0x555e974f1ba0 2023-11-08T05:20:28.660 INFO:tasks.ceph.cluster1.osd.2.smithi077.stderr:2023-11-08T05:20:28.658+0000 7f9900abf700 -1 reset not still connected to 0x555e97a1d520 2023-11-08T05:20:28.660 INFO:tasks.ceph.cluster1.osd.2.smithi077.stderr:2023-11-08T05:20:28.658+0000 7f9900abf700 -1 reset not still connected to 0x555e99341a00 2023-11-08T05:20:28.929 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 info mirror/image_46@snap 2023-11-08T05:20:28.968 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:rbd image 'image_46': 2023-11-08T05:20:28.968 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: size 128 MiB in 32 objects 2023-11-08T05:20:28.968 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: order 22 (4 MiB objects) 2023-11-08T05:20:28.969 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: snapshot_count: 2 2023-11-08T05:20:28.969 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: id: 12c3781dd12d 2023-11-08T05:20:28.969 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: block_name_prefix: rbd_data.12c3781dd12d 2023-11-08T05:20:28.969 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: format: 2 2023-11-08T05:20:28.969 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: features: layering, exclusive-lock, non-primary 2023-11-08T05:20:28.970 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: op_features: 2023-11-08T05:20:28.970 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: flags: 2023-11-08T05:20:28.970 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: create_timestamp: Wed Nov 8 05:12:31 2023 2023-11-08T05:20:28.970 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: access_timestamp: Wed Nov 8 05:12:31 2023 2023-11-08T05:20:28.970 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: modify_timestamp: Wed Nov 8 05:12:31 2023 2023-11-08T05:20:28.971 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: protected: False 2023-11-08T05:20:28.971 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring state: enabled 2023-11-08T05:20:28.971 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring mode: snapshot 2023-11-08T05:20:28.971 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring global id: ab12d762-6014-4867-a6ea-6c7d605eb202 2023-11-08T05:20:28.971 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring primary: false 2023-11-08T05:20:28.973 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:20:28.973 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ compare_image_snaps mirror image_46 snap 2023-11-08T05:20:28.973 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:20:28.973 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_46 2023-11-08T05:20:28.973 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap_name=snap 2023-11-08T05:20:28.974 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local ret=0 2023-11-08T05:20:28.974 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local rmt_export=/tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_46.export 2023-11-08T05:20:28.974 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local loc_export=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_46.export 2023-11-08T05:20:28.974 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rm -f /tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_46.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_46.export 2023-11-08T05:20:28.975 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 -p mirror export image_46@snap /tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_46.export 2023-11-08T05:20:29.110 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.346144505423309) > weight (0.25) 2023-11-08T05:20:29.111 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.1 2023-11-08T05:20:29.111 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Sent signal 15 2023-11-08T05:20:29.112 INFO:tasks.rbd_mirror_thrash:waiting for 3 secs before reviving daemons 2023-11-08T05:20:29.112 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi087.stderr:2023-11-08T05:20:29.110+0000 7f6552b5f700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 (PID: 142001) UID: 1000 2023-11-08T05:20:29.151 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-11-08T05:20:29.155 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 -p mirror export image_46@snap /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_46.export 2023-11-08T05:20:29.312 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi087.stderr:daemon-helper: command failed with exit status 1 2023-11-08T05:20:29.383 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-11-08T05:20:29.386 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ cmp /tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_46.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_46.export 2023-11-08T05:20:29.441 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rm -f /tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_46.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_46.export 2023-11-08T05:20:29.479 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:20:29.479 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-11-08T05:20:29.480 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image=image_47 2023-11-08T05:20:29.480 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ create_snap cluster2 mirror image_47 snap 2023-11-08T05:20:29.480 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:20:29.480 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:20:29.480 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_47 2023-11-08T05:20:29.481 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap_name=snap 2023-11-08T05:20:29.481 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 -p mirror snap create image_47@snap --debug-rbd=20 --debug-journaler=20 2023-11-08T05:20:31.349 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_image_replay_started cluster1 mirror image_47 2023-11-08T05:20:31.350 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:20:31.350 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:20:31.350 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_47 2023-11-08T05:20:31.350 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_image_replay_state cluster1 mirror image_47 started 2023-11-08T05:20:31.351 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:20:31.351 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:20:31.351 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_47 2023-11-08T05:20:31.351 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local state=started 2023-11-08T05:20:31.351 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local s 2023-11-08T05:20:31.351 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-11-08T05:20:31.352 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 1 2023-11-08T05:20:32.113 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.1 2023-11-08T05:20:32.113 INFO:teuthology.orchestra.run:waiting for 600 2023-11-08T05:20:32.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-08T05:20:32.114 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.1 2023-11-08T05:20:32.114 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Restarting daemon 2023-11-08T05:20:32.115 DEBUG:teuthology.orchestra.run.smithi087:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 2023-11-08T05:20:32.118 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Started 2023-11-08T05:20:32.118 INFO:tasks.rbd_mirror_thrash:waiting for 86 secs before thrashing 2023-11-08T05:20:32.352 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test_image_replay_state cluster1 mirror image_47 started 2023-11-08T05:20:32.352 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:20:32.352 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:20:32.352 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_47 2023-11-08T05:20:32.353 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local test_state=started 2023-11-08T05:20:32.353 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local status_result 2023-11-08T05:20:32.353 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local current_state=stopped 2023-11-08T05:20:32.353 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_47 2023-11-08T05:20:32.353 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ grep -i state 2023-11-08T05:20:32.354 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_instance=cluster1 2023-11-08T05:20:32.354 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T05:20:32.354 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1 2023-11-08T05:20:32.354 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance=cluster1 2023-11-08T05:20:32.354 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local loop_instance 2023-11-08T05:20:32.355 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-11-08T05:20:32.355 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sleep 0 2023-11-08T05:20:32.358 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-11-08T05:20:32.358 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ seq 0 3 2023-11-08T05:20:32.358 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T05:20:32.359 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_47 2023-11-08T05:20:32.359 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1:0 2023-11-08T05:20:32.359 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T05:20:32.359 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance 2023-11-08T05:20:32.359 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-11-08T05:20:32.360 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local val=cluster1:0 2023-11-08T05:20:32.361 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_var_name=cluster 2023-11-08T05:20:32.361 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance_var_name=instance 2023-11-08T05:20:32.362 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cluster=cluster1 2023-11-08T05:20:32.362 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ instance=0 2023-11-08T05:20:32.362 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' 0 = cluster1:0 ']' 2023-11-08T05:20:32.362 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval cluster=cluster1 2023-11-08T05:20:32.363 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:20:32.363 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval instance=0 2023-11-08T05:20:32.363 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=0 2023-11-08T05:20:32.363 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-11-08T05:20:32.363 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local local_cluster=cluster1:0 2023-11-08T05:20:32.364 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster=cluster1 2023-11-08T05:20:32.364 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance 2023-11-08T05:20:32.364 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-11-08T05:20:32.364 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local val=cluster1:0 2023-11-08T05:20:32.365 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster_var_name=local_cluster 2023-11-08T05:20:32.365 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance_var_name=instance 2023-11-08T05:20:32.365 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:20:32.365 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=0 2023-11-08T05:20:32.365 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ '[' 0 = cluster1:0 ']' 2023-11-08T05:20:32.365 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval local_cluster=cluster1 2023-11-08T05:20:32.366 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ local_cluster=cluster1 2023-11-08T05:20:32.366 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval instance=0 2023-11-08T05:20:32.366 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ instance=0 2023-11-08T05:20:32.366 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-11-08T05:20:32.369 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-08T05:20:32.370 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-08T05:20:32.370 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-08T05:20:32.370 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_47 2023-11-08T05:20:32.470 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:Invalid command: unused arguments: ['mirror/image_47'] 2023-11-08T05:20:32.470 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd mirror status : get status for rbd mirror 2023-11-08T05:20:32.470 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:admin_socket: invalid command 2023-11-08T05:20:32.473 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T05:20:32.473 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_47 2023-11-08T05:20:32.474 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1:1 2023-11-08T05:20:32.474 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T05:20:32.474 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance 2023-11-08T05:20:32.474 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-11-08T05:20:32.474 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local val=cluster1:1 2023-11-08T05:20:32.475 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_var_name=cluster 2023-11-08T05:20:32.475 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance_var_name=instance 2023-11-08T05:20:32.475 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cluster=cluster1 2023-11-08T05:20:32.475 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ instance=1 2023-11-08T05:20:32.476 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' 1 = cluster1:1 ']' 2023-11-08T05:20:32.476 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval cluster=cluster1 2023-11-08T05:20:32.476 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:20:32.476 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval instance=1 2023-11-08T05:20:32.476 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=1 2023-11-08T05:20:32.477 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-11-08T05:20:32.477 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local local_cluster=cluster1:1 2023-11-08T05:20:32.477 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster=cluster1 2023-11-08T05:20:32.477 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance 2023-11-08T05:20:32.477 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-11-08T05:20:32.478 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local val=cluster1:1 2023-11-08T05:20:32.478 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster_var_name=local_cluster 2023-11-08T05:20:32.478 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance_var_name=instance 2023-11-08T05:20:32.478 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:20:32.478 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=1 2023-11-08T05:20:32.479 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ '[' 1 = cluster1:1 ']' 2023-11-08T05:20:32.479 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval local_cluster=cluster1 2023-11-08T05:20:32.479 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ local_cluster=cluster1 2023-11-08T05:20:32.479 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval instance=1 2023-11-08T05:20:32.479 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ instance=1 2023-11-08T05:20:32.479 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-11-08T05:20:32.487 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-08T05:20:32.487 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-08T05:20:32.487 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-08T05:20:32.488 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_47 2023-11-08T05:20:32.596 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ return 0 2023-11-08T05:20:32.597 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ status_result=' "state": "Replaying"' 2023-11-08T05:20:32.597 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ echo ' "state": "Replaying"' 2023-11-08T05:20:32.597 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ grep -i Replaying 2023-11-08T05:20:32.598 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: "state": "Replaying" 2023-11-08T05:20:32.600 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ current_state=started 2023-11-08T05:20:32.600 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test started = started 2023-11-08T05:20:32.600 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:20:32.601 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_47 2023-11-08T05:20:32.601 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local local_cluster=cluster1 2023-11-08T05:20:32.601 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:20:32.601 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:20:32.601 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_47 2023-11-08T05:20:32.601 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' snapshot = journal ']' 2023-11-08T05:20:32.602 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' snapshot = snapshot ']' 2023-11-08T05:20:32.602 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_47 2023-11-08T05:20:32.602 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local local_cluster=cluster1 2023-11-08T05:20:32.602 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:20:32.602 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:20:32.603 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_47 2023-11-08T05:20:32.603 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ mkfname cluster2-mirror-image_47.status 2023-11-08T05:20:32.603 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ echo cluster2-mirror-image_47.status 2023-11-08T05:20:32.603 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -e 's|[/ ]|_|g' 2023-11-08T05:20:32.603 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local status_log=/tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_47.status 2023-11-08T05:20:32.603 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ mkfname cluster1-mirror-image_47.status 2023-11-08T05:20:32.604 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ echo cluster1-mirror-image_47.status 2023-11-08T05:20:32.604 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -e 's|[/ ]|_|g' 2023-11-08T05:20:32.605 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local local_status_log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_47.status 2023-11-08T05:20:32.605 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ mirror_image_snapshot cluster2 mirror image_47 2023-11-08T05:20:32.605 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:20:32.605 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:20:32.606 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_47 2023-11-08T05:20:32.606 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_47 2023-11-08T05:20:33.360 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:Snapshot ID: 211 2023-11-08T05:20:33.369 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_47 /tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_47.status 2023-11-08T05:20:33.369 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:20:33.369 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:20:33.370 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_47 2023-11-08T05:20:33.370 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_47.status 2023-11-08T05:20:33.370 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_47 --format xml 2023-11-08T05:20:33.370 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T05:20:33.407 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-11-08T05:20:33.408 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snapshot_id=211 2023-11-08T05:20:33.409 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ true 2023-11-08T05:20:33.409 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-11-08T05:20:33.409 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0.2 2023-11-08T05:20:33.610 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_47 /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_47.status 2023-11-08T05:20:33.611 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:20:33.611 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:20:33.611 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_47 2023-11-08T05:20:33.612 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_47.status 2023-11-08T05:20:33.612 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_47 --format xml 2023-11-08T05:20:33.612 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T05:20:33.655 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ true 2023-11-08T05:20:33.656 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-08T05:20:33.656 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:-:1.1: Document is empty 2023-11-08T05:20:33.657 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: 2023-11-08T05:20:33.657 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:^ 2023-11-08T05:20:33.658 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local primary_snapshot_id= 2023-11-08T05:20:33.658 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test 211 = '' 2023-11-08T05:20:33.658 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-11-08T05:20:33.658 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0.4 2023-11-08T05:20:34.058 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_47 /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_47.status 2023-11-08T05:20:34.059 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:20:34.059 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:20:34.059 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_47 2023-11-08T05:20:34.059 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_47.status 2023-11-08T05:20:34.060 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_47 --format xml 2023-11-08T05:20:34.060 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T05:20:34.097 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ true 2023-11-08T05:20:34.097 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-08T05:20:34.098 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:-:1.1: Document is empty 2023-11-08T05:20:34.098 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: 2023-11-08T05:20:34.099 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:^ 2023-11-08T05:20:34.099 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local primary_snapshot_id= 2023-11-08T05:20:34.099 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test 211 = '' 2023-11-08T05:20:34.100 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-11-08T05:20:34.100 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0.8 2023-11-08T05:20:34.900 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_47 /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_47.status 2023-11-08T05:20:34.900 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:20:34.901 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:20:34.901 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_47 2023-11-08T05:20:34.901 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_47.status 2023-11-08T05:20:34.901 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_47 --format xml 2023-11-08T05:20:34.901 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T05:20:34.948 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ true 2023-11-08T05:20:34.949 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-08T05:20:34.950 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:-:1.1: Document is empty 2023-11-08T05:20:34.950 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: 2023-11-08T05:20:34.950 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:^ 2023-11-08T05:20:34.951 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local primary_snapshot_id= 2023-11-08T05:20:34.951 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test 211 = '' 2023-11-08T05:20:34.951 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-11-08T05:20:34.951 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 1.6 2023-11-08T05:20:36.552 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_47 /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_47.status 2023-11-08T05:20:36.552 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:20:36.553 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:20:36.553 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_47 2023-11-08T05:20:36.553 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_47.status 2023-11-08T05:20:36.553 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_47 --format xml 2023-11-08T05:20:36.554 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T05:20:36.597 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-08T05:20:36.600 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local primary_snapshot_id=211 2023-11-08T05:20:36.600 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test 211 = 211 2023-11-08T05:20:36.600 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:20:36.600 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_snap_present cluster1 mirror image_47 snap 2023-11-08T05:20:36.600 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:20:36.601 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:20:36.601 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_47 2023-11-08T05:20:36.601 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap_name=snap 2023-11-08T05:20:36.601 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local s 2023-11-08T05:20:36.602 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-11-08T05:20:36.602 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 1 2023-11-08T05:20:37.600 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 info mirror/image_47@snap 2023-11-08T05:20:37.646 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:rbd image 'image_47': 2023-11-08T05:20:37.646 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: size 128 MiB in 32 objects 2023-11-08T05:20:37.646 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: order 22 (4 MiB objects) 2023-11-08T05:20:37.647 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: snapshot_count: 2 2023-11-08T05:20:37.647 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: id: 12b32d42e6ec 2023-11-08T05:20:37.647 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: block_name_prefix: rbd_data.12b32d42e6ec 2023-11-08T05:20:37.647 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: format: 2 2023-11-08T05:20:37.647 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: features: layering, exclusive-lock, non-primary 2023-11-08T05:20:37.648 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: op_features: 2023-11-08T05:20:37.648 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: flags: 2023-11-08T05:20:37.648 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: create_timestamp: Wed Nov 8 05:12:44 2023 2023-11-08T05:20:37.648 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: access_timestamp: Wed Nov 8 05:12:44 2023 2023-11-08T05:20:37.649 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: modify_timestamp: Wed Nov 8 05:12:44 2023 2023-11-08T05:20:37.649 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: protected: False 2023-11-08T05:20:37.649 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring state: enabled 2023-11-08T05:20:37.649 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring mode: snapshot 2023-11-08T05:20:37.649 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring global id: 8f6b2565-b6aa-4361-bc36-f1a120eb271e 2023-11-08T05:20:37.650 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring primary: false 2023-11-08T05:20:37.651 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:20:37.651 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ compare_image_snaps mirror image_47 snap 2023-11-08T05:20:37.651 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:20:37.652 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_47 2023-11-08T05:20:37.652 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap_name=snap 2023-11-08T05:20:37.652 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local ret=0 2023-11-08T05:20:37.653 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local rmt_export=/tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_47.export 2023-11-08T05:20:37.653 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local loc_export=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_47.export 2023-11-08T05:20:37.653 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rm -f /tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_47.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_47.export 2023-11-08T05:20:37.653 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 -p mirror export image_47@snap /tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_47.export 2023-11-08T05:20:37.822 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-11-08T05:20:37.825 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 -p mirror export image_47@snap /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_47.export 2023-11-08T05:20:38.053 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-11-08T05:20:38.056 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ cmp /tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_47.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_47.export 2023-11-08T05:20:38.111 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rm -f /tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_47.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_47.export 2023-11-08T05:20:38.147 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:20:38.148 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-11-08T05:20:38.148 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image=image_48 2023-11-08T05:20:38.148 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ create_snap cluster2 mirror image_48 snap 2023-11-08T05:20:38.148 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:20:38.149 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:20:38.149 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_48 2023-11-08T05:20:38.149 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap_name=snap 2023-11-08T05:20:38.149 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 -p mirror snap create image_48@snap --debug-rbd=20 --debug-journaler=20 2023-11-08T05:20:39.376 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_image_replay_started cluster1 mirror image_48 2023-11-08T05:20:39.376 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:20:39.376 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:20:39.377 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_48 2023-11-08T05:20:39.377 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_image_replay_state cluster1 mirror image_48 started 2023-11-08T05:20:39.377 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:20:39.377 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:20:39.377 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_48 2023-11-08T05:20:39.378 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local state=started 2023-11-08T05:20:39.378 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local s 2023-11-08T05:20:39.378 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-11-08T05:20:39.378 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 1 2023-11-08T05:20:40.377 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test_image_replay_state cluster1 mirror image_48 started 2023-11-08T05:20:40.378 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:20:40.378 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:20:40.379 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_48 2023-11-08T05:20:40.379 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local test_state=started 2023-11-08T05:20:40.379 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local status_result 2023-11-08T05:20:40.379 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local current_state=stopped 2023-11-08T05:20:40.382 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_48 2023-11-08T05:20:40.383 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_instance=cluster1 2023-11-08T05:20:40.384 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T05:20:40.384 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1 2023-11-08T05:20:40.384 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ grep -i state 2023-11-08T05:20:40.384 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance=cluster1 2023-11-08T05:20:40.385 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local loop_instance 2023-11-08T05:20:40.385 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-11-08T05:20:40.385 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sleep 0 2023-11-08T05:20:40.385 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-11-08T05:20:40.385 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ seq 0 3 2023-11-08T05:20:40.386 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T05:20:40.386 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_48 2023-11-08T05:20:40.386 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1:0 2023-11-08T05:20:40.386 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T05:20:40.386 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance 2023-11-08T05:20:40.387 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-11-08T05:20:40.387 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local val=cluster1:0 2023-11-08T05:20:40.387 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_var_name=cluster 2023-11-08T05:20:40.387 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance_var_name=instance 2023-11-08T05:20:40.387 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cluster=cluster1 2023-11-08T05:20:40.388 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ instance=0 2023-11-08T05:20:40.388 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' 0 = cluster1:0 ']' 2023-11-08T05:20:40.388 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval cluster=cluster1 2023-11-08T05:20:40.388 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:20:40.388 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval instance=0 2023-11-08T05:20:40.390 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=0 2023-11-08T05:20:40.390 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-11-08T05:20:40.391 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local local_cluster=cluster1:0 2023-11-08T05:20:40.391 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster=cluster1 2023-11-08T05:20:40.391 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance 2023-11-08T05:20:40.391 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-11-08T05:20:40.392 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local val=cluster1:0 2023-11-08T05:20:40.392 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster_var_name=local_cluster 2023-11-08T05:20:40.392 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance_var_name=instance 2023-11-08T05:20:40.392 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:20:40.392 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=0 2023-11-08T05:20:40.393 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ '[' 0 = cluster1:0 ']' 2023-11-08T05:20:40.393 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval local_cluster=cluster1 2023-11-08T05:20:40.393 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ local_cluster=cluster1 2023-11-08T05:20:40.393 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval instance=0 2023-11-08T05:20:40.393 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ instance=0 2023-11-08T05:20:40.393 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-11-08T05:20:40.399 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-08T05:20:40.399 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-08T05:20:40.399 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-08T05:20:40.400 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_48 2023-11-08T05:20:40.507 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:Invalid command: unused arguments: ['mirror/image_48'] 2023-11-08T05:20:40.507 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd mirror status : get status for rbd mirror 2023-11-08T05:20:40.508 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:admin_socket: invalid command 2023-11-08T05:20:40.512 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T05:20:40.512 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_48 2023-11-08T05:20:40.513 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1:1 2023-11-08T05:20:40.513 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T05:20:40.513 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance 2023-11-08T05:20:40.513 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-11-08T05:20:40.513 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local val=cluster1:1 2023-11-08T05:20:40.514 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_var_name=cluster 2023-11-08T05:20:40.514 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance_var_name=instance 2023-11-08T05:20:40.514 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cluster=cluster1 2023-11-08T05:20:40.514 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ instance=1 2023-11-08T05:20:40.515 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' 1 = cluster1:1 ']' 2023-11-08T05:20:40.515 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval cluster=cluster1 2023-11-08T05:20:40.515 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:20:40.515 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval instance=1 2023-11-08T05:20:40.515 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=1 2023-11-08T05:20:40.516 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-11-08T05:20:40.516 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local local_cluster=cluster1:1 2023-11-08T05:20:40.516 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster=cluster1 2023-11-08T05:20:40.516 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance 2023-11-08T05:20:40.516 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-11-08T05:20:40.516 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local val=cluster1:1 2023-11-08T05:20:40.517 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster_var_name=local_cluster 2023-11-08T05:20:40.517 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance_var_name=instance 2023-11-08T05:20:40.517 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:20:40.518 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=1 2023-11-08T05:20:40.518 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ '[' 1 = cluster1:1 ']' 2023-11-08T05:20:40.518 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval local_cluster=cluster1 2023-11-08T05:20:40.519 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ local_cluster=cluster1 2023-11-08T05:20:40.519 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval instance=1 2023-11-08T05:20:40.520 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ instance=1 2023-11-08T05:20:40.520 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-11-08T05:20:40.524 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-08T05:20:40.525 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-08T05:20:40.525 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-08T05:20:40.525 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_48 2023-11-08T05:20:40.634 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ return 0 2023-11-08T05:20:40.634 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ status_result=' "state": "Replaying"' 2023-11-08T05:20:40.635 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ echo ' "state": "Replaying"' 2023-11-08T05:20:40.635 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ grep -i Replaying 2023-11-08T05:20:40.636 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: "state": "Replaying" 2023-11-08T05:20:40.641 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ current_state=started 2023-11-08T05:20:40.642 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test started = started 2023-11-08T05:20:40.642 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:20:40.643 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_48 2023-11-08T05:20:40.643 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local local_cluster=cluster1 2023-11-08T05:20:40.643 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:20:40.644 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:20:40.644 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_48 2023-11-08T05:20:40.645 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' snapshot = journal ']' 2023-11-08T05:20:40.645 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' snapshot = snapshot ']' 2023-11-08T05:20:40.645 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_48 2023-11-08T05:20:40.646 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local local_cluster=cluster1 2023-11-08T05:20:40.646 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:20:40.646 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:20:40.647 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_48 2023-11-08T05:20:40.647 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ mkfname cluster2-mirror-image_48.status 2023-11-08T05:20:40.648 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ echo cluster2-mirror-image_48.status 2023-11-08T05:20:40.648 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -e 's|[/ ]|_|g' 2023-11-08T05:20:40.648 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local status_log=/tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_48.status 2023-11-08T05:20:40.649 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ mkfname cluster1-mirror-image_48.status 2023-11-08T05:20:40.649 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ echo cluster1-mirror-image_48.status 2023-11-08T05:20:40.649 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -e 's|[/ ]|_|g' 2023-11-08T05:20:40.650 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local local_status_log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_48.status 2023-11-08T05:20:40.650 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ mirror_image_snapshot cluster2 mirror image_48 2023-11-08T05:20:40.650 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:20:40.651 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:20:40.651 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_48 2023-11-08T05:20:40.651 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_48 2023-11-08T05:20:41.389 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:Snapshot ID: 213 2023-11-08T05:20:41.400 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_48 /tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_48.status 2023-11-08T05:20:41.400 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:20:41.400 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:20:41.400 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_48 2023-11-08T05:20:41.401 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_48.status 2023-11-08T05:20:41.401 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_48 --format xml 2023-11-08T05:20:41.401 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T05:20:41.439 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-11-08T05:20:41.441 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snapshot_id=213 2023-11-08T05:20:41.441 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ true 2023-11-08T05:20:41.441 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-11-08T05:20:41.441 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0.2 2023-11-08T05:20:41.642 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_48 /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_48.status 2023-11-08T05:20:41.643 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:20:41.643 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:20:41.643 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_48 2023-11-08T05:20:41.643 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_48.status 2023-11-08T05:20:41.643 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_48 --format xml 2023-11-08T05:20:41.644 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T05:20:41.683 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ true 2023-11-08T05:20:41.683 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-08T05:20:41.684 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:-:1.1: Document is empty 2023-11-08T05:20:41.684 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: 2023-11-08T05:20:41.684 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:^ 2023-11-08T05:20:41.685 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local primary_snapshot_id= 2023-11-08T05:20:41.685 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test 213 = '' 2023-11-08T05:20:41.686 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-11-08T05:20:41.686 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0.4 2023-11-08T05:20:42.086 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_48 /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_48.status 2023-11-08T05:20:42.086 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:20:42.086 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:20:42.087 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_48 2023-11-08T05:20:42.087 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_48.status 2023-11-08T05:20:42.087 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_48 --format xml 2023-11-08T05:20:42.087 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T05:20:42.127 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ true 2023-11-08T05:20:42.128 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-08T05:20:42.129 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:-:1.1: Document is empty 2023-11-08T05:20:42.129 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: 2023-11-08T05:20:42.129 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:^ 2023-11-08T05:20:42.130 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local primary_snapshot_id= 2023-11-08T05:20:42.130 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test 213 = '' 2023-11-08T05:20:42.131 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-11-08T05:20:42.131 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0.8 2023-11-08T05:20:42.931 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_48 /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_48.status 2023-11-08T05:20:42.931 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:20:42.932 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:20:42.932 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_48 2023-11-08T05:20:42.932 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_48.status 2023-11-08T05:20:42.932 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_48 --format xml 2023-11-08T05:20:42.932 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T05:20:42.970 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-08T05:20:42.974 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local primary_snapshot_id=213 2023-11-08T05:20:42.974 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test 213 = 213 2023-11-08T05:20:42.974 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:20:42.975 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_snap_present cluster1 mirror image_48 snap 2023-11-08T05:20:42.975 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:20:42.975 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:20:42.975 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_48 2023-11-08T05:20:42.976 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap_name=snap 2023-11-08T05:20:42.976 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local s 2023-11-08T05:20:42.976 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-11-08T05:20:42.976 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 1 2023-11-08T05:20:43.973 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 info mirror/image_48@snap 2023-11-08T05:20:44.214 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:rbd image 'image_48': 2023-11-08T05:20:44.215 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: size 128 MiB in 32 objects 2023-11-08T05:20:44.215 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: order 22 (4 MiB objects) 2023-11-08T05:20:44.215 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: snapshot_count: 2 2023-11-08T05:20:44.215 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: id: 12c33e13e64f 2023-11-08T05:20:44.216 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: block_name_prefix: rbd_data.12c33e13e64f 2023-11-08T05:20:44.216 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: format: 2 2023-11-08T05:20:44.216 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: features: layering, exclusive-lock, non-primary 2023-11-08T05:20:44.216 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: op_features: 2023-11-08T05:20:44.216 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: flags: 2023-11-08T05:20:44.217 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: create_timestamp: Wed Nov 8 05:12:58 2023 2023-11-08T05:20:44.217 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: access_timestamp: Wed Nov 8 05:12:58 2023 2023-11-08T05:20:44.217 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: modify_timestamp: Wed Nov 8 05:12:58 2023 2023-11-08T05:20:44.217 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: protected: False 2023-11-08T05:20:44.218 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring state: enabled 2023-11-08T05:20:44.218 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring mode: snapshot 2023-11-08T05:20:44.218 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring global id: c3d37bcb-5eef-4c9f-a31c-472ae9f8d4c1 2023-11-08T05:20:44.218 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring primary: false 2023-11-08T05:20:44.220 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:20:44.220 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ compare_image_snaps mirror image_48 snap 2023-11-08T05:20:44.220 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:20:44.220 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_48 2023-11-08T05:20:44.220 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap_name=snap 2023-11-08T05:20:44.221 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local ret=0 2023-11-08T05:20:44.221 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local rmt_export=/tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_48.export 2023-11-08T05:20:44.221 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local loc_export=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_48.export 2023-11-08T05:20:44.221 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rm -f /tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_48.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_48.export 2023-11-08T05:20:44.221 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 -p mirror export image_48@snap /tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_48.export 2023-11-08T05:20:44.397 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-11-08T05:20:44.401 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 -p mirror export image_48@snap /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_48.export 2023-11-08T05:20:44.857 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-11-08T05:20:44.860 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ cmp /tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_48.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_48.export 2023-11-08T05:20:44.914 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rm -f /tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_48.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_48.export 2023-11-08T05:20:44.951 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:20:44.951 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-11-08T05:20:44.951 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image=image_49 2023-11-08T05:20:44.951 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ create_snap cluster2 mirror image_49 snap 2023-11-08T05:20:44.952 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:20:44.952 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:20:44.952 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_49 2023-11-08T05:20:44.952 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap_name=snap 2023-11-08T05:20:44.952 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 -p mirror snap create image_49@snap --debug-rbd=20 --debug-journaler=20 2023-11-08T05:20:46.408 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_image_replay_started cluster1 mirror image_49 2023-11-08T05:20:46.409 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:20:46.409 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:20:46.409 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_49 2023-11-08T05:20:46.409 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_image_replay_state cluster1 mirror image_49 started 2023-11-08T05:20:46.410 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:20:46.410 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:20:46.410 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_49 2023-11-08T05:20:46.410 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local state=started 2023-11-08T05:20:46.411 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local s 2023-11-08T05:20:46.411 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-11-08T05:20:46.411 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 1 2023-11-08T05:20:47.410 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test_image_replay_state cluster1 mirror image_49 started 2023-11-08T05:20:47.411 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:20:47.411 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:20:47.411 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_49 2023-11-08T05:20:47.411 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local test_state=started 2023-11-08T05:20:47.412 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local status_result 2023-11-08T05:20:47.412 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local current_state=stopped 2023-11-08T05:20:47.412 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_49 2023-11-08T05:20:47.412 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_instance=cluster1 2023-11-08T05:20:47.413 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T05:20:47.413 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ grep -i state 2023-11-08T05:20:47.413 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1 2023-11-08T05:20:47.413 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance=cluster1 2023-11-08T05:20:47.413 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local loop_instance 2023-11-08T05:20:47.414 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-11-08T05:20:47.414 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sleep 0 2023-11-08T05:20:47.417 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-11-08T05:20:47.417 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ seq 0 3 2023-11-08T05:20:47.417 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T05:20:47.417 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_49 2023-11-08T05:20:47.418 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1:0 2023-11-08T05:20:47.418 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T05:20:47.418 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance 2023-11-08T05:20:47.418 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-11-08T05:20:47.419 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local val=cluster1:0 2023-11-08T05:20:47.419 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_var_name=cluster 2023-11-08T05:20:47.419 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance_var_name=instance 2023-11-08T05:20:47.419 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cluster=cluster1 2023-11-08T05:20:47.419 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ instance=0 2023-11-08T05:20:47.420 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' 0 = cluster1:0 ']' 2023-11-08T05:20:47.420 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval cluster=cluster1 2023-11-08T05:20:47.420 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:20:47.420 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval instance=0 2023-11-08T05:20:47.421 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=0 2023-11-08T05:20:47.421 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-11-08T05:20:47.421 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local local_cluster=cluster1:0 2023-11-08T05:20:47.421 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster=cluster1 2023-11-08T05:20:47.421 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance 2023-11-08T05:20:47.422 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-11-08T05:20:47.422 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local val=cluster1:0 2023-11-08T05:20:47.422 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster_var_name=local_cluster 2023-11-08T05:20:47.422 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance_var_name=instance 2023-11-08T05:20:47.422 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:20:47.423 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=0 2023-11-08T05:20:47.423 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ '[' 0 = cluster1:0 ']' 2023-11-08T05:20:47.423 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval local_cluster=cluster1 2023-11-08T05:20:47.423 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ local_cluster=cluster1 2023-11-08T05:20:47.423 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval instance=0 2023-11-08T05:20:47.424 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ instance=0 2023-11-08T05:20:47.424 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-11-08T05:20:47.431 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-08T05:20:47.432 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-08T05:20:47.432 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-08T05:20:47.432 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_49 2023-11-08T05:20:47.538 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:Invalid command: unused arguments: ['mirror/image_49'] 2023-11-08T05:20:47.539 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd mirror status : get status for rbd mirror 2023-11-08T05:20:47.539 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:admin_socket: invalid command 2023-11-08T05:20:47.542 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T05:20:47.542 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_49 2023-11-08T05:20:47.542 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1:1 2023-11-08T05:20:47.543 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T05:20:47.543 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance 2023-11-08T05:20:47.543 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-11-08T05:20:47.543 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local val=cluster1:1 2023-11-08T05:20:47.543 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_var_name=cluster 2023-11-08T05:20:47.544 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance_var_name=instance 2023-11-08T05:20:47.544 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cluster=cluster1 2023-11-08T05:20:47.544 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ instance=1 2023-11-08T05:20:47.544 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' 1 = cluster1:1 ']' 2023-11-08T05:20:47.545 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval cluster=cluster1 2023-11-08T05:20:47.545 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:20:47.545 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval instance=1 2023-11-08T05:20:47.545 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=1 2023-11-08T05:20:47.545 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-11-08T05:20:47.546 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local local_cluster=cluster1:1 2023-11-08T05:20:47.546 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster=cluster1 2023-11-08T05:20:47.546 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance 2023-11-08T05:20:47.546 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-11-08T05:20:47.547 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local val=cluster1:1 2023-11-08T05:20:47.547 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster_var_name=local_cluster 2023-11-08T05:20:47.547 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance_var_name=instance 2023-11-08T05:20:47.547 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:20:47.547 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=1 2023-11-08T05:20:47.548 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ '[' 1 = cluster1:1 ']' 2023-11-08T05:20:47.548 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval local_cluster=cluster1 2023-11-08T05:20:47.548 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ local_cluster=cluster1 2023-11-08T05:20:47.548 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval instance=1 2023-11-08T05:20:47.548 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ instance=1 2023-11-08T05:20:47.549 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-11-08T05:20:47.556 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-08T05:20:47.556 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-08T05:20:47.556 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-08T05:20:47.556 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_49 2023-11-08T05:20:47.665 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ return 0 2023-11-08T05:20:47.667 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ status_result=' "state": "Replaying"' 2023-11-08T05:20:47.667 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ echo ' "state": "Replaying"' 2023-11-08T05:20:47.667 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ grep -i Replaying 2023-11-08T05:20:47.667 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: "state": "Replaying" 2023-11-08T05:20:47.670 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ current_state=started 2023-11-08T05:20:47.670 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test started = started 2023-11-08T05:20:47.670 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:20:47.671 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_49 2023-11-08T05:20:47.671 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local local_cluster=cluster1 2023-11-08T05:20:47.671 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:20:47.671 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:20:47.672 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_49 2023-11-08T05:20:47.672 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' snapshot = journal ']' 2023-11-08T05:20:47.672 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' snapshot = snapshot ']' 2023-11-08T05:20:47.672 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_49 2023-11-08T05:20:47.672 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local local_cluster=cluster1 2023-11-08T05:20:47.672 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:20:47.673 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:20:47.673 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_49 2023-11-08T05:20:47.673 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ mkfname cluster2-mirror-image_49.status 2023-11-08T05:20:47.673 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ echo cluster2-mirror-image_49.status 2023-11-08T05:20:47.673 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -e 's|[/ ]|_|g' 2023-11-08T05:20:47.673 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local status_log=/tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_49.status 2023-11-08T05:20:47.674 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ mkfname cluster1-mirror-image_49.status 2023-11-08T05:20:47.674 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ echo cluster1-mirror-image_49.status 2023-11-08T05:20:47.674 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -e 's|[/ ]|_|g' 2023-11-08T05:20:47.675 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local local_status_log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_49.status 2023-11-08T05:20:47.675 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ mirror_image_snapshot cluster2 mirror image_49 2023-11-08T05:20:47.675 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:20:47.676 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:20:47.676 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_49 2023-11-08T05:20:47.676 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_49 2023-11-08T05:20:48.418 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:Snapshot ID: 215 2023-11-08T05:20:48.427 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_49 /tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_49.status 2023-11-08T05:20:48.427 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:20:48.427 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:20:48.428 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_49 2023-11-08T05:20:48.428 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_49.status 2023-11-08T05:20:48.429 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_49 --format xml 2023-11-08T05:20:48.429 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T05:20:48.463 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-11-08T05:20:48.465 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snapshot_id=215 2023-11-08T05:20:48.466 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ true 2023-11-08T05:20:48.466 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-11-08T05:20:48.467 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0.2 2023-11-08T05:20:48.666 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_49 /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_49.status 2023-11-08T05:20:48.667 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:20:48.667 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:20:48.667 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_49 2023-11-08T05:20:48.667 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_49.status 2023-11-08T05:20:48.667 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_49 --format xml 2023-11-08T05:20:48.668 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T05:20:48.707 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ true 2023-11-08T05:20:48.707 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-08T05:20:48.708 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:-:1.1: Document is empty 2023-11-08T05:20:48.708 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: 2023-11-08T05:20:48.708 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:^ 2023-11-08T05:20:48.709 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local primary_snapshot_id= 2023-11-08T05:20:48.709 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test 215 = '' 2023-11-08T05:20:48.709 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-11-08T05:20:48.710 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0.4 2023-11-08T05:20:49.110 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_49 /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_49.status 2023-11-08T05:20:49.110 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:20:49.110 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:20:49.111 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_49 2023-11-08T05:20:49.111 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_49.status 2023-11-08T05:20:49.111 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_49 --format xml 2023-11-08T05:20:49.111 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T05:20:49.151 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ true 2023-11-08T05:20:49.152 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-08T05:20:49.152 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:-:1.1: Document is empty 2023-11-08T05:20:49.153 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: 2023-11-08T05:20:49.153 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:^ 2023-11-08T05:20:49.154 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local primary_snapshot_id= 2023-11-08T05:20:49.154 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test 215 = '' 2023-11-08T05:20:49.154 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-11-08T05:20:49.155 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0.8 2023-11-08T05:20:49.954 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_49 /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_49.status 2023-11-08T05:20:49.955 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:20:49.955 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:20:49.955 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_49 2023-11-08T05:20:49.955 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_49.status 2023-11-08T05:20:49.956 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_49 --format xml 2023-11-08T05:20:49.956 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T05:20:49.994 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-08T05:20:49.996 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local primary_snapshot_id=215 2023-11-08T05:20:49.996 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test 215 = 215 2023-11-08T05:20:49.996 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:20:49.997 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_snap_present cluster1 mirror image_49 snap 2023-11-08T05:20:49.997 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:20:49.997 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:20:49.997 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_49 2023-11-08T05:20:49.997 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap_name=snap 2023-11-08T05:20:49.998 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local s 2023-11-08T05:20:49.998 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-11-08T05:20:49.998 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 1 2023-11-08T05:20:50.997 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 info mirror/image_49@snap 2023-11-08T05:20:51.037 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:rbd image 'image_49': 2023-11-08T05:20:51.038 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: size 128 MiB in 32 objects 2023-11-08T05:20:51.038 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: order 22 (4 MiB objects) 2023-11-08T05:20:51.038 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: snapshot_count: 2 2023-11-08T05:20:51.039 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: id: 11d8c5ae7c77 2023-11-08T05:20:51.039 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: block_name_prefix: rbd_data.11d8c5ae7c77 2023-11-08T05:20:51.039 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: format: 2 2023-11-08T05:20:51.039 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: features: layering, exclusive-lock, non-primary 2023-11-08T05:20:51.039 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: op_features: 2023-11-08T05:20:51.040 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: flags: 2023-11-08T05:20:51.040 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: create_timestamp: Wed Nov 8 05:13:08 2023 2023-11-08T05:20:51.040 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: access_timestamp: Wed Nov 8 05:13:08 2023 2023-11-08T05:20:51.040 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: modify_timestamp: Wed Nov 8 05:13:08 2023 2023-11-08T05:20:51.040 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: protected: False 2023-11-08T05:20:51.041 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring state: enabled 2023-11-08T05:20:51.041 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring mode: snapshot 2023-11-08T05:20:51.041 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring global id: e7efac79-0655-48fa-85f6-d32c1c0bf780 2023-11-08T05:20:51.041 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring primary: false 2023-11-08T05:20:51.042 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:20:51.043 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ compare_image_snaps mirror image_49 snap 2023-11-08T05:20:51.043 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:20:51.043 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_49 2023-11-08T05:20:51.043 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap_name=snap 2023-11-08T05:20:51.043 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local ret=0 2023-11-08T05:20:51.044 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local rmt_export=/tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_49.export 2023-11-08T05:20:51.044 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local loc_export=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_49.export 2023-11-08T05:20:51.044 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rm -f /tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_49.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_49.export 2023-11-08T05:20:51.044 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 -p mirror export image_49@snap /tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_49.export 2023-11-08T05:20:51.211 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-11-08T05:20:51.214 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 -p mirror export image_49@snap /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_49.export 2023-11-08T05:20:51.467 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-11-08T05:20:51.470 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ cmp /tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_49.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_49.export 2023-11-08T05:20:51.526 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rm -f /tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_49.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_49.export 2023-11-08T05:20:51.563 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:20:51.563 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-11-08T05:20:51.563 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image=image_50 2023-11-08T05:20:51.563 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ create_snap cluster2 mirror image_50 snap 2023-11-08T05:20:51.564 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:20:51.564 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:20:51.564 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_50 2023-11-08T05:20:51.564 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap_name=snap 2023-11-08T05:20:51.564 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 -p mirror snap create image_50@snap --debug-rbd=20 --debug-journaler=20 2023-11-08T05:20:53.431 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_image_replay_started cluster1 mirror image_50 2023-11-08T05:20:53.432 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:20:53.432 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:20:53.432 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_50 2023-11-08T05:20:53.432 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_image_replay_state cluster1 mirror image_50 started 2023-11-08T05:20:53.432 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:20:53.433 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:20:53.433 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_50 2023-11-08T05:20:53.433 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local state=started 2023-11-08T05:20:53.433 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local s 2023-11-08T05:20:53.433 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-11-08T05:20:53.434 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 1 2023-11-08T05:20:54.432 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test_image_replay_state cluster1 mirror image_50 started 2023-11-08T05:20:54.433 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:20:54.433 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:20:54.433 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_50 2023-11-08T05:20:54.433 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local test_state=started 2023-11-08T05:20:54.433 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local status_result 2023-11-08T05:20:54.434 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local current_state=stopped 2023-11-08T05:20:54.435 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_50 2023-11-08T05:20:54.435 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_instance=cluster1 2023-11-08T05:20:54.435 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T05:20:54.435 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ grep -i state 2023-11-08T05:20:54.436 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1 2023-11-08T05:20:54.436 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance=cluster1 2023-11-08T05:20:54.436 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local loop_instance 2023-11-08T05:20:54.436 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-11-08T05:20:54.436 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sleep 0 2023-11-08T05:20:54.437 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-11-08T05:20:54.437 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ seq 0 3 2023-11-08T05:20:54.439 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T05:20:54.440 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_50 2023-11-08T05:20:54.440 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1:0 2023-11-08T05:20:54.440 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T05:20:54.440 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance 2023-11-08T05:20:54.441 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-11-08T05:20:54.441 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local val=cluster1:0 2023-11-08T05:20:54.441 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_var_name=cluster 2023-11-08T05:20:54.441 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance_var_name=instance 2023-11-08T05:20:54.442 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cluster=cluster1 2023-11-08T05:20:54.442 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ instance=0 2023-11-08T05:20:54.442 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' 0 = cluster1:0 ']' 2023-11-08T05:20:54.442 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval cluster=cluster1 2023-11-08T05:20:54.442 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:20:54.443 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval instance=0 2023-11-08T05:20:54.443 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=0 2023-11-08T05:20:54.443 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-11-08T05:20:54.443 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local local_cluster=cluster1:0 2023-11-08T05:20:54.443 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster=cluster1 2023-11-08T05:20:54.444 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance 2023-11-08T05:20:54.444 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-11-08T05:20:54.444 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local val=cluster1:0 2023-11-08T05:20:54.444 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster_var_name=local_cluster 2023-11-08T05:20:54.444 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance_var_name=instance 2023-11-08T05:20:54.444 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:20:54.445 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=0 2023-11-08T05:20:54.445 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ '[' 0 = cluster1:0 ']' 2023-11-08T05:20:54.445 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval local_cluster=cluster1 2023-11-08T05:20:54.445 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ local_cluster=cluster1 2023-11-08T05:20:54.445 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval instance=0 2023-11-08T05:20:54.446 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ instance=0 2023-11-08T05:20:54.446 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-11-08T05:20:54.454 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-08T05:20:54.454 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-08T05:20:54.455 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-11-08T05:20:54.455 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_50 2023-11-08T05:20:54.561 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:Invalid command: unused arguments: ['mirror/image_50'] 2023-11-08T05:20:54.562 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd mirror status : get status for rbd mirror 2023-11-08T05:20:54.562 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:admin_socket: invalid command 2023-11-08T05:20:54.568 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T05:20:54.568 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_50 2023-11-08T05:20:54.569 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1:1 2023-11-08T05:20:54.569 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T05:20:54.570 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance 2023-11-08T05:20:54.570 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-11-08T05:20:54.570 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local val=cluster1:1 2023-11-08T05:20:54.571 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_var_name=cluster 2023-11-08T05:20:54.571 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance_var_name=instance 2023-11-08T05:20:54.571 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cluster=cluster1 2023-11-08T05:20:54.572 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ instance=1 2023-11-08T05:20:54.572 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' 1 = cluster1:1 ']' 2023-11-08T05:20:54.572 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval cluster=cluster1 2023-11-08T05:20:54.573 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:20:54.573 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval instance=1 2023-11-08T05:20:54.573 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=1 2023-11-08T05:20:54.574 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-11-08T05:20:54.574 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local local_cluster=cluster1:1 2023-11-08T05:20:54.575 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster=cluster1 2023-11-08T05:20:54.575 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance 2023-11-08T05:20:54.575 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-11-08T05:20:54.576 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local val=cluster1:1 2023-11-08T05:20:54.576 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster_var_name=local_cluster 2023-11-08T05:20:54.577 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance_var_name=instance 2023-11-08T05:20:54.577 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:20:54.578 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=1 2023-11-08T05:20:54.578 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ '[' 1 = cluster1:1 ']' 2023-11-08T05:20:54.578 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval local_cluster=cluster1 2023-11-08T05:20:54.579 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ local_cluster=cluster1 2023-11-08T05:20:54.579 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval instance=1 2023-11-08T05:20:54.579 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ instance=1 2023-11-08T05:20:54.580 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-11-08T05:20:54.581 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-08T05:20:54.582 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-08T05:20:54.582 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-11-08T05:20:54.582 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_50 2023-11-08T05:20:54.681 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:Invalid command: unused arguments: ['mirror/image_50'] 2023-11-08T05:20:54.682 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd mirror status : get status for rbd mirror 2023-11-08T05:20:54.682 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:admin_socket: invalid command 2023-11-08T05:20:54.685 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T05:20:54.685 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_50 2023-11-08T05:20:54.686 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1:2 2023-11-08T05:20:54.686 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T05:20:54.686 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance 2023-11-08T05:20:54.686 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-11-08T05:20:54.687 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local val=cluster1:2 2023-11-08T05:20:54.687 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_var_name=cluster 2023-11-08T05:20:54.687 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance_var_name=instance 2023-11-08T05:20:54.687 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cluster=cluster1 2023-11-08T05:20:54.687 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ instance=2 2023-11-08T05:20:54.688 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' 2 = cluster1:2 ']' 2023-11-08T05:20:54.688 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval cluster=cluster1 2023-11-08T05:20:54.688 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:20:54.688 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval instance=2 2023-11-08T05:20:54.688 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=2 2023-11-08T05:20:54.689 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-11-08T05:20:54.689 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local local_cluster=cluster1:2 2023-11-08T05:20:54.689 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster=cluster1 2023-11-08T05:20:54.689 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance 2023-11-08T05:20:54.689 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-11-08T05:20:54.689 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local val=cluster1:2 2023-11-08T05:20:54.690 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster_var_name=local_cluster 2023-11-08T05:20:54.690 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance_var_name=instance 2023-11-08T05:20:54.690 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:20:54.690 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=2 2023-11-08T05:20:54.691 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ '[' 2 = cluster1:2 ']' 2023-11-08T05:20:54.691 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval local_cluster=cluster1 2023-11-08T05:20:54.691 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ local_cluster=cluster1 2023-11-08T05:20:54.691 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval instance=2 2023-11-08T05:20:54.691 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ instance=2 2023-11-08T05:20:54.692 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-11-08T05:20:54.697 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-08T05:20:54.697 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-08T05:20:54.697 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-11-08T05:20:54.698 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_50 2023-11-08T05:20:54.796 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:Invalid command: unused arguments: ['mirror/image_50'] 2023-11-08T05:20:54.797 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd mirror status : get status for rbd mirror 2023-11-08T05:20:54.797 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:admin_socket: invalid command 2023-11-08T05:20:54.800 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T05:20:54.800 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_50 2023-11-08T05:20:54.800 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1:3 2023-11-08T05:20:54.800 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ shift 2023-11-08T05:20:54.800 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance 2023-11-08T05:20:54.801 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ set_cluster_instance cluster1:3 cluster instance 2023-11-08T05:20:54.801 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local val=cluster1:3 2023-11-08T05:20:54.801 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster_var_name=cluster 2023-11-08T05:20:54.801 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local instance_var_name=instance 2023-11-08T05:20:54.801 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cluster=cluster1 2023-11-08T05:20:54.802 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ instance=3 2023-11-08T05:20:54.802 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ '[' 3 = cluster1:3 ']' 2023-11-08T05:20:54.802 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval cluster=cluster1 2023-11-08T05:20:54.802 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:20:54.803 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ eval instance=3 2023-11-08T05:20:54.803 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=3 2023-11-08T05:20:54.803 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ daemon_asok_file cluster1:3 cluster1 2023-11-08T05:20:54.803 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local local_cluster=cluster1:3 2023-11-08T05:20:54.803 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster=cluster1 2023-11-08T05:20:54.803 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance 2023-11-08T05:20:54.804 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2023-11-08T05:20:54.804 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local val=cluster1:3 2023-11-08T05:20:54.804 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local cluster_var_name=local_cluster 2023-11-08T05:20:54.804 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ local instance_var_name=instance 2023-11-08T05:20:54.804 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ cluster=cluster1 2023-11-08T05:20:54.805 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ instance=3 2023-11-08T05:20:54.805 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ '[' 3 = cluster1:3 ']' 2023-11-08T05:20:54.805 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval local_cluster=cluster1 2023-11-08T05:20:54.805 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ local_cluster=cluster1 2023-11-08T05:20:54.805 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ eval instance=3 2023-11-08T05:20:54.806 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ instance=3 2023-11-08T05:20:54.806 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2023-11-08T05:20:54.813 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-11-08T05:20:54.814 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-11-08T05:20:54.814 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-11-08T05:20:54.814 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_50 2023-11-08T05:20:54.920 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ return 0 2023-11-08T05:20:54.921 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ status_result=' "state": "Replaying"' 2023-11-08T05:20:54.921 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ echo ' "state": "Replaying"' 2023-11-08T05:20:54.922 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ grep -i Replaying 2023-11-08T05:20:54.922 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: "state": "Replaying" 2023-11-08T05:20:54.924 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ current_state=started 2023-11-08T05:20:54.924 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test started = started 2023-11-08T05:20:54.925 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:20:54.925 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_50 2023-11-08T05:20:54.925 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local local_cluster=cluster1 2023-11-08T05:20:54.925 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:20:54.926 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:20:54.926 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_50 2023-11-08T05:20:54.926 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' snapshot = journal ']' 2023-11-08T05:20:54.926 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' snapshot = snapshot ']' 2023-11-08T05:20:54.926 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_50 2023-11-08T05:20:54.927 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local local_cluster=cluster1 2023-11-08T05:20:54.927 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:20:54.927 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:20:54.927 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_50 2023-11-08T05:20:54.927 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ mkfname cluster2-mirror-image_50.status 2023-11-08T05:20:54.927 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ echo cluster2-mirror-image_50.status 2023-11-08T05:20:54.928 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -e 's|[/ ]|_|g' 2023-11-08T05:20:54.928 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local status_log=/tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_50.status 2023-11-08T05:20:54.928 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ mkfname cluster1-mirror-image_50.status 2023-11-08T05:20:54.928 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ echo cluster1-mirror-image_50.status 2023-11-08T05:20:54.928 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -e 's|[/ ]|_|g' 2023-11-08T05:20:54.929 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local local_status_log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_50.status 2023-11-08T05:20:54.930 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ mirror_image_snapshot cluster2 mirror image_50 2023-11-08T05:20:54.930 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:20:54.930 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:20:54.930 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_50 2023-11-08T05:20:54.930 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_50 2023-11-08T05:20:55.436 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:Snapshot ID: 217 2023-11-08T05:20:55.447 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_50 /tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_50.status 2023-11-08T05:20:55.447 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:20:55.448 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:20:55.448 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_50 2023-11-08T05:20:55.448 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_50.status 2023-11-08T05:20:55.448 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_50 --format xml 2023-11-08T05:20:55.449 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T05:20:55.484 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-11-08T05:20:55.486 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snapshot_id=217 2023-11-08T05:20:55.486 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ true 2023-11-08T05:20:55.487 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-11-08T05:20:55.487 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0.2 2023-11-08T05:20:55.688 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_50 /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_50.status 2023-11-08T05:20:55.724 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:20:55.724 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:20:55.725 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_50 2023-11-08T05:20:55.725 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_50.status 2023-11-08T05:20:55.725 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_50 --format xml 2023-11-08T05:20:55.725 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T05:20:55.728 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ true 2023-11-08T05:20:55.728 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-08T05:20:55.729 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:-:1.1: Document is empty 2023-11-08T05:20:55.729 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: 2023-11-08T05:20:55.729 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:^ 2023-11-08T05:20:55.731 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local primary_snapshot_id= 2023-11-08T05:20:55.731 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test 217 = '' 2023-11-08T05:20:55.731 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-11-08T05:20:55.731 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0.4 2023-11-08T05:20:56.131 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_50 /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_50.status 2023-11-08T05:20:56.131 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:20:56.131 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:20:56.132 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_50 2023-11-08T05:20:56.132 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_50.status 2023-11-08T05:20:56.132 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_50 --format xml 2023-11-08T05:20:56.132 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T05:20:56.174 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ true 2023-11-08T05:20:56.174 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-08T05:20:56.175 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:-:1.1: Document is empty 2023-11-08T05:20:56.176 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: 2023-11-08T05:20:56.176 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:^ 2023-11-08T05:20:56.177 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local primary_snapshot_id= 2023-11-08T05:20:56.177 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test 217 = '' 2023-11-08T05:20:56.177 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-11-08T05:20:56.177 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0.8 2023-11-08T05:20:56.977 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_50 /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_50.status 2023-11-08T05:20:56.978 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:20:56.978 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:20:56.978 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_50 2023-11-08T05:20:56.979 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local log=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_50.status 2023-11-08T05:20:56.979 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_50 --format xml 2023-11-08T05:20:56.979 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-11-08T05:20:57.016 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-11-08T05:20:57.019 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local primary_snapshot_id=217 2023-11-08T05:20:57.019 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test 217 = 217 2023-11-08T05:20:57.019 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:20:57.020 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_snap_present cluster1 mirror image_50 snap 2023-11-08T05:20:57.020 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:20:57.020 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:20:57.020 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_50 2023-11-08T05:20:57.020 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap_name=snap 2023-11-08T05:20:57.021 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local s 2023-11-08T05:20:57.021 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-11-08T05:20:57.021 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 1 2023-11-08T05:20:58.020 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 info mirror/image_50@snap 2023-11-08T05:20:58.057 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:rbd image 'image_50': 2023-11-08T05:20:58.057 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: size 128 MiB in 32 objects 2023-11-08T05:20:58.058 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: order 22 (4 MiB objects) 2023-11-08T05:20:58.058 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: snapshot_count: 2 2023-11-08T05:20:58.058 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: id: 12a9c31aeb88 2023-11-08T05:20:58.058 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: block_name_prefix: rbd_data.12a9c31aeb88 2023-11-08T05:20:58.058 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: format: 2 2023-11-08T05:20:58.059 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: features: layering, exclusive-lock, non-primary 2023-11-08T05:20:58.059 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: op_features: 2023-11-08T05:20:58.059 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: flags: 2023-11-08T05:20:58.059 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: create_timestamp: Wed Nov 8 05:13:16 2023 2023-11-08T05:20:58.059 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: access_timestamp: Wed Nov 8 05:13:16 2023 2023-11-08T05:20:58.060 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: modify_timestamp: Wed Nov 8 05:13:16 2023 2023-11-08T05:20:58.060 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: protected: False 2023-11-08T05:20:58.060 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring state: enabled 2023-11-08T05:20:58.060 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring mode: snapshot 2023-11-08T05:20:58.060 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring global id: 39de3a57-ef7b-4ccc-a818-26ba71efeed5 2023-11-08T05:20:58.061 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring primary: false 2023-11-08T05:20:58.062 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:20:58.062 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ compare_image_snaps mirror image_50 snap 2023-11-08T05:20:58.062 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:20:58.063 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_50 2023-11-08T05:20:58.063 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap_name=snap 2023-11-08T05:20:58.063 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local ret=0 2023-11-08T05:20:58.063 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local rmt_export=/tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_50.export 2023-11-08T05:20:58.063 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local loc_export=/tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_50.export 2023-11-08T05:20:58.064 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rm -f /tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_50.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_50.export 2023-11-08T05:20:58.064 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 -p mirror export image_50@snap /tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_50.export 2023-11-08T05:20:58.239 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-11-08T05:20:58.242 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster1 -p mirror export image_50@snap /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_50.export 2023-11-08T05:20:58.502 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-11-08T05:20:58.505 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ cmp /tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_50.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_50.export 2023-11-08T05:20:58.560 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rm -f /tmp/tmp.Ludk8ikCAB/cluster2-mirror-image_50.export /tmp/tmp.Ludk8ikCAB/cluster1-mirror-image_50.export 2023-11-08T05:20:58.597 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:20:58.597 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ testlog 'TEST: delete many images' 2023-11-08T05:20:58.597 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ tee -a /tmp/tmp.Ludk8ikCAB/rbd-mirror.test.log 2023-11-08T05:20:58.598 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ date '+%F %T' 2023-11-08T05:20:58.598 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ echo 2023-11-08 05:20:58 TEST: delete many images 2023-11-08T05:20:58.599 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:2023-11-08 05:20:58 TEST: delete many images 2023-11-08T05:20:58.599 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ seq 1 50 2023-11-08T05:20:58.600 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-11-08T05:20:58.600 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image=image_1 2023-11-08T05:20:58.600 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ remove_snapshot cluster2 mirror image_1 snap 2023-11-08T05:20:58.601 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:20:58.601 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:20:58.601 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_1 2023-11-08T05:20:58.601 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap=snap 2023-11-08T05:20:58.601 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 snap rm mirror/image_1@snap 2023-11-08T05:20:59.437 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Removing snap: 100% complete...done. 2023-11-08T05:20:59.449 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ remove_image_retry cluster2 mirror image_1 2023-11-08T05:20:59.449 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:20:59.449 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:20:59.450 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_1 2023-11-08T05:20:59.450 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0 1 2 4 8 16 32 2023-11-08T05:20:59.450 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0 2023-11-08T05:20:59.451 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ remove_image cluster2 mirror image_1 2023-11-08T05:20:59.451 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:20:59.451 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:20:59.451 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_1 2023-11-08T05:20:59.452 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_1 2023-11-08T05:20:59.491 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster=cluster2 rm mirror/image_1 2023-11-08T05:20:59.629 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi087.stderr:2023-11-08T05:20:59.628+0000 7ff071387700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5631436fa1a0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-11-08T05:20:59.630 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi087.stderr:2023-11-08T05:20:59.628+0000 7ff077393700 -1 rbd::mirror::ImageReplayer: 0x563146d00500 [3/8f09f37c-3f8d-4d10-ad86-50b1dda83090] operator(): start failed: (85) Interrupted system call should be restarted 2023-11-08T05:21:00.595 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-11-08T05:21:00.598 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:21:00.599 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-11-08T05:21:00.599 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image=image_2 2023-11-08T05:21:00.599 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ remove_snapshot cluster2 mirror image_2 snap 2023-11-08T05:21:00.599 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:21:00.599 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:21:00.600 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_2 2023-11-08T05:21:00.600 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap=snap 2023-11-08T05:21:00.600 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 snap rm mirror/image_2@snap 2023-11-08T05:21:01.450 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Removing snap: 100% complete...done. 2023-11-08T05:21:01.461 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ remove_image_retry cluster2 mirror image_2 2023-11-08T05:21:01.461 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:21:01.461 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:21:01.461 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_2 2023-11-08T05:21:01.462 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0 1 2 4 8 16 32 2023-11-08T05:21:01.462 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0 2023-11-08T05:21:01.462 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ remove_image cluster2 mirror image_2 2023-11-08T05:21:01.462 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:21:01.463 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:21:01.463 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_2 2023-11-08T05:21:01.463 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_2 2023-11-08T05:21:01.504 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster=cluster2 rm mirror/image_2 2023-11-08T05:21:01.625 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi087.stderr:2023-11-08T05:21:01.624+0000 7fc4495f2700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55eab62e2680 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-11-08T05:21:01.626 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi087.stderr:2023-11-08T05:21:01.624+0000 7fc44fdff700 -1 rbd::mirror::ImageReplayer: 0x55eab5e80280 [3/c9cc5896-9129-4803-9473-1d795c399c55] operator(): start failed: (85) Interrupted system call should be restarted 2023-11-08T05:21:02.601 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-11-08T05:21:02.604 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:21:02.604 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-11-08T05:21:02.604 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image=image_3 2023-11-08T05:21:02.605 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ remove_snapshot cluster2 mirror image_3 snap 2023-11-08T05:21:02.605 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:21:02.605 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:21:02.605 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_3 2023-11-08T05:21:02.605 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap=snap 2023-11-08T05:21:02.606 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 snap rm mirror/image_3@snap 2023-11-08T05:21:03.235 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Removing snap: 100% complete...done. 2023-11-08T05:21:03.241 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ remove_image_retry cluster2 mirror image_3 2023-11-08T05:21:03.241 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:21:03.241 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:21:03.242 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_3 2023-11-08T05:21:03.242 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0 1 2 4 8 16 32 2023-11-08T05:21:03.242 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0 2023-11-08T05:21:03.243 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ remove_image cluster2 mirror image_3 2023-11-08T05:21:03.243 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:21:03.243 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:21:03.243 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_3 2023-11-08T05:21:03.244 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_3 2023-11-08T05:21:03.281 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster=cluster2 rm mirror/image_3 2023-11-08T05:21:03.398 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi087.stderr:2023-11-08T05:21:03.397+0000 7fc449df3700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55eab62e29c0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-11-08T05:21:03.399 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi087.stderr:2023-11-08T05:21:03.397+0000 7fc44fdff700 -1 rbd::mirror::ImageReplayer: 0x55eab1f48c80 [3/64454ccd-8f11-4a7c-ba33-492d579d2c3b] operator(): start failed: (85) Interrupted system call should be restarted 2023-11-08T05:21:04.364 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-11-08T05:21:04.367 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:21:04.367 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-11-08T05:21:04.368 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image=image_4 2023-11-08T05:21:04.368 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ remove_snapshot cluster2 mirror image_4 snap 2023-11-08T05:21:04.368 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:21:04.368 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:21:04.368 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_4 2023-11-08T05:21:04.369 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap=snap 2023-11-08T05:21:04.369 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 snap rm mirror/image_4@snap 2023-11-08T05:21:05.237 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Removing snap: 100% complete...done. 2023-11-08T05:21:05.247 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ remove_image_retry cluster2 mirror image_4 2023-11-08T05:21:05.247 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:21:05.248 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:21:05.248 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_4 2023-11-08T05:21:05.248 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0 1 2 4 8 16 32 2023-11-08T05:21:05.248 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0 2023-11-08T05:21:05.249 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ remove_image cluster2 mirror image_4 2023-11-08T05:21:05.249 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:21:05.249 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:21:05.249 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_4 2023-11-08T05:21:05.249 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_4 2023-11-08T05:21:05.292 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster=cluster2 rm mirror/image_4 2023-11-08T05:21:05.398 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi087.stderr:2023-11-08T05:21:05.396+0000 7fc4495f2700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55eab62e31e0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-11-08T05:21:05.398 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi087.stderr:2023-11-08T05:21:05.396+0000 7fc44fdff700 -1 rbd::mirror::ImageReplayer: 0x55eab6309680 [3/93d1c29c-aa4a-4563-b1cd-30608ea2b43f] operator(): start failed: (85) Interrupted system call should be restarted 2023-11-08T05:21:05.663 INFO:tasks.ceph.cluster1.osd.1.smithi077.stderr:2023-11-08T05:21:05.661+0000 7f63db900700 -1 reset not still connected to 0x5634cf09d380 2023-11-08T05:21:05.664 INFO:tasks.ceph.cluster1.osd.1.smithi077.stderr:2023-11-08T05:21:05.661+0000 7f63db900700 -1 reset not still connected to 0x5634df55b520 2023-11-08T05:21:05.664 INFO:tasks.ceph.cluster1.osd.1.smithi077.stderr:2023-11-08T05:21:05.661+0000 7f63db900700 -1 reset not still connected to 0x5634df8a8d00 2023-11-08T05:21:05.664 INFO:tasks.ceph.cluster1.osd.1.smithi077.stderr:2023-11-08T05:21:05.661+0000 7f63db900700 -1 reset not still connected to 0x5634e06b6dd0 2023-11-08T05:21:05.665 INFO:tasks.ceph.cluster1.osd.1.smithi077.stderr:2023-11-08T05:21:05.661+0000 7f63db900700 -1 reset not still connected to 0x5634e6618d00 2023-11-08T05:21:06.366 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-11-08T05:21:06.367 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:21:06.367 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-11-08T05:21:06.367 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image=image_5 2023-11-08T05:21:06.367 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ remove_snapshot cluster2 mirror image_5 snap 2023-11-08T05:21:06.368 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:21:06.368 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:21:06.368 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_5 2023-11-08T05:21:06.368 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap=snap 2023-11-08T05:21:06.368 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 snap rm mirror/image_5@snap 2023-11-08T05:21:07.244 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Removing snap: 100% complete...done. 2023-11-08T05:21:07.255 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ remove_image_retry cluster2 mirror image_5 2023-11-08T05:21:07.255 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:21:07.256 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:21:07.256 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_5 2023-11-08T05:21:07.256 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0 1 2 4 8 16 32 2023-11-08T05:21:07.256 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0 2023-11-08T05:21:07.257 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ remove_image cluster2 mirror image_5 2023-11-08T05:21:07.257 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:21:07.257 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:21:07.258 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_5 2023-11-08T05:21:07.258 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_5 2023-11-08T05:21:07.311 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster=cluster2 rm mirror/image_5 2023-11-08T05:21:07.411 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi087.stderr:2023-11-08T05:21:07.409+0000 7ff071387700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5631436f5ba0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-11-08T05:21:07.411 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi087.stderr:2023-11-08T05:21:07.409+0000 7ff077b94700 -1 rbd::mirror::ImageReplayer: 0x563142aaca00 [3/c80b0fb2-a1fa-458e-baf2-277b0fc2657c] operator(): start failed: (85) Interrupted system call should be restarted 2023-11-08T05:21:08.385 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-11-08T05:21:08.388 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:21:08.389 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-11-08T05:21:08.389 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image=image_6 2023-11-08T05:21:08.389 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ remove_snapshot cluster2 mirror image_6 snap 2023-11-08T05:21:08.389 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:21:08.389 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:21:08.390 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_6 2023-11-08T05:21:08.390 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap=snap 2023-11-08T05:21:08.390 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 snap rm mirror/image_6@snap 2023-11-08T05:21:09.260 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Removing snap: 100% complete...done. 2023-11-08T05:21:09.268 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ remove_image_retry cluster2 mirror image_6 2023-11-08T05:21:09.268 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:21:09.268 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:21:09.269 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_6 2023-11-08T05:21:09.269 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0 1 2 4 8 16 32 2023-11-08T05:21:09.269 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0 2023-11-08T05:21:09.270 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ remove_image cluster2 mirror image_6 2023-11-08T05:21:09.270 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:21:09.270 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:21:09.270 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_6 2023-11-08T05:21:09.271 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_6 2023-11-08T05:21:09.311 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster=cluster2 rm mirror/image_6 2023-11-08T05:21:09.420 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi087.stderr:2023-11-08T05:21:09.418+0000 7f45a41e9700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55b0ce6071e0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-11-08T05:21:09.421 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi087.stderr:2023-11-08T05:21:09.419+0000 7f45ab1f7700 -1 rbd::mirror::ImageReplayer: 0x55b0ca755900 [3/acb6ea51-6256-49db-9edb-d84ff9a099a2] operator(): start failed: (85) Interrupted system call should be restarted 2023-11-08T05:21:10.376 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-11-08T05:21:10.379 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:21:10.379 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-11-08T05:21:10.379 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image=image_7 2023-11-08T05:21:10.380 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ remove_snapshot cluster2 mirror image_7 snap 2023-11-08T05:21:10.380 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:21:10.380 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:21:10.380 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_7 2023-11-08T05:21:10.380 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap=snap 2023-11-08T05:21:10.381 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 snap rm mirror/image_7@snap 2023-11-08T05:21:11.254 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Removing snap: 100% complete...done. 2023-11-08T05:21:11.263 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ remove_image_retry cluster2 mirror image_7 2023-11-08T05:21:11.263 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:21:11.264 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:21:11.264 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_7 2023-11-08T05:21:11.264 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0 1 2 4 8 16 32 2023-11-08T05:21:11.264 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0 2023-11-08T05:21:11.265 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ remove_image cluster2 mirror image_7 2023-11-08T05:21:11.266 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:21:11.266 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:21:11.266 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_7 2023-11-08T05:21:11.267 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_7 2023-11-08T05:21:11.304 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster=cluster2 rm mirror/image_7 2023-11-08T05:21:11.410 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi087.stderr:2023-11-08T05:21:11.409+0000 7f4436702700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x560b01984d00 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-11-08T05:21:11.411 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi087.stderr:2023-11-08T05:21:11.409+0000 7f443cf0f700 -1 rbd::mirror::ImageReplayer: 0x560afd791b80 [3/baa8ce06-b6bc-450a-80c9-52c7757d4aa3] operator(): start failed: (85) Interrupted system call should be restarted 2023-11-08T05:21:12.396 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-11-08T05:21:12.399 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:21:12.400 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-11-08T05:21:12.400 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image=image_8 2023-11-08T05:21:12.400 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ remove_snapshot cluster2 mirror image_8 snap 2023-11-08T05:21:12.400 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:21:12.400 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:21:12.401 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_8 2023-11-08T05:21:12.401 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap=snap 2023-11-08T05:21:12.401 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 snap rm mirror/image_8@snap 2023-11-08T05:21:13.248 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Removing snap: 100% complete...done. 2023-11-08T05:21:13.261 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ remove_image_retry cluster2 mirror image_8 2023-11-08T05:21:13.261 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:21:13.262 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:21:13.262 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_8 2023-11-08T05:21:13.262 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0 1 2 4 8 16 32 2023-11-08T05:21:13.262 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0 2023-11-08T05:21:13.263 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ remove_image cluster2 mirror image_8 2023-11-08T05:21:13.263 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:21:13.263 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:21:13.264 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_8 2023-11-08T05:21:13.264 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_8 2023-11-08T05:21:13.304 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster=cluster2 rm mirror/image_8 2023-11-08T05:21:13.411 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi087.stderr:2023-11-08T05:21:13.410+0000 7fc449df3700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55eab63876c0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-11-08T05:21:13.412 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi087.stderr:2023-11-08T05:21:13.410+0000 7fc450600700 -1 rbd::mirror::ImageReplayer: 0x55eab1f48280 [3/c78a25c8-e255-457e-8ce9-bb570e06b394] operator(): start failed: (85) Interrupted system call should be restarted 2023-11-08T05:21:14.402 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-11-08T05:21:14.406 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:21:14.406 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-11-08T05:21:14.406 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image=image_9 2023-11-08T05:21:14.407 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ remove_snapshot cluster2 mirror image_9 snap 2023-11-08T05:21:14.407 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:21:14.407 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:21:14.407 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_9 2023-11-08T05:21:14.407 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap=snap 2023-11-08T05:21:14.408 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 snap rm mirror/image_9@snap 2023-11-08T05:21:15.276 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Removing snap: 100% complete...done. 2023-11-08T05:21:15.288 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ remove_image_retry cluster2 mirror image_9 2023-11-08T05:21:15.289 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:21:15.289 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:21:15.289 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_9 2023-11-08T05:21:15.289 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0 1 2 4 8 16 32 2023-11-08T05:21:15.290 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0 2023-11-08T05:21:15.290 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ remove_image cluster2 mirror image_9 2023-11-08T05:21:15.291 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:21:15.291 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:21:15.291 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_9 2023-11-08T05:21:15.291 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_9 2023-11-08T05:21:15.329 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster=cluster2 rm mirror/image_9 2023-11-08T05:21:15.436 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi087.stderr:2023-11-08T05:21:15.435+0000 7ff071387700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5631436f5d40 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-11-08T05:21:15.437 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi087.stderr:2023-11-08T05:21:15.435+0000 7ff077b94700 -1 rbd::mirror::ImageReplayer: 0x563142aacc80 [3/bd37ef4f-d302-424b-952f-0d71e891061b] operator(): start failed: (85) Interrupted system call should be restarted 2023-11-08T05:21:16.224 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.5575206196293955) > weight (0.25) 2023-11-08T05:21:16.224 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.7416981928795523) > weight (0.25) 2023-11-08T05:21:16.225 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.2 2023-11-08T05:21:16.225 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Sent signal 15 2023-11-08T05:21:16.226 INFO:tasks.rbd_mirror_thrash:waiting for 0 secs before reviving daemons 2023-11-08T05:21:16.226 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.2 2023-11-08T05:21:16.226 INFO:teuthology.orchestra.run:waiting for 600 2023-11-08T05:21:16.227 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi087.stderr:2023-11-08T05:21:16.223+0000 7fd634dd4700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 (PID: 137823) UID: 1000 2023-11-08T05:21:16.426 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi087.stderr:daemon-helper: command failed with exit status 1 2023-11-08T05:21:16.427 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-11-08T05:21:16.430 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:21:16.430 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-11-08T05:21:16.431 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image=image_10 2023-11-08T05:21:16.431 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ remove_snapshot cluster2 mirror image_10 snap 2023-11-08T05:21:16.431 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:21:16.431 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:21:16.432 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_10 2023-11-08T05:21:16.432 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap=snap 2023-11-08T05:21:16.432 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 snap rm mirror/image_10@snap 2023-11-08T05:21:17.292 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Removing snap: 100% complete...done. 2023-11-08T05:21:17.304 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ remove_image_retry cluster2 mirror image_10 2023-11-08T05:21:17.304 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:21:17.304 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:21:17.305 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_10 2023-11-08T05:21:17.305 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0 1 2 4 8 16 32 2023-11-08T05:21:17.305 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0 2023-11-08T05:21:17.306 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ remove_image cluster2 mirror image_10 2023-11-08T05:21:17.306 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:21:17.306 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:21:17.307 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_10 2023-11-08T05:21:17.307 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_10 2023-11-08T05:21:17.347 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster=cluster2 rm mirror/image_10 2023-11-08T05:21:17.461 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi087.stderr:2023-11-08T05:21:17.458+0000 7f4436702700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x560b01a13d40 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-11-08T05:21:17.461 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi087.stderr:2023-11-08T05:21:17.458+0000 7f443c70e700 -1 rbd::mirror::ImageReplayer: 0x560afd791680 [3/ab5153f3-f7f8-4e7b-ad36-9e427596786a] operator(): start failed: (85) Interrupted system call should be restarted 2023-11-08T05:21:18.379 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-11-08T05:21:18.383 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:21:18.383 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-11-08T05:21:18.383 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image=image_11 2023-11-08T05:21:18.383 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ remove_snapshot cluster2 mirror image_11 snap 2023-11-08T05:21:18.384 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:21:18.384 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:21:18.384 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_11 2023-11-08T05:21:18.384 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap=snap 2023-11-08T05:21:18.384 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 snap rm mirror/image_11@snap 2023-11-08T05:21:19.296 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Removing snap: 100% complete...done. 2023-11-08T05:21:19.306 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ remove_image_retry cluster2 mirror image_11 2023-11-08T05:21:19.306 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:21:19.307 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:21:19.307 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_11 2023-11-08T05:21:19.307 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0 1 2 4 8 16 32 2023-11-08T05:21:19.307 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0 2023-11-08T05:21:19.308 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ remove_image cluster2 mirror image_11 2023-11-08T05:21:19.308 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:21:19.308 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:21:19.309 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_11 2023-11-08T05:21:19.309 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_11 2023-11-08T05:21:19.357 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster=cluster2 rm mirror/image_11 2023-11-08T05:21:19.464 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi087.stderr:2023-11-08T05:21:19.461+0000 7f45a41e9700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55b0ce4a6b60 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-11-08T05:21:19.464 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi087.stderr:2023-11-08T05:21:19.461+0000 7f45ab1f7700 -1 rbd::mirror::ImageReplayer: 0x55b0ce53a000 [3/b31ed973-6d38-4d25-adab-1da0bc236902] operator(): start failed: (85) Interrupted system call should be restarted 2023-11-08T05:21:20.426 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-11-08T05:21:20.429 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:21:20.429 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-11-08T05:21:20.430 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image=image_12 2023-11-08T05:21:20.430 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ remove_snapshot cluster2 mirror image_12 snap 2023-11-08T05:21:20.430 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:21:20.430 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:21:20.431 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_12 2023-11-08T05:21:20.431 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap=snap 2023-11-08T05:21:20.431 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 snap rm mirror/image_12@snap 2023-11-08T05:21:21.310 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Removing snap: 100% complete...done. 2023-11-08T05:21:21.324 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ remove_image_retry cluster2 mirror image_12 2023-11-08T05:21:21.324 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:21:21.325 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:21:21.325 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_12 2023-11-08T05:21:21.325 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0 1 2 4 8 16 32 2023-11-08T05:21:21.325 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0 2023-11-08T05:21:21.326 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ remove_image cluster2 mirror image_12 2023-11-08T05:21:21.326 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:21:21.326 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:21:21.327 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_12 2023-11-08T05:21:21.327 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_12 2023-11-08T05:21:21.367 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster=cluster2 rm mirror/image_12 2023-11-08T05:21:21.479 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi087.stderr:2023-11-08T05:21:21.476+0000 7f4436702700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x560b01984ea0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-11-08T05:21:21.480 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi087.stderr:2023-11-08T05:21:21.476+0000 7f443cf0f700 -1 rbd::mirror::ImageReplayer: 0x560b015ce000 [3/c28fe69d-285b-4d38-a8a1-5a5367bf07ce] operator(): start failed: (85) Interrupted system call should be restarted 2023-11-08T05:21:22.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-08T05:21:22.329 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.2 2023-11-08T05:21:22.329 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Restarting daemon 2023-11-08T05:21:22.329 DEBUG:teuthology.orchestra.run.smithi087:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 2023-11-08T05:21:22.332 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Started 2023-11-08T05:21:22.332 INFO:tasks.rbd_mirror_thrash:waiting for 73 secs before thrashing 2023-11-08T05:21:22.455 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-11-08T05:21:22.458 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:21:22.459 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-11-08T05:21:22.459 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image=image_13 2023-11-08T05:21:22.459 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ remove_snapshot cluster2 mirror image_13 snap 2023-11-08T05:21:22.459 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:21:22.460 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:21:22.460 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_13 2023-11-08T05:21:22.460 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap=snap 2023-11-08T05:21:22.460 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 snap rm mirror/image_13@snap 2023-11-08T05:21:23.236 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Removing snap: 100% complete...done. 2023-11-08T05:21:23.245 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ remove_image_retry cluster2 mirror image_13 2023-11-08T05:21:23.245 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:21:23.245 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:21:23.245 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_13 2023-11-08T05:21:23.246 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0 1 2 4 8 16 32 2023-11-08T05:21:23.246 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0 2023-11-08T05:21:23.247 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ remove_image cluster2 mirror image_13 2023-11-08T05:21:23.247 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:21:23.247 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:21:23.247 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_13 2023-11-08T05:21:23.247 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_13 2023-11-08T05:21:23.295 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster=cluster2 rm mirror/image_13 2023-11-08T05:21:23.403 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi087.stderr:2023-11-08T05:21:23.401+0000 7fc449df3700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55eab63876c0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-11-08T05:21:23.404 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi087.stderr:2023-11-08T05:21:23.401+0000 7fc44fdff700 -1 rbd::mirror::ImageReplayer: 0x55eab5e80a00 [3/cb42c52c-8a81-4992-b8fb-d273572391a5] operator(): start failed: (85) Interrupted system call should be restarted 2023-11-08T05:21:23.508 INFO:tasks.ceph.cluster2.osd.2.smithi087.stderr:2023-11-08T05:21:23.506+0000 7fa243317700 -1 reset not still connected to 0x5613bcfb7450 2023-11-08T05:21:23.509 INFO:tasks.ceph.cluster2.osd.2.smithi087.stderr:2023-11-08T05:21:23.506+0000 7fa243317700 -1 reset not still connected to 0x5613bdabfa00 2023-11-08T05:21:23.509 INFO:tasks.ceph.cluster2.osd.2.smithi087.stderr:2023-11-08T05:21:23.506+0000 7fa243317700 -1 reset not still connected to 0x5613c8847110 2023-11-08T05:21:24.456 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-11-08T05:21:24.459 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:21:24.459 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-11-08T05:21:24.459 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image=image_14 2023-11-08T05:21:24.460 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ remove_snapshot cluster2 mirror image_14 snap 2023-11-08T05:21:24.460 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:21:24.460 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:21:24.460 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_14 2023-11-08T05:21:24.460 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap=snap 2023-11-08T05:21:24.461 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 snap rm mirror/image_14@snap 2023-11-08T05:21:25.326 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Removing snap: 100% complete...done. 2023-11-08T05:21:25.342 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ remove_image_retry cluster2 mirror image_14 2023-11-08T05:21:25.342 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:21:25.343 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:21:25.343 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_14 2023-11-08T05:21:25.343 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0 1 2 4 8 16 32 2023-11-08T05:21:25.343 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0 2023-11-08T05:21:25.344 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ remove_image cluster2 mirror image_14 2023-11-08T05:21:25.344 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:21:25.344 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:21:25.345 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_14 2023-11-08T05:21:25.345 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_14 2023-11-08T05:21:25.386 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster=cluster2 rm mirror/image_14 2023-11-08T05:21:25.443 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi087.stderr:2023-11-08T05:21:25.441+0000 7fb6b5f8d700 -1 rbd::mirror::ImageMap: 0x55dd2e7de7e0 handle_peer_ack_remove: failed to remove global_image_id=2e7c6dbb-61d7-4f4a-9510-adb4bfb2cb02 2023-11-08T05:21:26.472 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-11-08T05:21:26.475 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:21:26.475 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-11-08T05:21:26.476 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image=image_15 2023-11-08T05:21:26.476 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ remove_snapshot cluster2 mirror image_15 snap 2023-11-08T05:21:26.476 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:21:26.477 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:21:26.477 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_15 2023-11-08T05:21:26.477 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap=snap 2023-11-08T05:21:26.477 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 snap rm mirror/image_15@snap 2023-11-08T05:21:27.335 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Removing snap: 100% complete...done. 2023-11-08T05:21:27.344 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ remove_image_retry cluster2 mirror image_15 2023-11-08T05:21:27.345 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:21:27.345 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:21:27.345 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_15 2023-11-08T05:21:27.345 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0 1 2 4 8 16 32 2023-11-08T05:21:27.346 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0 2023-11-08T05:21:27.346 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ remove_image cluster2 mirror image_15 2023-11-08T05:21:27.347 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:21:27.347 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:21:27.347 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_15 2023-11-08T05:21:27.347 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_15 2023-11-08T05:21:27.389 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster=cluster2 rm mirror/image_15 2023-11-08T05:21:27.502 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi087.stderr:2023-11-08T05:21:27.500+0000 7ff070b86700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x563146e0f040 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-11-08T05:21:27.502 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi087.stderr:2023-11-08T05:21:27.500+0000 7ff077b94700 -1 rbd::mirror::ImageReplayer: 0x563146b98000 [3/c5f2bed1-9f45-45f4-aaf6-7a50884f15f8] operator(): start failed: (85) Interrupted system call should be restarted 2023-11-08T05:21:28.376 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-11-08T05:21:28.379 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:21:28.380 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-11-08T05:21:28.380 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image=image_16 2023-11-08T05:21:28.380 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ remove_snapshot cluster2 mirror image_16 snap 2023-11-08T05:21:28.380 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:21:28.380 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:21:28.381 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_16 2023-11-08T05:21:28.381 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap=snap 2023-11-08T05:21:28.381 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 snap rm mirror/image_16@snap 2023-11-08T05:21:29.341 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Removing snap: 100% complete...done. 2023-11-08T05:21:29.354 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ remove_image_retry cluster2 mirror image_16 2023-11-08T05:21:29.355 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:21:29.355 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:21:29.355 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_16 2023-11-08T05:21:29.356 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0 1 2 4 8 16 32 2023-11-08T05:21:29.356 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0 2023-11-08T05:21:29.357 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ remove_image cluster2 mirror image_16 2023-11-08T05:21:29.357 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:21:29.357 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:21:29.357 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_16 2023-11-08T05:21:29.357 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_16 2023-11-08T05:21:29.401 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster=cluster2 rm mirror/image_16 2023-11-08T05:21:29.535 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi087.stderr:2023-11-08T05:21:29.533+0000 7f4436702700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x560b01982000 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-11-08T05:21:29.535 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi087.stderr:2023-11-08T05:21:29.533+0000 7f443c70e700 -1 rbd::mirror::ImageReplayer: 0x560afd790f00 [3/6852ffd8-136b-481a-b662-fdde8fec627d] operator(): start failed: (85) Interrupted system call should be restarted 2023-11-08T05:21:30.489 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-11-08T05:21:30.493 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:21:30.493 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-11-08T05:21:30.493 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image=image_17 2023-11-08T05:21:30.493 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ remove_snapshot cluster2 mirror image_17 snap 2023-11-08T05:21:30.494 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:21:30.494 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:21:30.494 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_17 2023-11-08T05:21:30.494 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap=snap 2023-11-08T05:21:30.495 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 snap rm mirror/image_17@snap 2023-11-08T05:21:31.354 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Removing snap: 100% complete...done. 2023-11-08T05:21:31.364 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ remove_image_retry cluster2 mirror image_17 2023-11-08T05:21:31.364 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:21:31.365 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:21:31.365 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_17 2023-11-08T05:21:31.365 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0 1 2 4 8 16 32 2023-11-08T05:21:31.365 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0 2023-11-08T05:21:31.366 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ remove_image cluster2 mirror image_17 2023-11-08T05:21:31.366 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:21:31.366 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:21:31.366 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_17 2023-11-08T05:21:31.367 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_17 2023-11-08T05:21:31.411 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster=cluster2 rm mirror/image_17 2023-11-08T05:21:31.524 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi087.stderr:2023-11-08T05:21:31.522+0000 7f4436702700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x560b01985040 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-11-08T05:21:31.524 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi087.stderr:2023-11-08T05:21:31.522+0000 7f443cf0f700 -1 rbd::mirror::ImageReplayer: 0x560afd790500 [3/7b62298f-cd5f-402a-b9f9-d2211585565d] operator(): start failed: (85) Interrupted system call should be restarted 2023-11-08T05:21:32.495 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-11-08T05:21:32.498 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:21:32.499 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-11-08T05:21:32.499 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image=image_18 2023-11-08T05:21:32.499 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ remove_snapshot cluster2 mirror image_18 snap 2023-11-08T05:21:32.499 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:21:32.500 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:21:32.500 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_18 2023-11-08T05:21:32.500 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap=snap 2023-11-08T05:21:32.500 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 snap rm mirror/image_18@snap 2023-11-08T05:21:33.239 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Removing snap: 100% complete...done. 2023-11-08T05:21:33.250 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ remove_image_retry cluster2 mirror image_18 2023-11-08T05:21:33.250 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:21:33.251 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:21:33.251 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_18 2023-11-08T05:21:33.251 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0 1 2 4 8 16 32 2023-11-08T05:21:33.251 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0 2023-11-08T05:21:33.252 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ remove_image cluster2 mirror image_18 2023-11-08T05:21:33.252 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:21:33.252 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:21:33.253 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_18 2023-11-08T05:21:33.253 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_18 2023-11-08T05:21:33.303 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster=cluster2 rm mirror/image_18 2023-11-08T05:21:33.361 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi087.stderr:2023-11-08T05:21:33.359+0000 7fb6b578c700 -1 rbd::mirror::ImageMap: 0x55dd2e7de7e0 handle_peer_ack_remove: failed to remove global_image_id=2645822b-c23b-4b5d-bdd0-8114af333323 2023-11-08T05:21:34.502 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-11-08T05:21:34.505 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:21:34.505 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-11-08T05:21:34.506 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image=image_19 2023-11-08T05:21:34.506 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ remove_snapshot cluster2 mirror image_19 snap 2023-11-08T05:21:34.506 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:21:34.506 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:21:34.507 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_19 2023-11-08T05:21:34.507 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap=snap 2023-11-08T05:21:34.507 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 snap rm mirror/image_19@snap 2023-11-08T05:21:35.375 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Removing snap: 100% complete...done. 2023-11-08T05:21:35.385 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ remove_image_retry cluster2 mirror image_19 2023-11-08T05:21:35.385 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:21:35.386 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:21:35.386 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_19 2023-11-08T05:21:35.386 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0 1 2 4 8 16 32 2023-11-08T05:21:35.386 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0 2023-11-08T05:21:35.387 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ remove_image cluster2 mirror image_19 2023-11-08T05:21:35.387 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:21:35.388 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:21:35.388 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_19 2023-11-08T05:21:35.388 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_19 2023-11-08T05:21:35.427 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster=cluster2 rm mirror/image_19 2023-11-08T05:21:35.532 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi087.stderr:2023-11-08T05:21:35.530+0000 7f45a49ea700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55b0ce6e41a0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-11-08T05:21:35.532 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi087.stderr:2023-11-08T05:21:35.530+0000 7f45ab1f7700 -1 rbd::mirror::ImageReplayer: 0x55b0ce53af00 [3/783a1447-f0d7-471d-9452-7cb8dece9780] operator(): start failed: (85) Interrupted system call should be restarted 2023-11-08T05:21:36.509 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-11-08T05:21:36.512 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:21:36.512 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-11-08T05:21:36.513 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image=image_20 2023-11-08T05:21:36.513 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ remove_snapshot cluster2 mirror image_20 snap 2023-11-08T05:21:36.513 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:21:36.513 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:21:36.513 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_20 2023-11-08T05:21:36.514 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap=snap 2023-11-08T05:21:36.514 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 snap rm mirror/image_20@snap 2023-11-08T05:21:37.380 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Removing snap: 100% complete...done. 2023-11-08T05:21:37.387 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ remove_image_retry cluster2 mirror image_20 2023-11-08T05:21:37.387 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:21:37.388 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:21:37.388 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_20 2023-11-08T05:21:37.388 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0 1 2 4 8 16 32 2023-11-08T05:21:37.388 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0 2023-11-08T05:21:37.389 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ remove_image cluster2 mirror image_20 2023-11-08T05:21:37.389 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:21:37.390 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:21:37.390 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_20 2023-11-08T05:21:37.390 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_20 2023-11-08T05:21:37.429 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster=cluster2 rm mirror/image_20 2023-11-08T05:21:37.542 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi087.stderr:2023-11-08T05:21:37.540+0000 7ff070b86700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x563146e0f1e0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-11-08T05:21:37.542 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi087.stderr:2023-11-08T05:21:37.540+0000 7ff077b94700 -1 rbd::mirror::ImageReplayer: 0x563142aac280 [3/d1ee9201-2afc-4853-a96a-0fbcc08cbfe5] operator(): start failed: (85) Interrupted system call should be restarted 2023-11-08T05:21:38.384 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-11-08T05:21:38.388 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:21:38.388 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-11-08T05:21:38.388 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image=image_21 2023-11-08T05:21:38.388 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ remove_snapshot cluster2 mirror image_21 snap 2023-11-08T05:21:38.389 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:21:38.389 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:21:38.389 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_21 2023-11-08T05:21:38.389 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap=snap 2023-11-08T05:21:38.390 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 snap rm mirror/image_21@snap 2023-11-08T05:21:39.390 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Removing snap: 100% complete...done. 2023-11-08T05:21:39.400 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ remove_image_retry cluster2 mirror image_21 2023-11-08T05:21:39.400 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:21:39.400 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:21:39.400 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_21 2023-11-08T05:21:39.400 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0 1 2 4 8 16 32 2023-11-08T05:21:39.401 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0 2023-11-08T05:21:39.401 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ remove_image cluster2 mirror image_21 2023-11-08T05:21:39.402 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:21:39.402 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:21:39.402 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_21 2023-11-08T05:21:39.402 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_21 2023-11-08T05:21:39.442 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster=cluster2 rm mirror/image_21 2023-11-08T05:21:39.497 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi087.stderr:2023-11-08T05:21:39.496+0000 7fb6b578c700 -1 rbd::mirror::ImageMap: 0x55dd2e7de7e0 handle_peer_ack_remove: failed to remove global_image_id=1d89653a-15af-4c63-92d4-196f96656cd3 2023-11-08T05:21:40.522 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-11-08T05:21:40.525 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:21:40.525 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-11-08T05:21:40.525 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image=image_22 2023-11-08T05:21:40.526 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ remove_snapshot cluster2 mirror image_22 snap 2023-11-08T05:21:40.526 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:21:40.526 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:21:40.526 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_22 2023-11-08T05:21:40.526 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap=snap 2023-11-08T05:21:40.527 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 snap rm mirror/image_22@snap 2023-11-08T05:21:41.394 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Removing snap: 100% complete...done. 2023-11-08T05:21:41.405 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ remove_image_retry cluster2 mirror image_22 2023-11-08T05:21:41.405 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:21:41.406 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:21:41.406 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_22 2023-11-08T05:21:41.406 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0 1 2 4 8 16 32 2023-11-08T05:21:41.406 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0 2023-11-08T05:21:41.407 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ remove_image cluster2 mirror image_22 2023-11-08T05:21:41.407 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:21:41.407 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:21:41.408 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_22 2023-11-08T05:21:41.408 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_22 2023-11-08T05:21:41.452 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster=cluster2 rm mirror/image_22 2023-11-08T05:21:41.558 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi087.stderr:2023-11-08T05:21:41.557+0000 7fc449df3700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55eab634b6c0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-11-08T05:21:41.559 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi087.stderr:2023-11-08T05:21:41.557+0000 7fc450600700 -1 rbd::mirror::ImageReplayer: 0x55eab1f49680 [3/b8cf32bf-f325-43e1-a374-ee759365a5a1] operator(): start failed: (85) Interrupted system call should be restarted 2023-11-08T05:21:42.537 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-11-08T05:21:42.541 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:21:42.541 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-11-08T05:21:42.541 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image=image_23 2023-11-08T05:21:42.541 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ remove_snapshot cluster2 mirror image_23 snap 2023-11-08T05:21:42.541 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:21:42.542 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:21:42.542 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_23 2023-11-08T05:21:42.542 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap=snap 2023-11-08T05:21:42.542 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 snap rm mirror/image_23@snap 2023-11-08T05:21:43.239 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Removing snap: 100% complete...done. 2023-11-08T05:21:43.247 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ remove_image_retry cluster2 mirror image_23 2023-11-08T05:21:43.247 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:21:43.247 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:21:43.248 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_23 2023-11-08T05:21:43.248 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0 1 2 4 8 16 32 2023-11-08T05:21:43.248 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0 2023-11-08T05:21:43.249 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ remove_image cluster2 mirror image_23 2023-11-08T05:21:43.249 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:21:43.249 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:21:43.249 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_23 2023-11-08T05:21:43.249 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_23 2023-11-08T05:21:43.287 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster=cluster2 rm mirror/image_23 2023-11-08T05:21:43.400 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi087.stderr:2023-11-08T05:21:43.398+0000 7f4435f01700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x560b01982340 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-11-08T05:21:43.400 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi087.stderr:2023-11-08T05:21:43.398+0000 7f443c70e700 -1 rbd::mirror::ImageReplayer: 0x560afd791900 [3/b0c3eb50-dd81-436f-b795-2235c773f092] operator(): start failed: (85) Interrupted system call should be restarted 2023-11-08T05:21:44.378 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-11-08T05:21:44.382 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:21:44.382 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-11-08T05:21:44.382 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image=image_24 2023-11-08T05:21:44.382 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ remove_snapshot cluster2 mirror image_24 snap 2023-11-08T05:21:44.382 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:21:44.383 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:21:44.383 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_24 2023-11-08T05:21:44.383 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap=snap 2023-11-08T05:21:44.383 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 snap rm mirror/image_24@snap 2023-11-08T05:21:45.402 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Removing snap: 100% complete...done. 2023-11-08T05:21:45.414 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ remove_image_retry cluster2 mirror image_24 2023-11-08T05:21:45.415 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:21:45.415 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:21:45.415 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_24 2023-11-08T05:21:45.415 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0 1 2 4 8 16 32 2023-11-08T05:21:45.416 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0 2023-11-08T05:21:45.417 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ remove_image cluster2 mirror image_24 2023-11-08T05:21:45.417 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:21:45.417 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:21:45.418 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_24 2023-11-08T05:21:45.418 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_24 2023-11-08T05:21:45.458 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster=cluster2 rm mirror/image_24 2023-11-08T05:21:45.638 INFO:tasks.ceph.cluster1.osd.2.smithi077.stderr:2023-11-08T05:21:45.635+0000 7f9900abf700 -1 reset not still connected to 0x555e974f0680 2023-11-08T05:21:45.663 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi087.stderr:2023-11-08T05:21:45.659+0000 7ff071387700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x563146e0f380 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-11-08T05:21:45.663 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi087.stderr:2023-11-08T05:21:45.659+0000 7ff077b94700 -1 rbd::mirror::ImageReplayer: 0x563146b98f00 [3/4bf12ca9-9a70-496b-8c0e-da3173089291] operator(): start failed: (85) Interrupted system call should be restarted 2023-11-08T05:21:46.543 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-11-08T05:21:46.546 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:21:46.546 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-11-08T05:21:46.547 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image=image_25 2023-11-08T05:21:46.547 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ remove_snapshot cluster2 mirror image_25 snap 2023-11-08T05:21:46.547 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:21:46.547 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:21:46.547 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_25 2023-11-08T05:21:46.548 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap=snap 2023-11-08T05:21:46.548 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 snap rm mirror/image_25@snap 2023-11-08T05:21:47.404 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Removing snap: 100% complete...done. 2023-11-08T05:21:47.414 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ remove_image_retry cluster2 mirror image_25 2023-11-08T05:21:47.415 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:21:47.415 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:21:47.415 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_25 2023-11-08T05:21:47.415 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0 1 2 4 8 16 32 2023-11-08T05:21:47.416 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0 2023-11-08T05:21:47.416 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ remove_image cluster2 mirror image_25 2023-11-08T05:21:47.416 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:21:47.417 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:21:47.417 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_25 2023-11-08T05:21:47.417 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_25 2023-11-08T05:21:47.469 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster=cluster2 rm mirror/image_25 2023-11-08T05:21:47.596 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi087.stderr:2023-11-08T05:21:47.594+0000 7f4436702700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x560b01982680 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-11-08T05:21:47.596 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi087.stderr:2023-11-08T05:21:47.594+0000 7f443c70e700 -1 rbd::mirror::ImageReplayer: 0x560b018a8c80 [3/4f72c9b1-e028-4538-ade8-ae09b2818aa9] operator(): start failed: (85) Interrupted system call should be restarted 2023-11-08T05:21:48.385 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-11-08T05:21:48.388 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:21:48.388 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-11-08T05:21:48.389 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image=image_26 2023-11-08T05:21:48.389 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ remove_snapshot cluster2 mirror image_26 snap 2023-11-08T05:21:48.389 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:21:48.389 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:21:48.389 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_26 2023-11-08T05:21:48.390 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap=snap 2023-11-08T05:21:48.390 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 snap rm mirror/image_26@snap 2023-11-08T05:21:49.414 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Removing snap: 100% complete...done. 2023-11-08T05:21:49.428 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ remove_image_retry cluster2 mirror image_26 2023-11-08T05:21:49.428 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:21:49.429 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:21:49.429 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_26 2023-11-08T05:21:49.429 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0 1 2 4 8 16 32 2023-11-08T05:21:49.430 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0 2023-11-08T05:21:49.431 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ remove_image cluster2 mirror image_26 2023-11-08T05:21:49.431 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:21:49.431 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:21:49.432 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_26 2023-11-08T05:21:49.432 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_26 2023-11-08T05:21:49.473 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster=cluster2 rm mirror/image_26 2023-11-08T05:21:49.627 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi087.stderr:2023-11-08T05:21:49.625+0000 7ff070b86700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x563146e0f520 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-11-08T05:21:49.628 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi087.stderr:2023-11-08T05:21:49.625+0000 7ff077b94700 -1 rbd::mirror::ImageReplayer: 0x563142aac500 [3/fff00620-df68-4ab5-8679-1bb60d01c79a] operator(): start failed: (85) Interrupted system call should be restarted 2023-11-08T05:21:50.545 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-11-08T05:21:50.549 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:21:50.549 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-11-08T05:21:50.549 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image=image_27 2023-11-08T05:21:50.549 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ remove_snapshot cluster2 mirror image_27 snap 2023-11-08T05:21:50.550 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:21:50.550 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:21:50.550 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_27 2023-11-08T05:21:50.550 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap=snap 2023-11-08T05:21:50.550 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 snap rm mirror/image_27@snap 2023-11-08T05:21:51.429 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Removing snap: 100% complete...done. 2023-11-08T05:21:51.441 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ remove_image_retry cluster2 mirror image_27 2023-11-08T05:21:51.441 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:21:51.442 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:21:51.442 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_27 2023-11-08T05:21:51.442 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0 1 2 4 8 16 32 2023-11-08T05:21:51.442 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0 2023-11-08T05:21:51.443 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ remove_image cluster2 mirror image_27 2023-11-08T05:21:51.443 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:21:51.443 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:21:51.444 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_27 2023-11-08T05:21:51.444 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_27 2023-11-08T05:21:51.483 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster=cluster2 rm mirror/image_27 2023-11-08T05:21:51.596 INFO:tasks.ceph.cluster1.osd.0.smithi077.stderr:2023-11-08T05:21:51.595+0000 7f3987314700 -1 reset not still connected to 0x559af0c711e0 2023-11-08T05:21:51.597 INFO:tasks.ceph.cluster1.osd.0.smithi077.stderr:2023-11-08T05:21:51.595+0000 7f3987314700 -1 reset not still connected to 0x559af0c71e10 2023-11-08T05:21:51.597 INFO:tasks.ceph.cluster1.osd.0.smithi077.stderr:2023-11-08T05:21:51.595+0000 7f3987314700 -1 reset not still connected to 0x559af52c5450 2023-11-08T05:21:51.597 INFO:tasks.ceph.cluster1.osd.0.smithi077.stderr:2023-11-08T05:21:51.595+0000 7f3987314700 -1 reset not still connected to 0x559af557eea0 2023-11-08T05:21:51.597 INFO:tasks.ceph.cluster1.osd.0.smithi077.stderr:2023-11-08T05:21:51.595+0000 7f3987314700 -1 reset not still connected to 0x559af5a68680 2023-11-08T05:21:51.598 INFO:tasks.ceph.cluster1.osd.0.smithi077.stderr:2023-11-08T05:21:51.595+0000 7f3987314700 -1 reset not still connected to 0x559af699a410 2023-11-08T05:21:51.598 INFO:tasks.ceph.cluster1.osd.0.smithi077.stderr:2023-11-08T05:21:51.595+0000 7f3987314700 -1 reset not still connected to 0x559af699a8f0 2023-11-08T05:21:51.634 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi087.stderr:2023-11-08T05:21:51.632+0000 7ff071387700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x56314439da00 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-11-08T05:21:51.634 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi087.stderr:2023-11-08T05:21:51.632+0000 7ff077393700 -1 rbd::mirror::ImageReplayer: 0x563142aac780 [3/ebbdc03f-1275-4b8e-b5a6-94b379cd1440] operator(): start failed: (85) Interrupted system call should be restarted 2023-11-08T05:21:52.560 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-11-08T05:21:52.563 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:21:52.564 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-11-08T05:21:52.564 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image=image_28 2023-11-08T05:21:52.564 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ remove_snapshot cluster2 mirror image_28 snap 2023-11-08T05:21:52.564 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:21:52.565 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:21:52.565 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_28 2023-11-08T05:21:52.565 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap=snap 2023-11-08T05:21:52.566 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 snap rm mirror/image_28@snap 2023-11-08T05:21:53.242 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Removing snap: 100% complete...done. 2023-11-08T05:21:53.253 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ remove_image_retry cluster2 mirror image_28 2023-11-08T05:21:53.253 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:21:53.253 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:21:53.253 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_28 2023-11-08T05:21:53.254 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0 1 2 4 8 16 32 2023-11-08T05:21:53.254 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0 2023-11-08T05:21:53.255 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ remove_image cluster2 mirror image_28 2023-11-08T05:21:53.255 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:21:53.255 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:21:53.255 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_28 2023-11-08T05:21:53.255 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_28 2023-11-08T05:21:53.294 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster=cluster2 rm mirror/image_28 2023-11-08T05:21:53.358 INFO:tasks.ceph.cluster2.osd.2.smithi087.stderr:2023-11-08T05:21:53.356+0000 7fa243317700 -1 reset not still connected to 0x5613b8843860 2023-11-08T05:21:53.358 INFO:tasks.ceph.cluster2.osd.2.smithi087.stderr:2023-11-08T05:21:53.356+0000 7fa243317700 -1 reset not still connected to 0x5613b907d930 2023-11-08T05:21:53.359 INFO:tasks.ceph.cluster2.osd.2.smithi087.stderr:2023-11-08T05:21:53.356+0000 7fa243317700 -1 reset not still connected to 0x5613bcfb7450 2023-11-08T05:21:53.359 INFO:tasks.ceph.cluster2.osd.2.smithi087.stderr:2023-11-08T05:21:53.356+0000 7fa243317700 -1 reset not still connected to 0x5613bdabfa00 2023-11-08T05:21:53.359 INFO:tasks.ceph.cluster2.osd.2.smithi087.stderr:2023-11-08T05:21:53.356+0000 7fa243317700 -1 reset not still connected to 0x5613c8847110 2023-11-08T05:21:53.400 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi087.stderr:2023-11-08T05:21:53.398+0000 7f45a49ea700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55b0ce6cc9c0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-11-08T05:21:53.400 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi087.stderr:2023-11-08T05:21:53.398+0000 7f45ab1f7700 -1 rbd::mirror::ImageReplayer: 0x55b0ce53a780 [3/06978a7d-19c8-42c0-8df8-1e3a18982290] operator(): start failed: (85) Interrupted system call should be restarted 2023-11-08T05:21:54.558 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-11-08T05:21:54.561 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:21:54.561 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-11-08T05:21:54.561 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image=image_29 2023-11-08T05:21:54.561 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ remove_snapshot cluster2 mirror image_29 snap 2023-11-08T05:21:54.562 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:21:54.562 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:21:54.562 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_29 2023-11-08T05:21:54.562 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap=snap 2023-11-08T05:21:54.562 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 snap rm mirror/image_29@snap 2023-11-08T05:21:55.436 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Removing snap: 100% complete...done. 2023-11-08T05:21:55.450 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ remove_image_retry cluster2 mirror image_29 2023-11-08T05:21:55.451 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:21:55.451 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:21:55.451 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_29 2023-11-08T05:21:55.451 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0 1 2 4 8 16 32 2023-11-08T05:21:55.452 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0 2023-11-08T05:21:55.452 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ remove_image cluster2 mirror image_29 2023-11-08T05:21:55.453 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:21:55.453 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:21:55.453 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_29 2023-11-08T05:21:55.454 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_29 2023-11-08T05:21:55.491 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster=cluster2 rm mirror/image_29 2023-11-08T05:21:55.645 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi087.stderr:2023-11-08T05:21:55.643+0000 7f45a49ea700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55b0ce6ccd00 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-11-08T05:21:55.646 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi087.stderr:2023-11-08T05:21:55.643+0000 7f45ab1f7700 -1 rbd::mirror::ImageReplayer: 0x55b0ca754c80 [3/224b796b-9a1e-47f0-a771-84babb70a923] operator(): start failed: (85) Interrupted system call should be restarted 2023-11-08T05:21:56.571 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-11-08T05:21:56.574 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:21:56.574 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-11-08T05:21:56.574 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image=image_30 2023-11-08T05:21:56.575 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ remove_snapshot cluster2 mirror image_30 snap 2023-11-08T05:21:56.575 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:21:56.575 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:21:56.575 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_30 2023-11-08T05:21:56.575 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap=snap 2023-11-08T05:21:56.576 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 snap rm mirror/image_30@snap 2023-11-08T05:21:57.439 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Removing snap: 100% complete...done. 2023-11-08T05:21:57.448 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ remove_image_retry cluster2 mirror image_30 2023-11-08T05:21:57.449 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:21:57.449 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:21:57.449 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_30 2023-11-08T05:21:57.450 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0 1 2 4 8 16 32 2023-11-08T05:21:57.450 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0 2023-11-08T05:21:57.452 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ remove_image cluster2 mirror image_30 2023-11-08T05:21:57.452 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:21:57.452 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:21:57.453 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_30 2023-11-08T05:21:57.453 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_30 2023-11-08T05:21:57.490 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster=cluster2 rm mirror/image_30 2023-11-08T05:21:57.612 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi087.stderr:2023-11-08T05:21:57.610+0000 7ff070b86700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x563146e0f6c0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-11-08T05:21:57.612 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi087.stderr:2023-11-08T05:21:57.610+0000 7ff077b94700 -1 rbd::mirror::ImageReplayer: 0x563146b98280 [3/c78f7ab4-c752-49b4-8cd9-bc194f1db211] operator(): start failed: (85) Interrupted system call should be restarted 2023-11-08T05:21:58.119 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.43368282159695004) > weight (0.25) 2023-11-08T05:21:58.119 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.5320966000940618) > weight (0.25) 2023-11-08T05:21:58.120 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.8579615270708726) > weight (0.25) 2023-11-08T05:21:58.120 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.3 2023-11-08T05:21:58.120 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Sent signal 15 2023-11-08T05:21:58.121 INFO:tasks.rbd_mirror_thrash:waiting for 6 secs before reviving daemons 2023-11-08T05:21:58.121 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi087.stderr:2023-11-08T05:21:58.120+0000 7ff078395700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3 (PID: 132855) UID: 1000 2023-11-08T05:21:58.324 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi087.stderr:daemon-helper: command failed with exit status 1 2023-11-08T05:21:58.383 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-11-08T05:21:58.386 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:21:58.386 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-11-08T05:21:58.387 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image=image_31 2023-11-08T05:21:58.387 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ remove_snapshot cluster2 mirror image_31 snap 2023-11-08T05:21:58.387 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:21:58.387 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:21:58.387 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_31 2023-11-08T05:21:58.388 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap=snap 2023-11-08T05:21:58.388 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 snap rm mirror/image_31@snap 2023-11-08T05:21:59.438 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Removing snap: 100% complete...done. 2023-11-08T05:21:59.451 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ remove_image_retry cluster2 mirror image_31 2023-11-08T05:21:59.451 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:21:59.452 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:21:59.452 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_31 2023-11-08T05:21:59.452 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0 1 2 4 8 16 32 2023-11-08T05:21:59.452 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0 2023-11-08T05:21:59.454 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ remove_image cluster2 mirror image_31 2023-11-08T05:21:59.454 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:21:59.454 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:21:59.455 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_31 2023-11-08T05:21:59.455 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_31 2023-11-08T05:21:59.497 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster=cluster2 rm mirror/image_31 2023-11-08T05:21:59.635 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi087.stderr:2023-11-08T05:21:59.633+0000 7fc44fdff700 -1 rbd::mirror::ImageMap: 0x55eab5bad440 handle_peer_ack_remove: failed to remove global_image_id=50f6deef-3c6f-490a-bab8-f906a3136c2a 2023-11-08T05:22:00.588 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-11-08T05:22:00.591 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:22:00.592 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-11-08T05:22:00.592 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image=image_32 2023-11-08T05:22:00.592 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ remove_snapshot cluster2 mirror image_32 snap 2023-11-08T05:22:00.593 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:22:00.593 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:22:00.593 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_32 2023-11-08T05:22:00.594 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap=snap 2023-11-08T05:22:00.594 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 snap rm mirror/image_32@snap 2023-11-08T05:22:01.453 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Removing snap: 100% complete...done. 2023-11-08T05:22:01.464 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ remove_image_retry cluster2 mirror image_32 2023-11-08T05:22:01.464 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:22:01.465 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:22:01.465 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_32 2023-11-08T05:22:01.465 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0 1 2 4 8 16 32 2023-11-08T05:22:01.466 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0 2023-11-08T05:22:01.467 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ remove_image cluster2 mirror image_32 2023-11-08T05:22:01.467 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:22:01.467 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:22:01.468 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_32 2023-11-08T05:22:01.468 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_32 2023-11-08T05:22:01.509 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster=cluster2 rm mirror/image_32 2023-11-08T05:22:01.648 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi087.stderr:2023-11-08T05:22:01.645+0000 7fc44fdff700 -1 rbd::mirror::ImageMap: 0x55eab5bad440 handle_peer_ack_remove: failed to remove global_image_id=e6ec2241-9334-4ae3-ae69-78e6eb1a509b 2023-11-08T05:22:02.597 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-11-08T05:22:02.600 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:22:02.600 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-11-08T05:22:02.600 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image=image_33 2023-11-08T05:22:02.600 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ remove_snapshot cluster2 mirror image_33 snap 2023-11-08T05:22:02.600 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:22:02.601 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:22:02.601 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_33 2023-11-08T05:22:02.601 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap=snap 2023-11-08T05:22:02.601 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 snap rm mirror/image_33@snap 2023-11-08T05:22:03.250 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Removing snap: 100% complete...done. 2023-11-08T05:22:03.256 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ remove_image_retry cluster2 mirror image_33 2023-11-08T05:22:03.256 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:22:03.257 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:22:03.257 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_33 2023-11-08T05:22:03.257 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0 1 2 4 8 16 32 2023-11-08T05:22:03.257 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0 2023-11-08T05:22:03.258 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ remove_image cluster2 mirror image_33 2023-11-08T05:22:03.258 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:22:03.259 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:22:03.259 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_33 2023-11-08T05:22:03.259 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_33 2023-11-08T05:22:03.312 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster=cluster2 rm mirror/image_33 2023-11-08T05:22:03.418 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi087.stderr:2023-11-08T05:22:03.415+0000 7f45a49ea700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55b0ce6cd040 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-11-08T05:22:03.418 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi087.stderr:2023-11-08T05:22:03.415+0000 7f45ab1f7700 -1 rbd::mirror::ImageReplayer: 0x55b0ca755680 [3/6f088a8f-d677-4bda-b50c-7fba63dd233e] operator(): start failed: (85) Interrupted system call should be restarted 2023-11-08T05:22:04.121 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.3 2023-11-08T05:22:04.122 INFO:teuthology.orchestra.run:waiting for 600 2023-11-08T05:22:04.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-08T05:22:04.123 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.3 2023-11-08T05:22:04.124 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Restarting daemon 2023-11-08T05:22:04.124 DEBUG:teuthology.orchestra.run.smithi087:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3 2023-11-08T05:22:04.129 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Started 2023-11-08T05:22:04.129 INFO:tasks.rbd_mirror_thrash:waiting for 83 secs before thrashing 2023-11-08T05:22:04.410 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-11-08T05:22:04.413 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:22:04.414 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-11-08T05:22:04.414 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image=image_34 2023-11-08T05:22:04.414 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ remove_snapshot cluster2 mirror image_34 snap 2023-11-08T05:22:04.414 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:22:04.414 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:22:04.415 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_34 2023-11-08T05:22:04.415 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap=snap 2023-11-08T05:22:04.415 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 snap rm mirror/image_34@snap 2023-11-08T05:22:05.472 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Removing snap: 100% complete...done. 2023-11-08T05:22:05.483 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ remove_image_retry cluster2 mirror image_34 2023-11-08T05:22:05.483 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:22:05.483 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:22:05.484 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_34 2023-11-08T05:22:05.484 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0 1 2 4 8 16 32 2023-11-08T05:22:05.484 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0 2023-11-08T05:22:05.485 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ remove_image cluster2 mirror image_34 2023-11-08T05:22:05.485 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:22:05.485 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:22:05.486 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_34 2023-11-08T05:22:05.486 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_34 2023-11-08T05:22:05.528 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster=cluster2 rm mirror/image_34 2023-11-08T05:22:05.674 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi087.stderr:2023-11-08T05:22:05.671+0000 7f4435f01700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x560b019824e0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-11-08T05:22:05.674 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi087.stderr:2023-11-08T05:22:05.671+0000 7f443c70e700 -1 rbd::mirror::ImageReplayer: 0x560afd791400 [3/9fa92707-a7d6-4c46-967b-87ad9015703b] operator(): start failed: (85) Interrupted system call should be restarted 2023-11-08T05:22:06.606 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-11-08T05:22:06.609 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:22:06.610 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-11-08T05:22:06.610 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image=image_35 2023-11-08T05:22:06.610 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ remove_snapshot cluster2 mirror image_35 snap 2023-11-08T05:22:06.610 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:22:06.610 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:22:06.611 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_35 2023-11-08T05:22:06.611 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap=snap 2023-11-08T05:22:06.611 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 snap rm mirror/image_35@snap 2023-11-08T05:22:07.478 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Removing snap: 100% complete...done. 2023-11-08T05:22:07.485 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ remove_image_retry cluster2 mirror image_35 2023-11-08T05:22:07.486 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:22:07.486 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:22:07.486 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_35 2023-11-08T05:22:07.486 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0 1 2 4 8 16 32 2023-11-08T05:22:07.487 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0 2023-11-08T05:22:07.487 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ remove_image cluster2 mirror image_35 2023-11-08T05:22:07.488 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:22:07.488 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:22:07.488 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_35 2023-11-08T05:22:07.488 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_35 2023-11-08T05:22:07.528 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster=cluster2 rm mirror/image_35 2023-11-08T05:22:07.634 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi087.stderr:2023-11-08T05:22:07.632+0000 7fc4495f2700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55eab6159040 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-11-08T05:22:07.635 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi087.stderr:2023-11-08T05:22:07.632+0000 7fc450600700 -1 rbd::mirror::ImageReplayer: 0x55eab1f49180 [3/4f580749-5b1c-4916-870b-0406e147094b] operator(): start failed: (85) Interrupted system call should be restarted 2023-11-08T05:22:08.376 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-11-08T05:22:08.379 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:22:08.379 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-11-08T05:22:08.379 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image=image_36 2023-11-08T05:22:08.380 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ remove_snapshot cluster2 mirror image_36 snap 2023-11-08T05:22:08.380 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:22:08.380 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:22:08.380 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_36 2023-11-08T05:22:08.381 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap=snap 2023-11-08T05:22:08.381 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 snap rm mirror/image_36@snap 2023-11-08T05:22:09.253 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Removing snap: 100% complete...done. 2023-11-08T05:22:09.262 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ remove_image_retry cluster2 mirror image_36 2023-11-08T05:22:09.263 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:22:09.263 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:22:09.263 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_36 2023-11-08T05:22:09.263 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0 1 2 4 8 16 32 2023-11-08T05:22:09.263 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0 2023-11-08T05:22:09.264 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ remove_image cluster2 mirror image_36 2023-11-08T05:22:09.265 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:22:09.265 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:22:09.265 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_36 2023-11-08T05:22:09.265 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_36 2023-11-08T05:22:09.302 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster=cluster2 rm mirror/image_36 2023-11-08T05:22:09.403 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi087.stderr:2023-11-08T05:22:09.401+0000 7f4435f01700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x560b01983380 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-11-08T05:22:09.404 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi087.stderr:2023-11-08T05:22:09.401+0000 7f443c70e700 -1 rbd::mirror::ImageReplayer: 0x560afd791400 [3/a868743f-f258-4032-ae79-7cb04d60e193] operator(): start failed: (85) Interrupted system call should be restarted 2023-11-08T05:22:10.370 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-11-08T05:22:10.374 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:22:10.374 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-11-08T05:22:10.374 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image=image_37 2023-11-08T05:22:10.375 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ remove_snapshot cluster2 mirror image_37 snap 2023-11-08T05:22:10.375 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:22:10.375 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:22:10.375 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_37 2023-11-08T05:22:10.376 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap=snap 2023-11-08T05:22:10.376 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 snap rm mirror/image_37@snap 2023-11-08T05:22:11.260 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Removing snap: 100% complete...done. 2023-11-08T05:22:11.270 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ remove_image_retry cluster2 mirror image_37 2023-11-08T05:22:11.270 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:22:11.270 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:22:11.271 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_37 2023-11-08T05:22:11.271 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0 1 2 4 8 16 32 2023-11-08T05:22:11.271 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0 2023-11-08T05:22:11.272 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ remove_image cluster2 mirror image_37 2023-11-08T05:22:11.272 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:22:11.272 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:22:11.272 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_37 2023-11-08T05:22:11.273 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_37 2023-11-08T05:22:11.309 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster=cluster2 rm mirror/image_37 2023-11-08T05:22:11.415 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi087.stderr:2023-11-08T05:22:11.414+0000 7f5483b76700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55813d3b51e0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-11-08T05:22:11.416 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi087.stderr:2023-11-08T05:22:11.414+0000 7f548a383700 -1 rbd::mirror::ImageReplayer: 0x558139612a00 [3/7c36d62d-ec90-4bfc-b81d-9054fbd92513] operator(): start failed: (85) Interrupted system call should be restarted 2023-11-08T05:22:12.395 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-11-08T05:22:12.398 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:22:12.398 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-11-08T05:22:12.399 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image=image_38 2023-11-08T05:22:12.399 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ remove_snapshot cluster2 mirror image_38 snap 2023-11-08T05:22:12.399 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:22:12.399 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:22:12.399 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_38 2023-11-08T05:22:12.400 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap=snap 2023-11-08T05:22:12.400 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 snap rm mirror/image_38@snap 2023-11-08T05:22:13.257 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Removing snap: 100% complete...done. 2023-11-08T05:22:13.265 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ remove_image_retry cluster2 mirror image_38 2023-11-08T05:22:13.265 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:22:13.265 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:22:13.265 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_38 2023-11-08T05:22:13.266 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0 1 2 4 8 16 32 2023-11-08T05:22:13.266 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0 2023-11-08T05:22:13.267 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ remove_image cluster2 mirror image_38 2023-11-08T05:22:13.267 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:22:13.267 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:22:13.267 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_38 2023-11-08T05:22:13.267 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_38 2023-11-08T05:22:13.318 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster=cluster2 rm mirror/image_38 2023-11-08T05:22:13.424 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi087.stderr:2023-11-08T05:22:13.422+0000 7f5483375700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55813aedcb60 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-11-08T05:22:13.424 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi087.stderr:2023-11-08T05:22:13.422+0000 7f548a383700 -1 rbd::mirror::ImageReplayer: 0x558139613180 [3/c3a4ad2f-2430-4db4-a78e-1b1f4255422f] operator(): start failed: (85) Interrupted system call should be restarted 2023-11-08T05:22:14.392 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-11-08T05:22:14.395 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:22:14.395 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-11-08T05:22:14.396 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image=image_39 2023-11-08T05:22:14.396 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ remove_snapshot cluster2 mirror image_39 snap 2023-11-08T05:22:14.396 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:22:14.396 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:22:14.396 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_39 2023-11-08T05:22:14.397 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap=snap 2023-11-08T05:22:14.397 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 snap rm mirror/image_39@snap 2023-11-08T05:22:15.266 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Removing snap: 100% complete...done. 2023-11-08T05:22:15.277 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ remove_image_retry cluster2 mirror image_39 2023-11-08T05:22:15.277 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:22:15.277 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:22:15.278 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_39 2023-11-08T05:22:15.278 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0 1 2 4 8 16 32 2023-11-08T05:22:15.278 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0 2023-11-08T05:22:15.279 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ remove_image cluster2 mirror image_39 2023-11-08T05:22:15.279 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:22:15.279 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:22:15.279 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_39 2023-11-08T05:22:15.280 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_39 2023-11-08T05:22:15.318 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster=cluster2 rm mirror/image_39 2023-11-08T05:22:15.436 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi087.stderr:2023-11-08T05:22:15.434+0000 7fc44fdff700 -1 rbd::mirror::ImageMap: 0x55eab5bad440 handle_peer_ack_remove: failed to remove global_image_id=a4b03d0b-f737-4c8f-8a6c-6971fcf77e8a 2023-11-08T05:22:16.407 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-11-08T05:22:16.411 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:22:16.411 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-11-08T05:22:16.411 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image=image_40 2023-11-08T05:22:16.412 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ remove_snapshot cluster2 mirror image_40 snap 2023-11-08T05:22:16.412 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:22:16.412 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:22:16.412 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_40 2023-11-08T05:22:16.412 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap=snap 2023-11-08T05:22:16.413 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 snap rm mirror/image_40@snap 2023-11-08T05:22:17.273 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Removing snap: 100% complete...done. 2023-11-08T05:22:17.283 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ remove_image_retry cluster2 mirror image_40 2023-11-08T05:22:17.284 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:22:17.284 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:22:17.284 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_40 2023-11-08T05:22:17.284 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0 1 2 4 8 16 32 2023-11-08T05:22:17.284 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0 2023-11-08T05:22:17.285 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ remove_image cluster2 mirror image_40 2023-11-08T05:22:17.285 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:22:17.286 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:22:17.286 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_40 2023-11-08T05:22:17.286 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_40 2023-11-08T05:22:17.333 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster=cluster2 rm mirror/image_40 2023-11-08T05:22:17.439 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi087.stderr:2023-11-08T05:22:17.437+0000 7fc44fdff700 -1 rbd::mirror::ImageMap: 0x55eab5bad440 handle_peer_ack_remove: failed to remove global_image_id=17ed47a8-b4d3-4db4-8592-15e686f59163 2023-11-08T05:22:18.396 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-11-08T05:22:18.399 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:22:18.399 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-11-08T05:22:18.399 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image=image_41 2023-11-08T05:22:18.400 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ remove_snapshot cluster2 mirror image_41 snap 2023-11-08T05:22:18.400 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:22:18.400 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:22:18.400 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_41 2023-11-08T05:22:18.400 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap=snap 2023-11-08T05:22:18.401 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 snap rm mirror/image_41@snap 2023-11-08T05:22:19.292 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Removing snap: 100% complete...done. 2023-11-08T05:22:19.303 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ remove_image_retry cluster2 mirror image_41 2023-11-08T05:22:19.303 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:22:19.303 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:22:19.304 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_41 2023-11-08T05:22:19.304 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0 1 2 4 8 16 32 2023-11-08T05:22:19.304 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0 2023-11-08T05:22:19.305 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ remove_image cluster2 mirror image_41 2023-11-08T05:22:19.305 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:22:19.305 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:22:19.305 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_41 2023-11-08T05:22:19.306 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_41 2023-11-08T05:22:19.356 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster=cluster2 rm mirror/image_41 2023-11-08T05:22:19.462 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi087.stderr:2023-11-08T05:22:19.460+0000 7f4435f01700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x560b01a7e820 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-11-08T05:22:19.462 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi087.stderr:2023-11-08T05:22:19.460+0000 7f443c70e700 -1 rbd::mirror::ImageReplayer: 0x560b015ce280 [3/de215ca4-d87f-4583-9648-dab2d031119f] operator(): start failed: (85) Interrupted system call should be restarted 2023-11-08T05:22:20.437 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-11-08T05:22:20.441 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:22:20.441 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-11-08T05:22:20.441 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image=image_42 2023-11-08T05:22:20.441 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ remove_snapshot cluster2 mirror image_42 snap 2023-11-08T05:22:20.441 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:22:20.442 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:22:20.442 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_42 2023-11-08T05:22:20.442 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap=snap 2023-11-08T05:22:20.442 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 snap rm mirror/image_42@snap 2023-11-08T05:22:21.294 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Removing snap: 100% complete...done. 2023-11-08T05:22:21.306 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ remove_image_retry cluster2 mirror image_42 2023-11-08T05:22:21.307 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:22:21.307 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:22:21.307 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_42 2023-11-08T05:22:21.307 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0 1 2 4 8 16 32 2023-11-08T05:22:21.307 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0 2023-11-08T05:22:21.308 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ remove_image cluster2 mirror image_42 2023-11-08T05:22:21.308 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:22:21.308 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:22:21.309 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_42 2023-11-08T05:22:21.309 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_42 2023-11-08T05:22:21.347 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster=cluster2 rm mirror/image_42 2023-11-08T05:22:21.460 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi087.stderr:2023-11-08T05:22:21.459+0000 7f45a41e9700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55b0ce6cd860 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-11-08T05:22:21.461 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi087.stderr:2023-11-08T05:22:21.459+0000 7f45ab1f7700 -1 rbd::mirror::ImageReplayer: 0x55b0ce53a500 [3/2409fb55-b412-43a5-9d65-43c4ee962379] operator(): start failed: (85) Interrupted system call should be restarted 2023-11-08T05:22:22.432 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-11-08T05:22:22.435 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:22:22.435 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-11-08T05:22:22.436 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image=image_43 2023-11-08T05:22:22.436 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ remove_snapshot cluster2 mirror image_43 snap 2023-11-08T05:22:22.436 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:22:22.436 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:22:22.437 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_43 2023-11-08T05:22:22.437 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap=snap 2023-11-08T05:22:22.437 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 snap rm mirror/image_43@snap 2023-11-08T05:22:23.253 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Removing snap: 100% complete...done. 2023-11-08T05:22:23.265 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ remove_image_retry cluster2 mirror image_43 2023-11-08T05:22:23.265 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:22:23.265 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:22:23.266 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_43 2023-11-08T05:22:23.266 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0 1 2 4 8 16 32 2023-11-08T05:22:23.266 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0 2023-11-08T05:22:23.266 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ remove_image cluster2 mirror image_43 2023-11-08T05:22:23.267 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:22:23.267 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:22:23.267 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_43 2023-11-08T05:22:23.267 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_43 2023-11-08T05:22:23.310 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster=cluster2 rm mirror/image_43 2023-11-08T05:22:23.419 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi087.stderr:2023-11-08T05:22:23.418+0000 7f45a49ea700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55b0ce6cdba0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-11-08T05:22:23.420 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi087.stderr:2023-11-08T05:22:23.418+0000 7f45ab1f7700 -1 rbd::mirror::ImageReplayer: 0x55b0ce53a280 [3/0cc6b9f9-7eeb-414c-bd5f-685fa6d23f33] operator(): start failed: (85) Interrupted system call should be restarted 2023-11-08T05:22:24.430 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-11-08T05:22:24.433 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:22:24.433 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-11-08T05:22:24.433 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image=image_44 2023-11-08T05:22:24.433 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ remove_snapshot cluster2 mirror image_44 snap 2023-11-08T05:22:24.434 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:22:24.434 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:22:24.434 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_44 2023-11-08T05:22:24.434 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap=snap 2023-11-08T05:22:24.434 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 snap rm mirror/image_44@snap 2023-11-08T05:22:25.310 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Removing snap: 100% complete...done. 2023-11-08T05:22:25.319 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ remove_image_retry cluster2 mirror image_44 2023-11-08T05:22:25.320 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:22:25.320 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:22:25.320 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_44 2023-11-08T05:22:25.320 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0 1 2 4 8 16 32 2023-11-08T05:22:25.320 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0 2023-11-08T05:22:25.321 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ remove_image cluster2 mirror image_44 2023-11-08T05:22:25.321 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:22:25.321 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:22:25.322 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_44 2023-11-08T05:22:25.322 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_44 2023-11-08T05:22:25.562 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster=cluster2 rm mirror/image_44 2023-11-08T05:22:25.670 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi087.stderr:2023-11-08T05:22:25.668+0000 7f4436702700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x560b01a7eb60 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-11-08T05:22:25.670 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi087.stderr:2023-11-08T05:22:25.668+0000 7f443c70e700 -1 rbd::mirror::ImageReplayer: 0x560afd790c80 [3/211c5392-1879-4787-844d-06dee2447ed6] operator(): start failed: (85) Interrupted system call should be restarted 2023-11-08T05:22:26.454 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-11-08T05:22:26.458 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:22:26.458 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-11-08T05:22:26.459 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image=image_45 2023-11-08T05:22:26.459 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ remove_snapshot cluster2 mirror image_45 snap 2023-11-08T05:22:26.459 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:22:26.459 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:22:26.459 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_45 2023-11-08T05:22:26.460 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap=snap 2023-11-08T05:22:26.460 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 snap rm mirror/image_45@snap 2023-11-08T05:22:27.320 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Removing snap: 100% complete...done. 2023-11-08T05:22:27.331 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ remove_image_retry cluster2 mirror image_45 2023-11-08T05:22:27.331 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:22:27.332 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:22:27.332 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_45 2023-11-08T05:22:27.332 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0 1 2 4 8 16 32 2023-11-08T05:22:27.332 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0 2023-11-08T05:22:27.333 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ remove_image cluster2 mirror image_45 2023-11-08T05:22:27.333 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:22:27.333 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:22:27.334 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_45 2023-11-08T05:22:27.334 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_45 2023-11-08T05:22:27.374 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster=cluster2 rm mirror/image_45 2023-11-08T05:22:27.476 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi087.stderr:2023-11-08T05:22:27.474+0000 7f45a41e9700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55b0ce70b1e0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-11-08T05:22:27.476 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi087.stderr:2023-11-08T05:22:27.474+0000 7f45ab1f7700 -1 rbd::mirror::ImageReplayer: 0x55b0ca754a00 [3/20b7d4e5-6730-4919-b471-a15668fab78b] operator(): start failed: (85) Interrupted system call should be restarted 2023-11-08T05:22:28.390 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-11-08T05:22:28.393 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:22:28.394 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-11-08T05:22:28.394 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image=image_46 2023-11-08T05:22:28.394 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ remove_snapshot cluster2 mirror image_46 snap 2023-11-08T05:22:28.395 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:22:28.395 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:22:28.395 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_46 2023-11-08T05:22:28.395 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap=snap 2023-11-08T05:22:28.395 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 snap rm mirror/image_46@snap 2023-11-08T05:22:29.323 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Removing snap: 100% complete...done. 2023-11-08T05:22:29.335 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ remove_image_retry cluster2 mirror image_46 2023-11-08T05:22:29.336 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:22:29.336 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:22:29.336 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_46 2023-11-08T05:22:29.336 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0 1 2 4 8 16 32 2023-11-08T05:22:29.337 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0 2023-11-08T05:22:29.337 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ remove_image cluster2 mirror image_46 2023-11-08T05:22:29.338 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:22:29.338 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:22:29.338 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_46 2023-11-08T05:22:29.338 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_46 2023-11-08T05:22:29.381 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster=cluster2 rm mirror/image_46 2023-11-08T05:22:29.487 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi087.stderr:2023-11-08T05:22:29.485+0000 7fc4495f2700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55eab38284e0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-11-08T05:22:29.487 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi087.stderr:2023-11-08T05:22:29.485+0000 7fc44fdff700 -1 rbd::mirror::ImageReplayer: 0x55eab61bcc80 [3/ab12d762-6014-4867-a6ea-6c7d605eb202] operator(): start failed: (85) Interrupted system call should be restarted 2023-11-08T05:22:30.467 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-11-08T05:22:30.470 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:22:30.471 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-11-08T05:22:30.471 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image=image_47 2023-11-08T05:22:30.471 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ remove_snapshot cluster2 mirror image_47 snap 2023-11-08T05:22:30.471 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:22:30.471 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:22:30.472 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_47 2023-11-08T05:22:30.472 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap=snap 2023-11-08T05:22:30.472 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 snap rm mirror/image_47@snap 2023-11-08T05:22:31.343 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Removing snap: 100% complete...done. 2023-11-08T05:22:31.352 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ remove_image_retry cluster2 mirror image_47 2023-11-08T05:22:31.353 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:22:31.353 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:22:31.353 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_47 2023-11-08T05:22:31.353 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0 1 2 4 8 16 32 2023-11-08T05:22:31.354 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0 2023-11-08T05:22:31.354 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ remove_image cluster2 mirror image_47 2023-11-08T05:22:31.355 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:22:31.355 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:22:31.355 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_47 2023-11-08T05:22:31.355 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_47 2023-11-08T05:22:31.395 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster=cluster2 rm mirror/image_47 2023-11-08T05:22:31.496 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi087.stderr:2023-11-08T05:22:31.495+0000 7f5483b76700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55813d47c000 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-11-08T05:22:31.497 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi087.stderr:2023-11-08T05:22:31.495+0000 7f5489b82700 -1 rbd::mirror::ImageReplayer: 0x558139612c80 [3/8f6b2565-b6aa-4361-bc36-f1a120eb271e] operator(): start failed: (85) Interrupted system call should be restarted 2023-11-08T05:22:32.479 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-11-08T05:22:32.483 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:22:32.483 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-11-08T05:22:32.483 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image=image_48 2023-11-08T05:22:32.483 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ remove_snapshot cluster2 mirror image_48 snap 2023-11-08T05:22:32.484 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:22:32.484 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:22:32.484 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_48 2023-11-08T05:22:32.484 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap=snap 2023-11-08T05:22:32.485 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 snap rm mirror/image_48@snap 2023-11-08T05:22:33.257 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Removing snap: 100% complete...done. 2023-11-08T05:22:33.271 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ remove_image_retry cluster2 mirror image_48 2023-11-08T05:22:33.271 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:22:33.271 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:22:33.271 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_48 2023-11-08T05:22:33.271 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0 1 2 4 8 16 32 2023-11-08T05:22:33.272 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0 2023-11-08T05:22:33.272 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ remove_image cluster2 mirror image_48 2023-11-08T05:22:33.272 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:22:33.273 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:22:33.273 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_48 2023-11-08T05:22:33.273 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_48 2023-11-08T05:22:33.316 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster=cluster2 rm mirror/image_48 2023-11-08T05:22:33.441 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi087.stderr:2023-11-08T05:22:33.439+0000 7fc449df3700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55eab3827d40 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-11-08T05:22:33.441 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi087.stderr:2023-11-08T05:22:33.439+0000 7fc450600700 -1 rbd::mirror::ImageReplayer: 0x55eab1f48a00 [3/c3d37bcb-5eef-4c9f-a31c-472ae9f8d4c1] operator(): start failed: (85) Interrupted system call should be restarted 2023-11-08T05:22:34.483 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-11-08T05:22:34.486 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:22:34.486 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-11-08T05:22:34.487 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image=image_49 2023-11-08T05:22:34.487 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ remove_snapshot cluster2 mirror image_49 snap 2023-11-08T05:22:34.487 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:22:34.487 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:22:34.488 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_49 2023-11-08T05:22:34.488 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap=snap 2023-11-08T05:22:34.488 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 snap rm mirror/image_49@snap 2023-11-08T05:22:35.333 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.0 2023-11-08T05:22:35.334 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Sent signal 15 2023-11-08T05:22:35.334 INFO:tasks.rbd_mirror_thrash:waiting for 7 secs before reviving daemons 2023-11-08T05:22:35.335 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi087.stderr:2023-11-08T05:22:35.333+0000 7fe5db122700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 140831) UID: 1000 2023-11-08T05:22:35.350 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Removing snap: 100% complete...done. 2023-11-08T05:22:35.368 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ remove_image_retry cluster2 mirror image_49 2023-11-08T05:22:35.368 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:22:35.368 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:22:35.369 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_49 2023-11-08T05:22:35.369 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0 1 2 4 8 16 32 2023-11-08T05:22:35.369 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0 2023-11-08T05:22:35.370 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ remove_image cluster2 mirror image_49 2023-11-08T05:22:35.370 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:22:35.370 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:22:35.370 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_49 2023-11-08T05:22:35.371 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_49 2023-11-08T05:22:35.411 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster=cluster2 rm mirror/image_49 2023-11-08T05:22:35.516 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi087.stderr:2023-11-08T05:22:35.514+0000 7fc449df3700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55eab3828680 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-11-08T05:22:35.517 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi087.stderr:2023-11-08T05:22:35.514+0000 7fc44fdff700 -1 rbd::mirror::ImageReplayer: 0x55eab61bd400 [3/e7efac79-0655-48fa-85f6-d32c1c0bf780] operator(): start failed: (85) Interrupted system call should be restarted 2023-11-08T05:22:35.535 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi087.stderr:daemon-helper: command failed with exit status 1 2023-11-08T05:22:36.503 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-11-08T05:22:36.506 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:22:36.506 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-11-08T05:22:36.506 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image=image_50 2023-11-08T05:22:36.507 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ remove_snapshot cluster2 mirror image_50 snap 2023-11-08T05:22:36.507 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:22:36.507 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:22:36.507 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_50 2023-11-08T05:22:36.507 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local snap=snap 2023-11-08T05:22:36.508 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 snap rm mirror/image_50@snap 2023-11-08T05:22:37.359 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Removing snap: 100% complete...done. 2023-11-08T05:22:37.373 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ remove_image_retry cluster2 mirror image_50 2023-11-08T05:22:37.373 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:22:37.373 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:22:37.373 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_50 2023-11-08T05:22:37.374 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0 1 2 4 8 16 32 2023-11-08T05:22:37.374 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0 2023-11-08T05:22:37.375 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ remove_image cluster2 mirror image_50 2023-11-08T05:22:37.375 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:22:37.375 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:22:37.375 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_50 2023-11-08T05:22:37.376 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_50 2023-11-08T05:22:37.415 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster=cluster2 rm mirror/image_50 2023-11-08T05:22:37.552 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi087.stderr:2023-11-08T05:22:37.550+0000 7f45a41e9700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55b0cd88fd40 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-11-08T05:22:37.552 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi087.stderr:2023-11-08T05:22:37.550+0000 7f45aa9f6700 -1 rbd::mirror::ImageReplayer: 0x55b0ca754f00 [3/39de3a57-ef7b-4ccc-a818-26ba71efeed5] operator(): start failed: (85) Interrupted system call should be restarted 2023-11-08T05:22:38.385 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-11-08T05:22:38.388 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:22:38.389 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ testlog 'TEST: image deletions should propagate' 2023-11-08T05:22:38.389 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ tee -a /tmp/tmp.Ludk8ikCAB/rbd-mirror.test.log 2023-11-08T05:22:38.389 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ date '+%F %T' 2023-11-08T05:22:38.391 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ echo 2023-11-08 05:22:38 TEST: image deletions should propagate 2023-11-08T05:22:38.391 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:2023-11-08 05:22:38 TEST: image deletions should propagate 2023-11-08T05:22:38.391 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_pool_images cluster1 mirror 0 2023-11-08T05:22:38.392 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:22:38.392 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:22:38.392 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image_count=0 2023-11-08T05:22:38.392 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local s 2023-11-08T05:22:38.393 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local count 2023-11-08T05:22:38.393 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local last_count=0 2023-11-08T05:22:38.393 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ true 2023-11-08T05:22:38.393 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ seq 1 40 2023-11-08T05:22:38.393 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in `seq 1 40` 2023-11-08T05:22:38.394 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test 1 -ne 1 2023-11-08T05:22:38.394 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ rbd --cluster cluster1 -p mirror mirror pool status 2023-11-08T05:22:38.394 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ grep 'images: ' 2023-11-08T05:22:38.394 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cut '-d ' -f 2 2023-11-08T05:22:38.419 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd: failed to query services: (13) Permission denied 2023-11-08T05:22:38.422 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ count=0 2023-11-08T05:22:38.422 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test 0 = 0 2023-11-08T05:22:38.422 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:22:38.422 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_pool_healthy cluster1 mirror 0 2023-11-08T05:22:38.423 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:22:38.423 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:22:38.423 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local s 2023-11-08T05:22:38.423 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local state 2023-11-08T05:22:38.423 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ seq 1 40 2023-11-08T05:22:38.424 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in `seq 1 40` 2023-11-08T05:22:38.425 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test 1 -ne 1 2023-11-08T05:22:38.425 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ rbd --cluster cluster1 -p mirror mirror pool status 2023-11-08T05:22:38.425 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ grep 'image health:' 2023-11-08T05:22:38.425 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cut '-d ' -f 3 2023-11-08T05:22:38.450 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd: failed to query services: (13) Permission denied 2023-11-08T05:22:38.453 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ state=OK 2023-11-08T05:22:38.453 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test OK = ERROR 2023-11-08T05:22:38.453 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test OK = OK 2023-11-08T05:22:38.453 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:22:38.454 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ seq 1 50 2023-11-08T05:22:38.455 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-11-08T05:22:38.456 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image=image_1 2023-11-08T05:22:38.456 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_image_present cluster1 mirror image_1 deleted 2023-11-08T05:22:38.456 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:22:38.456 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:22:38.457 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_1 2023-11-08T05:22:38.457 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local state=deleted 2023-11-08T05:22:38.457 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image_id= 2023-11-08T05:22:38.457 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local s 2023-11-08T05:22:38.458 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test -n '' 2023-11-08T05:22:38.458 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ get_image_id cluster1 mirror image_1 2023-11-08T05:22:38.458 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1 2023-11-08T05:22:38.458 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local pool=mirror 2023-11-08T05:22:38.459 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local image=image_1 2023-11-08T05:22:38.459 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ rbd --cluster=cluster1 info mirror/image_1 2023-11-08T05:22:38.459 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-11-08T05:22:38.480 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd: error opening image image_1: (2) No such file or directory 2023-11-08T05:22:38.483 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image_id= 2023-11-08T05:22:38.483 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-11-08T05:22:38.483 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0.1 2023-11-08T05:22:38.585 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test_image_present cluster1 mirror image_1 deleted '' 2023-11-08T05:22:38.585 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:22:38.585 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:22:38.585 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_1 2023-11-08T05:22:38.586 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local test_state=deleted 2023-11-08T05:22:38.586 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image_id= 2023-11-08T05:22:38.586 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local current_state=deleted 2023-11-08T05:22:38.586 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local current_image_id 2023-11-08T05:22:38.587 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ get_image_id cluster1 mirror image_1 2023-11-08T05:22:38.587 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1 2023-11-08T05:22:38.589 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local pool=mirror 2023-11-08T05:22:38.589 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local image=image_1 2023-11-08T05:22:38.590 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ rbd --cluster=cluster1 info mirror/image_1 2023-11-08T05:22:38.590 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-11-08T05:22:38.613 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd: error opening image image_1: (2) No such file or directory 2023-11-08T05:22:38.617 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ current_image_id= 2023-11-08T05:22:38.618 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test -n '' 2023-11-08T05:22:38.618 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test deleted = deleted 2023-11-08T05:22:38.618 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:22:38.618 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-11-08T05:22:38.619 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image=image_2 2023-11-08T05:22:38.619 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_image_present cluster1 mirror image_2 deleted 2023-11-08T05:22:38.619 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:22:38.619 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:22:38.620 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_2 2023-11-08T05:22:38.620 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local state=deleted 2023-11-08T05:22:38.620 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image_id= 2023-11-08T05:22:38.620 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local s 2023-11-08T05:22:38.621 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test -n '' 2023-11-08T05:22:38.621 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ get_image_id cluster1 mirror image_2 2023-11-08T05:22:38.621 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1 2023-11-08T05:22:38.621 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local pool=mirror 2023-11-08T05:22:38.621 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local image=image_2 2023-11-08T05:22:38.622 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ rbd --cluster=cluster1 info mirror/image_2 2023-11-08T05:22:38.622 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-11-08T05:22:38.643 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd: error opening image image_2: (2) No such file or directory 2023-11-08T05:22:38.646 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image_id= 2023-11-08T05:22:38.646 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-11-08T05:22:38.646 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0.1 2023-11-08T05:22:38.748 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test_image_present cluster1 mirror image_2 deleted '' 2023-11-08T05:22:38.748 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:22:38.748 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:22:38.748 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_2 2023-11-08T05:22:38.749 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local test_state=deleted 2023-11-08T05:22:38.749 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image_id= 2023-11-08T05:22:38.749 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local current_state=deleted 2023-11-08T05:22:38.749 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local current_image_id 2023-11-08T05:22:38.749 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ get_image_id cluster1 mirror image_2 2023-11-08T05:22:38.750 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1 2023-11-08T05:22:38.750 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local pool=mirror 2023-11-08T05:22:38.750 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local image=image_2 2023-11-08T05:22:38.750 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ rbd --cluster=cluster1 info mirror/image_2 2023-11-08T05:22:38.751 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-11-08T05:22:38.774 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd: error opening image image_2: (2) No such file or directory 2023-11-08T05:22:38.778 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ current_image_id= 2023-11-08T05:22:38.778 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test -n '' 2023-11-08T05:22:38.778 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test deleted = deleted 2023-11-08T05:22:38.778 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:22:38.779 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-11-08T05:22:38.779 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image=image_3 2023-11-08T05:22:38.779 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_image_present cluster1 mirror image_3 deleted 2023-11-08T05:22:38.779 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:22:38.780 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:22:38.780 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_3 2023-11-08T05:22:38.780 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local state=deleted 2023-11-08T05:22:38.780 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image_id= 2023-11-08T05:22:38.780 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local s 2023-11-08T05:22:38.781 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test -n '' 2023-11-08T05:22:38.781 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ get_image_id cluster1 mirror image_3 2023-11-08T05:22:38.781 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1 2023-11-08T05:22:38.781 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local pool=mirror 2023-11-08T05:22:38.781 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local image=image_3 2023-11-08T05:22:38.782 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ rbd --cluster=cluster1 info mirror/image_3 2023-11-08T05:22:38.782 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-11-08T05:22:38.804 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd: error opening image image_3: (2) No such file or directory 2023-11-08T05:22:38.806 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image_id= 2023-11-08T05:22:38.807 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-11-08T05:22:38.807 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0.1 2023-11-08T05:22:38.908 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test_image_present cluster1 mirror image_3 deleted '' 2023-11-08T05:22:38.908 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:22:38.909 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:22:38.909 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_3 2023-11-08T05:22:38.909 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local test_state=deleted 2023-11-08T05:22:38.909 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image_id= 2023-11-08T05:22:38.909 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local current_state=deleted 2023-11-08T05:22:38.910 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local current_image_id 2023-11-08T05:22:38.910 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ get_image_id cluster1 mirror image_3 2023-11-08T05:22:38.910 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1 2023-11-08T05:22:38.910 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local pool=mirror 2023-11-08T05:22:38.910 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local image=image_3 2023-11-08T05:22:38.911 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ rbd --cluster=cluster1 info mirror/image_3 2023-11-08T05:22:38.911 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-11-08T05:22:38.934 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd: error opening image image_3: (2) No such file or directory 2023-11-08T05:22:38.938 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ current_image_id= 2023-11-08T05:22:38.938 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test -n '' 2023-11-08T05:22:38.939 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test deleted = deleted 2023-11-08T05:22:38.939 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:22:38.939 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-11-08T05:22:38.939 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image=image_4 2023-11-08T05:22:38.939 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_image_present cluster1 mirror image_4 deleted 2023-11-08T05:22:38.940 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:22:38.940 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:22:38.940 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_4 2023-11-08T05:22:38.940 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local state=deleted 2023-11-08T05:22:38.940 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image_id= 2023-11-08T05:22:38.941 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local s 2023-11-08T05:22:38.941 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test -n '' 2023-11-08T05:22:38.941 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ get_image_id cluster1 mirror image_4 2023-11-08T05:22:38.941 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1 2023-11-08T05:22:38.941 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local pool=mirror 2023-11-08T05:22:38.941 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local image=image_4 2023-11-08T05:22:38.942 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ rbd --cluster=cluster1 info mirror/image_4 2023-11-08T05:22:38.942 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-11-08T05:22:38.964 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd: error opening image image_4: (2) No such file or directory 2023-11-08T05:22:38.966 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image_id= 2023-11-08T05:22:38.967 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-11-08T05:22:38.967 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0.1 2023-11-08T05:22:39.068 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test_image_present cluster1 mirror image_4 deleted '' 2023-11-08T05:22:39.069 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:22:39.069 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:22:39.069 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_4 2023-11-08T05:22:39.069 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local test_state=deleted 2023-11-08T05:22:39.070 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image_id= 2023-11-08T05:22:39.070 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local current_state=deleted 2023-11-08T05:22:39.070 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local current_image_id 2023-11-08T05:22:39.070 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ get_image_id cluster1 mirror image_4 2023-11-08T05:22:39.070 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1 2023-11-08T05:22:39.071 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local pool=mirror 2023-11-08T05:22:39.071 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local image=image_4 2023-11-08T05:22:39.071 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ rbd --cluster=cluster1 info mirror/image_4 2023-11-08T05:22:39.071 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-11-08T05:22:39.095 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd: error opening image image_4: (2) No such file or directory 2023-11-08T05:22:39.099 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ current_image_id= 2023-11-08T05:22:39.099 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test -n '' 2023-11-08T05:22:39.099 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test deleted = deleted 2023-11-08T05:22:39.099 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:22:39.100 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-11-08T05:22:39.100 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image=image_5 2023-11-08T05:22:39.100 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_image_present cluster1 mirror image_5 deleted 2023-11-08T05:22:39.100 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:22:39.101 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:22:39.101 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_5 2023-11-08T05:22:39.101 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local state=deleted 2023-11-08T05:22:39.101 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image_id= 2023-11-08T05:22:39.101 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local s 2023-11-08T05:22:39.102 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test -n '' 2023-11-08T05:22:39.102 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ get_image_id cluster1 mirror image_5 2023-11-08T05:22:39.102 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1 2023-11-08T05:22:39.102 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local pool=mirror 2023-11-08T05:22:39.103 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local image=image_5 2023-11-08T05:22:39.103 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ rbd --cluster=cluster1 info mirror/image_5 2023-11-08T05:22:39.103 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-11-08T05:22:39.124 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd: error opening image image_5: (2) No such file or directory 2023-11-08T05:22:39.127 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image_id= 2023-11-08T05:22:39.127 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-11-08T05:22:39.128 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0.1 2023-11-08T05:22:39.229 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test_image_present cluster1 mirror image_5 deleted '' 2023-11-08T05:22:39.229 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:22:39.230 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:22:39.230 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_5 2023-11-08T05:22:39.230 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local test_state=deleted 2023-11-08T05:22:39.230 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image_id= 2023-11-08T05:22:39.230 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local current_state=deleted 2023-11-08T05:22:39.231 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local current_image_id 2023-11-08T05:22:39.231 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ get_image_id cluster1 mirror image_5 2023-11-08T05:22:39.231 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1 2023-11-08T05:22:39.231 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local pool=mirror 2023-11-08T05:22:39.231 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local image=image_5 2023-11-08T05:22:39.232 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ rbd --cluster=cluster1 info mirror/image_5 2023-11-08T05:22:39.232 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-11-08T05:22:39.256 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd: error opening image image_5: (2) No such file or directory 2023-11-08T05:22:39.260 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ current_image_id= 2023-11-08T05:22:39.260 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test -n '' 2023-11-08T05:22:39.260 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test deleted = deleted 2023-11-08T05:22:39.260 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:22:39.261 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-11-08T05:22:39.261 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image=image_6 2023-11-08T05:22:39.261 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_image_present cluster1 mirror image_6 deleted 2023-11-08T05:22:39.261 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:22:39.261 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:22:39.262 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_6 2023-11-08T05:22:39.262 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local state=deleted 2023-11-08T05:22:39.262 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image_id= 2023-11-08T05:22:39.262 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local s 2023-11-08T05:22:39.262 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test -n '' 2023-11-08T05:22:39.262 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ get_image_id cluster1 mirror image_6 2023-11-08T05:22:39.263 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1 2023-11-08T05:22:39.263 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local pool=mirror 2023-11-08T05:22:39.263 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local image=image_6 2023-11-08T05:22:39.263 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ rbd --cluster=cluster1 info mirror/image_6 2023-11-08T05:22:39.263 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-11-08T05:22:39.285 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd: error opening image image_6: (2) No such file or directory 2023-11-08T05:22:39.288 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image_id= 2023-11-08T05:22:39.288 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-11-08T05:22:39.289 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0.1 2023-11-08T05:22:39.390 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test_image_present cluster1 mirror image_6 deleted '' 2023-11-08T05:22:39.390 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:22:39.391 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:22:39.391 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_6 2023-11-08T05:22:39.391 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local test_state=deleted 2023-11-08T05:22:39.391 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image_id= 2023-11-08T05:22:39.392 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local current_state=deleted 2023-11-08T05:22:39.392 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local current_image_id 2023-11-08T05:22:39.392 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ get_image_id cluster1 mirror image_6 2023-11-08T05:22:39.392 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1 2023-11-08T05:22:39.392 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local pool=mirror 2023-11-08T05:22:39.392 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local image=image_6 2023-11-08T05:22:39.393 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ rbd --cluster=cluster1 info mirror/image_6 2023-11-08T05:22:39.393 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-11-08T05:22:39.418 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd: error opening image image_6: (2) No such file or directory 2023-11-08T05:22:39.420 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ current_image_id= 2023-11-08T05:22:39.420 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test -n '' 2023-11-08T05:22:39.420 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test deleted = deleted 2023-11-08T05:22:39.421 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:22:39.421 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-11-08T05:22:39.421 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image=image_7 2023-11-08T05:22:39.421 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_image_present cluster1 mirror image_7 deleted 2023-11-08T05:22:39.422 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:22:39.422 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:22:39.422 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_7 2023-11-08T05:22:39.422 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local state=deleted 2023-11-08T05:22:39.422 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image_id= 2023-11-08T05:22:39.423 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local s 2023-11-08T05:22:39.423 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test -n '' 2023-11-08T05:22:39.423 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ get_image_id cluster1 mirror image_7 2023-11-08T05:22:39.423 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1 2023-11-08T05:22:39.423 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local pool=mirror 2023-11-08T05:22:39.424 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local image=image_7 2023-11-08T05:22:39.424 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ rbd --cluster=cluster1 info mirror/image_7 2023-11-08T05:22:39.424 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-11-08T05:22:39.445 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd: error opening image image_7: (2) No such file or directory 2023-11-08T05:22:39.448 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image_id= 2023-11-08T05:22:39.448 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-11-08T05:22:39.448 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0.1 2023-11-08T05:22:39.550 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test_image_present cluster1 mirror image_7 deleted '' 2023-11-08T05:22:39.550 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:22:39.551 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:22:39.551 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_7 2023-11-08T05:22:39.551 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local test_state=deleted 2023-11-08T05:22:39.551 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image_id= 2023-11-08T05:22:39.552 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local current_state=deleted 2023-11-08T05:22:39.552 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local current_image_id 2023-11-08T05:22:39.552 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ get_image_id cluster1 mirror image_7 2023-11-08T05:22:39.552 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1 2023-11-08T05:22:39.552 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local pool=mirror 2023-11-08T05:22:39.553 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local image=image_7 2023-11-08T05:22:39.553 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ rbd --cluster=cluster1 info mirror/image_7 2023-11-08T05:22:39.553 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-11-08T05:22:39.575 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd: error opening image image_7: (2) No such file or directory 2023-11-08T05:22:39.580 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ current_image_id= 2023-11-08T05:22:39.580 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test -n '' 2023-11-08T05:22:39.580 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test deleted = deleted 2023-11-08T05:22:39.581 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:22:39.581 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-11-08T05:22:39.581 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image=image_8 2023-11-08T05:22:39.581 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_image_present cluster1 mirror image_8 deleted 2023-11-08T05:22:39.581 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:22:39.582 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:22:39.582 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_8 2023-11-08T05:22:39.582 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local state=deleted 2023-11-08T05:22:39.582 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image_id= 2023-11-08T05:22:39.582 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local s 2023-11-08T05:22:39.583 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test -n '' 2023-11-08T05:22:39.583 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ get_image_id cluster1 mirror image_8 2023-11-08T05:22:39.583 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1 2023-11-08T05:22:39.583 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local pool=mirror 2023-11-08T05:22:39.583 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local image=image_8 2023-11-08T05:22:39.584 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ rbd --cluster=cluster1 info mirror/image_8 2023-11-08T05:22:39.584 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-11-08T05:22:39.625 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd: error opening image image_8: (2) No such file or directory 2023-11-08T05:22:39.627 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image_id= 2023-11-08T05:22:39.627 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-11-08T05:22:39.628 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0.1 2023-11-08T05:22:39.730 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test_image_present cluster1 mirror image_8 deleted '' 2023-11-08T05:22:39.731 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:22:39.731 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:22:39.732 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_8 2023-11-08T05:22:39.732 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local test_state=deleted 2023-11-08T05:22:39.732 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image_id= 2023-11-08T05:22:39.732 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local current_state=deleted 2023-11-08T05:22:39.733 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local current_image_id 2023-11-08T05:22:39.733 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ get_image_id cluster1 mirror image_8 2023-11-08T05:22:39.733 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1 2023-11-08T05:22:39.733 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local pool=mirror 2023-11-08T05:22:39.733 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local image=image_8 2023-11-08T05:22:39.734 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ rbd --cluster=cluster1 info mirror/image_8 2023-11-08T05:22:39.734 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-11-08T05:22:39.755 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd: error opening image image_8: (2) No such file or directory 2023-11-08T05:22:39.759 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ current_image_id= 2023-11-08T05:22:39.759 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test -n '' 2023-11-08T05:22:39.760 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test deleted = deleted 2023-11-08T05:22:39.760 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:22:39.760 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-11-08T05:22:39.760 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image=image_9 2023-11-08T05:22:39.761 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_image_present cluster1 mirror image_9 deleted 2023-11-08T05:22:39.761 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:22:39.761 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:22:39.761 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_9 2023-11-08T05:22:39.761 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local state=deleted 2023-11-08T05:22:39.762 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image_id= 2023-11-08T05:22:39.762 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local s 2023-11-08T05:22:39.762 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test -n '' 2023-11-08T05:22:39.762 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ get_image_id cluster1 mirror image_9 2023-11-08T05:22:39.763 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1 2023-11-08T05:22:39.763 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local pool=mirror 2023-11-08T05:22:39.763 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local image=image_9 2023-11-08T05:22:39.763 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ rbd --cluster=cluster1 info mirror/image_9 2023-11-08T05:22:39.763 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-11-08T05:22:39.785 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd: error opening image image_9: (2) No such file or directory 2023-11-08T05:22:39.787 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image_id= 2023-11-08T05:22:39.787 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-11-08T05:22:39.788 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0.1 2023-11-08T05:22:39.889 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test_image_present cluster1 mirror image_9 deleted '' 2023-11-08T05:22:39.889 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:22:39.889 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:22:39.890 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_9 2023-11-08T05:22:39.890 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local test_state=deleted 2023-11-08T05:22:39.890 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image_id= 2023-11-08T05:22:39.890 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local current_state=deleted 2023-11-08T05:22:39.890 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local current_image_id 2023-11-08T05:22:39.891 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ get_image_id cluster1 mirror image_9 2023-11-08T05:22:39.891 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1 2023-11-08T05:22:39.891 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local pool=mirror 2023-11-08T05:22:39.891 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local image=image_9 2023-11-08T05:22:39.891 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ rbd --cluster=cluster1 info mirror/image_9 2023-11-08T05:22:39.892 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-11-08T05:22:39.915 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd: error opening image image_9: (2) No such file or directory 2023-11-08T05:22:39.918 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ current_image_id= 2023-11-08T05:22:39.919 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test -n '' 2023-11-08T05:22:39.919 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test deleted = deleted 2023-11-08T05:22:39.919 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:22:39.919 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-11-08T05:22:39.920 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image=image_10 2023-11-08T05:22:39.920 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_image_present cluster1 mirror image_10 deleted 2023-11-08T05:22:39.920 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:22:39.920 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:22:39.920 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_10 2023-11-08T05:22:39.920 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local state=deleted 2023-11-08T05:22:39.921 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image_id= 2023-11-08T05:22:39.921 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local s 2023-11-08T05:22:39.921 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test -n '' 2023-11-08T05:22:39.921 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ get_image_id cluster1 mirror image_10 2023-11-08T05:22:39.921 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1 2023-11-08T05:22:39.922 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local pool=mirror 2023-11-08T05:22:39.922 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local image=image_10 2023-11-08T05:22:39.922 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ rbd --cluster=cluster1 info mirror/image_10 2023-11-08T05:22:39.922 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-11-08T05:22:39.944 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd: error opening image image_10: (2) No such file or directory 2023-11-08T05:22:39.947 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image_id= 2023-11-08T05:22:39.947 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-11-08T05:22:39.947 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0.1 2023-11-08T05:22:40.049 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test_image_present cluster1 mirror image_10 deleted '' 2023-11-08T05:22:40.049 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:22:40.049 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:22:40.049 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_10 2023-11-08T05:22:40.050 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local test_state=deleted 2023-11-08T05:22:40.050 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image_id= 2023-11-08T05:22:40.050 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local current_state=deleted 2023-11-08T05:22:40.050 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local current_image_id 2023-11-08T05:22:40.050 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ get_image_id cluster1 mirror image_10 2023-11-08T05:22:40.051 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1 2023-11-08T05:22:40.051 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local pool=mirror 2023-11-08T05:22:40.051 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local image=image_10 2023-11-08T05:22:40.051 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ rbd --cluster=cluster1 info mirror/image_10 2023-11-08T05:22:40.051 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-11-08T05:22:40.075 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd: error opening image image_10: (2) No such file or directory 2023-11-08T05:22:40.078 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ current_image_id= 2023-11-08T05:22:40.079 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test -n '' 2023-11-08T05:22:40.079 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test deleted = deleted 2023-11-08T05:22:40.079 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:22:40.079 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-11-08T05:22:40.079 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image=image_11 2023-11-08T05:22:40.080 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_image_present cluster1 mirror image_11 deleted 2023-11-08T05:22:40.080 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:22:40.080 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:22:40.081 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_11 2023-11-08T05:22:40.081 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local state=deleted 2023-11-08T05:22:40.081 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image_id= 2023-11-08T05:22:40.081 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local s 2023-11-08T05:22:40.081 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test -n '' 2023-11-08T05:22:40.082 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ get_image_id cluster1 mirror image_11 2023-11-08T05:22:40.082 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1 2023-11-08T05:22:40.082 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local pool=mirror 2023-11-08T05:22:40.082 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local image=image_11 2023-11-08T05:22:40.082 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ rbd --cluster=cluster1 info mirror/image_11 2023-11-08T05:22:40.083 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-11-08T05:22:40.104 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd: error opening image image_11: (2) No such file or directory 2023-11-08T05:22:40.107 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image_id= 2023-11-08T05:22:40.107 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-11-08T05:22:40.107 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0.1 2023-11-08T05:22:40.209 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test_image_present cluster1 mirror image_11 deleted '' 2023-11-08T05:22:40.209 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:22:40.209 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:22:40.209 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_11 2023-11-08T05:22:40.210 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local test_state=deleted 2023-11-08T05:22:40.210 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image_id= 2023-11-08T05:22:40.210 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local current_state=deleted 2023-11-08T05:22:40.211 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local current_image_id 2023-11-08T05:22:40.211 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ get_image_id cluster1 mirror image_11 2023-11-08T05:22:40.211 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1 2023-11-08T05:22:40.211 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local pool=mirror 2023-11-08T05:22:40.211 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local image=image_11 2023-11-08T05:22:40.211 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ rbd --cluster=cluster1 info mirror/image_11 2023-11-08T05:22:40.212 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-11-08T05:22:40.235 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd: error opening image image_11: (2) No such file or directory 2023-11-08T05:22:40.238 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ current_image_id= 2023-11-08T05:22:40.239 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test -n '' 2023-11-08T05:22:40.239 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test deleted = deleted 2023-11-08T05:22:40.239 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:22:40.239 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-11-08T05:22:40.239 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image=image_12 2023-11-08T05:22:40.240 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_image_present cluster1 mirror image_12 deleted 2023-11-08T05:22:40.240 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:22:40.240 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:22:40.240 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_12 2023-11-08T05:22:40.240 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local state=deleted 2023-11-08T05:22:40.241 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image_id= 2023-11-08T05:22:40.241 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local s 2023-11-08T05:22:40.241 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test -n '' 2023-11-08T05:22:40.241 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ get_image_id cluster1 mirror image_12 2023-11-08T05:22:40.241 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1 2023-11-08T05:22:40.242 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local pool=mirror 2023-11-08T05:22:40.242 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local image=image_12 2023-11-08T05:22:40.242 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ rbd --cluster=cluster1 info mirror/image_12 2023-11-08T05:22:40.242 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-11-08T05:22:40.264 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd: error opening image image_12: (2) No such file or directory 2023-11-08T05:22:40.267 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image_id= 2023-11-08T05:22:40.267 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-11-08T05:22:40.267 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0.1 2023-11-08T05:22:40.369 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test_image_present cluster1 mirror image_12 deleted '' 2023-11-08T05:22:40.369 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:22:40.369 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:22:40.370 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_12 2023-11-08T05:22:40.370 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local test_state=deleted 2023-11-08T05:22:40.370 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image_id= 2023-11-08T05:22:40.370 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local current_state=deleted 2023-11-08T05:22:40.371 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local current_image_id 2023-11-08T05:22:40.371 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ get_image_id cluster1 mirror image_12 2023-11-08T05:22:40.371 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1 2023-11-08T05:22:40.371 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local pool=mirror 2023-11-08T05:22:40.371 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local image=image_12 2023-11-08T05:22:40.371 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ rbd --cluster=cluster1 info mirror/image_12 2023-11-08T05:22:40.372 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-11-08T05:22:40.395 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd: error opening image image_12: (2) No such file or directory 2023-11-08T05:22:40.399 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ current_image_id= 2023-11-08T05:22:40.400 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test -n '' 2023-11-08T05:22:40.400 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test deleted = deleted 2023-11-08T05:22:40.400 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:22:40.400 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-11-08T05:22:40.401 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image=image_13 2023-11-08T05:22:40.401 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_image_present cluster1 mirror image_13 deleted 2023-11-08T05:22:40.401 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:22:40.401 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:22:40.402 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_13 2023-11-08T05:22:40.402 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local state=deleted 2023-11-08T05:22:40.402 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image_id= 2023-11-08T05:22:40.402 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local s 2023-11-08T05:22:40.402 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test -n '' 2023-11-08T05:22:40.403 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ get_image_id cluster1 mirror image_13 2023-11-08T05:22:40.403 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1 2023-11-08T05:22:40.403 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local pool=mirror 2023-11-08T05:22:40.403 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local image=image_13 2023-11-08T05:22:40.404 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ rbd --cluster=cluster1 info mirror/image_13 2023-11-08T05:22:40.404 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-11-08T05:22:40.630 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd: error opening image image_13: (2) No such file or directory 2023-11-08T05:22:40.633 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image_id= 2023-11-08T05:22:40.633 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-11-08T05:22:40.633 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0.1 2023-11-08T05:22:40.735 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test_image_present cluster1 mirror image_13 deleted '' 2023-11-08T05:22:40.735 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:22:40.735 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:22:40.736 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_13 2023-11-08T05:22:40.736 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local test_state=deleted 2023-11-08T05:22:40.736 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image_id= 2023-11-08T05:22:40.736 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local current_state=deleted 2023-11-08T05:22:40.736 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local current_image_id 2023-11-08T05:22:40.737 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ get_image_id cluster1 mirror image_13 2023-11-08T05:22:40.737 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1 2023-11-08T05:22:40.737 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local pool=mirror 2023-11-08T05:22:40.737 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local image=image_13 2023-11-08T05:22:40.737 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ rbd --cluster=cluster1 info mirror/image_13 2023-11-08T05:22:40.737 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-11-08T05:22:40.762 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd: error opening image image_13: (2) No such file or directory 2023-11-08T05:22:40.766 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ current_image_id= 2023-11-08T05:22:40.766 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test -n '' 2023-11-08T05:22:40.766 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test deleted = deleted 2023-11-08T05:22:40.766 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:22:40.766 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-11-08T05:22:40.767 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image=image_14 2023-11-08T05:22:40.767 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_image_present cluster1 mirror image_14 deleted 2023-11-08T05:22:40.767 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:22:40.767 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:22:40.768 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_14 2023-11-08T05:22:40.768 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local state=deleted 2023-11-08T05:22:40.768 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image_id= 2023-11-08T05:22:40.768 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local s 2023-11-08T05:22:40.768 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test -n '' 2023-11-08T05:22:40.769 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ get_image_id cluster1 mirror image_14 2023-11-08T05:22:40.769 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1 2023-11-08T05:22:40.769 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local pool=mirror 2023-11-08T05:22:40.769 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local image=image_14 2023-11-08T05:22:40.770 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ rbd --cluster=cluster1 info mirror/image_14 2023-11-08T05:22:40.770 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-11-08T05:22:40.791 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd: error opening image image_14: (2) No such file or directory 2023-11-08T05:22:40.794 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image_id= 2023-11-08T05:22:40.794 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-11-08T05:22:40.794 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0.1 2023-11-08T05:22:40.896 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test_image_present cluster1 mirror image_14 deleted '' 2023-11-08T05:22:40.896 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:22:40.897 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:22:40.897 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_14 2023-11-08T05:22:40.897 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local test_state=deleted 2023-11-08T05:22:40.897 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image_id= 2023-11-08T05:22:40.897 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local current_state=deleted 2023-11-08T05:22:40.898 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local current_image_id 2023-11-08T05:22:40.898 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ get_image_id cluster1 mirror image_14 2023-11-08T05:22:40.898 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1 2023-11-08T05:22:40.898 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local pool=mirror 2023-11-08T05:22:40.898 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local image=image_14 2023-11-08T05:22:40.899 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ rbd --cluster=cluster1 info mirror/image_14 2023-11-08T05:22:40.899 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-11-08T05:22:40.922 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd: error opening image image_14: (2) No such file or directory 2023-11-08T05:22:40.925 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ current_image_id= 2023-11-08T05:22:40.926 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test -n '' 2023-11-08T05:22:40.926 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test deleted = deleted 2023-11-08T05:22:40.926 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:22:40.926 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-11-08T05:22:40.926 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image=image_15 2023-11-08T05:22:40.927 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_image_present cluster1 mirror image_15 deleted 2023-11-08T05:22:40.927 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:22:40.927 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:22:40.927 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_15 2023-11-08T05:22:40.927 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local state=deleted 2023-11-08T05:22:40.928 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image_id= 2023-11-08T05:22:40.928 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local s 2023-11-08T05:22:40.928 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test -n '' 2023-11-08T05:22:40.928 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ get_image_id cluster1 mirror image_15 2023-11-08T05:22:40.929 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1 2023-11-08T05:22:40.929 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local pool=mirror 2023-11-08T05:22:40.929 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local image=image_15 2023-11-08T05:22:40.929 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ rbd --cluster=cluster1 info mirror/image_15 2023-11-08T05:22:40.929 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-11-08T05:22:41.153 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd: error opening image image_15: (2) No such file or directory 2023-11-08T05:22:41.155 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image_id= 2023-11-08T05:22:41.155 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-11-08T05:22:41.155 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0.1 2023-11-08T05:22:41.257 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test_image_present cluster1 mirror image_15 deleted '' 2023-11-08T05:22:41.257 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:22:41.258 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:22:41.258 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_15 2023-11-08T05:22:41.258 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local test_state=deleted 2023-11-08T05:22:41.259 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image_id= 2023-11-08T05:22:41.259 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local current_state=deleted 2023-11-08T05:22:41.259 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local current_image_id 2023-11-08T05:22:41.259 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ get_image_id cluster1 mirror image_15 2023-11-08T05:22:41.260 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1 2023-11-08T05:22:41.260 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local pool=mirror 2023-11-08T05:22:41.260 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local image=image_15 2023-11-08T05:22:41.261 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ rbd --cluster=cluster1 info mirror/image_15 2023-11-08T05:22:41.261 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-11-08T05:22:41.284 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd: error opening image image_15: (2) No such file or directory 2023-11-08T05:22:41.288 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ current_image_id= 2023-11-08T05:22:41.289 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test -n '' 2023-11-08T05:22:41.289 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test deleted = deleted 2023-11-08T05:22:41.289 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:22:41.289 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-11-08T05:22:41.289 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image=image_16 2023-11-08T05:22:41.290 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_image_present cluster1 mirror image_16 deleted 2023-11-08T05:22:41.290 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:22:41.290 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:22:41.290 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_16 2023-11-08T05:22:41.290 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local state=deleted 2023-11-08T05:22:41.291 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image_id= 2023-11-08T05:22:41.291 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local s 2023-11-08T05:22:41.291 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test -n '' 2023-11-08T05:22:41.291 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ get_image_id cluster1 mirror image_16 2023-11-08T05:22:41.291 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1 2023-11-08T05:22:41.292 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local pool=mirror 2023-11-08T05:22:41.292 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local image=image_16 2023-11-08T05:22:41.292 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ rbd --cluster=cluster1 info mirror/image_16 2023-11-08T05:22:41.292 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-11-08T05:22:41.314 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd: error opening image image_16: (2) No such file or directory 2023-11-08T05:22:41.316 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image_id= 2023-11-08T05:22:41.317 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-11-08T05:22:41.317 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0.1 2023-11-08T05:22:41.418 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test_image_present cluster1 mirror image_16 deleted '' 2023-11-08T05:22:41.418 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:22:41.418 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:22:41.419 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_16 2023-11-08T05:22:41.419 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local test_state=deleted 2023-11-08T05:22:41.419 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image_id= 2023-11-08T05:22:41.419 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local current_state=deleted 2023-11-08T05:22:41.419 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local current_image_id 2023-11-08T05:22:41.420 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ get_image_id cluster1 mirror image_16 2023-11-08T05:22:41.420 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1 2023-11-08T05:22:41.420 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local pool=mirror 2023-11-08T05:22:41.420 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local image=image_16 2023-11-08T05:22:41.420 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ rbd --cluster=cluster1 info mirror/image_16 2023-11-08T05:22:41.421 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-11-08T05:22:41.444 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd: error opening image image_16: (2) No such file or directory 2023-11-08T05:22:41.448 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ current_image_id= 2023-11-08T05:22:41.448 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test -n '' 2023-11-08T05:22:41.448 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test deleted = deleted 2023-11-08T05:22:41.448 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:22:41.449 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-11-08T05:22:41.449 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image=image_17 2023-11-08T05:22:41.449 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_image_present cluster1 mirror image_17 deleted 2023-11-08T05:22:41.449 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:22:41.449 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:22:41.450 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_17 2023-11-08T05:22:41.450 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local state=deleted 2023-11-08T05:22:41.450 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image_id= 2023-11-08T05:22:41.450 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local s 2023-11-08T05:22:41.450 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test -n '' 2023-11-08T05:22:41.451 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ get_image_id cluster1 mirror image_17 2023-11-08T05:22:41.451 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1 2023-11-08T05:22:41.451 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local pool=mirror 2023-11-08T05:22:41.451 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local image=image_17 2023-11-08T05:22:41.451 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ rbd --cluster=cluster1 info mirror/image_17 2023-11-08T05:22:41.452 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-11-08T05:22:41.474 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd: error opening image image_17: (2) No such file or directory 2023-11-08T05:22:41.476 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image_id= 2023-11-08T05:22:41.476 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-11-08T05:22:41.477 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0.1 2023-11-08T05:22:41.578 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test_image_present cluster1 mirror image_17 deleted '' 2023-11-08T05:22:41.578 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:22:41.578 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:22:41.579 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_17 2023-11-08T05:22:41.579 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local test_state=deleted 2023-11-08T05:22:41.579 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image_id= 2023-11-08T05:22:41.579 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local current_state=deleted 2023-11-08T05:22:41.579 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local current_image_id 2023-11-08T05:22:41.580 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ get_image_id cluster1 mirror image_17 2023-11-08T05:22:41.580 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1 2023-11-08T05:22:41.580 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local pool=mirror 2023-11-08T05:22:41.580 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local image=image_17 2023-11-08T05:22:41.580 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ rbd --cluster=cluster1 info mirror/image_17 2023-11-08T05:22:41.581 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-11-08T05:22:41.604 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd: error opening image image_17: (2) No such file or directory 2023-11-08T05:22:41.608 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ current_image_id= 2023-11-08T05:22:41.608 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test -n '' 2023-11-08T05:22:41.608 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test deleted = deleted 2023-11-08T05:22:41.609 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:22:41.609 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-11-08T05:22:41.609 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image=image_18 2023-11-08T05:22:41.609 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_image_present cluster1 mirror image_18 deleted 2023-11-08T05:22:41.609 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:22:41.610 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:22:41.610 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_18 2023-11-08T05:22:41.610 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local state=deleted 2023-11-08T05:22:41.610 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image_id= 2023-11-08T05:22:41.611 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local s 2023-11-08T05:22:41.611 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test -n '' 2023-11-08T05:22:41.611 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ get_image_id cluster1 mirror image_18 2023-11-08T05:22:41.611 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1 2023-11-08T05:22:41.611 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local pool=mirror 2023-11-08T05:22:41.612 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local image=image_18 2023-11-08T05:22:41.612 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ rbd --cluster=cluster1 info mirror/image_18 2023-11-08T05:22:41.612 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-11-08T05:22:41.640 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd: error opening image image_18: (2) No such file or directory 2023-11-08T05:22:41.643 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image_id= 2023-11-08T05:22:41.643 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-11-08T05:22:41.644 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0.1 2023-11-08T05:22:41.745 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test_image_present cluster1 mirror image_18 deleted '' 2023-11-08T05:22:41.746 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:22:41.746 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:22:41.746 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_18 2023-11-08T05:22:41.746 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local test_state=deleted 2023-11-08T05:22:41.746 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image_id= 2023-11-08T05:22:41.747 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local current_state=deleted 2023-11-08T05:22:41.747 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local current_image_id 2023-11-08T05:22:41.747 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ get_image_id cluster1 mirror image_18 2023-11-08T05:22:41.747 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1 2023-11-08T05:22:41.747 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local pool=mirror 2023-11-08T05:22:41.747 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local image=image_18 2023-11-08T05:22:41.748 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ rbd --cluster=cluster1 info mirror/image_18 2023-11-08T05:22:41.748 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-11-08T05:22:41.771 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd: error opening image image_18: (2) No such file or directory 2023-11-08T05:22:41.775 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ current_image_id= 2023-11-08T05:22:41.775 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test -n '' 2023-11-08T05:22:41.775 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test deleted = deleted 2023-11-08T05:22:41.775 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:22:41.776 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-11-08T05:22:41.776 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image=image_19 2023-11-08T05:22:41.776 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_image_present cluster1 mirror image_19 deleted 2023-11-08T05:22:41.776 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:22:41.776 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:22:41.776 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_19 2023-11-08T05:22:41.777 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local state=deleted 2023-11-08T05:22:41.777 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image_id= 2023-11-08T05:22:41.777 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local s 2023-11-08T05:22:41.777 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test -n '' 2023-11-08T05:22:41.777 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ get_image_id cluster1 mirror image_19 2023-11-08T05:22:41.778 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1 2023-11-08T05:22:41.778 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local pool=mirror 2023-11-08T05:22:41.778 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local image=image_19 2023-11-08T05:22:41.778 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ rbd --cluster=cluster1 info mirror/image_19 2023-11-08T05:22:41.778 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-11-08T05:22:41.800 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd: error opening image image_19: (2) No such file or directory 2023-11-08T05:22:41.803 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image_id= 2023-11-08T05:22:41.803 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-11-08T05:22:41.803 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0.1 2023-11-08T05:22:41.905 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test_image_present cluster1 mirror image_19 deleted '' 2023-11-08T05:22:41.905 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:22:41.906 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:22:41.906 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_19 2023-11-08T05:22:41.906 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local test_state=deleted 2023-11-08T05:22:41.906 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image_id= 2023-11-08T05:22:41.907 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local current_state=deleted 2023-11-08T05:22:41.907 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local current_image_id 2023-11-08T05:22:41.907 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ get_image_id cluster1 mirror image_19 2023-11-08T05:22:41.907 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1 2023-11-08T05:22:41.907 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local pool=mirror 2023-11-08T05:22:41.908 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local image=image_19 2023-11-08T05:22:41.908 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ rbd --cluster=cluster1 info mirror/image_19 2023-11-08T05:22:41.908 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-11-08T05:22:41.931 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd: error opening image image_19: (2) No such file or directory 2023-11-08T05:22:41.935 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ current_image_id= 2023-11-08T05:22:41.936 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test -n '' 2023-11-08T05:22:41.936 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test deleted = deleted 2023-11-08T05:22:41.936 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:22:41.936 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-11-08T05:22:41.936 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image=image_20 2023-11-08T05:22:41.937 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_image_present cluster1 mirror image_20 deleted 2023-11-08T05:22:41.937 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:22:41.937 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:22:41.937 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_20 2023-11-08T05:22:41.937 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local state=deleted 2023-11-08T05:22:41.938 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image_id= 2023-11-08T05:22:41.938 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local s 2023-11-08T05:22:41.938 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test -n '' 2023-11-08T05:22:41.938 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ get_image_id cluster1 mirror image_20 2023-11-08T05:22:41.939 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1 2023-11-08T05:22:41.939 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local pool=mirror 2023-11-08T05:22:41.939 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local image=image_20 2023-11-08T05:22:41.939 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ rbd --cluster=cluster1 info mirror/image_20 2023-11-08T05:22:41.939 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-11-08T05:22:41.964 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd: error opening image image_20: (2) No such file or directory 2023-11-08T05:22:41.965 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image_id= 2023-11-08T05:22:41.965 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-11-08T05:22:41.966 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0.1 2023-11-08T05:22:42.064 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test_image_present cluster1 mirror image_20 deleted '' 2023-11-08T05:22:42.065 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:22:42.065 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:22:42.065 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_20 2023-11-08T05:22:42.065 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local test_state=deleted 2023-11-08T05:22:42.066 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image_id= 2023-11-08T05:22:42.066 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local current_state=deleted 2023-11-08T05:22:42.066 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local current_image_id 2023-11-08T05:22:42.066 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ get_image_id cluster1 mirror image_20 2023-11-08T05:22:42.066 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1 2023-11-08T05:22:42.067 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local pool=mirror 2023-11-08T05:22:42.067 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local image=image_20 2023-11-08T05:22:42.067 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ rbd --cluster=cluster1 info mirror/image_20 2023-11-08T05:22:42.067 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-11-08T05:22:42.090 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd: error opening image image_20: (2) No such file or directory 2023-11-08T05:22:42.093 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ current_image_id= 2023-11-08T05:22:42.094 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test -n '' 2023-11-08T05:22:42.094 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test deleted = deleted 2023-11-08T05:22:42.094 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:22:42.094 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-11-08T05:22:42.095 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image=image_21 2023-11-08T05:22:42.095 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_image_present cluster1 mirror image_21 deleted 2023-11-08T05:22:42.095 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:22:42.095 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:22:42.095 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_21 2023-11-08T05:22:42.096 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local state=deleted 2023-11-08T05:22:42.096 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image_id= 2023-11-08T05:22:42.096 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local s 2023-11-08T05:22:42.096 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test -n '' 2023-11-08T05:22:42.096 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ get_image_id cluster1 mirror image_21 2023-11-08T05:22:42.097 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1 2023-11-08T05:22:42.097 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local pool=mirror 2023-11-08T05:22:42.097 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local image=image_21 2023-11-08T05:22:42.097 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ rbd --cluster=cluster1 info mirror/image_21 2023-11-08T05:22:42.097 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-11-08T05:22:42.119 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd: error opening image image_21: (2) No such file or directory 2023-11-08T05:22:42.121 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image_id= 2023-11-08T05:22:42.122 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-11-08T05:22:42.122 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0.1 2023-11-08T05:22:42.223 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test_image_present cluster1 mirror image_21 deleted '' 2023-11-08T05:22:42.224 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:22:42.224 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:22:42.224 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_21 2023-11-08T05:22:42.224 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local test_state=deleted 2023-11-08T05:22:42.224 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image_id= 2023-11-08T05:22:42.225 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local current_state=deleted 2023-11-08T05:22:42.225 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local current_image_id 2023-11-08T05:22:42.225 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ get_image_id cluster1 mirror image_21 2023-11-08T05:22:42.225 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1 2023-11-08T05:22:42.225 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local pool=mirror 2023-11-08T05:22:42.226 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local image=image_21 2023-11-08T05:22:42.226 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ rbd --cluster=cluster1 info mirror/image_21 2023-11-08T05:22:42.226 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-11-08T05:22:42.249 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd: error opening image image_21: (2) No such file or directory 2023-11-08T05:22:42.253 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ current_image_id= 2023-11-08T05:22:42.254 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test -n '' 2023-11-08T05:22:42.254 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test deleted = deleted 2023-11-08T05:22:42.254 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:22:42.254 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-11-08T05:22:42.254 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image=image_22 2023-11-08T05:22:42.255 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_image_present cluster1 mirror image_22 deleted 2023-11-08T05:22:42.255 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:22:42.255 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:22:42.255 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_22 2023-11-08T05:22:42.255 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local state=deleted 2023-11-08T05:22:42.256 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image_id= 2023-11-08T05:22:42.256 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local s 2023-11-08T05:22:42.256 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test -n '' 2023-11-08T05:22:42.256 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ get_image_id cluster1 mirror image_22 2023-11-08T05:22:42.256 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1 2023-11-08T05:22:42.257 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local pool=mirror 2023-11-08T05:22:42.257 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local image=image_22 2023-11-08T05:22:42.257 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ rbd --cluster=cluster1 info mirror/image_22 2023-11-08T05:22:42.257 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-11-08T05:22:42.279 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd: error opening image image_22: (2) No such file or directory 2023-11-08T05:22:42.282 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image_id= 2023-11-08T05:22:42.282 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-11-08T05:22:42.282 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0.1 2023-11-08T05:22:42.335 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.0 2023-11-08T05:22:42.335 INFO:teuthology.orchestra.run:waiting for 600 2023-11-08T05:22:42.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-08T05:22:42.336 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.0 2023-11-08T05:22:42.336 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2023-11-08T05:22:42.337 DEBUG:teuthology.orchestra.run.smithi087:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2023-11-08T05:22:42.339 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2023-11-08T05:22:42.340 INFO:tasks.rbd_mirror_thrash:waiting for 108 secs before thrashing 2023-11-08T05:22:42.383 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test_image_present cluster1 mirror image_22 deleted '' 2023-11-08T05:22:42.384 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:22:42.384 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:22:42.384 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_22 2023-11-08T05:22:42.384 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local test_state=deleted 2023-11-08T05:22:42.385 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image_id= 2023-11-08T05:22:42.385 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local current_state=deleted 2023-11-08T05:22:42.385 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local current_image_id 2023-11-08T05:22:42.385 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ get_image_id cluster1 mirror image_22 2023-11-08T05:22:42.385 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1 2023-11-08T05:22:42.386 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local pool=mirror 2023-11-08T05:22:42.386 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local image=image_22 2023-11-08T05:22:42.386 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ rbd --cluster=cluster1 info mirror/image_22 2023-11-08T05:22:42.386 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-11-08T05:22:42.411 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd: error opening image image_22: (2) No such file or directory 2023-11-08T05:22:42.414 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ current_image_id= 2023-11-08T05:22:42.415 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test -n '' 2023-11-08T05:22:42.415 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test deleted = deleted 2023-11-08T05:22:42.415 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:22:42.415 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-11-08T05:22:42.415 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image=image_23 2023-11-08T05:22:42.416 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_image_present cluster1 mirror image_23 deleted 2023-11-08T05:22:42.416 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:22:42.416 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:22:42.416 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_23 2023-11-08T05:22:42.416 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local state=deleted 2023-11-08T05:22:42.417 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image_id= 2023-11-08T05:22:42.417 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local s 2023-11-08T05:22:42.417 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test -n '' 2023-11-08T05:22:42.417 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ get_image_id cluster1 mirror image_23 2023-11-08T05:22:42.417 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1 2023-11-08T05:22:42.417 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local pool=mirror 2023-11-08T05:22:42.418 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local image=image_23 2023-11-08T05:22:42.418 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ rbd --cluster=cluster1 info mirror/image_23 2023-11-08T05:22:42.418 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-11-08T05:22:42.441 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd: error opening image image_23: (2) No such file or directory 2023-11-08T05:22:42.444 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image_id= 2023-11-08T05:22:42.444 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-11-08T05:22:42.444 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0.1 2023-11-08T05:22:42.546 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test_image_present cluster1 mirror image_23 deleted '' 2023-11-08T05:22:42.546 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:22:42.546 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:22:42.546 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_23 2023-11-08T05:22:42.547 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local test_state=deleted 2023-11-08T05:22:42.547 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image_id= 2023-11-08T05:22:42.547 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local current_state=deleted 2023-11-08T05:22:42.547 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local current_image_id 2023-11-08T05:22:42.547 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ get_image_id cluster1 mirror image_23 2023-11-08T05:22:42.548 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1 2023-11-08T05:22:42.548 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local pool=mirror 2023-11-08T05:22:42.548 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local image=image_23 2023-11-08T05:22:42.548 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ rbd --cluster=cluster1 info mirror/image_23 2023-11-08T05:22:42.548 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-11-08T05:22:42.572 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd: error opening image image_23: (2) No such file or directory 2023-11-08T05:22:42.576 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ current_image_id= 2023-11-08T05:22:42.576 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test -n '' 2023-11-08T05:22:42.576 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test deleted = deleted 2023-11-08T05:22:42.576 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:22:42.577 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-11-08T05:22:42.577 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image=image_24 2023-11-08T05:22:42.577 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_image_present cluster1 mirror image_24 deleted 2023-11-08T05:22:42.577 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:22:42.577 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:22:42.578 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_24 2023-11-08T05:22:42.578 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local state=deleted 2023-11-08T05:22:42.578 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image_id= 2023-11-08T05:22:42.578 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local s 2023-11-08T05:22:42.578 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test -n '' 2023-11-08T05:22:42.579 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ get_image_id cluster1 mirror image_24 2023-11-08T05:22:42.579 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1 2023-11-08T05:22:42.579 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local pool=mirror 2023-11-08T05:22:42.579 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local image=image_24 2023-11-08T05:22:42.579 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ rbd --cluster=cluster1 info mirror/image_24 2023-11-08T05:22:42.580 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-11-08T05:22:42.601 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd: error opening image image_24: (2) No such file or directory 2023-11-08T05:22:42.603 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image_id= 2023-11-08T05:22:42.604 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-11-08T05:22:42.604 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0.1 2023-11-08T05:22:42.705 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test_image_present cluster1 mirror image_24 deleted '' 2023-11-08T05:22:42.706 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:22:42.706 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:22:42.706 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_24 2023-11-08T05:22:42.706 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local test_state=deleted 2023-11-08T05:22:42.707 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image_id= 2023-11-08T05:22:42.707 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local current_state=deleted 2023-11-08T05:22:42.707 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local current_image_id 2023-11-08T05:22:42.707 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ get_image_id cluster1 mirror image_24 2023-11-08T05:22:42.707 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1 2023-11-08T05:22:42.708 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local pool=mirror 2023-11-08T05:22:42.708 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local image=image_24 2023-11-08T05:22:42.708 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ rbd --cluster=cluster1 info mirror/image_24 2023-11-08T05:22:42.708 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-11-08T05:22:42.730 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd: error opening image image_24: (2) No such file or directory 2023-11-08T05:22:42.734 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ current_image_id= 2023-11-08T05:22:42.734 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test -n '' 2023-11-08T05:22:42.734 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test deleted = deleted 2023-11-08T05:22:42.735 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:22:42.735 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-11-08T05:22:42.735 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image=image_25 2023-11-08T05:22:42.735 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_image_present cluster1 mirror image_25 deleted 2023-11-08T05:22:42.735 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:22:42.736 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:22:42.736 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_25 2023-11-08T05:22:42.736 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local state=deleted 2023-11-08T05:22:42.736 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image_id= 2023-11-08T05:22:42.736 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local s 2023-11-08T05:22:42.737 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test -n '' 2023-11-08T05:22:42.737 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ get_image_id cluster1 mirror image_25 2023-11-08T05:22:42.737 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1 2023-11-08T05:22:42.737 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local pool=mirror 2023-11-08T05:22:42.737 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local image=image_25 2023-11-08T05:22:42.738 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ rbd --cluster=cluster1 info mirror/image_25 2023-11-08T05:22:42.738 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-11-08T05:22:42.759 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd: error opening image image_25: (2) No such file or directory 2023-11-08T05:22:42.762 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image_id= 2023-11-08T05:22:42.762 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-11-08T05:22:42.762 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0.1 2023-11-08T05:22:42.864 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test_image_present cluster1 mirror image_25 deleted '' 2023-11-08T05:22:42.864 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:22:42.864 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:22:42.865 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_25 2023-11-08T05:22:42.865 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local test_state=deleted 2023-11-08T05:22:42.865 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image_id= 2023-11-08T05:22:42.865 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local current_state=deleted 2023-11-08T05:22:42.865 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local current_image_id 2023-11-08T05:22:42.866 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ get_image_id cluster1 mirror image_25 2023-11-08T05:22:42.866 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1 2023-11-08T05:22:42.866 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local pool=mirror 2023-11-08T05:22:42.866 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local image=image_25 2023-11-08T05:22:42.866 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ rbd --cluster=cluster1 info mirror/image_25 2023-11-08T05:22:42.867 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-11-08T05:22:42.895 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd: error opening image image_25: (2) No such file or directory 2023-11-08T05:22:42.896 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ current_image_id= 2023-11-08T05:22:42.896 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test -n '' 2023-11-08T05:22:42.896 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test deleted = deleted 2023-11-08T05:22:42.897 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:22:42.897 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-11-08T05:22:42.897 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image=image_26 2023-11-08T05:22:42.897 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_image_present cluster1 mirror image_26 deleted 2023-11-08T05:22:42.898 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:22:42.898 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:22:42.898 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_26 2023-11-08T05:22:42.898 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local state=deleted 2023-11-08T05:22:42.899 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image_id= 2023-11-08T05:22:42.899 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local s 2023-11-08T05:22:42.899 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test -n '' 2023-11-08T05:22:42.899 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ get_image_id cluster1 mirror image_26 2023-11-08T05:22:42.899 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1 2023-11-08T05:22:42.900 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local pool=mirror 2023-11-08T05:22:42.900 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local image=image_26 2023-11-08T05:22:42.900 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ rbd --cluster=cluster1 info mirror/image_26 2023-11-08T05:22:42.900 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-11-08T05:22:42.920 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd: error opening image image_26: (2) No such file or directory 2023-11-08T05:22:42.922 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image_id= 2023-11-08T05:22:42.922 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-11-08T05:22:42.923 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0.1 2023-11-08T05:22:43.024 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test_image_present cluster1 mirror image_26 deleted '' 2023-11-08T05:22:43.024 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:22:43.024 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:22:43.025 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_26 2023-11-08T05:22:43.025 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local test_state=deleted 2023-11-08T05:22:43.025 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image_id= 2023-11-08T05:22:43.025 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local current_state=deleted 2023-11-08T05:22:43.025 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local current_image_id 2023-11-08T05:22:43.026 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ get_image_id cluster1 mirror image_26 2023-11-08T05:22:43.026 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1 2023-11-08T05:22:43.026 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local pool=mirror 2023-11-08T05:22:43.026 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local image=image_26 2023-11-08T05:22:43.027 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ rbd --cluster=cluster1 info mirror/image_26 2023-11-08T05:22:43.027 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-11-08T05:22:43.049 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd: error opening image image_26: (2) No such file or directory 2023-11-08T05:22:43.052 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ current_image_id= 2023-11-08T05:22:43.053 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test -n '' 2023-11-08T05:22:43.053 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test deleted = deleted 2023-11-08T05:22:43.053 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:22:43.053 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-11-08T05:22:43.054 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image=image_27 2023-11-08T05:22:43.054 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_image_present cluster1 mirror image_27 deleted 2023-11-08T05:22:43.054 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:22:43.054 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:22:43.054 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_27 2023-11-08T05:22:43.055 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local state=deleted 2023-11-08T05:22:43.055 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image_id= 2023-11-08T05:22:43.055 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local s 2023-11-08T05:22:43.055 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test -n '' 2023-11-08T05:22:43.055 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ get_image_id cluster1 mirror image_27 2023-11-08T05:22:43.056 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1 2023-11-08T05:22:43.056 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local pool=mirror 2023-11-08T05:22:43.056 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local image=image_27 2023-11-08T05:22:43.056 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ rbd --cluster=cluster1 info mirror/image_27 2023-11-08T05:22:43.056 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-11-08T05:22:43.078 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd: error opening image image_27: (2) No such file or directory 2023-11-08T05:22:43.081 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image_id= 2023-11-08T05:22:43.081 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-11-08T05:22:43.081 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0.1 2023-11-08T05:22:43.183 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test_image_present cluster1 mirror image_27 deleted '' 2023-11-08T05:22:43.183 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:22:43.183 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:22:43.184 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_27 2023-11-08T05:22:43.184 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local test_state=deleted 2023-11-08T05:22:43.184 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image_id= 2023-11-08T05:22:43.184 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local current_state=deleted 2023-11-08T05:22:43.184 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local current_image_id 2023-11-08T05:22:43.185 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ get_image_id cluster1 mirror image_27 2023-11-08T05:22:43.185 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1 2023-11-08T05:22:43.185 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local pool=mirror 2023-11-08T05:22:43.185 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local image=image_27 2023-11-08T05:22:43.185 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ rbd --cluster=cluster1 info mirror/image_27 2023-11-08T05:22:43.186 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-11-08T05:22:43.209 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd: error opening image image_27: (2) No such file or directory 2023-11-08T05:22:43.259 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ current_image_id= 2023-11-08T05:22:43.259 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test -n '' 2023-11-08T05:22:43.259 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test deleted = deleted 2023-11-08T05:22:43.259 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:22:43.260 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-11-08T05:22:43.260 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image=image_28 2023-11-08T05:22:43.260 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_image_present cluster1 mirror image_28 deleted 2023-11-08T05:22:43.260 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:22:43.260 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:22:43.261 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_28 2023-11-08T05:22:43.261 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local state=deleted 2023-11-08T05:22:43.261 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image_id= 2023-11-08T05:22:43.261 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local s 2023-11-08T05:22:43.262 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test -n '' 2023-11-08T05:22:43.262 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ get_image_id cluster1 mirror image_28 2023-11-08T05:22:43.262 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1 2023-11-08T05:22:43.262 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local pool=mirror 2023-11-08T05:22:43.262 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local image=image_28 2023-11-08T05:22:43.263 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ rbd --cluster=cluster1 info mirror/image_28 2023-11-08T05:22:43.263 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-11-08T05:22:43.263 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd: error opening image image_28: (2) No such file or directory 2023-11-08T05:22:43.263 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image_id= 2023-11-08T05:22:43.263 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-11-08T05:22:43.264 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0.1 2023-11-08T05:22:43.345 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test_image_present cluster1 mirror image_28 deleted '' 2023-11-08T05:22:43.346 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:22:43.346 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:22:43.346 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_28 2023-11-08T05:22:43.346 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local test_state=deleted 2023-11-08T05:22:43.346 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image_id= 2023-11-08T05:22:43.347 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local current_state=deleted 2023-11-08T05:22:43.347 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local current_image_id 2023-11-08T05:22:43.347 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ get_image_id cluster1 mirror image_28 2023-11-08T05:22:43.348 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1 2023-11-08T05:22:43.348 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local pool=mirror 2023-11-08T05:22:43.348 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local image=image_28 2023-11-08T05:22:43.354 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ rbd --cluster=cluster1 info mirror/image_28 2023-11-08T05:22:43.355 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-11-08T05:22:43.375 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd: error opening image image_28: (2) No such file or directory 2023-11-08T05:22:43.379 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ current_image_id= 2023-11-08T05:22:43.379 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test -n '' 2023-11-08T05:22:43.379 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test deleted = deleted 2023-11-08T05:22:43.380 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:22:43.380 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-11-08T05:22:43.380 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image=image_29 2023-11-08T05:22:43.380 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_image_present cluster1 mirror image_29 deleted 2023-11-08T05:22:43.380 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:22:43.381 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:22:43.381 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_29 2023-11-08T05:22:43.381 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local state=deleted 2023-11-08T05:22:43.381 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image_id= 2023-11-08T05:22:43.381 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local s 2023-11-08T05:22:43.381 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test -n '' 2023-11-08T05:22:43.382 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ get_image_id cluster1 mirror image_29 2023-11-08T05:22:43.382 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1 2023-11-08T05:22:43.382 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local pool=mirror 2023-11-08T05:22:43.383 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local image=image_29 2023-11-08T05:22:43.383 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ rbd --cluster=cluster1 info mirror/image_29 2023-11-08T05:22:43.383 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-11-08T05:22:43.405 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd: error opening image image_29: (2) No such file or directory 2023-11-08T05:22:43.407 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image_id= 2023-11-08T05:22:43.407 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-11-08T05:22:43.408 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0.1 2023-11-08T05:22:43.509 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test_image_present cluster1 mirror image_29 deleted '' 2023-11-08T05:22:43.509 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:22:43.509 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:22:43.510 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_29 2023-11-08T05:22:43.510 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local test_state=deleted 2023-11-08T05:22:43.510 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image_id= 2023-11-08T05:22:43.510 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local current_state=deleted 2023-11-08T05:22:43.510 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local current_image_id 2023-11-08T05:22:43.514 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ get_image_id cluster1 mirror image_29 2023-11-08T05:22:43.514 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1 2023-11-08T05:22:43.514 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local pool=mirror 2023-11-08T05:22:43.514 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local image=image_29 2023-11-08T05:22:43.514 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ rbd --cluster=cluster1 info mirror/image_29 2023-11-08T05:22:43.515 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-11-08T05:22:43.535 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd: error opening image image_29: (2) No such file or directory 2023-11-08T05:22:43.539 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ current_image_id= 2023-11-08T05:22:43.539 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test -n '' 2023-11-08T05:22:43.540 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test deleted = deleted 2023-11-08T05:22:43.540 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:22:43.540 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-11-08T05:22:43.540 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image=image_30 2023-11-08T05:22:43.540 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_image_present cluster1 mirror image_30 deleted 2023-11-08T05:22:43.541 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:22:43.541 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:22:43.541 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_30 2023-11-08T05:22:43.541 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local state=deleted 2023-11-08T05:22:43.541 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image_id= 2023-11-08T05:22:43.541 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local s 2023-11-08T05:22:43.542 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test -n '' 2023-11-08T05:22:43.542 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ get_image_id cluster1 mirror image_30 2023-11-08T05:22:43.542 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1 2023-11-08T05:22:43.542 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local pool=mirror 2023-11-08T05:22:43.543 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local image=image_30 2023-11-08T05:22:43.543 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ rbd --cluster=cluster1 info mirror/image_30 2023-11-08T05:22:43.543 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-11-08T05:22:43.567 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd: error opening image image_30: (2) No such file or directory 2023-11-08T05:22:43.570 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image_id= 2023-11-08T05:22:43.570 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-11-08T05:22:43.570 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0.1 2023-11-08T05:22:43.672 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test_image_present cluster1 mirror image_30 deleted '' 2023-11-08T05:22:43.672 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:22:43.672 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:22:43.673 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_30 2023-11-08T05:22:43.673 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local test_state=deleted 2023-11-08T05:22:43.673 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image_id= 2023-11-08T05:22:43.673 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local current_state=deleted 2023-11-08T05:22:43.673 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local current_image_id 2023-11-08T05:22:43.674 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ get_image_id cluster1 mirror image_30 2023-11-08T05:22:43.674 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1 2023-11-08T05:22:43.674 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local pool=mirror 2023-11-08T05:22:43.674 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local image=image_30 2023-11-08T05:22:43.674 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ rbd --cluster=cluster1 info mirror/image_30 2023-11-08T05:22:43.675 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-11-08T05:22:43.699 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd: error opening image image_30: (2) No such file or directory 2023-11-08T05:22:43.703 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ current_image_id= 2023-11-08T05:22:43.703 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test -n '' 2023-11-08T05:22:43.703 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test deleted = deleted 2023-11-08T05:22:43.704 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:22:43.704 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-11-08T05:22:43.704 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image=image_31 2023-11-08T05:22:43.704 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_image_present cluster1 mirror image_31 deleted 2023-11-08T05:22:43.705 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:22:43.705 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:22:43.705 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_31 2023-11-08T05:22:43.705 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local state=deleted 2023-11-08T05:22:43.705 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image_id= 2023-11-08T05:22:43.705 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local s 2023-11-08T05:22:43.706 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test -n '' 2023-11-08T05:22:43.706 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ get_image_id cluster1 mirror image_31 2023-11-08T05:22:43.706 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1 2023-11-08T05:22:43.706 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local pool=mirror 2023-11-08T05:22:43.706 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local image=image_31 2023-11-08T05:22:43.707 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ rbd --cluster=cluster1 info mirror/image_31 2023-11-08T05:22:43.707 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-11-08T05:22:43.728 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd: error opening image image_31: (2) No such file or directory 2023-11-08T05:22:43.731 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image_id= 2023-11-08T05:22:43.731 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-11-08T05:22:43.731 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0.1 2023-11-08T05:22:43.833 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test_image_present cluster1 mirror image_31 deleted '' 2023-11-08T05:22:43.833 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:22:43.833 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:22:43.834 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_31 2023-11-08T05:22:43.834 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local test_state=deleted 2023-11-08T05:22:43.834 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image_id= 2023-11-08T05:22:43.834 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local current_state=deleted 2023-11-08T05:22:43.834 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local current_image_id 2023-11-08T05:22:43.835 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ get_image_id cluster1 mirror image_31 2023-11-08T05:22:43.835 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1 2023-11-08T05:22:43.835 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local pool=mirror 2023-11-08T05:22:43.835 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local image=image_31 2023-11-08T05:22:43.835 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ rbd --cluster=cluster1 info mirror/image_31 2023-11-08T05:22:43.835 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-11-08T05:22:43.858 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd: error opening image image_31: (2) No such file or directory 2023-11-08T05:22:43.862 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ current_image_id= 2023-11-08T05:22:43.862 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test -n '' 2023-11-08T05:22:43.863 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test deleted = deleted 2023-11-08T05:22:43.863 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:22:43.863 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-11-08T05:22:43.863 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image=image_32 2023-11-08T05:22:43.863 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_image_present cluster1 mirror image_32 deleted 2023-11-08T05:22:43.864 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:22:43.864 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:22:43.864 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_32 2023-11-08T05:22:43.864 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local state=deleted 2023-11-08T05:22:43.864 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image_id= 2023-11-08T05:22:43.865 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local s 2023-11-08T05:22:43.865 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test -n '' 2023-11-08T05:22:43.865 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ get_image_id cluster1 mirror image_32 2023-11-08T05:22:43.865 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1 2023-11-08T05:22:43.865 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local pool=mirror 2023-11-08T05:22:43.866 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local image=image_32 2023-11-08T05:22:43.866 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ rbd --cluster=cluster1 info mirror/image_32 2023-11-08T05:22:43.866 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-11-08T05:22:43.887 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd: error opening image image_32: (2) No such file or directory 2023-11-08T05:22:43.890 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image_id= 2023-11-08T05:22:43.890 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-11-08T05:22:43.891 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0.1 2023-11-08T05:22:43.992 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test_image_present cluster1 mirror image_32 deleted '' 2023-11-08T05:22:43.992 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:22:43.993 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:22:43.993 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_32 2023-11-08T05:22:43.993 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local test_state=deleted 2023-11-08T05:22:43.993 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image_id= 2023-11-08T05:22:43.993 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local current_state=deleted 2023-11-08T05:22:43.994 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local current_image_id 2023-11-08T05:22:43.994 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ get_image_id cluster1 mirror image_32 2023-11-08T05:22:43.994 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1 2023-11-08T05:22:43.994 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local pool=mirror 2023-11-08T05:22:43.994 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local image=image_32 2023-11-08T05:22:43.995 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ rbd --cluster=cluster1 info mirror/image_32 2023-11-08T05:22:43.995 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-11-08T05:22:44.019 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd: error opening image image_32: (2) No such file or directory 2023-11-08T05:22:44.023 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ current_image_id= 2023-11-08T05:22:44.023 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test -n '' 2023-11-08T05:22:44.023 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test deleted = deleted 2023-11-08T05:22:44.024 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:22:44.024 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-11-08T05:22:44.024 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image=image_33 2023-11-08T05:22:44.024 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_image_present cluster1 mirror image_33 deleted 2023-11-08T05:22:44.025 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:22:44.025 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:22:44.025 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_33 2023-11-08T05:22:44.025 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local state=deleted 2023-11-08T05:22:44.025 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image_id= 2023-11-08T05:22:44.026 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local s 2023-11-08T05:22:44.026 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test -n '' 2023-11-08T05:22:44.026 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ get_image_id cluster1 mirror image_33 2023-11-08T05:22:44.026 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1 2023-11-08T05:22:44.026 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local pool=mirror 2023-11-08T05:22:44.027 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local image=image_33 2023-11-08T05:22:44.027 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ rbd --cluster=cluster1 info mirror/image_33 2023-11-08T05:22:44.027 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-11-08T05:22:44.048 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd: error opening image image_33: (2) No such file or directory 2023-11-08T05:22:44.050 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image_id= 2023-11-08T05:22:44.051 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-11-08T05:22:44.051 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0.1 2023-11-08T05:22:44.153 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test_image_present cluster1 mirror image_33 deleted '' 2023-11-08T05:22:44.154 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:22:44.154 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:22:44.156 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_33 2023-11-08T05:22:44.156 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local test_state=deleted 2023-11-08T05:22:44.156 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image_id= 2023-11-08T05:22:44.157 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local current_state=deleted 2023-11-08T05:22:44.157 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local current_image_id 2023-11-08T05:22:44.157 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ get_image_id cluster1 mirror image_33 2023-11-08T05:22:44.157 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1 2023-11-08T05:22:44.157 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local pool=mirror 2023-11-08T05:22:44.158 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local image=image_33 2023-11-08T05:22:44.158 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ rbd --cluster=cluster1 info mirror/image_33 2023-11-08T05:22:44.158 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-11-08T05:22:44.178 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd: error opening image image_33: (2) No such file or directory 2023-11-08T05:22:44.182 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ current_image_id= 2023-11-08T05:22:44.182 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test -n '' 2023-11-08T05:22:44.182 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test deleted = deleted 2023-11-08T05:22:44.182 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:22:44.183 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-11-08T05:22:44.183 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image=image_34 2023-11-08T05:22:44.183 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_image_present cluster1 mirror image_34 deleted 2023-11-08T05:22:44.183 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:22:44.183 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:22:44.183 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_34 2023-11-08T05:22:44.184 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local state=deleted 2023-11-08T05:22:44.184 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image_id= 2023-11-08T05:22:44.184 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local s 2023-11-08T05:22:44.184 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test -n '' 2023-11-08T05:22:44.184 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ get_image_id cluster1 mirror image_34 2023-11-08T05:22:44.185 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1 2023-11-08T05:22:44.185 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local pool=mirror 2023-11-08T05:22:44.185 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local image=image_34 2023-11-08T05:22:44.185 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ rbd --cluster=cluster1 info mirror/image_34 2023-11-08T05:22:44.185 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-11-08T05:22:44.207 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd: error opening image image_34: (2) No such file or directory 2023-11-08T05:22:44.210 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image_id= 2023-11-08T05:22:44.210 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-11-08T05:22:44.210 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0.1 2023-11-08T05:22:44.312 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test_image_present cluster1 mirror image_34 deleted '' 2023-11-08T05:22:44.313 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:22:44.313 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:22:44.313 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_34 2023-11-08T05:22:44.314 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local test_state=deleted 2023-11-08T05:22:44.314 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image_id= 2023-11-08T05:22:44.315 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local current_state=deleted 2023-11-08T05:22:44.315 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local current_image_id 2023-11-08T05:22:44.315 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ get_image_id cluster1 mirror image_34 2023-11-08T05:22:44.316 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1 2023-11-08T05:22:44.316 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local pool=mirror 2023-11-08T05:22:44.316 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local image=image_34 2023-11-08T05:22:44.317 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ rbd --cluster=cluster1 info mirror/image_34 2023-11-08T05:22:44.317 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-11-08T05:22:44.338 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd: error opening image image_34: (2) No such file or directory 2023-11-08T05:22:44.342 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ current_image_id= 2023-11-08T05:22:44.343 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test -n '' 2023-11-08T05:22:44.343 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test deleted = deleted 2023-11-08T05:22:44.343 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:22:44.344 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-11-08T05:22:44.344 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image=image_35 2023-11-08T05:22:44.345 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_image_present cluster1 mirror image_35 deleted 2023-11-08T05:22:44.345 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:22:44.345 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:22:44.346 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_35 2023-11-08T05:22:44.346 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local state=deleted 2023-11-08T05:22:44.346 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image_id= 2023-11-08T05:22:44.347 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local s 2023-11-08T05:22:44.348 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test -n '' 2023-11-08T05:22:44.348 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ get_image_id cluster1 mirror image_35 2023-11-08T05:22:44.348 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1 2023-11-08T05:22:44.349 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local pool=mirror 2023-11-08T05:22:44.349 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local image=image_35 2023-11-08T05:22:44.349 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ rbd --cluster=cluster1 info mirror/image_35 2023-11-08T05:22:44.350 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-11-08T05:22:44.368 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd: error opening image image_35: (2) No such file or directory 2023-11-08T05:22:44.371 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image_id= 2023-11-08T05:22:44.371 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-11-08T05:22:44.371 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0.1 2023-11-08T05:22:44.473 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test_image_present cluster1 mirror image_35 deleted '' 2023-11-08T05:22:44.474 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:22:44.474 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:22:44.474 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_35 2023-11-08T05:22:44.475 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local test_state=deleted 2023-11-08T05:22:44.475 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image_id= 2023-11-08T05:22:44.475 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local current_state=deleted 2023-11-08T05:22:44.476 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local current_image_id 2023-11-08T05:22:44.476 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ get_image_id cluster1 mirror image_35 2023-11-08T05:22:44.476 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1 2023-11-08T05:22:44.477 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local pool=mirror 2023-11-08T05:22:44.477 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local image=image_35 2023-11-08T05:22:44.478 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ rbd --cluster=cluster1 info mirror/image_35 2023-11-08T05:22:44.478 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-11-08T05:22:44.498 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd: error opening image image_35: (2) No such file or directory 2023-11-08T05:22:44.503 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ current_image_id= 2023-11-08T05:22:44.503 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test -n '' 2023-11-08T05:22:44.503 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test deleted = deleted 2023-11-08T05:22:44.504 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:22:44.504 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-11-08T05:22:44.504 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image=image_36 2023-11-08T05:22:44.505 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_image_present cluster1 mirror image_36 deleted 2023-11-08T05:22:44.505 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:22:44.506 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:22:44.506 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_36 2023-11-08T05:22:44.506 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local state=deleted 2023-11-08T05:22:44.507 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image_id= 2023-11-08T05:22:44.507 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local s 2023-11-08T05:22:44.508 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test -n '' 2023-11-08T05:22:44.508 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ get_image_id cluster1 mirror image_36 2023-11-08T05:22:44.508 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1 2023-11-08T05:22:44.509 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local pool=mirror 2023-11-08T05:22:44.509 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local image=image_36 2023-11-08T05:22:44.509 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ rbd --cluster=cluster1 info mirror/image_36 2023-11-08T05:22:44.510 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-11-08T05:22:44.529 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd: error opening image image_36: (2) No such file or directory 2023-11-08T05:22:44.532 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image_id= 2023-11-08T05:22:44.532 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-11-08T05:22:44.532 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0.1 2023-11-08T05:22:44.635 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test_image_present cluster1 mirror image_36 deleted '' 2023-11-08T05:22:44.635 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:22:44.636 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:22:44.636 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_36 2023-11-08T05:22:44.637 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local test_state=deleted 2023-11-08T05:22:44.637 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image_id= 2023-11-08T05:22:44.637 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local current_state=deleted 2023-11-08T05:22:44.638 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local current_image_id 2023-11-08T05:22:44.638 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ get_image_id cluster1 mirror image_36 2023-11-08T05:22:44.639 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1 2023-11-08T05:22:44.639 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local pool=mirror 2023-11-08T05:22:44.639 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local image=image_36 2023-11-08T05:22:44.640 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ rbd --cluster=cluster1 info mirror/image_36 2023-11-08T05:22:44.640 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-11-08T05:22:44.659 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd: error opening image image_36: (2) No such file or directory 2023-11-08T05:22:44.665 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ current_image_id= 2023-11-08T05:22:44.666 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test -n '' 2023-11-08T05:22:44.666 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test deleted = deleted 2023-11-08T05:22:44.667 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:22:44.667 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-11-08T05:22:44.667 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image=image_37 2023-11-08T05:22:44.668 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_image_present cluster1 mirror image_37 deleted 2023-11-08T05:22:44.668 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:22:44.668 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:22:44.669 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_37 2023-11-08T05:22:44.669 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local state=deleted 2023-11-08T05:22:44.669 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image_id= 2023-11-08T05:22:44.670 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local s 2023-11-08T05:22:44.670 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test -n '' 2023-11-08T05:22:44.670 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ get_image_id cluster1 mirror image_37 2023-11-08T05:22:44.671 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1 2023-11-08T05:22:44.671 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local pool=mirror 2023-11-08T05:22:44.671 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local image=image_37 2023-11-08T05:22:44.672 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ rbd --cluster=cluster1 info mirror/image_37 2023-11-08T05:22:44.672 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-11-08T05:22:44.690 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd: error opening image image_37: (2) No such file or directory 2023-11-08T05:22:44.692 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image_id= 2023-11-08T05:22:44.692 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-11-08T05:22:44.693 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0.1 2023-11-08T05:22:44.794 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test_image_present cluster1 mirror image_37 deleted '' 2023-11-08T05:22:44.794 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:22:44.795 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:22:44.795 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_37 2023-11-08T05:22:44.795 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local test_state=deleted 2023-11-08T05:22:44.795 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image_id= 2023-11-08T05:22:44.795 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local current_state=deleted 2023-11-08T05:22:44.796 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local current_image_id 2023-11-08T05:22:44.796 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ get_image_id cluster1 mirror image_37 2023-11-08T05:22:44.796 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1 2023-11-08T05:22:44.796 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local pool=mirror 2023-11-08T05:22:44.796 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local image=image_37 2023-11-08T05:22:44.797 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ rbd --cluster=cluster1 info mirror/image_37 2023-11-08T05:22:44.797 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-11-08T05:22:44.822 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd: error opening image image_37: (2) No such file or directory 2023-11-08T05:22:44.825 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ current_image_id= 2023-11-08T05:22:44.826 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test -n '' 2023-11-08T05:22:44.826 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test deleted = deleted 2023-11-08T05:22:44.826 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:22:44.826 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-11-08T05:22:44.826 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image=image_38 2023-11-08T05:22:44.827 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_image_present cluster1 mirror image_38 deleted 2023-11-08T05:22:44.827 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:22:44.827 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:22:44.827 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_38 2023-11-08T05:22:44.827 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local state=deleted 2023-11-08T05:22:44.828 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image_id= 2023-11-08T05:22:44.828 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local s 2023-11-08T05:22:44.828 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test -n '' 2023-11-08T05:22:44.828 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ get_image_id cluster1 mirror image_38 2023-11-08T05:22:44.828 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1 2023-11-08T05:22:44.829 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local pool=mirror 2023-11-08T05:22:44.829 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local image=image_38 2023-11-08T05:22:44.829 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ rbd --cluster=cluster1 info mirror/image_38 2023-11-08T05:22:44.829 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-11-08T05:22:44.851 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd: error opening image image_38: (2) No such file or directory 2023-11-08T05:22:44.853 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image_id= 2023-11-08T05:22:44.853 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-11-08T05:22:44.854 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0.1 2023-11-08T05:22:44.955 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test_image_present cluster1 mirror image_38 deleted '' 2023-11-08T05:22:44.955 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:22:44.956 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:22:44.956 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_38 2023-11-08T05:22:44.956 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local test_state=deleted 2023-11-08T05:22:44.956 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image_id= 2023-11-08T05:22:44.957 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local current_state=deleted 2023-11-08T05:22:44.957 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local current_image_id 2023-11-08T05:22:44.957 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ get_image_id cluster1 mirror image_38 2023-11-08T05:22:44.957 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1 2023-11-08T05:22:44.957 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local pool=mirror 2023-11-08T05:22:44.958 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local image=image_38 2023-11-08T05:22:44.958 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ rbd --cluster=cluster1 info mirror/image_38 2023-11-08T05:22:44.958 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-11-08T05:22:44.982 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd: error opening image image_38: (2) No such file or directory 2023-11-08T05:22:44.985 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ current_image_id= 2023-11-08T05:22:44.986 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test -n '' 2023-11-08T05:22:44.986 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test deleted = deleted 2023-11-08T05:22:44.986 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:22:44.986 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-11-08T05:22:44.987 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image=image_39 2023-11-08T05:22:44.987 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_image_present cluster1 mirror image_39 deleted 2023-11-08T05:22:44.987 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:22:44.987 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:22:44.987 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_39 2023-11-08T05:22:44.987 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local state=deleted 2023-11-08T05:22:44.988 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image_id= 2023-11-08T05:22:44.988 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local s 2023-11-08T05:22:44.988 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test -n '' 2023-11-08T05:22:44.988 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ get_image_id cluster1 mirror image_39 2023-11-08T05:22:44.988 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1 2023-11-08T05:22:44.989 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local pool=mirror 2023-11-08T05:22:44.989 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local image=image_39 2023-11-08T05:22:44.989 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ rbd --cluster=cluster1 info mirror/image_39 2023-11-08T05:22:44.989 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-11-08T05:22:45.012 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd: error opening image image_39: (2) No such file or directory 2023-11-08T05:22:45.014 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image_id= 2023-11-08T05:22:45.014 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-11-08T05:22:45.014 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0.1 2023-11-08T05:22:45.116 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test_image_present cluster1 mirror image_39 deleted '' 2023-11-08T05:22:45.116 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:22:45.116 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:22:45.117 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_39 2023-11-08T05:22:45.117 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local test_state=deleted 2023-11-08T05:22:45.117 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image_id= 2023-11-08T05:22:45.117 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local current_state=deleted 2023-11-08T05:22:45.117 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local current_image_id 2023-11-08T05:22:45.118 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ get_image_id cluster1 mirror image_39 2023-11-08T05:22:45.118 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1 2023-11-08T05:22:45.118 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local pool=mirror 2023-11-08T05:22:45.118 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local image=image_39 2023-11-08T05:22:45.119 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ rbd --cluster=cluster1 info mirror/image_39 2023-11-08T05:22:45.119 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-11-08T05:22:45.143 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd: error opening image image_39: (2) No such file or directory 2023-11-08T05:22:45.147 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ current_image_id= 2023-11-08T05:22:45.147 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test -n '' 2023-11-08T05:22:45.147 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test deleted = deleted 2023-11-08T05:22:45.147 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:22:45.148 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-11-08T05:22:45.148 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image=image_40 2023-11-08T05:22:45.148 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_image_present cluster1 mirror image_40 deleted 2023-11-08T05:22:45.148 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:22:45.148 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:22:45.149 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_40 2023-11-08T05:22:45.149 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local state=deleted 2023-11-08T05:22:45.150 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image_id= 2023-11-08T05:22:45.150 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local s 2023-11-08T05:22:45.150 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test -n '' 2023-11-08T05:22:45.150 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ get_image_id cluster1 mirror image_40 2023-11-08T05:22:45.150 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1 2023-11-08T05:22:45.151 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local pool=mirror 2023-11-08T05:22:45.151 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local image=image_40 2023-11-08T05:22:45.151 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ rbd --cluster=cluster1 info mirror/image_40 2023-11-08T05:22:45.151 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-11-08T05:22:45.173 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd: error opening image image_40: (2) No such file or directory 2023-11-08T05:22:45.175 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image_id= 2023-11-08T05:22:45.175 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-11-08T05:22:45.175 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0.1 2023-11-08T05:22:45.277 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test_image_present cluster1 mirror image_40 deleted '' 2023-11-08T05:22:45.277 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:22:45.277 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:22:45.278 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_40 2023-11-08T05:22:45.278 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local test_state=deleted 2023-11-08T05:22:45.278 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image_id= 2023-11-08T05:22:45.278 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local current_state=deleted 2023-11-08T05:22:45.279 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local current_image_id 2023-11-08T05:22:45.279 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ get_image_id cluster1 mirror image_40 2023-11-08T05:22:45.279 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1 2023-11-08T05:22:45.279 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local pool=mirror 2023-11-08T05:22:45.279 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local image=image_40 2023-11-08T05:22:45.279 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ rbd --cluster=cluster1 info mirror/image_40 2023-11-08T05:22:45.280 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-11-08T05:22:45.304 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd: error opening image image_40: (2) No such file or directory 2023-11-08T05:22:45.308 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ current_image_id= 2023-11-08T05:22:45.308 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test -n '' 2023-11-08T05:22:45.308 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test deleted = deleted 2023-11-08T05:22:45.308 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:22:45.309 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-11-08T05:22:45.309 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image=image_41 2023-11-08T05:22:45.309 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_image_present cluster1 mirror image_41 deleted 2023-11-08T05:22:45.309 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:22:45.310 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:22:45.310 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_41 2023-11-08T05:22:45.310 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local state=deleted 2023-11-08T05:22:45.310 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image_id= 2023-11-08T05:22:45.311 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local s 2023-11-08T05:22:45.311 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test -n '' 2023-11-08T05:22:45.311 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ get_image_id cluster1 mirror image_41 2023-11-08T05:22:45.311 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1 2023-11-08T05:22:45.311 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local pool=mirror 2023-11-08T05:22:45.312 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local image=image_41 2023-11-08T05:22:45.312 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ rbd --cluster=cluster1 info mirror/image_41 2023-11-08T05:22:45.312 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-11-08T05:22:45.333 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd: error opening image image_41: (2) No such file or directory 2023-11-08T05:22:45.336 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image_id= 2023-11-08T05:22:45.336 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-11-08T05:22:45.336 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0.1 2023-11-08T05:22:45.438 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test_image_present cluster1 mirror image_41 deleted '' 2023-11-08T05:22:45.438 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:22:45.439 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:22:45.439 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_41 2023-11-08T05:22:45.439 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local test_state=deleted 2023-11-08T05:22:45.439 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image_id= 2023-11-08T05:22:45.439 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local current_state=deleted 2023-11-08T05:22:45.440 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local current_image_id 2023-11-08T05:22:45.440 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ get_image_id cluster1 mirror image_41 2023-11-08T05:22:45.440 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1 2023-11-08T05:22:45.440 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local pool=mirror 2023-11-08T05:22:45.440 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local image=image_41 2023-11-08T05:22:45.441 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ rbd --cluster=cluster1 info mirror/image_41 2023-11-08T05:22:45.441 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-11-08T05:22:45.465 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd: error opening image image_41: (2) No such file or directory 2023-11-08T05:22:45.468 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ current_image_id= 2023-11-08T05:22:45.468 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test -n '' 2023-11-08T05:22:45.468 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test deleted = deleted 2023-11-08T05:22:45.469 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:22:45.469 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-11-08T05:22:45.469 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image=image_42 2023-11-08T05:22:45.469 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_image_present cluster1 mirror image_42 deleted 2023-11-08T05:22:45.469 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:22:45.470 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:22:45.470 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_42 2023-11-08T05:22:45.470 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local state=deleted 2023-11-08T05:22:45.470 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image_id= 2023-11-08T05:22:45.470 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local s 2023-11-08T05:22:45.471 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test -n '' 2023-11-08T05:22:45.471 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ get_image_id cluster1 mirror image_42 2023-11-08T05:22:45.471 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1 2023-11-08T05:22:45.471 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local pool=mirror 2023-11-08T05:22:45.471 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local image=image_42 2023-11-08T05:22:45.472 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ rbd --cluster=cluster1 info mirror/image_42 2023-11-08T05:22:45.472 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-11-08T05:22:45.493 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd: error opening image image_42: (2) No such file or directory 2023-11-08T05:22:45.496 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image_id= 2023-11-08T05:22:45.496 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-11-08T05:22:45.496 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0.1 2023-11-08T05:22:45.598 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test_image_present cluster1 mirror image_42 deleted '' 2023-11-08T05:22:45.598 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:22:45.598 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:22:45.599 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_42 2023-11-08T05:22:45.599 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local test_state=deleted 2023-11-08T05:22:45.599 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image_id= 2023-11-08T05:22:45.599 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local current_state=deleted 2023-11-08T05:22:45.599 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local current_image_id 2023-11-08T05:22:45.599 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ get_image_id cluster1 mirror image_42 2023-11-08T05:22:45.600 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1 2023-11-08T05:22:45.600 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local pool=mirror 2023-11-08T05:22:45.600 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local image=image_42 2023-11-08T05:22:45.600 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ rbd --cluster=cluster1 info mirror/image_42 2023-11-08T05:22:45.600 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-11-08T05:22:45.624 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd: error opening image image_42: (2) No such file or directory 2023-11-08T05:22:45.628 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ current_image_id= 2023-11-08T05:22:45.628 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test -n '' 2023-11-08T05:22:45.629 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test deleted = deleted 2023-11-08T05:22:45.629 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:22:45.629 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-11-08T05:22:45.629 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image=image_43 2023-11-08T05:22:45.629 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_image_present cluster1 mirror image_43 deleted 2023-11-08T05:22:45.630 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:22:45.630 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:22:45.630 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_43 2023-11-08T05:22:45.630 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local state=deleted 2023-11-08T05:22:45.631 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image_id= 2023-11-08T05:22:45.631 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local s 2023-11-08T05:22:45.631 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test -n '' 2023-11-08T05:22:45.631 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ get_image_id cluster1 mirror image_43 2023-11-08T05:22:45.632 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1 2023-11-08T05:22:45.632 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local pool=mirror 2023-11-08T05:22:45.632 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local image=image_43 2023-11-08T05:22:45.632 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ rbd --cluster=cluster1 info mirror/image_43 2023-11-08T05:22:45.632 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-11-08T05:22:45.654 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd: error opening image image_43: (2) No such file or directory 2023-11-08T05:22:45.656 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image_id= 2023-11-08T05:22:45.657 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-11-08T05:22:45.657 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0.1 2023-11-08T05:22:45.758 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test_image_present cluster1 mirror image_43 deleted '' 2023-11-08T05:22:45.759 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:22:45.759 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:22:45.759 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_43 2023-11-08T05:22:45.759 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local test_state=deleted 2023-11-08T05:22:45.759 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image_id= 2023-11-08T05:22:45.760 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local current_state=deleted 2023-11-08T05:22:45.760 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local current_image_id 2023-11-08T05:22:45.760 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ get_image_id cluster1 mirror image_43 2023-11-08T05:22:45.760 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1 2023-11-08T05:22:45.760 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local pool=mirror 2023-11-08T05:22:45.761 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local image=image_43 2023-11-08T05:22:45.761 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ rbd --cluster=cluster1 info mirror/image_43 2023-11-08T05:22:45.761 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-11-08T05:22:45.784 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd: error opening image image_43: (2) No such file or directory 2023-11-08T05:22:45.788 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ current_image_id= 2023-11-08T05:22:45.788 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test -n '' 2023-11-08T05:22:45.788 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test deleted = deleted 2023-11-08T05:22:45.788 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:22:45.788 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-11-08T05:22:45.789 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image=image_44 2023-11-08T05:22:45.789 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_image_present cluster1 mirror image_44 deleted 2023-11-08T05:22:45.789 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:22:45.789 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:22:45.789 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_44 2023-11-08T05:22:45.790 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local state=deleted 2023-11-08T05:22:45.790 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image_id= 2023-11-08T05:22:45.790 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local s 2023-11-08T05:22:45.790 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test -n '' 2023-11-08T05:22:45.790 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ get_image_id cluster1 mirror image_44 2023-11-08T05:22:45.791 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1 2023-11-08T05:22:45.791 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local pool=mirror 2023-11-08T05:22:45.791 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local image=image_44 2023-11-08T05:22:45.791 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ rbd --cluster=cluster1 info mirror/image_44 2023-11-08T05:22:45.791 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-11-08T05:22:45.812 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd: error opening image image_44: (2) No such file or directory 2023-11-08T05:22:45.815 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image_id= 2023-11-08T05:22:45.815 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-11-08T05:22:45.815 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0.1 2023-11-08T05:22:45.918 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test_image_present cluster1 mirror image_44 deleted '' 2023-11-08T05:22:45.919 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:22:45.919 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:22:45.919 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_44 2023-11-08T05:22:45.919 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local test_state=deleted 2023-11-08T05:22:45.920 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image_id= 2023-11-08T05:22:45.920 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local current_state=deleted 2023-11-08T05:22:45.920 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local current_image_id 2023-11-08T05:22:45.920 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ get_image_id cluster1 mirror image_44 2023-11-08T05:22:45.920 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1 2023-11-08T05:22:45.920 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local pool=mirror 2023-11-08T05:22:45.921 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local image=image_44 2023-11-08T05:22:45.921 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ rbd --cluster=cluster1 info mirror/image_44 2023-11-08T05:22:45.921 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-11-08T05:22:45.943 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd: error opening image image_44: (2) No such file or directory 2023-11-08T05:22:45.950 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ current_image_id= 2023-11-08T05:22:45.950 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test -n '' 2023-11-08T05:22:45.951 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test deleted = deleted 2023-11-08T05:22:45.951 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:22:45.951 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-11-08T05:22:45.952 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image=image_45 2023-11-08T05:22:45.952 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_image_present cluster1 mirror image_45 deleted 2023-11-08T05:22:45.952 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:22:45.953 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:22:45.953 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_45 2023-11-08T05:22:45.954 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local state=deleted 2023-11-08T05:22:45.954 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image_id= 2023-11-08T05:22:45.954 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local s 2023-11-08T05:22:45.955 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test -n '' 2023-11-08T05:22:45.955 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ get_image_id cluster1 mirror image_45 2023-11-08T05:22:45.956 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1 2023-11-08T05:22:45.956 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local pool=mirror 2023-11-08T05:22:45.956 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local image=image_45 2023-11-08T05:22:45.957 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ rbd --cluster=cluster1 info mirror/image_45 2023-11-08T05:22:45.957 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-11-08T05:22:45.972 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd: error opening image image_45: (2) No such file or directory 2023-11-08T05:22:45.975 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image_id= 2023-11-08T05:22:45.975 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-11-08T05:22:45.975 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0.1 2023-11-08T05:22:46.076 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test_image_present cluster1 mirror image_45 deleted '' 2023-11-08T05:22:46.076 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:22:46.077 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:22:46.077 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_45 2023-11-08T05:22:46.077 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local test_state=deleted 2023-11-08T05:22:46.077 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image_id= 2023-11-08T05:22:46.077 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local current_state=deleted 2023-11-08T05:22:46.078 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local current_image_id 2023-11-08T05:22:46.078 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ get_image_id cluster1 mirror image_45 2023-11-08T05:22:46.078 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1 2023-11-08T05:22:46.078 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local pool=mirror 2023-11-08T05:22:46.079 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local image=image_45 2023-11-08T05:22:46.079 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ rbd --cluster=cluster1 info mirror/image_45 2023-11-08T05:22:46.079 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-11-08T05:22:46.102 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd: error opening image image_45: (2) No such file or directory 2023-11-08T05:22:46.106 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ current_image_id= 2023-11-08T05:22:46.106 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test -n '' 2023-11-08T05:22:46.106 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test deleted = deleted 2023-11-08T05:22:46.107 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:22:46.107 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-11-08T05:22:46.107 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image=image_46 2023-11-08T05:22:46.107 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_image_present cluster1 mirror image_46 deleted 2023-11-08T05:22:46.107 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:22:46.108 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:22:46.108 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_46 2023-11-08T05:22:46.108 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local state=deleted 2023-11-08T05:22:46.108 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image_id= 2023-11-08T05:22:46.108 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local s 2023-11-08T05:22:46.108 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test -n '' 2023-11-08T05:22:46.109 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ get_image_id cluster1 mirror image_46 2023-11-08T05:22:46.109 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1 2023-11-08T05:22:46.109 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local pool=mirror 2023-11-08T05:22:46.109 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local image=image_46 2023-11-08T05:22:46.109 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ rbd --cluster=cluster1 info mirror/image_46 2023-11-08T05:22:46.110 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-11-08T05:22:46.132 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd: error opening image image_46: (2) No such file or directory 2023-11-08T05:22:46.134 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image_id= 2023-11-08T05:22:46.134 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-11-08T05:22:46.135 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0.1 2023-11-08T05:22:46.236 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test_image_present cluster1 mirror image_46 deleted '' 2023-11-08T05:22:46.236 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:22:46.237 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:22:46.237 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_46 2023-11-08T05:22:46.237 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local test_state=deleted 2023-11-08T05:22:46.238 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image_id= 2023-11-08T05:22:46.238 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local current_state=deleted 2023-11-08T05:22:46.239 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local current_image_id 2023-11-08T05:22:46.239 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ get_image_id cluster1 mirror image_46 2023-11-08T05:22:46.239 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1 2023-11-08T05:22:46.240 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local pool=mirror 2023-11-08T05:22:46.240 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local image=image_46 2023-11-08T05:22:46.240 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ rbd --cluster=cluster1 info mirror/image_46 2023-11-08T05:22:46.241 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-11-08T05:22:46.264 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd: error opening image image_46: (2) No such file or directory 2023-11-08T05:22:46.269 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ current_image_id= 2023-11-08T05:22:46.269 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test -n '' 2023-11-08T05:22:46.270 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test deleted = deleted 2023-11-08T05:22:46.270 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:22:46.271 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-11-08T05:22:46.271 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image=image_47 2023-11-08T05:22:46.272 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_image_present cluster1 mirror image_47 deleted 2023-11-08T05:22:46.272 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:22:46.272 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:22:46.273 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_47 2023-11-08T05:22:46.273 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local state=deleted 2023-11-08T05:22:46.273 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image_id= 2023-11-08T05:22:46.274 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local s 2023-11-08T05:22:46.274 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test -n '' 2023-11-08T05:22:46.275 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ get_image_id cluster1 mirror image_47 2023-11-08T05:22:46.275 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1 2023-11-08T05:22:46.275 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local pool=mirror 2023-11-08T05:22:46.276 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local image=image_47 2023-11-08T05:22:46.276 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ rbd --cluster=cluster1 info mirror/image_47 2023-11-08T05:22:46.276 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-11-08T05:22:46.292 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd: error opening image image_47: (2) No such file or directory 2023-11-08T05:22:46.295 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image_id= 2023-11-08T05:22:46.295 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-11-08T05:22:46.295 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0.1 2023-11-08T05:22:46.397 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test_image_present cluster1 mirror image_47 deleted '' 2023-11-08T05:22:46.397 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:22:46.398 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:22:46.398 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_47 2023-11-08T05:22:46.398 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local test_state=deleted 2023-11-08T05:22:46.399 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image_id= 2023-11-08T05:22:46.399 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local current_state=deleted 2023-11-08T05:22:46.399 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local current_image_id 2023-11-08T05:22:46.400 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ get_image_id cluster1 mirror image_47 2023-11-08T05:22:46.400 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1 2023-11-08T05:22:46.400 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local pool=mirror 2023-11-08T05:22:46.401 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local image=image_47 2023-11-08T05:22:46.401 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ rbd --cluster=cluster1 info mirror/image_47 2023-11-08T05:22:46.402 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-11-08T05:22:46.422 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd: error opening image image_47: (2) No such file or directory 2023-11-08T05:22:46.427 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ current_image_id= 2023-11-08T05:22:46.428 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test -n '' 2023-11-08T05:22:46.428 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test deleted = deleted 2023-11-08T05:22:46.429 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:22:46.429 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-11-08T05:22:46.429 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image=image_48 2023-11-08T05:22:46.430 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_image_present cluster1 mirror image_48 deleted 2023-11-08T05:22:46.430 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:22:46.431 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:22:46.431 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_48 2023-11-08T05:22:46.431 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local state=deleted 2023-11-08T05:22:46.432 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image_id= 2023-11-08T05:22:46.432 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local s 2023-11-08T05:22:46.432 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test -n '' 2023-11-08T05:22:46.432 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ get_image_id cluster1 mirror image_48 2023-11-08T05:22:46.433 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1 2023-11-08T05:22:46.433 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local pool=mirror 2023-11-08T05:22:46.433 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local image=image_48 2023-11-08T05:22:46.433 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ rbd --cluster=cluster1 info mirror/image_48 2023-11-08T05:22:46.434 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-11-08T05:22:46.453 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd: error opening image image_48: (2) No such file or directory 2023-11-08T05:22:46.455 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image_id= 2023-11-08T05:22:46.455 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-11-08T05:22:46.456 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0.1 2023-11-08T05:22:46.557 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test_image_present cluster1 mirror image_48 deleted '' 2023-11-08T05:22:46.557 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:22:46.558 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:22:46.558 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_48 2023-11-08T05:22:46.558 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local test_state=deleted 2023-11-08T05:22:46.558 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image_id= 2023-11-08T05:22:46.558 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local current_state=deleted 2023-11-08T05:22:46.559 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local current_image_id 2023-11-08T05:22:46.559 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ get_image_id cluster1 mirror image_48 2023-11-08T05:22:46.559 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1 2023-11-08T05:22:46.559 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local pool=mirror 2023-11-08T05:22:46.560 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local image=image_48 2023-11-08T05:22:46.560 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ rbd --cluster=cluster1 info mirror/image_48 2023-11-08T05:22:46.560 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-11-08T05:22:46.584 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd: error opening image image_48: (2) No such file or directory 2023-11-08T05:22:46.588 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ current_image_id= 2023-11-08T05:22:46.588 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test -n '' 2023-11-08T05:22:46.588 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test deleted = deleted 2023-11-08T05:22:46.588 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:22:46.588 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-11-08T05:22:46.589 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image=image_49 2023-11-08T05:22:46.589 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_image_present cluster1 mirror image_49 deleted 2023-11-08T05:22:46.589 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:22:46.589 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:22:46.589 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_49 2023-11-08T05:22:46.590 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local state=deleted 2023-11-08T05:22:46.590 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image_id= 2023-11-08T05:22:46.590 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local s 2023-11-08T05:22:46.590 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test -n '' 2023-11-08T05:22:46.590 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ get_image_id cluster1 mirror image_49 2023-11-08T05:22:46.591 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1 2023-11-08T05:22:46.591 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local pool=mirror 2023-11-08T05:22:46.591 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local image=image_49 2023-11-08T05:22:46.591 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ rbd --cluster=cluster1 info mirror/image_49 2023-11-08T05:22:46.591 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-11-08T05:22:46.613 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd: error opening image image_49: (2) No such file or directory 2023-11-08T05:22:46.616 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image_id= 2023-11-08T05:22:46.616 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-11-08T05:22:46.616 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0.1 2023-11-08T05:22:46.718 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test_image_present cluster1 mirror image_49 deleted '' 2023-11-08T05:22:46.718 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:22:46.718 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:22:46.719 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_49 2023-11-08T05:22:46.719 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local test_state=deleted 2023-11-08T05:22:46.719 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image_id= 2023-11-08T05:22:46.719 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local current_state=deleted 2023-11-08T05:22:46.720 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local current_image_id 2023-11-08T05:22:46.720 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ get_image_id cluster1 mirror image_49 2023-11-08T05:22:46.720 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1 2023-11-08T05:22:46.720 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local pool=mirror 2023-11-08T05:22:46.720 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local image=image_49 2023-11-08T05:22:46.721 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ rbd --cluster=cluster1 info mirror/image_49 2023-11-08T05:22:46.721 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-11-08T05:22:46.743 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd: error opening image image_49: (2) No such file or directory 2023-11-08T05:22:46.747 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ current_image_id= 2023-11-08T05:22:46.747 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test -n '' 2023-11-08T05:22:46.747 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test deleted = deleted 2023-11-08T05:22:46.748 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:22:46.748 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-11-08T05:22:46.748 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image=image_50 2023-11-08T05:22:46.748 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_image_present cluster1 mirror image_50 deleted 2023-11-08T05:22:46.748 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:22:46.749 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:22:46.749 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_50 2023-11-08T05:22:46.749 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local state=deleted 2023-11-08T05:22:46.749 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image_id= 2023-11-08T05:22:46.749 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local s 2023-11-08T05:22:46.750 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test -n '' 2023-11-08T05:22:46.750 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ get_image_id cluster1 mirror image_50 2023-11-08T05:22:46.750 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1 2023-11-08T05:22:46.750 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local pool=mirror 2023-11-08T05:22:46.750 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local image=image_50 2023-11-08T05:22:46.751 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ rbd --cluster=cluster1 info mirror/image_50 2023-11-08T05:22:46.751 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-11-08T05:22:46.772 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd: error opening image image_50: (2) No such file or directory 2023-11-08T05:22:46.775 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image_id= 2023-11-08T05:22:46.775 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-11-08T05:22:46.776 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0.1 2023-11-08T05:22:46.877 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test_image_present cluster1 mirror image_50 deleted '' 2023-11-08T05:22:46.877 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:22:46.878 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:22:46.878 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_50 2023-11-08T05:22:46.878 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local test_state=deleted 2023-11-08T05:22:46.878 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image_id= 2023-11-08T05:22:46.878 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local current_state=deleted 2023-11-08T05:22:46.879 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local current_image_id 2023-11-08T05:22:46.879 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ get_image_id cluster1 mirror image_50 2023-11-08T05:22:46.879 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1 2023-11-08T05:22:46.879 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local pool=mirror 2023-11-08T05:22:46.879 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local image=image_50 2023-11-08T05:22:46.880 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ rbd --cluster=cluster1 info mirror/image_50 2023-11-08T05:22:46.880 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-11-08T05:22:46.903 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd: error opening image image_50: (2) No such file or directory 2023-11-08T05:22:46.906 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ current_image_id= 2023-11-08T05:22:46.906 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test -n '' 2023-11-08T05:22:46.906 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test deleted = deleted 2023-11-08T05:22:46.906 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:22:46.907 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ testlog 'TEST: delete images during bootstrap' 2023-11-08T05:22:46.907 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ tee -a /tmp/tmp.Ludk8ikCAB/rbd-mirror.test.log 2023-11-08T05:22:46.907 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ date '+%F %T' 2023-11-08T05:22:46.908 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ echo 2023-11-08 05:22:46 TEST: delete images during bootstrap 2023-11-08T05:22:46.908 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:2023-11-08 05:22:46 TEST: delete images during bootstrap 2023-11-08T05:22:46.909 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ set_pool_mirror_mode cluster1 mirror image 2023-11-08T05:22:46.909 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:22:46.909 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:22:46.909 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local mode=image 2023-11-08T05:22:46.909 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster=cluster1 mirror pool enable mirror image 2023-11-08T05:22:46.933 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:rbd: mirroring is already configured for image mode 2023-11-08T05:22:46.935 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ set_pool_mirror_mode cluster2 mirror image 2023-11-08T05:22:46.936 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:22:46.936 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:22:46.936 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local mode=image 2023-11-08T05:22:46.936 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster=cluster2 mirror pool enable mirror image 2023-11-08T05:22:46.961 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:rbd: mirroring is already configured for image mode 2023-11-08T05:22:46.966 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ start_mirror cluster1 2023-11-08T05:22:46.966 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:22:46.966 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local instance 2023-11-08T05:22:46.966 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ set_cluster_instance cluster1 cluster instance 2023-11-08T05:22:46.967 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local val=cluster1 2023-11-08T05:22:46.967 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster_var_name=cluster 2023-11-08T05:22:46.967 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local instance_var_name=instance 2023-11-08T05:22:46.967 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ cluster=cluster1 2023-11-08T05:22:46.967 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ instance=cluster1 2023-11-08T05:22:46.968 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' cluster1 = cluster1 ']' 2023-11-08T05:22:46.968 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ instance=0 2023-11-08T05:22:46.968 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ eval cluster=cluster1 2023-11-08T05:22:46.968 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ cluster=cluster1 2023-11-08T05:22:46.968 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ eval instance=0 2023-11-08T05:22:46.969 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ instance=0 2023-11-08T05:22:46.969 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test -n 1 2023-11-08T05:22:46.969 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 2023-11-08T05:22:46.969 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image=test 2023-11-08T05:22:46.969 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ seq 1 10 2023-11-08T05:22:46.970 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for i in `seq 1 10` 2023-11-08T05:22:46.970 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image=image_1 2023-11-08T05:22:46.970 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ create_image cluster2 mirror image_1 512M 2023-11-08T05:22:46.970 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:22:46.970 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:22:46.971 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:22:46.971 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:22:46.971 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_1 2023-11-08T05:22:46.971 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:22:46.971 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local size=128 2023-11-08T05:22:46.972 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' -n 512M ']' 2023-11-08T05:22:46.972 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ size=512M 2023-11-08T05:22:46.972 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:22:46.972 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock mirror/image_1 2023-11-08T05:22:47.005 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ enable_mirror cluster2 mirror image_1 2023-11-08T05:22:47.005 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:22:47.005 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:22:47.006 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_1 2023-11-08T05:22:47.006 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local mode=snapshot 2023-11-08T05:22:47.006 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_1 snapshot 2023-11-08T05:22:47.383 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:Mirroring enabled 2023-11-08T05:22:47.391 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster=cluster2 info mirror/image_1 2023-11-08T05:22:47.421 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:rbd image 'image_1': 2023-11-08T05:22:47.422 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: size 512 MiB in 128 objects 2023-11-08T05:22:47.422 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: order 22 (4 MiB objects) 2023-11-08T05:22:47.422 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: snapshot_count: 1 2023-11-08T05:22:47.423 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: id: 19053f9f65ed 2023-11-08T05:22:47.423 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: block_name_prefix: rbd_data.19053f9f65ed 2023-11-08T05:22:47.423 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: format: 2 2023-11-08T05:22:47.423 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: features: layering, exclusive-lock 2023-11-08T05:22:47.423 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: op_features: 2023-11-08T05:22:47.426 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: flags: 2023-11-08T05:22:47.426 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: create_timestamp: Wed Nov 8 05:22:46 2023 2023-11-08T05:22:47.427 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: access_timestamp: Wed Nov 8 05:22:46 2023 2023-11-08T05:22:47.427 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: modify_timestamp: Wed Nov 8 05:22:46 2023 2023-11-08T05:22:47.427 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring state: enabled 2023-11-08T05:22:47.427 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring mode: snapshot 2023-11-08T05:22:47.427 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring global id: 0cf561a1-2432-4634-9511-3e976fb93118 2023-11-08T05:22:47.428 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring primary: true 2023-11-08T05:22:47.429 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ stress_write_image cluster2 mirror image_1 2023-11-08T05:22:47.429 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:22:47.429 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:22:47.430 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_1 2023-11-08T05:22:47.430 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-11-08T05:22:47.430 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local duration=7 2023-11-08T05:22:47.430 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ set +e 2023-11-08T05:22:47.430 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_1 --debug-rbd=20 --debug-journaler=20 2023-11-08T05:22:47.628 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:100 IOs, 31537152 bytes 2023-11-08T05:22:47.768 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:200 IOs, 46437376 bytes 2023-11-08T05:22:48.001 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:300 IOs, 111896576 bytes 2023-11-08T05:22:48.147 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:400 IOs, 139978752 bytes 2023-11-08T05:22:48.293 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:500 IOs, 163518464 bytes 2023-11-08T05:22:48.444 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:600 IOs, 188431360 bytes 2023-11-08T05:22:48.628 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:700 IOs, 231370752 bytes 2023-11-08T05:22:48.746 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:800 IOs, 238015488 bytes 2023-11-08T05:22:48.881 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:900 IOs, 263767040 bytes 2023-11-08T05:22:49.041 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1000 IOs, 286491648 bytes 2023-11-08T05:22:49.188 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1100 IOs, 305729536 bytes 2023-11-08T05:22:49.341 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1200 IOs, 332085248 bytes 2023-11-08T05:22:49.605 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1300 IOs, 409600000 bytes 2023-11-08T05:22:49.817 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1400 IOs, 471568384 bytes 2023-11-08T05:22:50.161 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1500 IOs, 528080896 bytes 2023-11-08T05:22:50.381 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1600 IOs, 567263232 bytes 2023-11-08T05:22:50.614 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1700 IOs, 634126336 bytes 2023-11-08T05:22:50.845 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1800 IOs, 697655296 bytes 2023-11-08T05:22:51.055 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1900 IOs, 748821504 bytes 2023-11-08T05:22:51.306 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2000 IOs, 848096256 bytes 2023-11-08T05:22:51.602 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2100 IOs, 904536064 bytes 2023-11-08T05:22:51.737 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2200 IOs, 932747264 bytes 2023-11-08T05:22:51.997 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2300 IOs, 986041344 bytes 2023-11-08T05:22:52.131 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2400 IOs, 1000912896 bytes 2023-11-08T05:22:52.346 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2500 IOs, 1056269312 bytes 2023-11-08T05:22:52.581 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2600 IOs, 1104902144 bytes 2023-11-08T05:22:52.690 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2700 IOs, 1105521664 bytes 2023-11-08T05:22:52.844 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2800 IOs, 1134665728 bytes 2023-11-08T05:22:52.968 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2900 IOs, 1159243776 bytes 2023-11-08T05:22:53.271 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3000 IOs, 1245077504 bytes 2023-11-08T05:22:53.454 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3100 IOs, 1279320064 bytes 2023-11-08T05:22:53.654 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3200 IOs, 1334682624 bytes 2023-11-08T05:22:53.826 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3300 IOs, 1367030784 bytes 2023-11-08T05:22:54.093 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3400 IOs, 1449745408 bytes 2023-11-08T05:22:54.339 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3500 IOs, 1521603584 bytes 2023-11-08T05:22:54.432 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ error_code=124 2023-11-08T05:22:54.432 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ set -e 2023-11-08T05:22:54.432 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' 124 -eq 124 ']' 2023-11-08T05:22:54.433 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:22:54.433 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_image_present cluster1 mirror image_1 present 2023-11-08T05:22:54.433 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:22:54.434 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:22:54.434 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_1 2023-11-08T05:22:54.434 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local state=present 2023-11-08T05:22:54.434 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image_id= 2023-11-08T05:22:54.435 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local s 2023-11-08T05:22:54.435 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test -n '' 2023-11-08T05:22:54.435 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ get_image_id cluster1 mirror image_1 2023-11-08T05:22:54.435 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1 2023-11-08T05:22:54.435 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local pool=mirror 2023-11-08T05:22:54.436 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local image=image_1 2023-11-08T05:22:54.436 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ rbd --cluster=cluster1 info mirror/image_1 2023-11-08T05:22:54.436 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-11-08T05:22:54.468 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image_id=12a97e77ad41 2023-11-08T05:22:54.469 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-11-08T05:22:54.469 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0.1 2023-11-08T05:22:54.570 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test_image_present cluster1 mirror image_1 present 12a97e77ad41 2023-11-08T05:22:54.570 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:22:54.570 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:22:54.570 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_1 2023-11-08T05:22:54.571 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local test_state=present 2023-11-08T05:22:54.571 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image_id=12a97e77ad41 2023-11-08T05:22:54.571 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local current_state=deleted 2023-11-08T05:22:54.571 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local current_image_id 2023-11-08T05:22:54.571 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ get_image_id cluster1 mirror image_1 2023-11-08T05:22:54.572 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1 2023-11-08T05:22:54.572 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local pool=mirror 2023-11-08T05:22:54.572 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local image=image_1 2023-11-08T05:22:54.572 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ rbd --cluster=cluster1 info mirror/image_1 2023-11-08T05:22:54.572 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-11-08T05:22:54.607 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ current_image_id=12a97e77ad41 2023-11-08T05:22:54.607 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test -n 12a97e77ad41 2023-11-08T05:22:54.607 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test -z 12a97e77ad41 -o 12a97e77ad41 = 12a97e77ad41 2023-11-08T05:22:54.607 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ current_state=present 2023-11-08T05:22:54.608 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test present = present 2023-11-08T05:22:54.608 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:22:54.608 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ disable_mirror cluster2 mirror image_1 2023-11-08T05:22:54.608 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:22:54.608 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:22:54.609 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_1 2023-11-08T05:22:54.609 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_1 2023-11-08T05:22:55.421 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi087.stderr:2023-11-08T05:22:55.419+0000 7fb6a8772700 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55dd31a02400 scan_remote_mirror_snapshots: failed to locate remote start snapshot: snap_id=368 2023-11-08T05:22:55.422 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi087.stderr:2023-11-08T05:22:55.419+0000 7fb6a8772700 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55dd31a02400 scan_remote_mirror_snapshots: split-brain detected: failed to find matching non-primary snapshot in remote image: local_snap_id_start=368, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=06977a9b-46f8-44a8-aea5-8992e82c582a, primary_snap_id=170, last_copied_object_number=128, snap_seqs={368=18446744073709551614}] 2023-11-08T05:23:00.424 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:Mirroring disabled 2023-11-08T05:23:00.432 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_image_present cluster1 mirror image_1 deleted 2023-11-08T05:23:00.432 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:23:00.433 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:23:00.433 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_1 2023-11-08T05:23:00.433 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local state=deleted 2023-11-08T05:23:00.434 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image_id= 2023-11-08T05:23:00.434 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local s 2023-11-08T05:23:00.434 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test -n '' 2023-11-08T05:23:00.434 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ get_image_id cluster1 mirror image_1 2023-11-08T05:23:00.435 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1 2023-11-08T05:23:00.435 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local pool=mirror 2023-11-08T05:23:00.435 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local image=image_1 2023-11-08T05:23:00.435 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ rbd --cluster=cluster1 info mirror/image_1 2023-11-08T05:23:00.435 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-11-08T05:23:00.463 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd: error opening image image_1: (2) No such file or directory 2023-11-08T05:23:00.466 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image_id= 2023-11-08T05:23:00.466 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-11-08T05:23:00.467 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0.1 2023-11-08T05:23:00.568 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test_image_present cluster1 mirror image_1 deleted '' 2023-11-08T05:23:00.568 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:23:00.569 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:23:00.569 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_1 2023-11-08T05:23:00.569 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local test_state=deleted 2023-11-08T05:23:00.569 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image_id= 2023-11-08T05:23:00.569 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local current_state=deleted 2023-11-08T05:23:00.570 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local current_image_id 2023-11-08T05:23:00.570 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ get_image_id cluster1 mirror image_1 2023-11-08T05:23:00.570 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1 2023-11-08T05:23:00.570 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local pool=mirror 2023-11-08T05:23:00.570 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local image=image_1 2023-11-08T05:23:00.571 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ rbd --cluster=cluster1 info mirror/image_1 2023-11-08T05:23:00.571 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-11-08T05:23:00.598 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd: error opening image image_1: (2) No such file or directory 2023-11-08T05:23:00.601 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ current_image_id= 2023-11-08T05:23:00.601 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test -n '' 2023-11-08T05:23:00.601 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test deleted = deleted 2023-11-08T05:23:00.601 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:23:00.601 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ purge_snapshots cluster2 mirror image_1 2023-11-08T05:23:00.602 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:23:00.602 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:23:00.602 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_1 2023-11-08T05:23:00.602 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 snap purge mirror/image_1 2023-11-08T05:23:00.640 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ remove_image_retry cluster2 mirror image_1 2023-11-08T05:23:00.640 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:23:00.641 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:23:00.641 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_1 2023-11-08T05:23:00.642 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0 1 2 4 8 16 32 2023-11-08T05:23:00.642 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0 2023-11-08T05:23:00.642 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ remove_image cluster2 mirror image_1 2023-11-08T05:23:00.643 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:23:00.643 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:23:00.643 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_1 2023-11-08T05:23:00.644 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_1 2023-11-08T05:23:00.677 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster=cluster2 rm mirror/image_1 2023-11-08T05:23:24.912 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2023-11-08T05:23:24.916 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:23:24.916 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for i in `seq 1 10` 2023-11-08T05:23:24.916 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image=image_2 2023-11-08T05:23:24.917 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ create_image cluster2 mirror image_2 512M 2023-11-08T05:23:24.917 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:23:24.917 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:23:24.917 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:23:24.918 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:23:24.918 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_2 2023-11-08T05:23:24.918 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:23:24.918 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local size=128 2023-11-08T05:23:24.919 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' -n 512M ']' 2023-11-08T05:23:24.919 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ size=512M 2023-11-08T05:23:24.919 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:23:24.919 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock mirror/image_2 2023-11-08T05:23:24.956 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ enable_mirror cluster2 mirror image_2 2023-11-08T05:23:24.956 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:23:24.956 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:23:24.957 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_2 2023-11-08T05:23:24.957 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local mode=snapshot 2023-11-08T05:23:24.957 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_2 snapshot 2023-11-08T05:23:25.467 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:Mirroring enabled 2023-11-08T05:23:25.474 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster=cluster2 info mirror/image_2 2023-11-08T05:23:25.504 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:rbd image 'image_2': 2023-11-08T05:23:25.505 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: size 512 MiB in 128 objects 2023-11-08T05:23:25.505 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: order 22 (4 MiB objects) 2023-11-08T05:23:25.505 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: snapshot_count: 1 2023-11-08T05:23:25.505 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: id: 191dd50cbe32 2023-11-08T05:23:25.506 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: block_name_prefix: rbd_data.191dd50cbe32 2023-11-08T05:23:25.506 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: format: 2 2023-11-08T05:23:25.506 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: features: layering, exclusive-lock 2023-11-08T05:23:25.506 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: op_features: 2023-11-08T05:23:25.506 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: flags: 2023-11-08T05:23:25.507 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: create_timestamp: Wed Nov 8 05:23:24 2023 2023-11-08T05:23:25.507 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: access_timestamp: Wed Nov 8 05:23:24 2023 2023-11-08T05:23:25.507 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: modify_timestamp: Wed Nov 8 05:23:24 2023 2023-11-08T05:23:25.507 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring state: enabled 2023-11-08T05:23:25.508 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring mode: snapshot 2023-11-08T05:23:25.508 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring global id: d28028bb-94e2-4cfb-aeff-f31a873a50bc 2023-11-08T05:23:25.508 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring primary: true 2023-11-08T05:23:25.509 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ stress_write_image cluster2 mirror image_2 2023-11-08T05:23:25.509 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:23:25.509 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:23:25.510 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_2 2023-11-08T05:23:25.510 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-11-08T05:23:25.510 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local duration=5 2023-11-08T05:23:25.510 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ set +e 2023-11-08T05:23:25.510 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_2 --debug-rbd=20 --debug-journaler=20 2023-11-08T05:23:25.684 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:100 IOs, 17754112 bytes 2023-11-08T05:23:25.840 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:200 IOs, 50432000 bytes 2023-11-08T05:23:26.010 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:300 IOs, 85583872 bytes 2023-11-08T05:23:26.193 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:400 IOs, 136176640 bytes 2023-11-08T05:23:26.407 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:500 IOs, 188952576 bytes 2023-11-08T05:23:26.746 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:600 IOs, 304884736 bytes 2023-11-08T05:23:26.902 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:700 IOs, 332480512 bytes 2023-11-08T05:23:27.061 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:800 IOs, 356718592 bytes 2023-11-08T05:23:27.130 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.32846885770481526) > weight (0.25) 2023-11-08T05:23:27.131 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.499787387947189) > weight (0.25) 2023-11-08T05:23:27.131 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.5298583006045847) > weight (0.25) 2023-11-08T05:23:27.131 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.3 with skip (0.552229302531597) > weight (0.25) 2023-11-08T05:23:27.131 INFO:tasks.rbd_mirror_thrash:waiting for 108 secs before thrashing 2023-11-08T05:23:27.185 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:900 IOs, 369489920 bytes 2023-11-08T05:23:27.379 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1000 IOs, 416334848 bytes 2023-11-08T05:23:27.515 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1100 IOs, 432399360 bytes 2023-11-08T05:23:27.798 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1200 IOs, 529075200 bytes 2023-11-08T05:23:27.990 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1300 IOs, 573389824 bytes 2023-11-08T05:23:28.178 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1400 IOs, 616674304 bytes 2023-11-08T05:23:28.321 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1500 IOs, 640550912 bytes 2023-11-08T05:23:28.647 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1600 IOs, 753036288 bytes 2023-11-08T05:23:28.793 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1700 IOs, 776938496 bytes 2023-11-08T05:23:28.966 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1800 IOs, 811830272 bytes 2023-11-08T05:23:29.170 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1900 IOs, 872599552 bytes 2023-11-08T05:23:29.440 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2000 IOs, 950793216 bytes 2023-11-08T05:23:29.711 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2100 IOs, 1035604992 bytes 2023-11-08T05:23:29.856 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2200 IOs, 1052435456 bytes 2023-11-08T05:23:30.081 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2300 IOs, 1130338304 bytes 2023-11-08T05:23:30.256 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2400 IOs, 1150487552 bytes 2023-11-08T05:23:30.439 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2500 IOs, 1201539072 bytes 2023-11-08T05:23:30.517 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ error_code=124 2023-11-08T05:23:30.517 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ set -e 2023-11-08T05:23:30.517 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' 124 -eq 124 ']' 2023-11-08T05:23:30.517 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:23:30.518 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_image_present cluster1 mirror image_2 present 2023-11-08T05:23:30.518 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:23:30.518 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:23:30.518 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_2 2023-11-08T05:23:30.518 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local state=present 2023-11-08T05:23:30.519 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image_id= 2023-11-08T05:23:30.519 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local s 2023-11-08T05:23:30.519 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test -n '' 2023-11-08T05:23:30.519 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ get_image_id cluster1 mirror image_2 2023-11-08T05:23:30.519 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1 2023-11-08T05:23:30.520 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local pool=mirror 2023-11-08T05:23:30.520 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local image=image_2 2023-11-08T05:23:30.520 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ rbd --cluster=cluster1 info mirror/image_2 2023-11-08T05:23:30.520 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-11-08T05:23:30.553 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image_id=12a939311f57 2023-11-08T05:23:30.554 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-11-08T05:23:30.554 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0.1 2023-11-08T05:23:30.655 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test_image_present cluster1 mirror image_2 present 12a939311f57 2023-11-08T05:23:30.655 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:23:30.655 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:23:30.656 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_2 2023-11-08T05:23:30.656 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local test_state=present 2023-11-08T05:23:30.656 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image_id=12a939311f57 2023-11-08T05:23:30.656 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local current_state=deleted 2023-11-08T05:23:30.656 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local current_image_id 2023-11-08T05:23:30.656 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ get_image_id cluster1 mirror image_2 2023-11-08T05:23:30.657 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1 2023-11-08T05:23:30.657 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local pool=mirror 2023-11-08T05:23:30.657 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local image=image_2 2023-11-08T05:23:30.657 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ rbd --cluster=cluster1 info mirror/image_2 2023-11-08T05:23:30.657 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-11-08T05:23:30.692 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ current_image_id=12a939311f57 2023-11-08T05:23:30.692 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test -n 12a939311f57 2023-11-08T05:23:30.693 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test -z 12a939311f57 -o 12a939311f57 = 12a939311f57 2023-11-08T05:23:30.693 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ current_state=present 2023-11-08T05:23:30.693 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test present = present 2023-11-08T05:23:30.693 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:23:30.693 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ disable_mirror cluster2 mirror image_2 2023-11-08T05:23:30.693 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:23:30.694 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:23:30.694 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_2 2023-11-08T05:23:30.694 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_2 2023-11-08T05:23:31.517 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi087.stderr:2023-11-08T05:23:31.514+0000 7fb6a8772700 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55dd319d8c00 scan_remote_mirror_snapshots: failed to locate remote start snapshot: snap_id=370 2023-11-08T05:23:31.518 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi087.stderr:2023-11-08T05:23:31.514+0000 7fb6a8772700 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55dd319d8c00 scan_remote_mirror_snapshots: split-brain detected: failed to find matching non-primary snapshot in remote image: local_snap_id_start=370, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=06977a9b-46f8-44a8-aea5-8992e82c582a, primary_snap_id=172, last_copied_object_number=128, snap_seqs={370=18446744073709551614}] 2023-11-08T05:23:36.518 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:Mirroring disabled 2023-11-08T05:23:36.526 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_image_present cluster1 mirror image_2 deleted 2023-11-08T05:23:36.526 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:23:36.527 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:23:36.527 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_2 2023-11-08T05:23:36.527 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local state=deleted 2023-11-08T05:23:36.527 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image_id= 2023-11-08T05:23:36.527 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local s 2023-11-08T05:23:36.528 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test -n '' 2023-11-08T05:23:36.528 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ get_image_id cluster1 mirror image_2 2023-11-08T05:23:36.528 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1 2023-11-08T05:23:36.528 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local pool=mirror 2023-11-08T05:23:36.529 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local image=image_2 2023-11-08T05:23:36.529 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ rbd --cluster=cluster1 info mirror/image_2 2023-11-08T05:23:36.530 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-11-08T05:23:36.557 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd: error opening image image_2: (2) No such file or directory 2023-11-08T05:23:36.560 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image_id= 2023-11-08T05:23:36.561 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-11-08T05:23:36.561 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0.1 2023-11-08T05:23:36.662 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test_image_present cluster1 mirror image_2 deleted '' 2023-11-08T05:23:36.663 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:23:36.663 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:23:36.663 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_2 2023-11-08T05:23:36.663 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local test_state=deleted 2023-11-08T05:23:36.663 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image_id= 2023-11-08T05:23:36.664 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local current_state=deleted 2023-11-08T05:23:36.664 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local current_image_id 2023-11-08T05:23:36.664 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ get_image_id cluster1 mirror image_2 2023-11-08T05:23:36.664 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1 2023-11-08T05:23:36.665 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local pool=mirror 2023-11-08T05:23:36.665 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local image=image_2 2023-11-08T05:23:36.665 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ rbd --cluster=cluster1 info mirror/image_2 2023-11-08T05:23:36.665 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-11-08T05:23:36.691 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd: error opening image image_2: (2) No such file or directory 2023-11-08T05:23:36.694 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ current_image_id= 2023-11-08T05:23:36.694 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test -n '' 2023-11-08T05:23:36.695 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test deleted = deleted 2023-11-08T05:23:36.695 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:23:36.695 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ purge_snapshots cluster2 mirror image_2 2023-11-08T05:23:36.696 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:23:36.696 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:23:36.696 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_2 2023-11-08T05:23:36.696 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 snap purge mirror/image_2 2023-11-08T05:23:36.734 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ remove_image_retry cluster2 mirror image_2 2023-11-08T05:23:36.734 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:23:36.735 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:23:36.735 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_2 2023-11-08T05:23:36.735 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0 1 2 4 8 16 32 2023-11-08T05:23:36.735 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0 2023-11-08T05:23:36.736 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ remove_image cluster2 mirror image_2 2023-11-08T05:23:36.736 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:23:36.737 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:23:36.737 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_2 2023-11-08T05:23:36.737 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_2 2023-11-08T05:23:36.773 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster=cluster2 rm mirror/image_2 2023-11-08T05:23:56.768 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2023-11-08T05:23:56.772 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:23:56.772 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for i in `seq 1 10` 2023-11-08T05:23:56.772 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image=image_3 2023-11-08T05:23:56.772 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ create_image cluster2 mirror image_3 512M 2023-11-08T05:23:56.772 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:23:56.773 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:23:56.773 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:23:56.773 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:23:56.773 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_3 2023-11-08T05:23:56.773 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:23:56.774 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local size=128 2023-11-08T05:23:56.774 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' -n 512M ']' 2023-11-08T05:23:56.774 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ size=512M 2023-11-08T05:23:56.774 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:23:56.774 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock mirror/image_3 2023-11-08T05:23:56.811 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ enable_mirror cluster2 mirror image_3 2023-11-08T05:23:56.811 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:23:56.812 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:23:56.812 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_3 2023-11-08T05:23:56.812 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local mode=snapshot 2023-11-08T05:23:56.812 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_3 snapshot 2023-11-08T05:23:57.551 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:Mirroring enabled 2023-11-08T05:23:57.559 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster=cluster2 info mirror/image_3 2023-11-08T05:23:57.590 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:rbd image 'image_3': 2023-11-08T05:23:57.591 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: size 512 MiB in 128 objects 2023-11-08T05:23:57.591 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: order 22 (4 MiB objects) 2023-11-08T05:23:57.591 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: snapshot_count: 1 2023-11-08T05:23:57.591 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: id: 1935759a9a57 2023-11-08T05:23:57.591 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: block_name_prefix: rbd_data.1935759a9a57 2023-11-08T05:23:57.592 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: format: 2 2023-11-08T05:23:57.592 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: features: layering, exclusive-lock 2023-11-08T05:23:57.592 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: op_features: 2023-11-08T05:23:57.592 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: flags: 2023-11-08T05:23:57.592 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: create_timestamp: Wed Nov 8 05:23:56 2023 2023-11-08T05:23:57.593 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: access_timestamp: Wed Nov 8 05:23:56 2023 2023-11-08T05:23:57.593 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: modify_timestamp: Wed Nov 8 05:23:56 2023 2023-11-08T05:23:57.593 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring state: enabled 2023-11-08T05:23:57.593 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring mode: snapshot 2023-11-08T05:23:57.593 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring global id: 72dabb4f-492e-434b-815c-725a89819475 2023-11-08T05:23:57.594 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring primary: true 2023-11-08T05:23:57.594 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ stress_write_image cluster2 mirror image_3 2023-11-08T05:23:57.595 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:23:57.595 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:23:57.595 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_3 2023-11-08T05:23:57.595 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-11-08T05:23:57.596 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local duration=10 2023-11-08T05:23:57.596 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ set +e 2023-11-08T05:23:57.596 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_3 --debug-rbd=20 --debug-journaler=20 2023-11-08T05:23:57.831 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:100 IOs, 52198400 bytes 2023-11-08T05:23:57.970 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:200 IOs, 73649152 bytes 2023-11-08T05:23:58.231 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:300 IOs, 161836032 bytes 2023-11-08T05:23:58.366 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:400 IOs, 178470912 bytes 2023-11-08T05:23:58.526 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:500 IOs, 233600000 bytes 2023-11-08T05:23:58.805 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:600 IOs, 309571584 bytes 2023-11-08T05:23:58.915 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:700 IOs, 312525824 bytes 2023-11-08T05:23:59.052 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:800 IOs, 326688768 bytes 2023-11-08T05:23:59.270 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:900 IOs, 381721600 bytes 2023-11-08T05:23:59.436 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1000 IOs, 404972544 bytes 2023-11-08T05:23:59.604 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1100 IOs, 452531200 bytes 2023-11-08T05:23:59.923 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1200 IOs, 553601024 bytes 2023-11-08T05:24:00.104 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1300 IOs, 584889344 bytes 2023-11-08T05:24:00.320 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1400 IOs, 652750848 bytes 2023-11-08T05:24:00.552 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1500 IOs, 707776512 bytes 2023-11-08T05:24:00.746 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1600 IOs, 758012928 bytes 2023-11-08T05:24:00.952 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1700 IOs, 811997184 bytes 2023-11-08T05:24:01.126 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1800 IOs, 850129920 bytes 2023-11-08T05:24:01.255 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1900 IOs, 863527936 bytes 2023-11-08T05:24:01.519 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2000 IOs, 949172224 bytes 2023-11-08T05:24:01.794 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2100 IOs, 1024875520 bytes 2023-11-08T05:24:01.960 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2200 IOs, 1052910592 bytes 2023-11-08T05:24:02.135 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2300 IOs, 1085622272 bytes 2023-11-08T05:24:02.353 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2400 IOs, 1140582400 bytes 2023-11-08T05:24:02.652 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2500 IOs, 1248424960 bytes 2023-11-08T05:24:02.804 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2600 IOs, 1262029824 bytes 2023-11-08T05:24:03.015 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2700 IOs, 1312407552 bytes 2023-11-08T05:24:03.164 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2800 IOs, 1349604352 bytes 2023-11-08T05:24:03.408 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2900 IOs, 1421720576 bytes 2023-11-08T05:24:03.625 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3000 IOs, 1460346880 bytes 2023-11-08T05:24:03.792 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3100 IOs, 1496223744 bytes 2023-11-08T05:24:03.937 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3200 IOs, 1531750400 bytes 2023-11-08T05:24:04.159 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3300 IOs, 1576020992 bytes 2023-11-08T05:24:04.366 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3400 IOs, 1621012480 bytes 2023-11-08T05:24:04.577 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3500 IOs, 1684992000 bytes 2023-11-08T05:24:04.749 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3600 IOs, 1716673536 bytes 2023-11-08T05:24:04.915 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3700 IOs, 1743789056 bytes 2023-11-08T05:24:05.098 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3800 IOs, 1782327296 bytes 2023-11-08T05:24:05.280 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3900 IOs, 1820242944 bytes 2023-11-08T05:24:05.502 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4000 IOs, 1879116800 bytes 2023-11-08T05:24:05.704 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4100 IOs, 1929410560 bytes 2023-11-08T05:24:05.865 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4200 IOs, 1955718144 bytes 2023-11-08T05:24:06.013 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4300 IOs, 1988031488 bytes 2023-11-08T05:24:06.251 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4400 IOs, 2041206784 bytes 2023-11-08T05:24:06.411 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4500 IOs, 2076895232 bytes 2023-11-08T05:24:06.595 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4600 IOs, 2108481536 bytes 2023-11-08T05:24:06.747 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4700 IOs, 2133390336 bytes 2023-11-08T05:24:06.888 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4800 IOs, 2161668096 bytes 2023-11-08T05:24:07.174 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4900 IOs, 2243602432 bytes 2023-11-08T05:24:07.389 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5000 IOs, 2300590080 bytes 2023-11-08T05:24:07.603 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ error_code=124 2023-11-08T05:24:07.603 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ set -e 2023-11-08T05:24:07.603 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' 124 -eq 124 ']' 2023-11-08T05:24:07.604 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:24:07.604 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_image_present cluster1 mirror image_3 present 2023-11-08T05:24:07.604 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:24:07.604 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:24:07.604 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_3 2023-11-08T05:24:07.604 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local state=present 2023-11-08T05:24:07.605 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image_id= 2023-11-08T05:24:07.605 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local s 2023-11-08T05:24:07.605 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test -n '' 2023-11-08T05:24:07.605 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ get_image_id cluster1 mirror image_3 2023-11-08T05:24:07.606 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1 2023-11-08T05:24:07.606 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local pool=mirror 2023-11-08T05:24:07.606 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local image=image_3 2023-11-08T05:24:07.606 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ rbd --cluster=cluster1 info mirror/image_3 2023-11-08T05:24:07.606 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-11-08T05:24:07.638 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image_id=12a949805b1b 2023-11-08T05:24:07.638 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-11-08T05:24:07.638 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0.1 2023-11-08T05:24:07.740 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test_image_present cluster1 mirror image_3 present 12a949805b1b 2023-11-08T05:24:07.740 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:24:07.740 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:24:07.741 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_3 2023-11-08T05:24:07.741 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local test_state=present 2023-11-08T05:24:07.741 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image_id=12a949805b1b 2023-11-08T05:24:07.741 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local current_state=deleted 2023-11-08T05:24:07.741 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local current_image_id 2023-11-08T05:24:07.741 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ get_image_id cluster1 mirror image_3 2023-11-08T05:24:07.742 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1 2023-11-08T05:24:07.742 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local pool=mirror 2023-11-08T05:24:07.742 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local image=image_3 2023-11-08T05:24:07.742 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ rbd --cluster=cluster1 info mirror/image_3 2023-11-08T05:24:07.743 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-11-08T05:24:07.775 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ current_image_id=12a949805b1b 2023-11-08T05:24:07.775 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test -n 12a949805b1b 2023-11-08T05:24:07.775 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test -z 12a949805b1b -o 12a949805b1b = 12a949805b1b 2023-11-08T05:24:07.776 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ current_state=present 2023-11-08T05:24:07.776 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test present = present 2023-11-08T05:24:07.776 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:24:07.776 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ disable_mirror cluster2 mirror image_3 2023-11-08T05:24:07.776 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:24:07.777 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:24:07.777 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_3 2023-11-08T05:24:07.777 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_3 2023-11-08T05:24:08.682 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi087.stderr:2023-11-08T05:24:08.678+0000 7fb6a8772700 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55dd31a11400 scan_remote_mirror_snapshots: failed to locate remote start snapshot: snap_id=372 2023-11-08T05:24:08.683 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi087.stderr:2023-11-08T05:24:08.678+0000 7fb6a8772700 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55dd31a11400 scan_remote_mirror_snapshots: split-brain detected: failed to find matching non-primary snapshot in remote image: local_snap_id_start=372, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=06977a9b-46f8-44a8-aea5-8992e82c582a, primary_snap_id=174, last_copied_object_number=128, snap_seqs={372=18446744073709551614}] 2023-11-08T05:24:13.686 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:Mirroring disabled 2023-11-08T05:24:13.694 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_image_present cluster1 mirror image_3 deleted 2023-11-08T05:24:13.694 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:24:13.695 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:24:13.695 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_3 2023-11-08T05:24:13.695 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local state=deleted 2023-11-08T05:24:13.695 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image_id= 2023-11-08T05:24:13.695 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local s 2023-11-08T05:24:13.696 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test -n '' 2023-11-08T05:24:13.696 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ get_image_id cluster1 mirror image_3 2023-11-08T05:24:13.696 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1 2023-11-08T05:24:13.696 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local pool=mirror 2023-11-08T05:24:13.696 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local image=image_3 2023-11-08T05:24:13.697 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ rbd --cluster=cluster1 info mirror/image_3 2023-11-08T05:24:13.697 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-11-08T05:24:13.725 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd: error opening image image_3: (2) No such file or directory 2023-11-08T05:24:13.729 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image_id= 2023-11-08T05:24:13.729 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-11-08T05:24:13.729 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0.1 2023-11-08T05:24:13.831 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test_image_present cluster1 mirror image_3 deleted '' 2023-11-08T05:24:13.831 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:24:13.831 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:24:13.832 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_3 2023-11-08T05:24:13.832 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local test_state=deleted 2023-11-08T05:24:13.832 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image_id= 2023-11-08T05:24:13.832 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local current_state=deleted 2023-11-08T05:24:13.832 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local current_image_id 2023-11-08T05:24:13.832 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ get_image_id cluster1 mirror image_3 2023-11-08T05:24:13.833 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1 2023-11-08T05:24:13.833 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local pool=mirror 2023-11-08T05:24:13.833 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local image=image_3 2023-11-08T05:24:13.833 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ rbd --cluster=cluster1 info mirror/image_3 2023-11-08T05:24:13.833 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-11-08T05:24:13.860 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd: error opening image image_3: (2) No such file or directory 2023-11-08T05:24:13.863 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ current_image_id= 2023-11-08T05:24:13.863 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test -n '' 2023-11-08T05:24:13.863 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test deleted = deleted 2023-11-08T05:24:13.864 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:24:13.864 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ purge_snapshots cluster2 mirror image_3 2023-11-08T05:24:13.864 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:24:13.864 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:24:13.864 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_3 2023-11-08T05:24:13.865 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 snap purge mirror/image_3 2023-11-08T05:24:13.903 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ remove_image_retry cluster2 mirror image_3 2023-11-08T05:24:13.903 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:24:13.903 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:24:13.903 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_3 2023-11-08T05:24:13.904 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0 1 2 4 8 16 32 2023-11-08T05:24:13.904 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0 2023-11-08T05:24:13.905 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ remove_image cluster2 mirror image_3 2023-11-08T05:24:13.905 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:24:13.905 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:24:13.905 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_3 2023-11-08T05:24:13.905 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_3 2023-11-08T05:24:13.942 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster=cluster2 rm mirror/image_3 2023-11-08T05:24:30.341 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.31101829643931755) > weight (0.25) 2023-11-08T05:24:30.342 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.1 2023-11-08T05:24:30.342 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Sent signal 15 2023-11-08T05:24:30.343 INFO:tasks.rbd_mirror_thrash:waiting for 6 secs before reviving daemons 2023-11-08T05:24:30.343 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi087.stderr:2023-11-08T05:24:30.342+0000 7fb6b678e700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 (PID: 133083) UID: 1000 2023-11-08T05:24:30.544 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi087.stderr:daemon-helper: command failed with exit status 1 2023-11-08T05:24:36.344 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.1 2023-11-08T05:24:36.344 INFO:teuthology.orchestra.run:waiting for 600 2023-11-08T05:24:36.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-08T05:24:36.345 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.1 2023-11-08T05:24:36.345 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Restarting daemon 2023-11-08T05:24:36.346 DEBUG:teuthology.orchestra.run.smithi087:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 2023-11-08T05:24:36.349 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Started 2023-11-08T05:24:36.349 INFO:tasks.rbd_mirror_thrash:waiting for 102 secs before thrashing 2023-11-08T05:24:37.946 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2023-11-08T05:24:37.950 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:24:37.950 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for i in `seq 1 10` 2023-11-08T05:24:37.951 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image=image_4 2023-11-08T05:24:37.951 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ create_image cluster2 mirror image_4 512M 2023-11-08T05:24:37.951 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:24:37.951 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:24:37.952 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:24:37.952 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:24:37.952 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_4 2023-11-08T05:24:37.952 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:24:37.952 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local size=128 2023-11-08T05:24:37.953 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' -n 512M ']' 2023-11-08T05:24:37.953 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ size=512M 2023-11-08T05:24:37.953 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:24:37.953 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock mirror/image_4 2023-11-08T05:24:37.990 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ enable_mirror cluster2 mirror image_4 2023-11-08T05:24:37.990 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:24:37.990 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:24:37.991 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_4 2023-11-08T05:24:37.991 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local mode=snapshot 2023-11-08T05:24:37.991 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_4 snapshot 2023-11-08T05:24:38.725 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:Mirroring enabled 2023-11-08T05:24:38.734 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster=cluster2 info mirror/image_4 2023-11-08T05:24:38.766 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:rbd image 'image_4': 2023-11-08T05:24:38.767 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: size 512 MiB in 128 objects 2023-11-08T05:24:38.767 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: order 22 (4 MiB objects) 2023-11-08T05:24:38.767 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: snapshot_count: 1 2023-11-08T05:24:38.768 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: id: 1953a35b3294 2023-11-08T05:24:38.768 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: block_name_prefix: rbd_data.1953a35b3294 2023-11-08T05:24:38.768 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: format: 2 2023-11-08T05:24:38.768 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: features: layering, exclusive-lock 2023-11-08T05:24:38.768 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: op_features: 2023-11-08T05:24:38.769 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: flags: 2023-11-08T05:24:38.769 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: create_timestamp: Wed Nov 8 05:24:37 2023 2023-11-08T05:24:38.769 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: access_timestamp: Wed Nov 8 05:24:37 2023 2023-11-08T05:24:38.769 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: modify_timestamp: Wed Nov 8 05:24:37 2023 2023-11-08T05:24:38.770 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring state: enabled 2023-11-08T05:24:38.770 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring mode: snapshot 2023-11-08T05:24:38.770 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring global id: 65e92d00-b2e4-4e41-8eb5-a8b88af042f6 2023-11-08T05:24:38.770 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring primary: true 2023-11-08T05:24:38.771 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ stress_write_image cluster2 mirror image_4 2023-11-08T05:24:38.772 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:24:38.772 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:24:38.772 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_4 2023-11-08T05:24:38.772 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-11-08T05:24:38.773 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local duration=14 2023-11-08T05:24:38.773 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ set +e 2023-11-08T05:24:38.773 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_4 --debug-rbd=20 --debug-journaler=20 2023-11-08T05:24:39.027 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:100 IOs, 54923264 bytes 2023-11-08T05:24:39.195 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:200 IOs, 89476096 bytes 2023-11-08T05:24:39.475 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:300 IOs, 181452800 bytes 2023-11-08T05:24:39.604 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:400 IOs, 197414912 bytes 2023-11-08T05:24:39.749 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:500 IOs, 216367104 bytes 2023-11-08T05:24:39.877 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:600 IOs, 234439680 bytes 2023-11-08T05:24:40.035 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:700 IOs, 268865536 bytes 2023-11-08T05:24:40.255 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:800 IOs, 335200256 bytes 2023-11-08T05:24:40.420 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:900 IOs, 391038976 bytes 2023-11-08T05:24:40.569 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1000 IOs, 407288832 bytes 2023-11-08T05:24:40.739 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1100 IOs, 447459328 bytes 2023-11-08T05:24:40.862 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1200 IOs, 462377984 bytes 2023-11-08T05:24:40.960 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1300 IOs, 463367168 bytes 2023-11-08T05:24:41.127 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1400 IOs, 499828736 bytes 2023-11-08T05:24:41.354 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1500 IOs, 574551040 bytes 2023-11-08T05:24:41.531 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1600 IOs, 615903232 bytes 2023-11-08T05:24:41.705 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1700 IOs, 670384128 bytes 2023-11-08T05:24:41.877 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1800 IOs, 740496384 bytes 2023-11-08T05:24:42.173 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1900 IOs, 802990080 bytes 2023-11-08T05:24:42.314 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2000 IOs, 823641088 bytes 2023-11-08T05:24:42.598 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2100 IOs, 916864000 bytes 2023-11-08T05:24:42.748 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2200 IOs, 970775552 bytes 2023-11-08T05:24:42.925 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2300 IOs, 985333760 bytes 2023-11-08T05:24:43.083 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2400 IOs, 1014953984 bytes 2023-11-08T05:24:43.252 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2500 IOs, 1051447296 bytes 2023-11-08T05:24:43.507 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2600 IOs, 1125624832 bytes 2023-11-08T05:24:43.659 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2700 IOs, 1147399168 bytes 2023-11-08T05:24:43.854 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2800 IOs, 1193395200 bytes 2023-11-08T05:24:44.006 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2900 IOs, 1221286912 bytes 2023-11-08T05:24:44.150 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3000 IOs, 1244129280 bytes 2023-11-08T05:24:44.323 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3100 IOs, 1269478400 bytes 2023-11-08T05:24:44.465 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3200 IOs, 1287137280 bytes 2023-11-08T05:24:44.593 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3300 IOs, 1298057216 bytes 2023-11-08T05:24:44.761 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3400 IOs, 1327636480 bytes 2023-11-08T05:24:45.009 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3500 IOs, 1395088384 bytes 2023-11-08T05:24:45.165 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3600 IOs, 1423444992 bytes 2023-11-08T05:24:45.310 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3700 IOs, 1437262848 bytes 2023-11-08T05:24:45.499 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3800 IOs, 1477608448 bytes 2023-11-08T05:24:45.607 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3900 IOs, 1479424000 bytes 2023-11-08T05:24:45.890 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4000 IOs, 1566094336 bytes 2023-11-08T05:24:46.054 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4100 IOs, 1594539008 bytes 2023-11-08T05:24:46.202 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4200 IOs, 1609487360 bytes 2023-11-08T05:24:46.381 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4300 IOs, 1657315328 bytes 2023-11-08T05:24:46.528 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4400 IOs, 1666720768 bytes 2023-11-08T05:24:46.676 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4500 IOs, 1687786496 bytes 2023-11-08T05:24:46.792 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4600 IOs, 1713197056 bytes 2023-11-08T05:24:47.042 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4700 IOs, 1763755008 bytes 2023-11-08T05:24:47.200 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4800 IOs, 1792088064 bytes 2023-11-08T05:24:47.434 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4900 IOs, 1857035264 bytes 2023-11-08T05:24:47.732 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5000 IOs, 1958995968 bytes 2023-11-08T05:24:47.945 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5100 IOs, 2015581184 bytes 2023-11-08T05:24:48.197 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5200 IOs, 2102082560 bytes 2023-11-08T05:24:48.399 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5300 IOs, 2139318272 bytes 2023-11-08T05:24:48.554 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5400 IOs, 2158362624 bytes 2023-11-08T05:24:48.758 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5500 IOs, 2202401792 bytes 2023-11-08T05:24:48.980 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5600 IOs, 2259349504 bytes 2023-11-08T05:24:49.161 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5700 IOs, 2292715520 bytes 2023-11-08T05:24:49.300 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5800 IOs, 2302603264 bytes 2023-11-08T05:24:49.422 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5900 IOs, 2311366656 bytes 2023-11-08T05:24:49.546 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:6000 IOs, 2321136640 bytes 2023-11-08T05:24:49.773 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:6100 IOs, 2382543872 bytes 2023-11-08T05:24:49.931 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:6200 IOs, 2404870144 bytes 2023-11-08T05:24:50.126 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:6300 IOs, 2464374784 bytes 2023-11-08T05:24:50.287 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:6400 IOs, 2482471936 bytes 2023-11-08T05:24:50.496 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:6500 IOs, 2530695168 bytes 2023-11-08T05:24:50.799 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:6600 IOs, 2638990336 bytes 2023-11-08T05:24:50.911 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:6700 IOs, 2639601664 bytes 2023-11-08T05:24:51.151 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:6800 IOs, 2704845824 bytes 2023-11-08T05:24:51.343 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:6900 IOs, 2758094848 bytes 2023-11-08T05:24:51.522 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:7000 IOs, 2789721088 bytes 2023-11-08T05:24:51.677 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:7100 IOs, 2814615552 bytes 2023-11-08T05:24:51.883 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:7200 IOs, 2860686336 bytes 2023-11-08T05:24:52.012 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:7300 IOs, 2873825280 bytes 2023-11-08T05:24:52.200 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:7400 IOs, 2912566272 bytes 2023-11-08T05:24:52.502 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:7500 IOs, 3006925824 bytes 2023-11-08T05:24:52.674 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:7600 IOs, 3042358272 bytes 2023-11-08T05:24:52.777 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ error_code=124 2023-11-08T05:24:52.777 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ set -e 2023-11-08T05:24:52.777 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' 124 -eq 124 ']' 2023-11-08T05:24:52.777 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:24:52.778 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_image_present cluster1 mirror image_4 present 2023-11-08T05:24:52.778 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:24:52.778 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:24:52.778 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_4 2023-11-08T05:24:52.779 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local state=present 2023-11-08T05:24:52.779 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image_id= 2023-11-08T05:24:52.779 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local s 2023-11-08T05:24:52.779 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test -n '' 2023-11-08T05:24:52.779 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ get_image_id cluster1 mirror image_4 2023-11-08T05:24:52.780 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1 2023-11-08T05:24:52.780 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local pool=mirror 2023-11-08T05:24:52.780 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local image=image_4 2023-11-08T05:24:52.780 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ rbd --cluster=cluster1 info mirror/image_4 2023-11-08T05:24:52.780 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-11-08T05:24:52.803 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd: error opening image image_4: (2) No such file or directory 2023-11-08T05:24:52.806 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image_id= 2023-11-08T05:24:52.806 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-11-08T05:24:52.806 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0.1 2023-11-08T05:24:52.908 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test_image_present cluster1 mirror image_4 present '' 2023-11-08T05:24:52.908 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:24:52.908 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:24:52.909 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_4 2023-11-08T05:24:52.909 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local test_state=present 2023-11-08T05:24:52.909 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image_id= 2023-11-08T05:24:52.909 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local current_state=deleted 2023-11-08T05:24:52.909 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local current_image_id 2023-11-08T05:24:52.909 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ get_image_id cluster1 mirror image_4 2023-11-08T05:24:52.910 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1 2023-11-08T05:24:52.910 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local pool=mirror 2023-11-08T05:24:52.910 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local image=image_4 2023-11-08T05:24:52.910 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ rbd --cluster=cluster1 info mirror/image_4 2023-11-08T05:24:52.911 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-11-08T05:24:52.934 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd: error opening image image_4: (2) No such file or directory 2023-11-08T05:24:52.937 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ current_image_id= 2023-11-08T05:24:52.937 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test -n '' 2023-11-08T05:24:52.937 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test present = deleted 2023-11-08T05:24:52.938 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-11-08T05:24:52.938 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 1 2023-11-08T05:24:53.939 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test_image_present cluster1 mirror image_4 present '' 2023-11-08T05:24:53.939 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:24:53.939 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:24:53.939 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_4 2023-11-08T05:24:53.940 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local test_state=present 2023-11-08T05:24:53.940 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image_id= 2023-11-08T05:24:53.940 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local current_state=deleted 2023-11-08T05:24:53.940 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local current_image_id 2023-11-08T05:24:53.940 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ get_image_id cluster1 mirror image_4 2023-11-08T05:24:53.941 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1 2023-11-08T05:24:53.941 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local pool=mirror 2023-11-08T05:24:53.941 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local image=image_4 2023-11-08T05:24:53.941 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ rbd --cluster=cluster1 info mirror/image_4 2023-11-08T05:24:53.941 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-11-08T05:24:53.970 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd: error opening image image_4: (2) No such file or directory 2023-11-08T05:24:53.973 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ current_image_id= 2023-11-08T05:24:53.973 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test -n '' 2023-11-08T05:24:53.973 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test present = deleted 2023-11-08T05:24:53.974 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-11-08T05:24:53.974 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 2 2023-11-08T05:24:55.975 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test_image_present cluster1 mirror image_4 present '' 2023-11-08T05:24:55.975 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:24:55.976 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:24:55.976 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_4 2023-11-08T05:24:55.976 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local test_state=present 2023-11-08T05:24:55.976 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image_id= 2023-11-08T05:24:55.976 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local current_state=deleted 2023-11-08T05:24:55.977 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local current_image_id 2023-11-08T05:24:55.977 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ get_image_id cluster1 mirror image_4 2023-11-08T05:24:55.977 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1 2023-11-08T05:24:55.977 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local pool=mirror 2023-11-08T05:24:55.977 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local image=image_4 2023-11-08T05:24:55.978 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ rbd --cluster=cluster1 info mirror/image_4 2023-11-08T05:24:55.978 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-11-08T05:24:56.006 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd: error opening image image_4: (2) No such file or directory 2023-11-08T05:24:56.009 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ current_image_id= 2023-11-08T05:24:56.010 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test -n '' 2023-11-08T05:24:56.010 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test present = deleted 2023-11-08T05:24:56.010 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-11-08T05:24:56.010 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 4 2023-11-08T05:25:00.011 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test_image_present cluster1 mirror image_4 present '' 2023-11-08T05:25:00.011 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:25:00.011 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:25:00.012 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_4 2023-11-08T05:25:00.012 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local test_state=present 2023-11-08T05:25:00.012 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image_id= 2023-11-08T05:25:00.012 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local current_state=deleted 2023-11-08T05:25:00.012 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local current_image_id 2023-11-08T05:25:00.013 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ get_image_id cluster1 mirror image_4 2023-11-08T05:25:00.013 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1 2023-11-08T05:25:00.013 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local pool=mirror 2023-11-08T05:25:00.013 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local image=image_4 2023-11-08T05:25:00.014 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ rbd --cluster=cluster1 info mirror/image_4 2023-11-08T05:25:00.014 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-11-08T05:25:00.045 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd: error opening image image_4: (2) No such file or directory 2023-11-08T05:25:00.047 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ current_image_id= 2023-11-08T05:25:00.047 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test -n '' 2023-11-08T05:25:00.047 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test present = deleted 2023-11-08T05:25:00.048 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-11-08T05:25:00.048 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 8 2023-11-08T05:25:08.049 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test_image_present cluster1 mirror image_4 present '' 2023-11-08T05:25:08.049 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:25:08.050 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:25:08.050 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_4 2023-11-08T05:25:08.050 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local test_state=present 2023-11-08T05:25:08.050 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image_id= 2023-11-08T05:25:08.051 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local current_state=deleted 2023-11-08T05:25:08.051 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local current_image_id 2023-11-08T05:25:08.051 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ get_image_id cluster1 mirror image_4 2023-11-08T05:25:08.051 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1 2023-11-08T05:25:08.051 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local pool=mirror 2023-11-08T05:25:08.051 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local image=image_4 2023-11-08T05:25:08.052 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ rbd --cluster=cluster1 info mirror/image_4 2023-11-08T05:25:08.052 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-11-08T05:25:08.092 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ current_image_id=1561231a4a2e 2023-11-08T05:25:08.093 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test -n 1561231a4a2e 2023-11-08T05:25:08.093 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test -z '' -o '' = 1561231a4a2e 2023-11-08T05:25:08.093 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ current_state=present 2023-11-08T05:25:08.093 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test present = present 2023-11-08T05:25:08.093 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:25:08.094 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ disable_mirror cluster2 mirror image_4 2023-11-08T05:25:08.094 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:25:08.094 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:25:08.094 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_4 2023-11-08T05:25:08.094 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_4 2023-11-08T05:25:08.333 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi087.stderr:2023-11-08T05:25:08.331+0000 7fc158a67700 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x562591f3cc00 scan_remote_mirror_snapshots: failed to locate remote start snapshot: snap_id=374 2023-11-08T05:25:08.333 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi087.stderr:2023-11-08T05:25:08.331+0000 7fc158a67700 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x562591f3cc00 scan_remote_mirror_snapshots: split-brain detected: failed to find matching non-primary snapshot in remote image: local_snap_id_start=374, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=06977a9b-46f8-44a8-aea5-8992e82c582a, primary_snap_id=176, last_copied_object_number=128, snap_seqs={374=18446744073709551614}] 2023-11-08T05:25:13.340 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:Mirroring disabled 2023-11-08T05:25:13.347 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_image_present cluster1 mirror image_4 deleted 2023-11-08T05:25:13.347 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:25:13.348 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:25:13.348 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_4 2023-11-08T05:25:13.348 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local state=deleted 2023-11-08T05:25:13.348 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image_id= 2023-11-08T05:25:13.348 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local s 2023-11-08T05:25:13.349 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test -n '' 2023-11-08T05:25:13.349 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ get_image_id cluster1 mirror image_4 2023-11-08T05:25:13.349 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1 2023-11-08T05:25:13.349 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local pool=mirror 2023-11-08T05:25:13.349 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local image=image_4 2023-11-08T05:25:13.350 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ rbd --cluster=cluster1 info mirror/image_4 2023-11-08T05:25:13.350 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-11-08T05:25:13.379 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd: error opening image image_4: (2) No such file or directory 2023-11-08T05:25:13.382 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image_id= 2023-11-08T05:25:13.382 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-11-08T05:25:13.382 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0.1 2023-11-08T05:25:13.484 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test_image_present cluster1 mirror image_4 deleted '' 2023-11-08T05:25:13.484 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:25:13.484 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:25:13.485 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_4 2023-11-08T05:25:13.485 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local test_state=deleted 2023-11-08T05:25:13.485 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image_id= 2023-11-08T05:25:13.485 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local current_state=deleted 2023-11-08T05:25:13.485 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local current_image_id 2023-11-08T05:25:13.486 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ get_image_id cluster1 mirror image_4 2023-11-08T05:25:13.486 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1 2023-11-08T05:25:13.486 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local pool=mirror 2023-11-08T05:25:13.486 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local image=image_4 2023-11-08T05:25:13.486 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ rbd --cluster=cluster1 info mirror/image_4 2023-11-08T05:25:13.487 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-11-08T05:25:13.513 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd: error opening image image_4: (2) No such file or directory 2023-11-08T05:25:13.516 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ current_image_id= 2023-11-08T05:25:13.516 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test -n '' 2023-11-08T05:25:13.517 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test deleted = deleted 2023-11-08T05:25:13.517 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:25:13.517 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ purge_snapshots cluster2 mirror image_4 2023-11-08T05:25:13.517 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:25:13.518 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:25:13.518 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_4 2023-11-08T05:25:13.518 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 snap purge mirror/image_4 2023-11-08T05:25:13.556 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ remove_image_retry cluster2 mirror image_4 2023-11-08T05:25:13.556 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:25:13.557 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:25:13.557 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_4 2023-11-08T05:25:13.557 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0 1 2 4 8 16 32 2023-11-08T05:25:13.557 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0 2023-11-08T05:25:13.558 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ remove_image cluster2 mirror image_4 2023-11-08T05:25:13.558 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:25:13.558 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:25:13.559 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_4 2023-11-08T05:25:13.559 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_4 2023-11-08T05:25:13.595 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster=cluster2 rm mirror/image_4 2023-11-08T05:25:15.133 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.7184005487462501) > weight (0.25) 2023-11-08T05:25:15.133 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.7465607933462143) > weight (0.25) 2023-11-08T05:25:15.133 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.6998341895830111) > weight (0.25) 2023-11-08T05:25:15.133 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.3 with skip (0.5303098344933073) > weight (0.25) 2023-11-08T05:25:15.134 INFO:tasks.rbd_mirror_thrash:waiting for 102 secs before thrashing 2023-11-08T05:25:20.573 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2023-11-08T05:25:20.577 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:25:20.577 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for i in `seq 1 10` 2023-11-08T05:25:20.577 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image=image_5 2023-11-08T05:25:20.577 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ create_image cluster2 mirror image_5 512M 2023-11-08T05:25:20.577 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:25:20.578 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:25:20.578 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:25:20.578 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:25:20.578 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_5 2023-11-08T05:25:20.578 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:25:20.579 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local size=128 2023-11-08T05:25:20.579 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' -n 512M ']' 2023-11-08T05:25:20.579 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ size=512M 2023-11-08T05:25:20.579 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:25:20.580 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock mirror/image_5 2023-11-08T05:25:20.617 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ enable_mirror cluster2 mirror image_5 2023-11-08T05:25:20.617 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:25:20.618 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:25:20.618 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_5 2023-11-08T05:25:20.618 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local mode=snapshot 2023-11-08T05:25:20.618 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_5 snapshot 2023-11-08T05:25:21.355 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:Mirroring enabled 2023-11-08T05:25:21.363 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster=cluster2 info mirror/image_5 2023-11-08T05:25:21.394 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:rbd image 'image_5': 2023-11-08T05:25:21.394 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: size 512 MiB in 128 objects 2023-11-08T05:25:21.395 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: order 22 (4 MiB objects) 2023-11-08T05:25:21.395 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: snapshot_count: 1 2023-11-08T05:25:21.395 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: id: 196b67d3e244 2023-11-08T05:25:21.395 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: block_name_prefix: rbd_data.196b67d3e244 2023-11-08T05:25:21.395 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: format: 2 2023-11-08T05:25:21.396 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: features: layering, exclusive-lock 2023-11-08T05:25:21.396 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: op_features: 2023-11-08T05:25:21.396 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: flags: 2023-11-08T05:25:21.396 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: create_timestamp: Wed Nov 8 05:25:20 2023 2023-11-08T05:25:21.396 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: access_timestamp: Wed Nov 8 05:25:20 2023 2023-11-08T05:25:21.396 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: modify_timestamp: Wed Nov 8 05:25:20 2023 2023-11-08T05:25:21.397 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring state: enabled 2023-11-08T05:25:21.397 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring mode: snapshot 2023-11-08T05:25:21.397 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring global id: 042be858-f38c-4af0-9890-7210a8160fd5 2023-11-08T05:25:21.397 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring primary: true 2023-11-08T05:25:21.398 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ stress_write_image cluster2 mirror image_5 2023-11-08T05:25:21.398 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:25:21.398 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:25:21.398 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_5 2023-11-08T05:25:21.399 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-11-08T05:25:21.399 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local duration=14 2023-11-08T05:25:21.399 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ set +e 2023-11-08T05:25:21.400 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_5 --debug-rbd=20 --debug-journaler=20 2023-11-08T05:25:21.618 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:100 IOs, 29824000 bytes 2023-11-08T05:25:21.836 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:200 IOs, 62098432 bytes 2023-11-08T05:25:22.011 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:300 IOs, 101536768 bytes 2023-11-08T05:25:22.152 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:400 IOs, 121032704 bytes 2023-11-08T05:25:22.274 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:500 IOs, 135410688 bytes 2023-11-08T05:25:22.507 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:600 IOs, 209234944 bytes 2023-11-08T05:25:22.666 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:700 IOs, 231868416 bytes 2023-11-08T05:25:22.799 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:800 IOs, 248001536 bytes 2023-11-08T05:25:23.081 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:900 IOs, 341994496 bytes 2023-11-08T05:25:23.401 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1000 IOs, 428359680 bytes 2023-11-08T05:25:23.579 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1100 IOs, 464656384 bytes 2023-11-08T05:25:23.697 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1200 IOs, 473600000 bytes 2023-11-08T05:25:23.990 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1300 IOs, 572356608 bytes 2023-11-08T05:25:24.218 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1400 IOs, 634013696 bytes 2023-11-08T05:25:24.504 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1500 IOs, 728068096 bytes 2023-11-08T05:25:24.762 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1600 IOs, 809404416 bytes 2023-11-08T05:25:24.929 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1700 IOs, 840956928 bytes 2023-11-08T05:25:25.088 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1800 IOs, 862916608 bytes 2023-11-08T05:25:25.235 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1900 IOs, 881068032 bytes 2023-11-08T05:25:25.491 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2000 IOs, 959554560 bytes 2023-11-08T05:25:25.609 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2100 IOs, 967972864 bytes 2023-11-08T05:25:25.720 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2200 IOs, 971515904 bytes 2023-11-08T05:25:25.969 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2300 IOs, 1042683904 bytes 2023-11-08T05:25:26.111 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2400 IOs, 1084158976 bytes 2023-11-08T05:25:26.277 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2500 IOs, 1097856000 bytes 2023-11-08T05:25:26.470 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2600 IOs, 1138792448 bytes 2023-11-08T05:25:26.738 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2700 IOs, 1208960000 bytes 2023-11-08T05:25:26.923 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2800 IOs, 1266829312 bytes 2023-11-08T05:25:27.191 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2900 IOs, 1306621952 bytes 2023-11-08T05:25:27.372 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3000 IOs, 1336189952 bytes 2023-11-08T05:25:27.522 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3100 IOs, 1373815808 bytes 2023-11-08T05:25:27.918 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3200 IOs, 1445839872 bytes 2023-11-08T05:25:28.198 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3300 IOs, 1538914304 bytes 2023-11-08T05:25:28.449 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3400 IOs, 1605789696 bytes 2023-11-08T05:25:28.715 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3500 IOs, 1686155264 bytes 2023-11-08T05:25:28.892 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3600 IOs, 1721823232 bytes 2023-11-08T05:25:29.104 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3700 IOs, 1774918656 bytes 2023-11-08T05:25:29.261 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3800 IOs, 1792839680 bytes 2023-11-08T05:25:29.449 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3900 IOs, 1848536064 bytes 2023-11-08T05:25:29.597 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4000 IOs, 1857570816 bytes 2023-11-08T05:25:29.761 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4100 IOs, 1882462208 bytes 2023-11-08T05:25:29.886 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4200 IOs, 1899926528 bytes 2023-11-08T05:25:30.104 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4300 IOs, 1948745728 bytes 2023-11-08T05:25:30.335 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4400 IOs, 2008571904 bytes 2023-11-08T05:25:30.490 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4500 IOs, 2029091840 bytes 2023-11-08T05:25:30.679 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4600 IOs, 2087838720 bytes 2023-11-08T05:25:30.906 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4700 IOs, 2139897856 bytes 2023-11-08T05:25:31.098 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4800 IOs, 2174895104 bytes 2023-11-08T05:25:31.217 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4900 IOs, 2181453824 bytes 2023-11-08T05:25:31.352 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5000 IOs, 2198340608 bytes 2023-11-08T05:25:31.548 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5100 IOs, 2247631872 bytes 2023-11-08T05:25:31.848 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5200 IOs, 2346177536 bytes 2023-11-08T05:25:31.999 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5300 IOs, 2378531840 bytes 2023-11-08T05:25:32.171 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5400 IOs, 2413443072 bytes 2023-11-08T05:25:32.344 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5500 IOs, 2447622144 bytes 2023-11-08T05:25:32.468 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5600 IOs, 2454855680 bytes 2023-11-08T05:25:32.581 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5700 IOs, 2455459840 bytes 2023-11-08T05:25:32.740 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5800 IOs, 2485443584 bytes 2023-11-08T05:25:32.860 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5900 IOs, 2486075392 bytes 2023-11-08T05:25:33.085 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:6000 IOs, 2547077120 bytes 2023-11-08T05:25:33.270 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:6100 IOs, 2588964864 bytes 2023-11-08T05:25:33.512 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:6200 IOs, 2669850624 bytes 2023-11-08T05:25:33.792 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:6300 IOs, 2745686016 bytes 2023-11-08T05:25:33.941 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:6400 IOs, 2759651328 bytes 2023-11-08T05:25:34.147 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:6500 IOs, 2831763456 bytes 2023-11-08T05:25:34.423 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:6600 IOs, 2913260544 bytes 2023-11-08T05:25:34.592 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:6700 IOs, 2926206976 bytes 2023-11-08T05:25:34.778 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:6800 IOs, 2972622848 bytes 2023-11-08T05:25:35.033 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:6900 IOs, 3051919360 bytes 2023-11-08T05:25:35.286 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:7000 IOs, 3105130496 bytes 2023-11-08T05:25:35.406 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ error_code=124 2023-11-08T05:25:35.407 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ set -e 2023-11-08T05:25:35.407 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' 124 -eq 124 ']' 2023-11-08T05:25:35.407 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:25:35.407 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_image_present cluster1 mirror image_5 present 2023-11-08T05:25:35.407 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:25:35.408 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:25:35.408 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_5 2023-11-08T05:25:35.408 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local state=present 2023-11-08T05:25:35.408 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image_id= 2023-11-08T05:25:35.408 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local s 2023-11-08T05:25:35.409 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test -n '' 2023-11-08T05:25:35.409 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ get_image_id cluster1 mirror image_5 2023-11-08T05:25:35.409 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1 2023-11-08T05:25:35.409 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local pool=mirror 2023-11-08T05:25:35.409 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local image=image_5 2023-11-08T05:25:35.409 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ rbd --cluster=cluster1 info mirror/image_5 2023-11-08T05:25:35.410 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-11-08T05:25:35.444 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image_id=15613b4e7177 2023-11-08T05:25:35.444 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-11-08T05:25:35.444 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0.1 2023-11-08T05:25:35.546 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test_image_present cluster1 mirror image_5 present 15613b4e7177 2023-11-08T05:25:35.546 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:25:35.546 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:25:35.547 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_5 2023-11-08T05:25:35.547 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local test_state=present 2023-11-08T05:25:35.547 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image_id=15613b4e7177 2023-11-08T05:25:35.547 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local current_state=deleted 2023-11-08T05:25:35.547 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local current_image_id 2023-11-08T05:25:35.548 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ get_image_id cluster1 mirror image_5 2023-11-08T05:25:35.548 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1 2023-11-08T05:25:35.548 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local pool=mirror 2023-11-08T05:25:35.548 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local image=image_5 2023-11-08T05:25:35.548 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ rbd --cluster=cluster1 info mirror/image_5 2023-11-08T05:25:35.549 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-11-08T05:25:35.582 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ current_image_id=15613b4e7177 2023-11-08T05:25:35.582 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test -n 15613b4e7177 2023-11-08T05:25:35.583 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test -z 15613b4e7177 -o 15613b4e7177 = 15613b4e7177 2023-11-08T05:25:35.583 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ current_state=present 2023-11-08T05:25:35.583 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test present = present 2023-11-08T05:25:35.583 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:25:35.583 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ disable_mirror cluster2 mirror image_5 2023-11-08T05:25:35.584 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:25:35.584 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:25:35.584 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_5 2023-11-08T05:25:35.584 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_5 2023-11-08T05:25:36.529 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi087.stderr:2023-11-08T05:25:36.527+0000 7fc158a67700 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x562591804c00 scan_remote_mirror_snapshots: failed to locate remote start snapshot: snap_id=376 2023-11-08T05:25:36.529 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi087.stderr:2023-11-08T05:25:36.527+0000 7fc158a67700 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x562591804c00 scan_remote_mirror_snapshots: split-brain detected: failed to find matching non-primary snapshot in remote image: local_snap_id_start=376, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=06977a9b-46f8-44a8-aea5-8992e82c582a, primary_snap_id=178, last_copied_object_number=128, snap_seqs={376=18446744073709551614}] 2023-11-08T05:25:37.283 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi087.stderr:2023-11-08T05:25:37.281+0000 7fc449df3700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55eab62e3520 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-11-08T05:25:37.284 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi087.stderr:2023-11-08T05:25:37.281+0000 7fc44fdff700 -1 rbd::mirror::ImageReplayer: 0x55eab1f49680 [3/042be858-f38c-4af0-9890-7210a8160fd5] operator(): start failed: (85) Interrupted system call should be restarted 2023-11-08T05:25:41.532 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:Mirroring disabled 2023-11-08T05:25:41.539 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_image_present cluster1 mirror image_5 deleted 2023-11-08T05:25:41.539 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:25:41.539 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:25:41.540 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_5 2023-11-08T05:25:41.540 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local state=deleted 2023-11-08T05:25:41.540 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image_id= 2023-11-08T05:25:41.540 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local s 2023-11-08T05:25:41.540 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test -n '' 2023-11-08T05:25:41.541 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ get_image_id cluster1 mirror image_5 2023-11-08T05:25:41.541 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1 2023-11-08T05:25:41.542 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local pool=mirror 2023-11-08T05:25:41.542 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local image=image_5 2023-11-08T05:25:41.542 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ rbd --cluster=cluster1 info mirror/image_5 2023-11-08T05:25:41.542 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-11-08T05:25:41.572 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd: error opening image image_5: (2) No such file or directory 2023-11-08T05:25:41.575 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image_id= 2023-11-08T05:25:41.575 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-11-08T05:25:41.576 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0.1 2023-11-08T05:25:41.677 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test_image_present cluster1 mirror image_5 deleted '' 2023-11-08T05:25:41.677 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:25:41.677 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:25:41.678 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_5 2023-11-08T05:25:41.678 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local test_state=deleted 2023-11-08T05:25:41.678 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image_id= 2023-11-08T05:25:41.678 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local current_state=deleted 2023-11-08T05:25:41.678 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local current_image_id 2023-11-08T05:25:41.679 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ get_image_id cluster1 mirror image_5 2023-11-08T05:25:41.679 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1 2023-11-08T05:25:41.679 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local pool=mirror 2023-11-08T05:25:41.679 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local image=image_5 2023-11-08T05:25:41.679 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ rbd --cluster=cluster1 info mirror/image_5 2023-11-08T05:25:41.680 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-11-08T05:25:41.704 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd: error opening image image_5: (2) No such file or directory 2023-11-08T05:25:41.708 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ current_image_id= 2023-11-08T05:25:41.708 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test -n '' 2023-11-08T05:25:41.708 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test deleted = deleted 2023-11-08T05:25:41.708 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:25:41.709 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ purge_snapshots cluster2 mirror image_5 2023-11-08T05:25:41.709 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:25:41.709 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:25:41.709 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_5 2023-11-08T05:25:41.709 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 snap purge mirror/image_5 2023-11-08T05:25:41.746 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ remove_image_retry cluster2 mirror image_5 2023-11-08T05:25:41.746 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:25:41.746 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:25:41.747 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_5 2023-11-08T05:25:41.747 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0 1 2 4 8 16 32 2023-11-08T05:25:41.747 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0 2023-11-08T05:25:41.748 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ remove_image cluster2 mirror image_5 2023-11-08T05:25:41.748 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:25:41.748 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:25:41.749 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_5 2023-11-08T05:25:41.749 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_5 2023-11-08T05:25:41.785 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster=cluster2 rm mirror/image_5 2023-11-08T05:26:04.582 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2023-11-08T05:26:04.585 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:26:04.585 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for i in `seq 1 10` 2023-11-08T05:26:04.585 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image=image_6 2023-11-08T05:26:04.586 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ create_image cluster2 mirror image_6 512M 2023-11-08T05:26:04.586 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:26:04.586 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:26:04.586 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:26:04.586 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:26:04.587 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_6 2023-11-08T05:26:04.587 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:26:04.587 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local size=128 2023-11-08T05:26:04.587 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' -n 512M ']' 2023-11-08T05:26:04.587 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ size=512M 2023-11-08T05:26:04.588 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:26:04.588 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock mirror/image_6 2023-11-08T05:26:04.625 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ enable_mirror cluster2 mirror image_6 2023-11-08T05:26:04.625 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:26:04.626 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:26:04.626 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_6 2023-11-08T05:26:04.626 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local mode=snapshot 2023-11-08T05:26:04.626 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_6 snapshot 2023-11-08T05:26:05.576 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:Mirroring enabled 2023-11-08T05:26:05.584 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster=cluster2 info mirror/image_6 2023-11-08T05:26:05.616 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:rbd image 'image_6': 2023-11-08T05:26:05.617 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: size 512 MiB in 128 objects 2023-11-08T05:26:05.617 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: order 22 (4 MiB objects) 2023-11-08T05:26:05.617 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: snapshot_count: 1 2023-11-08T05:26:05.617 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: id: 198330f301ff 2023-11-08T05:26:05.618 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: block_name_prefix: rbd_data.198330f301ff 2023-11-08T05:26:05.618 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: format: 2 2023-11-08T05:26:05.618 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: features: layering, exclusive-lock 2023-11-08T05:26:05.618 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: op_features: 2023-11-08T05:26:05.618 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: flags: 2023-11-08T05:26:05.619 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: create_timestamp: Wed Nov 8 05:26:04 2023 2023-11-08T05:26:05.619 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: access_timestamp: Wed Nov 8 05:26:04 2023 2023-11-08T05:26:05.619 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: modify_timestamp: Wed Nov 8 05:26:04 2023 2023-11-08T05:26:05.619 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring state: enabled 2023-11-08T05:26:05.619 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring mode: snapshot 2023-11-08T05:26:05.620 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring global id: 34fbf3f7-7dba-470c-a7cb-63a0465b5652 2023-11-08T05:26:05.620 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring primary: true 2023-11-08T05:26:05.621 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ stress_write_image cluster2 mirror image_6 2023-11-08T05:26:05.621 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:26:05.621 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:26:05.621 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_6 2023-11-08T05:26:05.622 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-11-08T05:26:05.622 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local duration=8 2023-11-08T05:26:05.622 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ set +e 2023-11-08T05:26:05.622 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_6 --debug-rbd=20 --debug-journaler=20 2023-11-08T05:26:05.775 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:100 IOs, 7827456 bytes 2023-11-08T05:26:05.929 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:200 IOs, 36504576 bytes 2023-11-08T05:26:06.096 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:300 IOs, 71480320 bytes 2023-11-08T05:26:06.300 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:400 IOs, 127457280 bytes 2023-11-08T05:26:06.522 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:500 IOs, 208736256 bytes 2023-11-08T05:26:06.707 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:600 IOs, 235123712 bytes 2023-11-08T05:26:06.871 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:700 IOs, 278283264 bytes 2023-11-08T05:26:07.077 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:800 IOs, 328012800 bytes 2023-11-08T05:26:07.255 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:900 IOs, 364140544 bytes 2023-11-08T05:26:07.411 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1000 IOs, 407879680 bytes 2023-11-08T05:26:07.664 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1100 IOs, 466635776 bytes 2023-11-08T05:26:07.767 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1200 IOs, 467250176 bytes 2023-11-08T05:26:07.922 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1300 IOs, 495651840 bytes 2023-11-08T05:26:08.106 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1400 IOs, 537545728 bytes 2023-11-08T05:26:08.280 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1500 IOs, 590181376 bytes 2023-11-08T05:26:08.650 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1600 IOs, 716020736 bytes 2023-11-08T05:26:08.871 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1700 IOs, 771763200 bytes 2023-11-08T05:26:09.111 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1800 IOs, 837565440 bytes 2023-11-08T05:26:09.298 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1900 IOs, 889611264 bytes 2023-11-08T05:26:09.432 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2000 IOs, 903334912 bytes 2023-11-08T05:26:09.586 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2100 IOs, 929317888 bytes 2023-11-08T05:26:09.705 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2200 IOs, 960357376 bytes 2023-11-08T05:26:09.928 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2300 IOs, 1007852544 bytes 2023-11-08T05:26:10.094 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2400 IOs, 1038666752 bytes 2023-11-08T05:26:10.358 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2500 IOs, 1116265472 bytes 2023-11-08T05:26:10.501 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2600 IOs, 1135790080 bytes 2023-11-08T05:26:10.669 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2700 IOs, 1175808000 bytes 2023-11-08T05:26:10.963 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2800 IOs, 1262744576 bytes 2023-11-08T05:26:11.104 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2900 IOs, 1291346944 bytes 2023-11-08T05:26:11.378 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3000 IOs, 1369445376 bytes 2023-11-08T05:26:11.613 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3100 IOs, 1432750080 bytes 2023-11-08T05:26:11.842 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3200 IOs, 1497753600 bytes 2023-11-08T05:26:12.003 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3300 IOs, 1519941632 bytes 2023-11-08T05:26:12.214 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3400 IOs, 1568031744 bytes 2023-11-08T05:26:12.374 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3500 IOs, 1603919872 bytes 2023-11-08T05:26:12.747 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3600 IOs, 1727602688 bytes 2023-11-08T05:26:12.931 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3700 IOs, 1755826176 bytes 2023-11-08T05:26:13.122 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3800 IOs, 1804428288 bytes 2023-11-08T05:26:13.365 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3900 IOs, 1884780544 bytes 2023-11-08T05:26:13.567 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4000 IOs, 1936301056 bytes 2023-11-08T05:26:13.630 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ error_code=124 2023-11-08T05:26:13.630 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ set -e 2023-11-08T05:26:13.630 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' 124 -eq 124 ']' 2023-11-08T05:26:13.630 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:26:13.630 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_image_present cluster1 mirror image_6 present 2023-11-08T05:26:13.631 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:26:13.631 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:26:13.631 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_6 2023-11-08T05:26:13.631 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local state=present 2023-11-08T05:26:13.631 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image_id= 2023-11-08T05:26:13.632 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local s 2023-11-08T05:26:13.632 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test -n '' 2023-11-08T05:26:13.632 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ get_image_id cluster1 mirror image_6 2023-11-08T05:26:13.632 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1 2023-11-08T05:26:13.632 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local pool=mirror 2023-11-08T05:26:13.632 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local image=image_6 2023-11-08T05:26:13.633 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ rbd --cluster=cluster1 info mirror/image_6 2023-11-08T05:26:13.633 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-11-08T05:26:13.668 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image_id=1561ae44da09 2023-11-08T05:26:13.669 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-11-08T05:26:13.669 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0.1 2023-11-08T05:26:13.770 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test_image_present cluster1 mirror image_6 present 1561ae44da09 2023-11-08T05:26:13.771 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:26:13.771 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:26:13.771 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_6 2023-11-08T05:26:13.771 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local test_state=present 2023-11-08T05:26:13.771 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image_id=1561ae44da09 2023-11-08T05:26:13.772 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local current_state=deleted 2023-11-08T05:26:13.772 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local current_image_id 2023-11-08T05:26:13.772 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ get_image_id cluster1 mirror image_6 2023-11-08T05:26:13.772 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1 2023-11-08T05:26:13.772 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local pool=mirror 2023-11-08T05:26:13.773 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local image=image_6 2023-11-08T05:26:13.773 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ rbd --cluster=cluster1 info mirror/image_6 2023-11-08T05:26:13.773 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-11-08T05:26:13.807 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ current_image_id=1561ae44da09 2023-11-08T05:26:13.807 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test -n 1561ae44da09 2023-11-08T05:26:13.808 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test -z 1561ae44da09 -o 1561ae44da09 = 1561ae44da09 2023-11-08T05:26:13.808 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ current_state=present 2023-11-08T05:26:13.808 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test present = present 2023-11-08T05:26:13.809 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:26:13.809 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ disable_mirror cluster2 mirror image_6 2023-11-08T05:26:13.809 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:26:13.810 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:26:13.810 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_6 2023-11-08T05:26:13.810 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_6 2023-11-08T05:26:14.615 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi087.stderr:2023-11-08T05:26:14.613+0000 7fc158a67700 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x56259199b000 scan_remote_mirror_snapshots: failed to locate remote start snapshot: snap_id=378 2023-11-08T05:26:14.616 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi087.stderr:2023-11-08T05:26:14.613+0000 7fc158a67700 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x56259199b000 scan_remote_mirror_snapshots: split-brain detected: failed to find matching non-primary snapshot in remote image: local_snap_id_start=378, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=06977a9b-46f8-44a8-aea5-8992e82c582a, primary_snap_id=17a, last_copied_object_number=128, snap_seqs={378=18446744073709551614}] 2023-11-08T05:26:18.350 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.0 2023-11-08T05:26:18.351 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Sent signal 15 2023-11-08T05:26:18.352 INFO:tasks.rbd_mirror_thrash:waiting for 3 secs before reviving daemons 2023-11-08T05:26:18.353 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi087.stderr:2023-11-08T05:26:18.350+0000 7fb87ab65700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 153341) UID: 1000 2023-11-08T05:26:18.552 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi087.stderr:daemon-helper: command failed with exit status 1 2023-11-08T05:26:19.617 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:Mirroring disabled 2023-11-08T05:26:19.625 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_image_present cluster1 mirror image_6 deleted 2023-11-08T05:26:19.625 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:26:19.626 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:26:19.626 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_6 2023-11-08T05:26:19.626 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local state=deleted 2023-11-08T05:26:19.626 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image_id= 2023-11-08T05:26:19.626 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local s 2023-11-08T05:26:19.627 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test -n '' 2023-11-08T05:26:19.627 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ get_image_id cluster1 mirror image_6 2023-11-08T05:26:19.627 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1 2023-11-08T05:26:19.627 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local pool=mirror 2023-11-08T05:26:19.627 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local image=image_6 2023-11-08T05:26:19.628 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ rbd --cluster=cluster1 info mirror/image_6 2023-11-08T05:26:19.628 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-11-08T05:26:19.656 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd: error opening image image_6: (2) No such file or directory 2023-11-08T05:26:19.659 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image_id= 2023-11-08T05:26:19.659 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-11-08T05:26:19.660 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0.1 2023-11-08T05:26:19.761 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test_image_present cluster1 mirror image_6 deleted '' 2023-11-08T05:26:19.762 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:26:19.762 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:26:19.762 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_6 2023-11-08T05:26:19.762 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local test_state=deleted 2023-11-08T05:26:19.763 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image_id= 2023-11-08T05:26:19.763 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local current_state=deleted 2023-11-08T05:26:19.763 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local current_image_id 2023-11-08T05:26:19.763 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ get_image_id cluster1 mirror image_6 2023-11-08T05:26:19.763 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1 2023-11-08T05:26:19.764 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local pool=mirror 2023-11-08T05:26:19.764 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local image=image_6 2023-11-08T05:26:19.764 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ rbd --cluster=cluster1 info mirror/image_6 2023-11-08T05:26:19.764 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-11-08T05:26:19.791 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd: error opening image image_6: (2) No such file or directory 2023-11-08T05:26:19.794 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ current_image_id= 2023-11-08T05:26:19.794 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test -n '' 2023-11-08T05:26:19.795 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test deleted = deleted 2023-11-08T05:26:19.795 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:26:19.795 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ purge_snapshots cluster2 mirror image_6 2023-11-08T05:26:19.795 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:26:19.796 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:26:19.796 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_6 2023-11-08T05:26:19.796 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 snap purge mirror/image_6 2023-11-08T05:26:19.834 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ remove_image_retry cluster2 mirror image_6 2023-11-08T05:26:19.835 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:26:19.835 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:26:19.835 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_6 2023-11-08T05:26:19.835 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0 1 2 4 8 16 32 2023-11-08T05:26:19.835 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0 2023-11-08T05:26:19.836 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ remove_image cluster2 mirror image_6 2023-11-08T05:26:19.836 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:26:19.837 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:26:19.837 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_6 2023-11-08T05:26:19.837 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_6 2023-11-08T05:26:19.873 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster=cluster2 rm mirror/image_6 2023-11-08T05:26:21.353 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.0 2023-11-08T05:26:21.353 INFO:teuthology.orchestra.run:waiting for 600 2023-11-08T05:26:21.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-08T05:26:21.354 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.0 2023-11-08T05:26:21.355 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2023-11-08T05:26:21.355 DEBUG:teuthology.orchestra.run.smithi087:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2023-11-08T05:26:21.358 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2023-11-08T05:26:21.359 INFO:tasks.rbd_mirror_thrash:waiting for 70 secs before thrashing 2023-11-08T05:26:42.894 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2023-11-08T05:26:42.898 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:26:42.898 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for i in `seq 1 10` 2023-11-08T05:26:42.898 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image=image_7 2023-11-08T05:26:42.898 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ create_image cluster2 mirror image_7 512M 2023-11-08T05:26:42.899 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:26:42.899 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:26:42.899 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:26:42.899 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:26:42.899 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_7 2023-11-08T05:26:42.900 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:26:42.900 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local size=128 2023-11-08T05:26:42.900 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' -n 512M ']' 2023-11-08T05:26:42.900 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ size=512M 2023-11-08T05:26:42.900 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:26:42.901 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock mirror/image_7 2023-11-08T05:26:42.938 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ enable_mirror cluster2 mirror image_7 2023-11-08T05:26:42.938 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:26:42.938 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:26:42.939 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_7 2023-11-08T05:26:42.939 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local mode=snapshot 2023-11-08T05:26:42.939 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_7 snapshot 2023-11-08T05:26:43.662 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:Mirroring enabled 2023-11-08T05:26:43.669 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster=cluster2 info mirror/image_7 2023-11-08T05:26:43.701 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:rbd image 'image_7': 2023-11-08T05:26:43.701 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: size 512 MiB in 128 objects 2023-11-08T05:26:43.702 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: order 22 (4 MiB objects) 2023-11-08T05:26:43.702 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: snapshot_count: 1 2023-11-08T05:26:43.702 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: id: 199f47a0c10b 2023-11-08T05:26:43.702 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: block_name_prefix: rbd_data.199f47a0c10b 2023-11-08T05:26:43.703 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: format: 2 2023-11-08T05:26:43.703 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: features: layering, exclusive-lock 2023-11-08T05:26:43.703 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: op_features: 2023-11-08T05:26:43.703 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: flags: 2023-11-08T05:26:43.704 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: create_timestamp: Wed Nov 8 05:26:42 2023 2023-11-08T05:26:43.704 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: access_timestamp: Wed Nov 8 05:26:42 2023 2023-11-08T05:26:43.704 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: modify_timestamp: Wed Nov 8 05:26:42 2023 2023-11-08T05:26:43.704 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring state: enabled 2023-11-08T05:26:43.704 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring mode: snapshot 2023-11-08T05:26:43.705 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring global id: a2425254-f9ac-47f0-9224-589d0c3c89c4 2023-11-08T05:26:43.705 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring primary: true 2023-11-08T05:26:43.706 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ stress_write_image cluster2 mirror image_7 2023-11-08T05:26:43.706 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:26:43.706 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:26:43.707 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_7 2023-11-08T05:26:43.707 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-11-08T05:26:43.707 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local duration=11 2023-11-08T05:26:43.707 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ set +e 2023-11-08T05:26:43.707 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_7 --debug-rbd=20 --debug-journaler=20 2023-11-08T05:26:43.965 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:100 IOs, 68569088 bytes 2023-11-08T05:26:44.098 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:200 IOs, 81356800 bytes 2023-11-08T05:26:44.206 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:300 IOs, 85482496 bytes 2023-11-08T05:26:44.402 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:400 IOs, 138300416 bytes 2023-11-08T05:26:44.568 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:500 IOs, 171385856 bytes 2023-11-08T05:26:44.880 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:600 IOs, 275082240 bytes 2023-11-08T05:26:45.061 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:700 IOs, 313536512 bytes 2023-11-08T05:26:45.196 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:800 IOs, 320989184 bytes 2023-11-08T05:26:45.396 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:900 IOs, 371417088 bytes 2023-11-08T05:26:45.537 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1000 IOs, 390383616 bytes 2023-11-08T05:26:45.735 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1100 IOs, 435739648 bytes 2023-11-08T05:26:45.908 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1200 IOs, 483902464 bytes 2023-11-08T05:26:46.064 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1300 IOs, 497907712 bytes 2023-11-08T05:26:46.281 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1400 IOs, 564775936 bytes 2023-11-08T05:26:46.564 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1500 IOs, 667390976 bytes 2023-11-08T05:26:46.788 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1600 IOs, 724904960 bytes 2023-11-08T05:26:46.955 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1700 IOs, 758285312 bytes 2023-11-08T05:26:47.189 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1800 IOs, 827500544 bytes 2023-11-08T05:26:47.372 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1900 IOs, 888080384 bytes 2023-11-08T05:26:47.632 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2000 IOs, 948234240 bytes 2023-11-08T05:26:47.860 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2100 IOs, 1007316992 bytes 2023-11-08T05:26:48.129 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2200 IOs, 1087118336 bytes 2023-11-08T05:26:48.340 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2300 IOs, 1144066048 bytes 2023-11-08T05:26:48.473 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2400 IOs, 1156367360 bytes 2023-11-08T05:26:48.588 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2500 IOs, 1166408704 bytes 2023-11-08T05:26:48.841 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2600 IOs, 1237114880 bytes 2023-11-08T05:26:49.023 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2700 IOs, 1283386368 bytes 2023-11-08T05:26:49.141 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2800 IOs, 1284004864 bytes 2023-11-08T05:26:49.391 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2900 IOs, 1349480448 bytes 2023-11-08T05:26:49.545 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3000 IOs, 1370630144 bytes 2023-11-08T05:26:49.776 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3100 IOs, 1458131968 bytes 2023-11-08T05:26:50.014 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3200 IOs, 1523835904 bytes 2023-11-08T05:26:50.233 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3300 IOs, 1561077760 bytes 2023-11-08T05:26:50.355 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3400 IOs, 1563816960 bytes 2023-11-08T05:26:50.572 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3500 IOs, 1620702208 bytes 2023-11-08T05:26:50.933 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3600 IOs, 1743859712 bytes 2023-11-08T05:26:51.269 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3700 IOs, 1849214976 bytes 2023-11-08T05:26:51.379 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3800 IOs, 1849848832 bytes 2023-11-08T05:26:51.548 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3900 IOs, 1879730176 bytes 2023-11-08T05:26:51.784 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4000 IOs, 1942913024 bytes 2023-11-08T05:26:51.963 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4100 IOs, 1980139520 bytes 2023-11-08T05:26:52.194 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4200 IOs, 2052888576 bytes 2023-11-08T05:26:52.324 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4300 IOs, 2066518016 bytes 2023-11-08T05:26:52.465 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4400 IOs, 2082610176 bytes 2023-11-08T05:26:52.660 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4500 IOs, 2122604544 bytes 2023-11-08T05:26:52.832 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4600 IOs, 2156601344 bytes 2023-11-08T05:26:52.999 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4700 IOs, 2202728448 bytes 2023-11-08T05:26:53.197 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4800 IOs, 2235086848 bytes 2023-11-08T05:26:53.400 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4900 IOs, 2281988096 bytes 2023-11-08T05:26:53.553 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5000 IOs, 2300989440 bytes 2023-11-08T05:26:53.664 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5100 IOs, 2301637632 bytes 2023-11-08T05:26:53.819 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5200 IOs, 2329409536 bytes 2023-11-08T05:26:53.982 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5300 IOs, 2364273664 bytes 2023-11-08T05:26:54.185 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5400 IOs, 2412368896 bytes 2023-11-08T05:26:54.354 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5500 IOs, 2426514432 bytes 2023-11-08T05:26:54.551 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5600 IOs, 2473080832 bytes 2023-11-08T05:26:54.679 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5700 IOs, 2473871360 bytes 2023-11-08T05:26:54.710 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ error_code=124 2023-11-08T05:26:54.711 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ set -e 2023-11-08T05:26:54.711 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' 124 -eq 124 ']' 2023-11-08T05:26:54.711 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:26:54.711 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_image_present cluster1 mirror image_7 present 2023-11-08T05:26:54.712 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:26:54.712 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:26:54.712 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_7 2023-11-08T05:26:54.712 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local state=present 2023-11-08T05:26:54.712 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image_id= 2023-11-08T05:26:54.712 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local s 2023-11-08T05:26:54.713 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test -n '' 2023-11-08T05:26:54.713 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ get_image_id cluster1 mirror image_7 2023-11-08T05:26:54.713 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1 2023-11-08T05:26:54.713 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local pool=mirror 2023-11-08T05:26:54.713 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local image=image_7 2023-11-08T05:26:54.714 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ rbd --cluster=cluster1 info mirror/image_7 2023-11-08T05:26:54.714 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-11-08T05:26:54.748 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image_id=156169f35e34 2023-11-08T05:26:54.749 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-11-08T05:26:54.749 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0.1 2023-11-08T05:26:54.850 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test_image_present cluster1 mirror image_7 present 156169f35e34 2023-11-08T05:26:54.850 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:26:54.851 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:26:54.851 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_7 2023-11-08T05:26:54.851 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local test_state=present 2023-11-08T05:26:54.851 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image_id=156169f35e34 2023-11-08T05:26:54.851 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local current_state=deleted 2023-11-08T05:26:54.851 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local current_image_id 2023-11-08T05:26:54.852 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ get_image_id cluster1 mirror image_7 2023-11-08T05:26:54.852 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1 2023-11-08T05:26:54.852 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local pool=mirror 2023-11-08T05:26:54.852 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local image=image_7 2023-11-08T05:26:54.852 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ rbd --cluster=cluster1 info mirror/image_7 2023-11-08T05:26:54.853 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-11-08T05:26:54.887 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ current_image_id=156169f35e34 2023-11-08T05:26:54.887 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test -n 156169f35e34 2023-11-08T05:26:54.888 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test -z 156169f35e34 -o 156169f35e34 = 156169f35e34 2023-11-08T05:26:54.888 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ current_state=present 2023-11-08T05:26:54.888 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test present = present 2023-11-08T05:26:54.888 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:26:54.888 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ disable_mirror cluster2 mirror image_7 2023-11-08T05:26:54.888 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:26:54.889 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:26:54.889 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_7 2023-11-08T05:26:54.889 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_7 2023-11-08T05:26:55.772 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi087.stderr:2023-11-08T05:26:55.770+0000 7fc158a67700 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x562591805400 scan_remote_mirror_snapshots: failed to locate remote start snapshot: snap_id=380 2023-11-08T05:26:55.772 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi087.stderr:2023-11-08T05:26:55.770+0000 7fc158a67700 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x562591805400 scan_remote_mirror_snapshots: split-brain detected: failed to find matching non-primary snapshot in remote image: local_snap_id_start=380, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=06977a9b-46f8-44a8-aea5-8992e82c582a, primary_snap_id=17c, last_copied_object_number=128, snap_seqs={380=18446744073709551614}] 2023-11-08T05:26:57.135 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.8425776765471634) > weight (0.25) 2023-11-08T05:26:57.135 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.1 2023-11-08T05:26:57.136 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Sent signal 15 2023-11-08T05:26:57.137 INFO:tasks.rbd_mirror_thrash:waiting for 5 secs before reviving daemons 2023-11-08T05:26:57.137 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi087.stderr:2023-11-08T05:26:57.136+0000 7f45ab9f8700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 (PID: 147938) UID: 1000 2023-11-08T05:26:57.338 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi087.stderr:daemon-helper: command failed with exit status 1 2023-11-08T05:26:58.402 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi087.stderr:2023-11-08T05:26:58.400+0000 7fc4495f2700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55eab634dba0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-11-08T05:26:58.402 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi087.stderr:2023-11-08T05:26:58.400+0000 7fc450600700 -1 rbd::mirror::ImageReplayer: 0x55eab1f49680 [3/a2425254-f9ac-47f0-9224-589d0c3c89c4] operator(): start failed: (85) Interrupted system call should be restarted 2023-11-08T05:27:00.775 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:Mirroring disabled 2023-11-08T05:27:00.784 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_image_present cluster1 mirror image_7 deleted 2023-11-08T05:27:00.784 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:27:00.785 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:27:00.785 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_7 2023-11-08T05:27:00.785 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local state=deleted 2023-11-08T05:27:00.785 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image_id= 2023-11-08T05:27:00.785 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local s 2023-11-08T05:27:00.786 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test -n '' 2023-11-08T05:27:00.786 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ get_image_id cluster1 mirror image_7 2023-11-08T05:27:00.786 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1 2023-11-08T05:27:00.786 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local pool=mirror 2023-11-08T05:27:00.787 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local image=image_7 2023-11-08T05:27:00.787 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ rbd --cluster=cluster1 info mirror/image_7 2023-11-08T05:27:00.787 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-11-08T05:27:00.814 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd: error opening image image_7: (2) No such file or directory 2023-11-08T05:27:00.816 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image_id= 2023-11-08T05:27:00.817 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-11-08T05:27:00.817 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0.1 2023-11-08T05:27:00.919 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test_image_present cluster1 mirror image_7 deleted '' 2023-11-08T05:27:00.919 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:27:00.919 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:27:00.919 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_7 2023-11-08T05:27:00.920 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local test_state=deleted 2023-11-08T05:27:00.920 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image_id= 2023-11-08T05:27:00.920 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local current_state=deleted 2023-11-08T05:27:00.920 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local current_image_id 2023-11-08T05:27:00.920 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ get_image_id cluster1 mirror image_7 2023-11-08T05:27:00.920 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1 2023-11-08T05:27:00.921 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local pool=mirror 2023-11-08T05:27:00.921 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local image=image_7 2023-11-08T05:27:00.921 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ rbd --cluster=cluster1 info mirror/image_7 2023-11-08T05:27:00.921 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-11-08T05:27:00.948 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd: error opening image image_7: (2) No such file or directory 2023-11-08T05:27:00.952 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ current_image_id= 2023-11-08T05:27:00.952 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test -n '' 2023-11-08T05:27:00.952 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test deleted = deleted 2023-11-08T05:27:00.952 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:27:00.952 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ purge_snapshots cluster2 mirror image_7 2023-11-08T05:27:00.953 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:27:00.953 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:27:00.953 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_7 2023-11-08T05:27:00.953 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 snap purge mirror/image_7 2023-11-08T05:27:00.992 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ remove_image_retry cluster2 mirror image_7 2023-11-08T05:27:00.992 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:27:00.992 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:27:00.992 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_7 2023-11-08T05:27:00.993 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0 1 2 4 8 16 32 2023-11-08T05:27:00.993 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0 2023-11-08T05:27:00.994 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ remove_image cluster2 mirror image_7 2023-11-08T05:27:00.994 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:27:00.994 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:27:00.994 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_7 2023-11-08T05:27:00.995 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_7 2023-11-08T05:27:01.031 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster=cluster2 rm mirror/image_7 2023-11-08T05:27:02.138 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.1 2023-11-08T05:27:02.138 INFO:teuthology.orchestra.run:waiting for 600 2023-11-08T05:27:02.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-08T05:27:02.139 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.1 2023-11-08T05:27:02.139 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Restarting daemon 2023-11-08T05:27:02.139 DEBUG:teuthology.orchestra.run.smithi087:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 2023-11-08T05:27:02.142 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Started 2023-11-08T05:27:02.143 INFO:tasks.rbd_mirror_thrash:waiting for 72 secs before thrashing 2023-11-08T05:27:24.636 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2023-11-08T05:27:24.639 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:27:24.640 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for i in `seq 1 10` 2023-11-08T05:27:24.640 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image=image_8 2023-11-08T05:27:24.640 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ create_image cluster2 mirror image_8 512M 2023-11-08T05:27:24.640 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:27:24.641 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:27:24.641 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:27:24.641 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:27:24.641 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_8 2023-11-08T05:27:24.642 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:27:24.642 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local size=128 2023-11-08T05:27:24.642 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' -n 512M ']' 2023-11-08T05:27:24.643 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ size=512M 2023-11-08T05:27:24.643 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:27:24.643 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock mirror/image_8 2023-11-08T05:27:24.680 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ enable_mirror cluster2 mirror image_8 2023-11-08T05:27:24.681 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:27:24.681 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:27:24.681 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_8 2023-11-08T05:27:24.681 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local mode=snapshot 2023-11-08T05:27:24.682 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_8 snapshot 2023-11-08T05:27:25.396 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:Mirroring enabled 2023-11-08T05:27:25.404 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster=cluster2 info mirror/image_8 2023-11-08T05:27:25.435 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:rbd image 'image_8': 2023-11-08T05:27:25.435 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: size 512 MiB in 128 objects 2023-11-08T05:27:25.435 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: order 22 (4 MiB objects) 2023-11-08T05:27:25.435 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: snapshot_count: 1 2023-11-08T05:27:25.436 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: id: 19bdf13c9f3c 2023-11-08T05:27:25.436 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: block_name_prefix: rbd_data.19bdf13c9f3c 2023-11-08T05:27:25.436 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: format: 2 2023-11-08T05:27:25.436 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: features: layering, exclusive-lock 2023-11-08T05:27:25.436 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: op_features: 2023-11-08T05:27:25.436 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: flags: 2023-11-08T05:27:25.437 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: create_timestamp: Wed Nov 8 05:27:24 2023 2023-11-08T05:27:25.437 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: access_timestamp: Wed Nov 8 05:27:24 2023 2023-11-08T05:27:25.437 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: modify_timestamp: Wed Nov 8 05:27:24 2023 2023-11-08T05:27:25.437 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring state: enabled 2023-11-08T05:27:25.437 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring mode: snapshot 2023-11-08T05:27:25.438 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring global id: 19f6acdc-5fae-4614-b16e-064d0012c6b2 2023-11-08T05:27:25.438 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring primary: true 2023-11-08T05:27:25.439 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ stress_write_image cluster2 mirror image_8 2023-11-08T05:27:25.439 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:27:25.439 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:27:25.439 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_8 2023-11-08T05:27:25.439 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-11-08T05:27:25.440 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local duration=5 2023-11-08T05:27:25.440 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ set +e 2023-11-08T05:27:25.440 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_8 --debug-rbd=20 --debug-journaler=20 2023-11-08T05:27:25.705 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:100 IOs, 62161920 bytes 2023-11-08T05:27:25.904 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:200 IOs, 114021376 bytes 2023-11-08T05:27:26.094 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:300 IOs, 159972352 bytes 2023-11-08T05:27:26.258 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:400 IOs, 189595648 bytes 2023-11-08T05:27:26.502 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:500 IOs, 261624832 bytes 2023-11-08T05:27:26.779 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:600 IOs, 357896192 bytes 2023-11-08T05:27:26.951 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:700 IOs, 416541696 bytes 2023-11-08T05:27:27.200 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:800 IOs, 471528448 bytes 2023-11-08T05:27:27.431 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:900 IOs, 549102592 bytes 2023-11-08T05:27:27.601 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1000 IOs, 558938112 bytes 2023-11-08T05:27:27.933 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1100 IOs, 672000000 bytes 2023-11-08T05:27:28.108 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1200 IOs, 730506240 bytes 2023-11-08T05:27:28.295 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1300 IOs, 753624064 bytes 2023-11-08T05:27:28.494 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1400 IOs, 802333696 bytes 2023-11-08T05:27:28.657 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1500 IOs, 832218112 bytes 2023-11-08T05:27:28.990 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1600 IOs, 965706752 bytes 2023-11-08T05:27:29.218 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1700 IOs, 1014141952 bytes 2023-11-08T05:27:29.448 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1800 IOs, 1075124224 bytes 2023-11-08T05:27:29.699 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1900 IOs, 1154182144 bytes 2023-11-08T05:27:29.805 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2000 IOs, 1154805760 bytes 2023-11-08T05:27:29.984 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2100 IOs, 1192727552 bytes 2023-11-08T05:27:30.112 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2200 IOs, 1203563520 bytes 2023-11-08T05:27:30.294 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2300 IOs, 1242302464 bytes 2023-11-08T05:27:30.439 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2400 IOs, 1261807616 bytes 2023-11-08T05:27:30.448 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ error_code=124 2023-11-08T05:27:30.448 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ set -e 2023-11-08T05:27:30.448 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' 124 -eq 124 ']' 2023-11-08T05:27:30.448 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:27:30.449 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_image_present cluster1 mirror image_8 present 2023-11-08T05:27:30.449 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:27:30.449 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:27:30.449 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_8 2023-11-08T05:27:30.449 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local state=present 2023-11-08T05:27:30.450 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image_id= 2023-11-08T05:27:30.450 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local s 2023-11-08T05:27:30.450 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test -n '' 2023-11-08T05:27:30.450 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ get_image_id cluster1 mirror image_8 2023-11-08T05:27:30.450 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1 2023-11-08T05:27:30.451 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local pool=mirror 2023-11-08T05:27:30.451 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local image=image_8 2023-11-08T05:27:30.451 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ rbd --cluster=cluster1 info mirror/image_8 2023-11-08T05:27:30.451 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-11-08T05:27:30.482 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image_id=1561c88eb492 2023-11-08T05:27:30.483 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-11-08T05:27:30.483 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0.1 2023-11-08T05:27:30.584 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test_image_present cluster1 mirror image_8 present 1561c88eb492 2023-11-08T05:27:30.584 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:27:30.585 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:27:30.585 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_8 2023-11-08T05:27:30.585 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local test_state=present 2023-11-08T05:27:30.585 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image_id=1561c88eb492 2023-11-08T05:27:30.585 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local current_state=deleted 2023-11-08T05:27:30.586 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local current_image_id 2023-11-08T05:27:30.586 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ get_image_id cluster1 mirror image_8 2023-11-08T05:27:30.586 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1 2023-11-08T05:27:30.586 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local pool=mirror 2023-11-08T05:27:30.586 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local image=image_8 2023-11-08T05:27:30.587 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ rbd --cluster=cluster1 info mirror/image_8 2023-11-08T05:27:30.587 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-11-08T05:27:30.622 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ current_image_id=1561c88eb492 2023-11-08T05:27:30.622 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test -n 1561c88eb492 2023-11-08T05:27:30.622 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test -z 1561c88eb492 -o 1561c88eb492 = 1561c88eb492 2023-11-08T05:27:30.623 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ current_state=present 2023-11-08T05:27:30.623 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test present = present 2023-11-08T05:27:30.623 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:27:30.623 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ disable_mirror cluster2 mirror image_8 2023-11-08T05:27:30.623 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:27:30.624 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:27:30.624 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_8 2023-11-08T05:27:30.624 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_8 2023-11-08T05:27:31.360 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.8779659048652751) > weight (0.25) 2023-11-08T05:27:31.360 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.28229720526584146) > weight (0.25) 2023-11-08T05:27:31.360 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.5956538127628046) > weight (0.25) 2023-11-08T05:27:31.360 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.3 with skip (0.6476775232153841) > weight (0.25) 2023-11-08T05:27:31.361 INFO:tasks.rbd_mirror_thrash:waiting for 89 secs before thrashing 2023-11-08T05:27:31.441 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi087.stderr:2023-11-08T05:27:31.439+0000 7fc158a67700 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x562591c5bc00 scan_remote_mirror_snapshots: failed to locate remote start snapshot: snap_id=382 2023-11-08T05:27:31.441 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi087.stderr:2023-11-08T05:27:31.439+0000 7fc158a67700 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x562591c5bc00 scan_remote_mirror_snapshots: split-brain detected: failed to find matching non-primary snapshot in remote image: local_snap_id_start=382, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=06977a9b-46f8-44a8-aea5-8992e82c582a, primary_snap_id=17e, last_copied_object_number=128, snap_seqs={382=18446744073709551614}] 2023-11-08T05:27:36.440 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:Mirroring disabled 2023-11-08T05:27:36.451 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_image_present cluster1 mirror image_8 deleted 2023-11-08T05:27:36.451 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:27:36.451 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:27:36.451 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_8 2023-11-08T05:27:36.452 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local state=deleted 2023-11-08T05:27:36.452 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image_id= 2023-11-08T05:27:36.452 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local s 2023-11-08T05:27:36.452 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test -n '' 2023-11-08T05:27:36.452 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ get_image_id cluster1 mirror image_8 2023-11-08T05:27:36.453 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1 2023-11-08T05:27:36.453 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local pool=mirror 2023-11-08T05:27:36.453 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local image=image_8 2023-11-08T05:27:36.453 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ rbd --cluster=cluster1 info mirror/image_8 2023-11-08T05:27:36.453 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-11-08T05:27:36.482 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd: error opening image image_8: (2) No such file or directory 2023-11-08T05:27:36.485 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image_id= 2023-11-08T05:27:36.485 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-11-08T05:27:36.486 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0.1 2023-11-08T05:27:36.587 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test_image_present cluster1 mirror image_8 deleted '' 2023-11-08T05:27:36.587 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:27:36.588 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:27:36.588 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_8 2023-11-08T05:27:36.588 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local test_state=deleted 2023-11-08T05:27:36.588 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image_id= 2023-11-08T05:27:36.589 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local current_state=deleted 2023-11-08T05:27:36.589 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local current_image_id 2023-11-08T05:27:36.589 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ get_image_id cluster1 mirror image_8 2023-11-08T05:27:36.589 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1 2023-11-08T05:27:36.590 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local pool=mirror 2023-11-08T05:27:36.590 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local image=image_8 2023-11-08T05:27:36.590 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ rbd --cluster=cluster1 info mirror/image_8 2023-11-08T05:27:36.590 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-11-08T05:27:36.615 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd: error opening image image_8: (2) No such file or directory 2023-11-08T05:27:36.618 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ current_image_id= 2023-11-08T05:27:36.618 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test -n '' 2023-11-08T05:27:36.619 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test deleted = deleted 2023-11-08T05:27:36.619 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:27:36.619 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ purge_snapshots cluster2 mirror image_8 2023-11-08T05:27:36.620 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:27:36.620 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:27:36.620 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_8 2023-11-08T05:27:36.620 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 snap purge mirror/image_8 2023-11-08T05:27:36.658 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ remove_image_retry cluster2 mirror image_8 2023-11-08T05:27:36.658 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:27:36.658 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:27:36.658 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_8 2023-11-08T05:27:36.659 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0 1 2 4 8 16 32 2023-11-08T05:27:36.659 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0 2023-11-08T05:27:36.660 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ remove_image cluster2 mirror image_8 2023-11-08T05:27:36.660 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:27:36.660 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:27:36.660 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_8 2023-11-08T05:27:36.660 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_8 2023-11-08T05:27:36.696 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster=cluster2 rm mirror/image_8 2023-11-08T05:28:00.697 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2023-11-08T05:28:00.700 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:28:00.701 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for i in `seq 1 10` 2023-11-08T05:28:00.701 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image=image_9 2023-11-08T05:28:00.701 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ create_image cluster2 mirror image_9 512M 2023-11-08T05:28:00.701 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:28:00.701 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:28:00.702 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:28:00.702 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:28:00.702 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_9 2023-11-08T05:28:00.702 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:28:00.702 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local size=128 2023-11-08T05:28:00.703 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' -n 512M ']' 2023-11-08T05:28:00.703 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ size=512M 2023-11-08T05:28:00.703 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:28:00.703 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock mirror/image_9 2023-11-08T05:28:00.741 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ enable_mirror cluster2 mirror image_9 2023-11-08T05:28:00.741 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:28:00.741 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:28:00.742 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_9 2023-11-08T05:28:00.742 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local mode=snapshot 2023-11-08T05:28:00.742 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_9 snapshot 2023-11-08T05:28:01.479 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:Mirroring enabled 2023-11-08T05:28:01.487 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster=cluster2 info mirror/image_9 2023-11-08T05:28:01.521 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:rbd image 'image_9': 2023-11-08T05:28:01.521 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: size 512 MiB in 128 objects 2023-11-08T05:28:01.521 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: order 22 (4 MiB objects) 2023-11-08T05:28:01.521 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: snapshot_count: 1 2023-11-08T05:28:01.522 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: id: 19d53a969189 2023-11-08T05:28:01.522 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: block_name_prefix: rbd_data.19d53a969189 2023-11-08T05:28:01.522 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: format: 2 2023-11-08T05:28:01.522 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: features: layering, exclusive-lock 2023-11-08T05:28:01.522 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: op_features: 2023-11-08T05:28:01.523 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: flags: 2023-11-08T05:28:01.523 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: create_timestamp: Wed Nov 8 05:28:00 2023 2023-11-08T05:28:01.523 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: access_timestamp: Wed Nov 8 05:28:00 2023 2023-11-08T05:28:01.523 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: modify_timestamp: Wed Nov 8 05:28:00 2023 2023-11-08T05:28:01.523 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring state: enabled 2023-11-08T05:28:01.524 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring mode: snapshot 2023-11-08T05:28:01.524 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring global id: cb5d0973-ae2a-4623-82f1-7094e06cebe1 2023-11-08T05:28:01.524 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring primary: true 2023-11-08T05:28:01.525 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ stress_write_image cluster2 mirror image_9 2023-11-08T05:28:01.525 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:28:01.525 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:28:01.525 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_9 2023-11-08T05:28:01.526 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-11-08T05:28:01.526 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local duration=11 2023-11-08T05:28:01.526 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ set +e 2023-11-08T05:28:01.527 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_9 --debug-rbd=20 --debug-journaler=20 2023-11-08T05:28:01.821 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:100 IOs, 77030400 bytes 2023-11-08T05:28:02.007 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:200 IOs, 122444800 bytes 2023-11-08T05:28:02.234 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:300 IOs, 180145152 bytes 2023-11-08T05:28:02.389 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:400 IOs, 203951104 bytes 2023-11-08T05:28:02.571 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:500 IOs, 244094976 bytes 2023-11-08T05:28:02.785 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:600 IOs, 302097408 bytes 2023-11-08T05:28:03.042 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:700 IOs, 398135296 bytes 2023-11-08T05:28:03.282 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:800 IOs, 458050560 bytes 2023-11-08T05:28:03.429 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:900 IOs, 483461120 bytes 2023-11-08T05:28:03.680 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1000 IOs, 580517888 bytes 2023-11-08T05:28:03.853 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1100 IOs, 596109312 bytes 2023-11-08T05:28:04.018 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1200 IOs, 641756160 bytes 2023-11-08T05:28:04.201 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1300 IOs, 678140928 bytes 2023-11-08T05:28:04.443 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1400 IOs, 735696896 bytes 2023-11-08T05:28:04.584 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1500 IOs, 754804736 bytes 2023-11-08T05:28:04.690 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1600 IOs, 755430400 bytes 2023-11-08T05:28:04.952 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1700 IOs, 837064704 bytes 2023-11-08T05:28:05.156 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1800 IOs, 881966080 bytes 2023-11-08T05:28:05.382 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1900 IOs, 944785408 bytes 2023-11-08T05:28:05.541 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2000 IOs, 975294464 bytes 2023-11-08T05:28:05.750 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2100 IOs, 1028696064 bytes 2023-11-08T05:28:05.900 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2200 IOs, 1069883392 bytes 2023-11-08T05:28:06.136 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2300 IOs, 1121165312 bytes 2023-11-08T05:28:06.269 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2400 IOs, 1135647744 bytes 2023-11-08T05:28:06.448 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2500 IOs, 1185049600 bytes 2023-11-08T05:28:06.598 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2600 IOs, 1185746944 bytes 2023-11-08T05:28:06.828 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2700 IOs, 1257390080 bytes 2023-11-08T05:28:07.024 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2800 IOs, 1316360192 bytes 2023-11-08T05:28:07.206 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2900 IOs, 1343012864 bytes 2023-11-08T05:28:07.398 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3000 IOs, 1381366784 bytes 2023-11-08T05:28:07.662 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3100 IOs, 1423546368 bytes 2023-11-08T05:28:07.927 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3200 IOs, 1484130304 bytes 2023-11-08T05:28:08.123 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3300 IOs, 1533911040 bytes 2023-11-08T05:28:08.327 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3400 IOs, 1582217216 bytes 2023-11-08T05:28:08.527 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3500 IOs, 1640649728 bytes 2023-11-08T05:28:08.785 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3600 IOs, 1693286400 bytes 2023-11-08T05:28:08.980 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3700 IOs, 1734365184 bytes 2023-11-08T05:28:09.115 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3800 IOs, 1749363712 bytes 2023-11-08T05:28:09.265 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3900 IOs, 1781326848 bytes 2023-11-08T05:28:09.635 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4000 IOs, 1914417152 bytes 2023-11-08T05:28:09.843 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4100 IOs, 1966656512 bytes 2023-11-08T05:28:09.967 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4200 IOs, 1974330368 bytes 2023-11-08T05:28:10.104 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4300 IOs, 1997570048 bytes 2023-11-08T05:28:10.274 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4400 IOs, 2020844544 bytes 2023-11-08T05:28:10.418 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4500 IOs, 2053892096 bytes 2023-11-08T05:28:10.572 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4600 IOs, 2054507520 bytes 2023-11-08T05:28:10.684 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4700 IOs, 2055106560 bytes 2023-11-08T05:28:10.799 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4800 IOs, 2062034944 bytes 2023-11-08T05:28:10.959 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4900 IOs, 2090306560 bytes 2023-11-08T05:28:11.122 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5000 IOs, 2117244928 bytes 2023-11-08T05:28:11.369 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5100 IOs, 2180737024 bytes 2023-11-08T05:28:11.702 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5200 IOs, 2292370432 bytes 2023-11-08T05:28:11.977 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5300 IOs, 2356158464 bytes 2023-11-08T05:28:12.109 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5400 IOs, 2385997824 bytes 2023-11-08T05:28:12.218 INFO:tasks.ceph.cluster2.osd.0.smithi087.stderr:2023-11-08T05:28:12.216+0000 7f1acef4d700 -1 bdev(0x55f9dfd12400 /var/lib/ceph/osd/cluster2-0/block) _aio_thread got r=-28 ((28) No space left on device) 2023-11-08T05:28:12.219 INFO:tasks.ceph.cluster2.osd.0.smithi087.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/16.2.14-129-gcfb6fd52/rpm/el8/BUILD/ceph-16.2.14-129-gcfb6fd52/src/blk/kernel/KernelDevice.cc: In function 'void KernelDevice::_aio_thread()' thread 7f1acef4d700 time 2023-11-08T05:28:12.217518+0000 2023-11-08T05:28:12.220 INFO:tasks.ceph.cluster2.osd.0.smithi087.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/16.2.14-129-gcfb6fd52/rpm/el8/BUILD/ceph-16.2.14-129-gcfb6fd52/src/blk/kernel/KernelDevice.cc: 600: ceph_abort_msg("Unexpected IO error. This may suggest a hardware issue. Please check your kernel log!") 2023-11-08T05:28:12.220 INFO:tasks.ceph.cluster2.osd.0.smithi087.stderr: ceph version 16.2.14-129-gcfb6fd52 (cfb6fd52113be6a0af3fe7fb89b2e08474eadc0f) pacific (stable) 2023-11-08T05:28:12.220 INFO:tasks.ceph.cluster2.osd.0.smithi087.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xe5) [0x55f9d36ae9b4] 2023-11-08T05:28:12.220 INFO:tasks.ceph.cluster2.osd.0.smithi087.stderr: 2: (KernelDevice::_aio_thread()+0x1254) [0x55f9d421c7c4] 2023-11-08T05:28:12.220 INFO:tasks.ceph.cluster2.osd.0.smithi087.stderr: 3: (KernelDevice::AioCompletionThread::entry()+0x11) [0x55f9d4227ac1] 2023-11-08T05:28:12.221 INFO:tasks.ceph.cluster2.osd.0.smithi087.stderr: 4: /lib64/libpthread.so.0(+0x81ca) [0x7f1adadc81ca] 2023-11-08T05:28:12.221 INFO:tasks.ceph.cluster2.osd.0.smithi087.stderr: 5: clone() 2023-11-08T05:28:12.221 INFO:tasks.ceph.cluster2.osd.0.smithi087.stderr:2023-11-08T05:28:12.218+0000 7f1acef4d700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/16.2.14-129-gcfb6fd52/rpm/el8/BUILD/ceph-16.2.14-129-gcfb6fd52/src/blk/kernel/KernelDevice.cc: In function 'void KernelDevice::_aio_thread()' thread 7f1acef4d700 time 2023-11-08T05:28:12.217518+0000 2023-11-08T05:28:12.221 INFO:tasks.ceph.cluster2.osd.0.smithi087.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/16.2.14-129-gcfb6fd52/rpm/el8/BUILD/ceph-16.2.14-129-gcfb6fd52/src/blk/kernel/KernelDevice.cc: 600: ceph_abort_msg("Unexpected IO error. This may suggest a hardware issue. Please check your kernel log!") 2023-11-08T05:28:12.221 INFO:tasks.ceph.cluster2.osd.0.smithi087.stderr: 2023-11-08T05:28:12.222 INFO:tasks.ceph.cluster2.osd.0.smithi087.stderr: ceph version 16.2.14-129-gcfb6fd52 (cfb6fd52113be6a0af3fe7fb89b2e08474eadc0f) pacific (stable) 2023-11-08T05:28:12.222 INFO:tasks.ceph.cluster2.osd.0.smithi087.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xe5) [0x55f9d36ae9b4] 2023-11-08T05:28:12.222 INFO:tasks.ceph.cluster2.osd.0.smithi087.stderr: 2: (KernelDevice::_aio_thread()+0x1254) [0x55f9d421c7c4] 2023-11-08T05:28:12.222 INFO:tasks.ceph.cluster2.osd.0.smithi087.stderr: 3: (KernelDevice::AioCompletionThread::entry()+0x11) [0x55f9d4227ac1] 2023-11-08T05:28:12.222 INFO:tasks.ceph.cluster2.osd.0.smithi087.stderr: 4: /lib64/libpthread.so.0(+0x81ca) [0x7f1adadc81ca] 2023-11-08T05:28:12.223 INFO:tasks.ceph.cluster2.osd.0.smithi087.stderr: 5: clone() 2023-11-08T05:28:12.223 INFO:tasks.ceph.cluster2.osd.0.smithi087.stderr: 2023-11-08T05:28:12.223 INFO:tasks.ceph.cluster2.osd.0.smithi087.stderr:*** Caught signal (Aborted) ** 2023-11-08T05:28:12.223 INFO:tasks.ceph.cluster2.osd.0.smithi087.stderr: in thread 7f1acef4d700 thread_name:bstore_aio 2023-11-08T05:28:12.224 INFO:tasks.ceph.cluster2.osd.0.smithi087.stderr: ceph version 16.2.14-129-gcfb6fd52 (cfb6fd52113be6a0af3fe7fb89b2e08474eadc0f) pacific (stable) 2023-11-08T05:28:12.224 INFO:tasks.ceph.cluster2.osd.0.smithi087.stderr: 1: /lib64/libpthread.so.0(+0x12cf0) [0x7f1adadd2cf0] 2023-11-08T05:28:12.224 INFO:tasks.ceph.cluster2.osd.0.smithi087.stderr: 2: gsignal() 2023-11-08T05:28:12.224 INFO:tasks.ceph.cluster2.osd.0.smithi087.stderr: 3: abort() 2023-11-08T05:28:12.225 INFO:tasks.ceph.cluster2.osd.0.smithi087.stderr: 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x1b6) [0x55f9d36aea85] 2023-11-08T05:28:12.225 INFO:tasks.ceph.cluster2.osd.0.smithi087.stderr: 5: (KernelDevice::_aio_thread()+0x1254) [0x55f9d421c7c4] 2023-11-08T05:28:12.225 INFO:tasks.ceph.cluster2.osd.0.smithi087.stderr: 6: (KernelDevice::AioCompletionThread::entry()+0x11) [0x55f9d4227ac1] 2023-11-08T05:28:12.225 INFO:tasks.ceph.cluster2.osd.0.smithi087.stderr: 7: /lib64/libpthread.so.0(+0x81ca) [0x7f1adadc81ca] 2023-11-08T05:28:12.225 INFO:tasks.ceph.cluster2.osd.0.smithi087.stderr: 8: clone() 2023-11-08T05:28:12.226 INFO:tasks.ceph.cluster2.osd.0.smithi087.stderr:2023-11-08T05:28:12.220+0000 7f1acef4d700 -1 *** Caught signal (Aborted) ** 2023-11-08T05:28:12.226 INFO:tasks.ceph.cluster2.osd.0.smithi087.stderr: in thread 7f1acef4d700 thread_name:bstore_aio 2023-11-08T05:28:12.226 INFO:tasks.ceph.cluster2.osd.0.smithi087.stderr: 2023-11-08T05:28:12.226 INFO:tasks.ceph.cluster2.osd.0.smithi087.stderr: ceph version 16.2.14-129-gcfb6fd52 (cfb6fd52113be6a0af3fe7fb89b2e08474eadc0f) pacific (stable) 2023-11-08T05:28:12.226 INFO:tasks.ceph.cluster2.osd.0.smithi087.stderr: 1: /lib64/libpthread.so.0(+0x12cf0) [0x7f1adadd2cf0] 2023-11-08T05:28:12.227 INFO:tasks.ceph.cluster2.osd.0.smithi087.stderr: 2: gsignal() 2023-11-08T05:28:12.227 INFO:tasks.ceph.cluster2.osd.0.smithi087.stderr: 3: abort() 2023-11-08T05:28:12.227 INFO:tasks.ceph.cluster2.osd.0.smithi087.stderr: 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x1b6) [0x55f9d36aea85] 2023-11-08T05:28:12.227 INFO:tasks.ceph.cluster2.osd.0.smithi087.stderr: 5: (KernelDevice::_aio_thread()+0x1254) [0x55f9d421c7c4] 2023-11-08T05:28:12.227 INFO:tasks.ceph.cluster2.osd.0.smithi087.stderr: 6: (KernelDevice::AioCompletionThread::entry()+0x11) [0x55f9d4227ac1] 2023-11-08T05:28:12.228 INFO:tasks.ceph.cluster2.osd.0.smithi087.stderr: 7: /lib64/libpthread.so.0(+0x81ca) [0x7f1adadc81ca] 2023-11-08T05:28:12.228 INFO:tasks.ceph.cluster2.osd.0.smithi087.stderr: 8: clone() 2023-11-08T05:28:12.228 INFO:tasks.ceph.cluster2.osd.0.smithi087.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-11-08T05:28:12.228 INFO:tasks.ceph.cluster2.osd.0.smithi087.stderr: 2023-11-08T05:28:12.234 INFO:tasks.ceph.cluster2.osd.0.smithi087.stderr: -188> 2023-11-08T05:28:12.216+0000 7f1acef4d700 -1 bdev(0x55f9dfd12400 /var/lib/ceph/osd/cluster2-0/block) _aio_thread got r=-28 ((28) No space left on device) 2023-11-08T05:28:12.234 INFO:tasks.ceph.cluster2.osd.0.smithi087.stderr: -1> 2023-11-08T05:28:12.218+0000 7f1acef4d700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/16.2.14-129-gcfb6fd52/rpm/el8/BUILD/ceph-16.2.14-129-gcfb6fd52/src/blk/kernel/KernelDevice.cc: In function 'void KernelDevice::_aio_thread()' thread 7f1acef4d700 time 2023-11-08T05:28:12.217518+0000 2023-11-08T05:28:12.234 INFO:tasks.ceph.cluster2.osd.0.smithi087.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/16.2.14-129-gcfb6fd52/rpm/el8/BUILD/ceph-16.2.14-129-gcfb6fd52/src/blk/kernel/KernelDevice.cc: 600: ceph_abort_msg("Unexpected IO error. This may suggest a hardware issue. Please check your kernel log!") 2023-11-08T05:28:12.235 INFO:tasks.ceph.cluster2.osd.0.smithi087.stderr: 2023-11-08T05:28:12.235 INFO:tasks.ceph.cluster2.osd.0.smithi087.stderr: ceph version 16.2.14-129-gcfb6fd52 (cfb6fd52113be6a0af3fe7fb89b2e08474eadc0f) pacific (stable) 2023-11-08T05:28:12.235 INFO:tasks.ceph.cluster2.osd.0.smithi087.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xe5) [0x55f9d36ae9b4] 2023-11-08T05:28:12.235 INFO:tasks.ceph.cluster2.osd.0.smithi087.stderr: 2: (KernelDevice::_aio_thread()+0x1254) [0x55f9d421c7c4] 2023-11-08T05:28:12.235 INFO:tasks.ceph.cluster2.osd.0.smithi087.stderr: 3: (KernelDevice::AioCompletionThread::entry()+0x11) [0x55f9d4227ac1] 2023-11-08T05:28:12.236 INFO:tasks.ceph.cluster2.osd.0.smithi087.stderr: 4: /lib64/libpthread.so.0(+0x81ca) [0x7f1adadc81ca] 2023-11-08T05:28:12.236 INFO:tasks.ceph.cluster2.osd.0.smithi087.stderr: 5: clone() 2023-11-08T05:28:12.236 INFO:tasks.ceph.cluster2.osd.0.smithi087.stderr: 2023-11-08T05:28:12.236 INFO:tasks.ceph.cluster2.osd.0.smithi087.stderr: 0> 2023-11-08T05:28:12.220+0000 7f1acef4d700 -1 *** Caught signal (Aborted) ** 2023-11-08T05:28:12.236 INFO:tasks.ceph.cluster2.osd.0.smithi087.stderr: in thread 7f1acef4d700 thread_name:bstore_aio 2023-11-08T05:28:12.237 INFO:tasks.ceph.cluster2.osd.0.smithi087.stderr: 2023-11-08T05:28:12.237 INFO:tasks.ceph.cluster2.osd.0.smithi087.stderr: ceph version 16.2.14-129-gcfb6fd52 (cfb6fd52113be6a0af3fe7fb89b2e08474eadc0f) pacific (stable) 2023-11-08T05:28:12.237 INFO:tasks.ceph.cluster2.osd.0.smithi087.stderr: 1: /lib64/libpthread.so.0(+0x12cf0) [0x7f1adadd2cf0] 2023-11-08T05:28:12.237 INFO:tasks.ceph.cluster2.osd.0.smithi087.stderr: 2: gsignal() 2023-11-08T05:28:12.237 INFO:tasks.ceph.cluster2.osd.0.smithi087.stderr: 3: abort() 2023-11-08T05:28:12.237 INFO:tasks.ceph.cluster2.osd.0.smithi087.stderr: 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x1b6) [0x55f9d36aea85] 2023-11-08T05:28:12.238 INFO:tasks.ceph.cluster2.osd.0.smithi087.stderr: 5: (KernelDevice::_aio_thread()+0x1254) [0x55f9d421c7c4] 2023-11-08T05:28:12.238 INFO:tasks.ceph.cluster2.osd.0.smithi087.stderr: 6: (KernelDevice::AioCompletionThread::entry()+0x11) [0x55f9d4227ac1] 2023-11-08T05:28:12.238 INFO:tasks.ceph.cluster2.osd.0.smithi087.stderr: 7: /lib64/libpthread.so.0(+0x81ca) [0x7f1adadc81ca] 2023-11-08T05:28:12.238 INFO:tasks.ceph.cluster2.osd.0.smithi087.stderr: 8: clone() 2023-11-08T05:28:12.238 INFO:tasks.ceph.cluster2.osd.0.smithi087.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-11-08T05:28:12.239 INFO:tasks.ceph.cluster2.osd.0.smithi087.stderr: 2023-11-08T05:28:12.246 INFO:tasks.ceph.cluster2.osd.0.smithi087.stderr: -339> 2023-11-08T05:28:12.216+0000 7f1acef4d700 -1 bdev(0x55f9dfd12400 /var/lib/ceph/osd/cluster2-0/block) _aio_thread got r=-28 ((28) No space left on device) 2023-11-08T05:28:12.246 INFO:tasks.ceph.cluster2.osd.0.smithi087.stderr: -152> 2023-11-08T05:28:12.218+0000 7f1acef4d700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/16.2.14-129-gcfb6fd52/rpm/el8/BUILD/ceph-16.2.14-129-gcfb6fd52/src/blk/kernel/KernelDevice.cc: In function 'void KernelDevice::_aio_thread()' thread 7f1acef4d700 time 2023-11-08T05:28:12.217518+0000 2023-11-08T05:28:12.246 INFO:tasks.ceph.cluster2.osd.0.smithi087.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/16.2.14-129-gcfb6fd52/rpm/el8/BUILD/ceph-16.2.14-129-gcfb6fd52/src/blk/kernel/KernelDevice.cc: 600: ceph_abort_msg("Unexpected IO error. This may suggest a hardware issue. Please check your kernel log!") 2023-11-08T05:28:12.246 INFO:tasks.ceph.cluster2.osd.0.smithi087.stderr: 2023-11-08T05:28:12.247 INFO:tasks.ceph.cluster2.osd.0.smithi087.stderr: ceph version 16.2.14-129-gcfb6fd52 (cfb6fd52113be6a0af3fe7fb89b2e08474eadc0f) pacific (stable) 2023-11-08T05:28:12.247 INFO:tasks.ceph.cluster2.osd.0.smithi087.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xe5) [0x55f9d36ae9b4] 2023-11-08T05:28:12.247 INFO:tasks.ceph.cluster2.osd.0.smithi087.stderr: 2: (KernelDevice::_aio_thread()+0x1254) [0x55f9d421c7c4] 2023-11-08T05:28:12.247 INFO:tasks.ceph.cluster2.osd.0.smithi087.stderr: 3: (KernelDevice::AioCompletionThread::entry()+0x11) [0x55f9d4227ac1] 2023-11-08T05:28:12.247 INFO:tasks.ceph.cluster2.osd.0.smithi087.stderr: 4: /lib64/libpthread.so.0(+0x81ca) [0x7f1adadc81ca] 2023-11-08T05:28:12.248 INFO:tasks.ceph.cluster2.osd.0.smithi087.stderr: 5: clone() 2023-11-08T05:28:12.248 INFO:tasks.ceph.cluster2.osd.0.smithi087.stderr: 2023-11-08T05:28:12.248 INFO:tasks.ceph.cluster2.osd.0.smithi087.stderr: -151> 2023-11-08T05:28:12.220+0000 7f1acef4d700 -1 *** Caught signal (Aborted) ** 2023-11-08T05:28:12.248 INFO:tasks.ceph.cluster2.osd.0.smithi087.stderr: in thread 7f1acef4d700 thread_name:bstore_aio 2023-11-08T05:28:12.248 INFO:tasks.ceph.cluster2.osd.0.smithi087.stderr: 2023-11-08T05:28:12.249 INFO:tasks.ceph.cluster2.osd.0.smithi087.stderr: ceph version 16.2.14-129-gcfb6fd52 (cfb6fd52113be6a0af3fe7fb89b2e08474eadc0f) pacific (stable) 2023-11-08T05:28:12.249 INFO:tasks.ceph.cluster2.osd.0.smithi087.stderr: 1: /lib64/libpthread.so.0(+0x12cf0) [0x7f1adadd2cf0] 2023-11-08T05:28:12.249 INFO:tasks.ceph.cluster2.osd.0.smithi087.stderr: 2: gsignal() 2023-11-08T05:28:12.249 INFO:tasks.ceph.cluster2.osd.0.smithi087.stderr: 3: abort() 2023-11-08T05:28:12.249 INFO:tasks.ceph.cluster2.osd.0.smithi087.stderr: 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x1b6) [0x55f9d36aea85] 2023-11-08T05:28:12.250 INFO:tasks.ceph.cluster2.osd.0.smithi087.stderr: 5: (KernelDevice::_aio_thread()+0x1254) [0x55f9d421c7c4] 2023-11-08T05:28:12.250 INFO:tasks.ceph.cluster2.osd.0.smithi087.stderr: 6: (KernelDevice::AioCompletionThread::entry()+0x11) [0x55f9d4227ac1] 2023-11-08T05:28:12.250 INFO:tasks.ceph.cluster2.osd.0.smithi087.stderr: 7: /lib64/libpthread.so.0(+0x81ca) [0x7f1adadc81ca] 2023-11-08T05:28:12.250 INFO:tasks.ceph.cluster2.osd.0.smithi087.stderr: 8: clone() 2023-11-08T05:28:12.250 INFO:tasks.ceph.cluster2.osd.0.smithi087.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-11-08T05:28:12.250 INFO:tasks.ceph.cluster2.osd.0.smithi087.stderr: 2023-11-08T05:28:12.394 INFO:tasks.ceph.cluster2.osd.0.smithi087.stderr:daemon-helper: command failed with exit status 5 2023-11-08T05:28:12.531 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ error_code=124 2023-11-08T05:28:12.531 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ set -e 2023-11-08T05:28:12.531 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' 124 -eq 124 ']' 2023-11-08T05:28:12.532 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:28:12.532 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_image_present cluster1 mirror image_9 present 2023-11-08T05:28:12.532 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:28:12.532 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:28:12.533 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_9 2023-11-08T05:28:12.533 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local state=present 2023-11-08T05:28:12.533 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image_id= 2023-11-08T05:28:12.533 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local s 2023-11-08T05:28:12.533 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test -n '' 2023-11-08T05:28:12.533 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ get_image_id cluster1 mirror image_9 2023-11-08T05:28:12.534 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1 2023-11-08T05:28:12.534 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local pool=mirror 2023-11-08T05:28:12.534 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local image=image_9 2023-11-08T05:28:12.534 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ rbd --cluster=cluster1 info mirror/image_9 2023-11-08T05:28:12.535 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-11-08T05:28:12.576 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image_id=1561b817b3a 2023-11-08T05:28:12.577 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-11-08T05:28:12.577 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0.1 2023-11-08T05:28:12.678 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test_image_present cluster1 mirror image_9 present 1561b817b3a 2023-11-08T05:28:12.678 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:28:12.678 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:28:12.679 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_9 2023-11-08T05:28:12.679 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local test_state=present 2023-11-08T05:28:12.679 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image_id=1561b817b3a 2023-11-08T05:28:12.679 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local current_state=deleted 2023-11-08T05:28:12.679 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local current_image_id 2023-11-08T05:28:12.680 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ get_image_id cluster1 mirror image_9 2023-11-08T05:28:12.680 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1 2023-11-08T05:28:12.680 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local pool=mirror 2023-11-08T05:28:12.680 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local image=image_9 2023-11-08T05:28:12.680 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ rbd --cluster=cluster1 info mirror/image_9 2023-11-08T05:28:12.681 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-11-08T05:28:12.715 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ current_image_id=1561b817b3a 2023-11-08T05:28:12.715 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test -n 1561b817b3a 2023-11-08T05:28:12.715 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test -z 1561b817b3a -o 1561b817b3a = 1561b817b3a 2023-11-08T05:28:12.716 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ current_state=present 2023-11-08T05:28:12.716 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test present = present 2023-11-08T05:28:12.716 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:28:12.716 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ disable_mirror cluster2 mirror image_9 2023-11-08T05:28:12.717 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:28:12.717 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:28:12.717 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_9 2023-11-08T05:28:12.717 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_9 2023-11-08T05:28:14.144 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.6292312751545713) > weight (0.25) 2023-11-08T05:28:14.144 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.8511970392271938) > weight (0.25) 2023-11-08T05:28:14.144 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.2654818210586126) > weight (0.25) 2023-11-08T05:28:14.145 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.3 with skip (0.8183089170418417) > weight (0.25) 2023-11-08T05:28:14.145 INFO:tasks.rbd_mirror_thrash:waiting for 106 secs before thrashing 2023-11-08T05:28:14.557 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi087.stderr:2023-11-08T05:28:14.555+0000 7fc159268700 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x56258f108800 scan_remote_mirror_snapshots: failed to locate remote start snapshot: snap_id=384 2023-11-08T05:28:14.557 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi087.stderr:2023-11-08T05:28:14.555+0000 7fc159268700 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x56258f108800 scan_remote_mirror_snapshots: split-brain detected: failed to find matching non-primary snapshot in remote image: local_snap_id_start=384, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=06977a9b-46f8-44a8-aea5-8992e82c582a, primary_snap_id=180, last_copied_object_number=128, snap_seqs={384=18446744073709551614}] 2023-11-08T05:28:18.047 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~0s 2023-11-08T05:28:19.560 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:Mirroring disabled 2023-11-08T05:28:19.568 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_image_present cluster1 mirror image_9 deleted 2023-11-08T05:28:19.568 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:28:19.569 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:28:19.569 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_9 2023-11-08T05:28:19.569 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local state=deleted 2023-11-08T05:28:19.569 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image_id= 2023-11-08T05:28:19.569 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local s 2023-11-08T05:28:19.569 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test -n '' 2023-11-08T05:28:19.570 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ get_image_id cluster1 mirror image_9 2023-11-08T05:28:19.570 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1 2023-11-08T05:28:19.570 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local pool=mirror 2023-11-08T05:28:19.570 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local image=image_9 2023-11-08T05:28:19.570 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ rbd --cluster=cluster1 info mirror/image_9 2023-11-08T05:28:19.571 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-11-08T05:28:19.600 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd: error opening image image_9: (2) No such file or directory 2023-11-08T05:28:19.605 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image_id= 2023-11-08T05:28:19.605 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-11-08T05:28:19.605 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0.1 2023-11-08T05:28:19.706 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test_image_present cluster1 mirror image_9 deleted '' 2023-11-08T05:28:19.706 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:28:19.707 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:28:19.707 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_9 2023-11-08T05:28:19.707 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local test_state=deleted 2023-11-08T05:28:19.707 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image_id= 2023-11-08T05:28:19.707 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local current_state=deleted 2023-11-08T05:28:19.708 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local current_image_id 2023-11-08T05:28:19.708 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ get_image_id cluster1 mirror image_9 2023-11-08T05:28:19.708 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1 2023-11-08T05:28:19.708 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local pool=mirror 2023-11-08T05:28:19.708 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local image=image_9 2023-11-08T05:28:19.708 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ rbd --cluster=cluster1 info mirror/image_9 2023-11-08T05:28:19.709 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-11-08T05:28:19.734 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd: error opening image image_9: (2) No such file or directory 2023-11-08T05:28:19.737 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ current_image_id= 2023-11-08T05:28:19.737 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test -n '' 2023-11-08T05:28:19.737 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test deleted = deleted 2023-11-08T05:28:19.738 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:28:19.738 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ purge_snapshots cluster2 mirror image_9 2023-11-08T05:28:19.738 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:28:19.738 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:28:19.738 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_9 2023-11-08T05:28:19.739 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 snap purge mirror/image_9 2023-11-08T05:28:19.773 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ remove_image_retry cluster2 mirror image_9 2023-11-08T05:28:19.774 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:28:19.774 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:28:19.774 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_9 2023-11-08T05:28:19.774 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0 1 2 4 8 16 32 2023-11-08T05:28:19.775 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0 2023-11-08T05:28:19.776 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ remove_image cluster2 mirror image_9 2023-11-08T05:28:19.776 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:28:19.776 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:28:19.776 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_9 2023-11-08T05:28:19.776 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_9 2023-11-08T05:28:19.812 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster=cluster2 rm mirror/image_9 2023-11-08T05:28:23.451 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~5s 2023-11-08T05:28:28.856 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~11s 2023-11-08T05:28:34.260 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~16s 2023-11-08T05:28:39.663 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~22s 2023-11-08T05:28:44.579 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2023-11-08T05:28:44.583 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:28:44.583 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for i in `seq 1 10` 2023-11-08T05:28:44.583 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image=image_10 2023-11-08T05:28:44.583 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ create_image cluster2 mirror image_10 512M 2023-11-08T05:28:44.584 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:28:44.584 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:28:44.584 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:28:44.584 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:28:44.584 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_10 2023-11-08T05:28:44.585 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:28:44.585 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local size=128 2023-11-08T05:28:44.585 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' -n 512M ']' 2023-11-08T05:28:44.585 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ size=512M 2023-11-08T05:28:44.585 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ shift 2023-11-08T05:28:44.586 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock mirror/image_10 2023-11-08T05:28:44.623 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ enable_mirror cluster2 mirror image_10 2023-11-08T05:28:44.623 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:28:44.624 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:28:44.624 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_10 2023-11-08T05:28:44.624 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local mode=snapshot 2023-11-08T05:28:44.624 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_10 snapshot 2023-11-08T05:28:45.068 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~27s 2023-11-08T05:28:45.652 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:Mirroring enabled 2023-11-08T05:28:45.659 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster=cluster2 info mirror/image_10 2023-11-08T05:28:45.689 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:rbd image 'image_10': 2023-11-08T05:28:45.690 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: size 512 MiB in 128 objects 2023-11-08T05:28:45.690 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: order 22 (4 MiB objects) 2023-11-08T05:28:45.690 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: snapshot_count: 1 2023-11-08T05:28:45.690 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: id: 19ed3580263f 2023-11-08T05:28:45.691 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: block_name_prefix: rbd_data.19ed3580263f 2023-11-08T05:28:45.691 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: format: 2 2023-11-08T05:28:45.691 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: features: layering, exclusive-lock 2023-11-08T05:28:45.691 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: op_features: 2023-11-08T05:28:45.691 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: flags: 2023-11-08T05:28:45.692 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: create_timestamp: Wed Nov 8 05:28:44 2023 2023-11-08T05:28:45.692 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: access_timestamp: Wed Nov 8 05:28:44 2023 2023-11-08T05:28:45.692 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: modify_timestamp: Wed Nov 8 05:28:44 2023 2023-11-08T05:28:45.692 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring state: enabled 2023-11-08T05:28:45.693 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring mode: snapshot 2023-11-08T05:28:45.693 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring global id: 150e7600-0392-48b4-9a16-174dcf79140c 2023-11-08T05:28:45.693 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout: mirroring primary: true 2023-11-08T05:28:45.694 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ stress_write_image cluster2 mirror image_10 2023-11-08T05:28:45.694 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:28:45.695 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:28:45.695 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_10 2023-11-08T05:28:45.695 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-11-08T05:28:45.695 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local duration=11 2023-11-08T05:28:45.695 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ set +e 2023-11-08T05:28:45.696 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_10 --debug-rbd=20 --debug-journaler=20 2023-11-08T05:28:45.880 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:100 IOs, 64784384 bytes 2023-11-08T05:28:45.980 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:200 IOs, 81386496 bytes 2023-11-08T05:28:46.088 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:300 IOs, 122714112 bytes 2023-11-08T05:28:46.221 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:400 IOs, 164647936 bytes 2023-11-08T05:28:46.301 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:500 IOs, 172141568 bytes 2023-11-08T05:28:46.422 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:600 IOs, 209542144 bytes 2023-11-08T05:28:46.535 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:700 IOs, 247389184 bytes 2023-11-08T05:28:46.607 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:800 IOs, 249336832 bytes 2023-11-08T05:28:46.703 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:900 IOs, 274769920 bytes 2023-11-08T05:28:46.864 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1000 IOs, 326585344 bytes 2023-11-08T05:28:46.968 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1100 IOs, 357310464 bytes 2023-11-08T05:28:47.127 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1200 IOs, 436803584 bytes 2023-11-08T05:28:47.262 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1300 IOs, 477426688 bytes 2023-11-08T05:28:47.363 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1400 IOs, 511825920 bytes 2023-11-08T05:28:47.486 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1500 IOs, 574197760 bytes 2023-11-08T05:28:47.656 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1600 IOs, 640500736 bytes 2023-11-08T05:28:47.776 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1700 IOs, 682168320 bytes 2023-11-08T05:28:47.927 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1800 IOs, 756884480 bytes 2023-11-08T05:28:48.138 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:1900 IOs, 849625088 bytes 2023-11-08T05:28:48.220 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2000 IOs, 865838080 bytes 2023-11-08T05:28:48.376 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2100 IOs, 940520448 bytes 2023-11-08T05:28:48.465 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2200 IOs, 958812160 bytes 2023-11-08T05:28:48.572 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2300 IOs, 985646080 bytes 2023-11-08T05:28:48.690 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2400 IOs, 1020557312 bytes 2023-11-08T05:28:48.830 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2500 IOs, 1093521408 bytes 2023-11-08T05:28:49.030 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2600 IOs, 1186995200 bytes 2023-11-08T05:28:49.157 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2700 IOs, 1236427776 bytes 2023-11-08T05:28:49.252 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2800 IOs, 1252592640 bytes 2023-11-08T05:28:49.370 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:2900 IOs, 1288950784 bytes 2023-11-08T05:28:49.513 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3000 IOs, 1351479296 bytes 2023-11-08T05:28:49.613 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3100 IOs, 1375249408 bytes 2023-11-08T05:28:49.802 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3200 IOs, 1455360000 bytes 2023-11-08T05:28:49.916 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3300 IOs, 1489017856 bytes 2023-11-08T05:28:50.021 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3400 IOs, 1513909248 bytes 2023-11-08T05:28:50.176 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3500 IOs, 1579180032 bytes 2023-11-08T05:28:50.307 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3600 IOs, 1613104128 bytes 2023-11-08T05:28:50.425 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3700 IOs, 1636251648 bytes 2023-11-08T05:28:50.471 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~32s 2023-11-08T05:28:50.551 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3800 IOs, 1662035968 bytes 2023-11-08T05:28:50.719 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:3900 IOs, 1740066816 bytes 2023-11-08T05:28:50.828 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4000 IOs, 1746649088 bytes 2023-11-08T05:28:50.993 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4100 IOs, 1804754944 bytes 2023-11-08T05:28:51.184 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4200 IOs, 1878836224 bytes 2023-11-08T05:28:51.295 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4300 IOs, 1905667072 bytes 2023-11-08T05:28:51.377 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4400 IOs, 1910264832 bytes 2023-11-08T05:28:51.534 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4500 IOs, 1965512704 bytes 2023-11-08T05:28:51.644 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4600 IOs, 1987153920 bytes 2023-11-08T05:28:51.746 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4700 IOs, 2011185152 bytes 2023-11-08T05:28:51.879 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4800 IOs, 2076938240 bytes 2023-11-08T05:28:52.050 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:4900 IOs, 2129825792 bytes 2023-11-08T05:28:52.165 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5000 IOs, 2150372352 bytes 2023-11-08T05:28:52.387 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5100 IOs, 2255210496 bytes 2023-11-08T05:28:52.544 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5200 IOs, 2333232128 bytes 2023-11-08T05:28:52.728 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5300 IOs, 2378238976 bytes 2023-11-08T05:28:52.815 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5400 IOs, 2380334080 bytes 2023-11-08T05:28:52.926 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5500 IOs, 2398820352 bytes 2023-11-08T05:28:53.071 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5600 IOs, 2450968576 bytes 2023-11-08T05:28:53.209 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5700 IOs, 2493956096 bytes 2023-11-08T05:28:53.420 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5800 IOs, 2584622080 bytes 2023-11-08T05:28:53.591 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:5900 IOs, 2663153664 bytes 2023-11-08T05:28:53.710 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:6000 IOs, 2709913600 bytes 2023-11-08T05:28:53.902 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:6100 IOs, 2823479296 bytes 2023-11-08T05:28:54.021 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:6200 IOs, 2855892992 bytes 2023-11-08T05:28:54.165 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:6300 IOs, 2903492608 bytes 2023-11-08T05:28:54.330 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:6400 IOs, 2962365440 bytes 2023-11-08T05:28:54.435 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:6500 IOs, 2975263744 bytes 2023-11-08T05:28:54.563 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:6600 IOs, 3026909184 bytes 2023-11-08T05:28:54.689 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:6700 IOs, 3066002432 bytes 2023-11-08T05:28:54.811 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:6800 IOs, 3095599104 bytes 2023-11-08T05:28:54.923 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:6900 IOs, 3117229056 bytes 2023-11-08T05:28:55.064 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:7000 IOs, 3165661184 bytes 2023-11-08T05:28:55.222 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:7100 IOs, 3251789824 bytes 2023-11-08T05:28:55.306 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:7200 IOs, 3252392960 bytes 2023-11-08T05:28:55.420 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:7300 IOs, 3276569600 bytes 2023-11-08T05:28:55.545 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:7400 IOs, 3321885696 bytes 2023-11-08T05:28:55.719 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:7500 IOs, 3393596416 bytes 2023-11-08T05:28:55.875 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~38s 2023-11-08T05:28:55.899 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:7600 IOs, 3472333824 bytes 2023-11-08T05:28:56.056 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:7700 IOs, 3507771392 bytes 2023-11-08T05:28:56.166 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:7800 IOs, 3536081920 bytes 2023-11-08T05:28:56.274 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:7900 IOs, 3563348992 bytes 2023-11-08T05:28:56.352 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:8000 IOs, 3563967488 bytes 2023-11-08T05:28:56.499 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:8100 IOs, 3609839616 bytes 2023-11-08T05:28:56.605 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:8200 IOs, 3627998208 bytes 2023-11-08T05:28:56.701 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ error_code=124 2023-11-08T05:28:56.701 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ set -e 2023-11-08T05:28:56.701 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' 124 -eq 124 ']' 2023-11-08T05:28:56.701 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:28:56.702 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_image_present cluster1 mirror image_10 present 2023-11-08T05:28:56.702 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:28:56.702 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:28:56.702 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_10 2023-11-08T05:28:56.702 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local state=present 2023-11-08T05:28:56.703 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image_id= 2023-11-08T05:28:56.703 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local s 2023-11-08T05:28:56.703 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test -n '' 2023-11-08T05:28:56.703 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ get_image_id cluster1 mirror image_10 2023-11-08T05:28:56.703 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1 2023-11-08T05:28:56.704 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local pool=mirror 2023-11-08T05:28:56.704 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local image=image_10 2023-11-08T05:28:56.704 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ rbd --cluster=cluster1 info mirror/image_10 2023-11-08T05:28:56.704 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-11-08T05:28:56.738 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image_id=1561dd40c5ca 2023-11-08T05:28:56.738 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-11-08T05:28:56.739 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0.1 2023-11-08T05:28:56.840 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test_image_present cluster1 mirror image_10 present 1561dd40c5ca 2023-11-08T05:28:56.840 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:28:56.840 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:28:56.841 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_10 2023-11-08T05:28:56.841 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local test_state=present 2023-11-08T05:28:56.841 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image_id=1561dd40c5ca 2023-11-08T05:28:56.841 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local current_state=deleted 2023-11-08T05:28:56.841 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local current_image_id 2023-11-08T05:28:56.841 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ get_image_id cluster1 mirror image_10 2023-11-08T05:28:56.842 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1 2023-11-08T05:28:56.842 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local pool=mirror 2023-11-08T05:28:56.842 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local image=image_10 2023-11-08T05:28:56.842 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ rbd --cluster=cluster1 info mirror/image_10 2023-11-08T05:28:56.843 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-11-08T05:28:56.879 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ current_image_id=1561dd40c5ca 2023-11-08T05:28:56.879 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test -n 1561dd40c5ca 2023-11-08T05:28:56.879 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test -z 1561dd40c5ca -o 1561dd40c5ca = 1561dd40c5ca 2023-11-08T05:28:56.879 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ current_state=present 2023-11-08T05:28:56.880 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test present = present 2023-11-08T05:28:56.880 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:28:56.880 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ disable_mirror cluster2 mirror image_10 2023-11-08T05:28:56.880 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:28:56.880 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:28:56.881 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_10 2023-11-08T05:28:56.881 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_10 2023-11-08T05:28:57.783 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi087.stderr:2023-11-08T05:28:57.782+0000 7fc159268700 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x562591cf9800 scan_remote_mirror_snapshots: failed to locate remote start snapshot: snap_id=386 2023-11-08T05:28:57.784 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi087.stderr:2023-11-08T05:28:57.782+0000 7fc159268700 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x562591cf9800 scan_remote_mirror_snapshots: split-brain detected: failed to find matching non-primary snapshot in remote image: local_snap_id_start=386, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=06977a9b-46f8-44a8-aea5-8992e82c582a, primary_snap_id=182, last_copied_object_number=128, snap_seqs={386=18446744073709551614}] 2023-11-08T05:28:58.403 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi087.stderr:2023-11-08T05:28:58.401+0000 7fc4495f2700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55eab5e71860 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-11-08T05:28:58.403 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi087.stderr:2023-11-08T05:28:58.401+0000 7fc44fdff700 -1 rbd::mirror::ImageReplayer: 0x55eab1f49680 [3/150e7600-0392-48b4-9a16-174dcf79140c] operator(): start failed: (85) Interrupted system call should be restarted 2023-11-08T05:29:00.362 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.6130916391566826) > weight (0.25) 2023-11-08T05:29:00.362 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.1 2023-11-08T05:29:00.362 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Sent signal 15 2023-11-08T05:29:00.363 INFO:tasks.rbd_mirror_thrash:waiting for 7 secs before reviving daemons 2023-11-08T05:29:00.363 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi087.stderr:2023-11-08T05:29:00.362+0000 7fb3cdfbc700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 (PID: 156780) UID: 1000 2023-11-08T05:29:00.564 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi087.stderr:daemon-helper: command failed with exit status 1 2023-11-08T05:29:01.279 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~43s 2023-11-08T05:29:02.787 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:Mirroring disabled 2023-11-08T05:29:02.795 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_image_present cluster1 mirror image_10 deleted 2023-11-08T05:29:02.795 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:29:02.796 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:29:02.796 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_10 2023-11-08T05:29:02.796 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local state=deleted 2023-11-08T05:29:02.796 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image_id= 2023-11-08T05:29:02.796 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local s 2023-11-08T05:29:02.797 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test -n '' 2023-11-08T05:29:02.797 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ get_image_id cluster1 mirror image_10 2023-11-08T05:29:02.797 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1 2023-11-08T05:29:02.797 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local pool=mirror 2023-11-08T05:29:02.797 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local image=image_10 2023-11-08T05:29:02.798 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ rbd --cluster=cluster1 info mirror/image_10 2023-11-08T05:29:02.798 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-11-08T05:29:02.826 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd: error opening image image_10: (2) No such file or directory 2023-11-08T05:29:02.829 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ image_id= 2023-11-08T05:29:02.830 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-11-08T05:29:02.830 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0.1 2023-11-08T05:29:02.931 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test_image_present cluster1 mirror image_10 deleted '' 2023-11-08T05:29:02.931 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:29:02.932 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:29:02.932 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_10 2023-11-08T05:29:02.932 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local test_state=deleted 2023-11-08T05:29:02.932 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image_id= 2023-11-08T05:29:02.933 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local current_state=deleted 2023-11-08T05:29:02.933 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local current_image_id 2023-11-08T05:29:02.933 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ get_image_id cluster1 mirror image_10 2023-11-08T05:29:02.933 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1 2023-11-08T05:29:02.934 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local pool=mirror 2023-11-08T05:29:02.934 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local image=image_10 2023-11-08T05:29:02.934 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ rbd --cluster=cluster1 info mirror/image_10 2023-11-08T05:29:02.934 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-11-08T05:29:02.960 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:rbd: error opening image image_10: (2) No such file or directory 2023-11-08T05:29:02.963 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ current_image_id= 2023-11-08T05:29:02.964 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test -n '' 2023-11-08T05:29:02.964 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test deleted = deleted 2023-11-08T05:29:02.964 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:29:02.964 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ purge_snapshots cluster2 mirror image_10 2023-11-08T05:29:02.964 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:29:02.965 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:29:02.965 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_10 2023-11-08T05:29:02.965 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster cluster2 snap purge mirror/image_10 2023-11-08T05:29:03.003 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ remove_image_retry cluster2 mirror image_10 2023-11-08T05:29:03.003 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:29:03.003 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:29:03.003 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_10 2023-11-08T05:29:03.004 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0 1 2 4 8 16 32 2023-11-08T05:29:03.004 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0 2023-11-08T05:29:03.005 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ remove_image cluster2 mirror image_10 2023-11-08T05:29:03.005 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:29:03.005 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:29:03.005 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local image=image_10 2023-11-08T05:29:03.006 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_10 2023-11-08T05:29:03.041 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rbd --cluster=cluster2 rm mirror/image_10 2023-11-08T05:29:06.683 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~49s 2023-11-08T05:29:07.364 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.1 2023-11-08T05:29:07.365 INFO:teuthology.orchestra.run:waiting for 600 2023-11-08T05:29:07.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-08T05:29:07.365 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.1 2023-11-08T05:29:07.366 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Restarting daemon 2023-11-08T05:29:07.366 DEBUG:teuthology.orchestra.run.smithi087:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 2023-11-08T05:29:07.369 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Started 2023-11-08T05:29:07.369 INFO:tasks.rbd_mirror_thrash:waiting for 82 secs before thrashing 2023-11-08T05:29:12.087 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~54s 2023-11-08T05:29:17.491 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~59s 2023-11-08T05:29:22.895 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~65s 2023-11-08T05:29:26.658 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2023-11-08T05:29:26.660 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:29:26.661 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ testlog 'TEST: check if removed images'\'' OMAP are removed' 2023-11-08T05:29:26.661 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ tee -a /tmp/tmp.Ludk8ikCAB/rbd-mirror.test.log 2023-11-08T05:29:26.661 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ date '+%F %T' 2023-11-08T05:29:26.664 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ echo 2023-11-08 05:29:26 TEST: check if removed 'images'\''' OMAP are removed 2023-11-08T05:29:26.664 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:2023-11-08 05:29:26 TEST: check if removed images' OMAP are removed 2023-11-08T05:29:26.664 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_image_in_omap cluster1 mirror 2023-11-08T05:29:26.664 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:29:26.664 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:29:26.665 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_omap_keys cluster1 mirror rbd_mirroring status_global 2023-11-08T05:29:26.665 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:29:26.665 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:29:26.665 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local obj_name=rbd_mirroring 2023-11-08T05:29:26.665 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local filter=status_global 2023-11-08T05:29:26.666 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32 2023-11-08T05:29:26.666 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0 2023-11-08T05:29:26.666 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ set +e 2023-11-08T05:29:26.666 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ count_omap_keys_with_filter cluster1 mirror rbd_mirroring status_global 2023-11-08T05:29:26.666 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1 2023-11-08T05:29:26.667 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local pool=mirror 2023-11-08T05:29:26.667 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local obj_name=rbd_mirroring 2023-11-08T05:29:26.667 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local filter=status_global 2023-11-08T05:29:26.667 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ list_omap_keys cluster1 mirror rbd_mirroring 2023-11-08T05:29:26.667 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1 2023-11-08T05:29:26.668 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local pool=mirror 2023-11-08T05:29:26.668 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local obj_name=rbd_mirroring 2023-11-08T05:29:26.668 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ grep -c status_global 2023-11-08T05:29:26.668 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ rados --cluster cluster1 -p mirror listomapkeys rbd_mirroring 2023-11-08T05:29:26.692 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test 0 = 0 2023-11-08T05:29:26.692 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ error_code=0 2023-11-08T05:29:26.692 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ set -e 2023-11-08T05:29:26.692 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' 0 -eq 0 ']' 2023-11-08T05:29:26.693 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:29:26.693 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_omap_keys cluster1 mirror rbd_mirroring image_ 2023-11-08T05:29:26.693 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:29:26.693 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:29:26.693 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local obj_name=rbd_mirroring 2023-11-08T05:29:26.694 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local filter=image_ 2023-11-08T05:29:26.694 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32 2023-11-08T05:29:26.694 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0 2023-11-08T05:29:26.694 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ set +e 2023-11-08T05:29:26.695 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ count_omap_keys_with_filter cluster1 mirror rbd_mirroring image_ 2023-11-08T05:29:26.696 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1 2023-11-08T05:29:26.696 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local pool=mirror 2023-11-08T05:29:26.696 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local obj_name=rbd_mirroring 2023-11-08T05:29:26.696 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local filter=image_ 2023-11-08T05:29:26.696 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ list_omap_keys cluster1 mirror rbd_mirroring 2023-11-08T05:29:26.697 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1 2023-11-08T05:29:26.697 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local pool=mirror 2023-11-08T05:29:26.697 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local obj_name=rbd_mirroring 2023-11-08T05:29:26.697 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ grep -c image_ 2023-11-08T05:29:26.697 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ rados --cluster cluster1 -p mirror listomapkeys rbd_mirroring 2023-11-08T05:29:26.720 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test 0 = 0 2023-11-08T05:29:26.720 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ error_code=0 2023-11-08T05:29:26.720 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ set -e 2023-11-08T05:29:26.721 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' 0 -eq 0 ']' 2023-11-08T05:29:26.721 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:29:26.721 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_omap_keys cluster1 mirror rbd_mirror_leader image_map 2023-11-08T05:29:26.721 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:29:26.721 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:29:26.722 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local obj_name=rbd_mirror_leader 2023-11-08T05:29:26.722 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local filter=image_map 2023-11-08T05:29:26.722 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32 2023-11-08T05:29:26.722 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0 2023-11-08T05:29:26.723 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ set +e 2023-11-08T05:29:26.724 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ count_omap_keys_with_filter cluster1 mirror rbd_mirror_leader image_map 2023-11-08T05:29:26.724 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1 2023-11-08T05:29:26.724 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local pool=mirror 2023-11-08T05:29:26.724 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local obj_name=rbd_mirror_leader 2023-11-08T05:29:26.725 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local filter=image_map 2023-11-08T05:29:26.725 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ list_omap_keys cluster1 mirror rbd_mirror_leader 2023-11-08T05:29:26.725 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster1 2023-11-08T05:29:26.725 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local pool=mirror 2023-11-08T05:29:26.725 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local obj_name=rbd_mirror_leader 2023-11-08T05:29:26.725 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ grep -c image_map 2023-11-08T05:29:26.726 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ rados --cluster cluster1 -p mirror listomapkeys rbd_mirror_leader 2023-11-08T05:29:26.750 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test 0 = 0 2023-11-08T05:29:26.751 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ error_code=0 2023-11-08T05:29:26.751 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ set -e 2023-11-08T05:29:26.751 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' 0 -eq 0 ']' 2023-11-08T05:29:26.751 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:29:26.751 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_image_in_omap cluster2 mirror 2023-11-08T05:29:26.752 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:29:26.752 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:29:26.752 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_omap_keys cluster2 mirror rbd_mirroring status_global 2023-11-08T05:29:26.752 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:29:26.752 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:29:26.753 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local obj_name=rbd_mirroring 2023-11-08T05:29:26.753 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local filter=status_global 2023-11-08T05:29:26.753 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32 2023-11-08T05:29:26.753 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0 2023-11-08T05:29:26.753 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ set +e 2023-11-08T05:29:26.754 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ count_omap_keys_with_filter cluster2 mirror rbd_mirroring status_global 2023-11-08T05:29:26.754 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster2 2023-11-08T05:29:26.754 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local pool=mirror 2023-11-08T05:29:26.754 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local obj_name=rbd_mirroring 2023-11-08T05:29:26.754 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local filter=status_global 2023-11-08T05:29:26.755 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ list_omap_keys cluster2 mirror rbd_mirroring 2023-11-08T05:29:26.755 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster2 2023-11-08T05:29:26.755 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local pool=mirror 2023-11-08T05:29:26.755 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local obj_name=rbd_mirroring 2023-11-08T05:29:26.755 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ grep -c status_global 2023-11-08T05:29:26.755 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ rados --cluster cluster2 -p mirror listomapkeys rbd_mirroring 2023-11-08T05:29:26.778 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test 0 = 0 2023-11-08T05:29:26.778 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ error_code=0 2023-11-08T05:29:26.778 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ set -e 2023-11-08T05:29:26.778 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' 0 -eq 0 ']' 2023-11-08T05:29:26.779 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:29:26.779 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_omap_keys cluster2 mirror rbd_mirroring image_ 2023-11-08T05:29:26.779 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:29:26.779 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:29:26.779 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local obj_name=rbd_mirroring 2023-11-08T05:29:26.779 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local filter=image_ 2023-11-08T05:29:26.780 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32 2023-11-08T05:29:26.780 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0 2023-11-08T05:29:26.780 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ set +e 2023-11-08T05:29:26.781 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ count_omap_keys_with_filter cluster2 mirror rbd_mirroring image_ 2023-11-08T05:29:26.782 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster2 2023-11-08T05:29:26.782 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local pool=mirror 2023-11-08T05:29:26.782 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local obj_name=rbd_mirroring 2023-11-08T05:29:26.782 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local filter=image_ 2023-11-08T05:29:26.782 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ list_omap_keys cluster2 mirror rbd_mirroring 2023-11-08T05:29:26.783 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster2 2023-11-08T05:29:26.783 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local pool=mirror 2023-11-08T05:29:26.783 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local obj_name=rbd_mirroring 2023-11-08T05:29:26.783 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ rados --cluster cluster2 -p mirror listomapkeys rbd_mirroring 2023-11-08T05:29:26.783 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ grep -c image_ 2023-11-08T05:29:26.807 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test 0 = 0 2023-11-08T05:29:26.807 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ error_code=0 2023-11-08T05:29:26.807 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ set -e 2023-11-08T05:29:26.807 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' 0 -eq 0 ']' 2023-11-08T05:29:26.808 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:29:26.808 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ wait_for_omap_keys cluster2 mirror rbd_mirror_leader image_map 2023-11-08T05:29:26.808 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:29:26.808 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local pool=mirror 2023-11-08T05:29:26.808 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local obj_name=rbd_mirror_leader 2023-11-08T05:29:26.809 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local filter=image_map 2023-11-08T05:29:26.809 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32 2023-11-08T05:29:26.809 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ sleep 0 2023-11-08T05:29:26.809 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ set +e 2023-11-08T05:29:26.809 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ count_omap_keys_with_filter cluster2 mirror rbd_mirror_leader image_map 2023-11-08T05:29:26.810 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster2 2023-11-08T05:29:26.810 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local pool=mirror 2023-11-08T05:29:26.810 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local obj_name=rbd_mirror_leader 2023-11-08T05:29:26.810 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local filter=image_map 2023-11-08T05:29:26.810 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ list_omap_keys cluster2 mirror rbd_mirror_leader 2023-11-08T05:29:26.811 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local cluster=cluster2 2023-11-08T05:29:26.811 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local pool=mirror 2023-11-08T05:29:26.811 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ local obj_name=rbd_mirror_leader 2023-11-08T05:29:26.811 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ grep -c image_map 2023-11-08T05:29:26.811 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ rados --cluster cluster2 -p mirror listomapkeys rbd_mirror_leader 2023-11-08T05:29:26.834 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test 0 = 0 2023-11-08T05:29:26.834 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ error_code=0 2023-11-08T05:29:26.834 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ set -e 2023-11-08T05:29:26.835 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' 0 -eq 0 ']' 2023-11-08T05:29:26.835 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 0 2023-11-08T05:29:26.835 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ cleanup 0 2023-11-08T05:29:26.835 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local error_code=0 2023-11-08T05:29:26.835 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ set +e 2023-11-08T05:29:26.835 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' 0 -ne 0 ']' 2023-11-08T05:29:26.836 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' -z '' ']' 2023-11-08T05:29:26.836 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster instance 2023-11-08T05:29:26.836 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ CEPH_ARGS= 2023-11-08T05:29:26.836 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ ceph --cluster cluster1 osd pool rm mirror mirror --yes-i-really-really-mean-it 2023-11-08T05:29:27.763 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:pool 'mirror' does not exist 2023-11-08T05:29:27.772 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ CEPH_ARGS= 2023-11-08T05:29:27.773 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ ceph --cluster cluster2 osd pool rm mirror mirror --yes-i-really-really-mean-it 2023-11-08T05:29:28.299 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~70s 2023-11-08T05:29:28.449 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi087.stderr:2023-11-08T05:29:28.447+0000 7fc15f274700 -1 rbd::mirror::image_deleter::TrashWatcher: 0x562591972000 handle_unregister_watcher: error unregistering watcher for trash directory: (2) No such file or directory 2023-11-08T05:29:28.450 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi087.stderr:2023-11-08T05:29:28.447+0000 7fc15f274700 -1 rbd::mirror::image_deleter::TrashWatcher: 0x562591973d40 handle_unregister_watcher: error unregistering watcher for trash directory: (2) No such file or directory 2023-11-08T05:29:28.450 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi087.stderr:2023-11-08T05:29:28.447+0000 7fc15fa75700 -1 rbd::mirror::PoolWatcher: 0x562591c681e0 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2023-11-08T05:29:28.451 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi087.stderr:2023-11-08T05:29:28.447+0000 7fc15fa75700 -1 rbd::mirror::PoolWatcher: 0x562591c68000 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2023-11-08T05:29:28.452 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi087.stderr:2023-11-08T05:29:28.450+0000 7fc165a81700 -1 rbd::mirror::InstanceWatcher: 0x56258e55d180 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.5473 object: (2) No such file or directory 2023-11-08T05:29:28.454 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi087.stderr:2023-11-08T05:29:28.451+0000 7fc15fa75700 -1 rbd::mirror::InstanceWatcher: 0x56258e55d180 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2023-11-08T05:29:28.454 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi087.stderr:2023-11-08T05:29:28.451+0000 7fc165a81700 -1 rbd::mirror::InstanceWatcher: 0x56258e55ce00 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.5473 object: (2) No such file or directory 2023-11-08T05:29:28.456 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi087.stderr:2023-11-08T05:29:28.453+0000 7fc15fa75700 -1 rbd::mirror::MirrorStatusUpdater 0x56258fdf0120 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-11-08T05:29:28.456 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi087.stderr:2023-11-08T05:29:28.453+0000 7fc165a81700 -1 rbd::mirror::NamespaceReplayer: 0x56258fdee1a0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2023-11-08T05:29:28.457 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi087.stderr:2023-11-08T05:29:28.454+0000 7fc15fa75700 -1 rbd::mirror::InstanceWatcher: 0x56258e55ce00 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2023-11-08T05:29:28.459 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi087.stderr:2023-11-08T05:29:28.457+0000 7fc15f274700 -1 rbd::mirror::MirrorStatusUpdater 0x56258fdf0000 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-11-08T05:29:28.460 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi087.stderr:2023-11-08T05:29:28.457+0000 7fc166282700 -1 rbd::mirror::NamespaceReplayer: 0x56258fdee000 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2023-11-08T05:29:28.498 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi087.stderr:2023-11-08T05:29:28.495+0000 7fc159268700 -1 rbd::mirror::RemotePoolPoller: 0x56258d7d0d00 handle_mirror_peer_ping: failed to ping remote mirror peer: (2) No such file or directory 2023-11-08T05:29:28.499 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi087.stderr:2023-11-08T05:29:28.495+0000 7fc158a67700 -1 rbd::mirror::RemotePoolPoller: 0x56258d7d0d00 handle_mirror_peer_list: failed to retrieve mirror peers: (2) No such file or directory 2023-11-08T05:29:28.499 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi087.stderr:2023-11-08T05:29:28.496+0000 7fc158a67700 -1 rbd::mirror::PoolReplayer: 0x56258e4b0300 handle_remote_pool_meta_updated: remote pool metadata updated unexpectedly 2023-11-08T05:29:28.669 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi087.stderr:2023-11-08T05:29:28.666+0000 7fc15fa75700 -1 rbd::mirror::image_deleter::TrashWatcher: 0x562591973200 handle_unregister_watcher: error unregistering watcher for trash directory: (2) No such file or directory 2023-11-08T05:29:28.669 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi087.stderr:2023-11-08T05:29:28.666+0000 7fc15fa75700 -1 rbd::mirror::PoolWatcher: 0x562591759680 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2023-11-08T05:29:28.673 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi087.stderr:2023-11-08T05:29:28.670+0000 7fc158a67700 -1 rbd::mirror::PoolWatcher: 0x562591c685a0 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2023-11-08T05:29:28.676 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi087.stderr:2023-11-08T05:29:28.673+0000 7fc165a81700 -1 rbd::mirror::LeaderWatcher: 0x56258fc6ac00 handle_notify_lock_released: error notifying leader lock released: (2) No such file or directory 2023-11-08T05:29:28.676 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi087.stderr:2023-11-08T05:29:28.673+0000 7fc165a81700 -1 rbd::mirror::LeaderWatcher: 0x56258fc6ac00 handle_shut_down_leader_lock: error shutting down leader lock: (2) No such file or directory 2023-11-08T05:29:28.677 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi087.stderr:2023-11-08T05:29:28.674+0000 7fc166282700 -1 rbd::mirror::LeaderWatcher: 0x56258fc6ac00 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2023-11-08T05:29:28.678 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi087.stderr:2023-11-08T05:29:28.674+0000 7fc166282700 -1 rbd::mirror::InstanceWatcher: 0x56258e55ca80 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.5473 object: (2) No such file or directory 2023-11-08T05:29:28.678 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi087.stderr:2023-11-08T05:29:28.674+0000 7fc15f274700 -1 rbd::mirror::InstanceWatcher: 0x56258e55ca80 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2023-11-08T05:29:28.679 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi087.stderr:2023-11-08T05:29:28.675+0000 7fc158a67700 -1 rbd::mirror::MirrorStatusUpdater 0x56258fda0000 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-11-08T05:29:28.679 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi087.stderr:2023-11-08T05:29:28.675+0000 7fc166282700 -1 rbd::mirror::NamespaceReplayer: 0x56258fcde340 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2023-11-08T05:29:28.679 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi087.stderr:2023-11-08T05:29:28.675+0000 7fc15fa75700 -1 rbd::mirror::MirrorStatusUpdater 0x56258f135560 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-11-08T05:29:28.680 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi087.stderr:2023-11-08T05:29:28.675+0000 7fc165a81700 -1 rbd::mirror::NamespaceReplayer: 0x56258fcde340 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2023-11-08T05:29:28.680 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:pool 'mirror' does not exist 2023-11-08T05:29:28.690 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ CEPH_ARGS= 2023-11-08T05:29:28.690 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ ceph --cluster cluster1 osd pool rm mirror_parent mirror_parent --yes-i-really-really-mean-it 2023-11-08T05:29:29.810 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:pool 'mirror_parent' does not exist 2023-11-08T05:29:29.820 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ CEPH_ARGS= 2023-11-08T05:29:29.820 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ ceph --cluster cluster2 osd pool rm mirror_parent mirror_parent --yes-i-really-really-mean-it 2023-11-08T05:29:30.586 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:pool 'mirror_parent' does not exist 2023-11-08T05:29:30.596 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for cluster in "${CLUSTER1}" "${CLUSTER2}" 2023-11-08T05:29:30.596 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ stop_mirrors cluster1 2023-11-08T05:29:30.596 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1 2023-11-08T05:29:30.596 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local sig= 2023-11-08T05:29:30.596 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ seq 0 3 2023-11-08T05:29:30.600 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T05:29:30.601 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ stop_mirror cluster1:0 '' 2023-11-08T05:29:30.601 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1:0 2023-11-08T05:29:30.601 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local sig= 2023-11-08T05:29:30.601 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test -n 1 2023-11-08T05:29:30.601 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 2023-11-08T05:29:30.602 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T05:29:30.602 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ stop_mirror cluster1:1 '' 2023-11-08T05:29:30.602 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1:1 2023-11-08T05:29:30.603 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local sig= 2023-11-08T05:29:30.603 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test -n 1 2023-11-08T05:29:30.606 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 2023-11-08T05:29:30.606 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T05:29:30.606 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ stop_mirror cluster1:2 '' 2023-11-08T05:29:30.606 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1:2 2023-11-08T05:29:30.607 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local sig= 2023-11-08T05:29:30.607 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test -n 1 2023-11-08T05:29:30.607 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 2023-11-08T05:29:30.607 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T05:29:30.607 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ stop_mirror cluster1:3 '' 2023-11-08T05:29:30.608 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster1:3 2023-11-08T05:29:30.608 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local sig= 2023-11-08T05:29:30.608 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test -n 1 2023-11-08T05:29:30.608 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 2023-11-08T05:29:30.608 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for cluster in "${CLUSTER1}" "${CLUSTER2}" 2023-11-08T05:29:30.609 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ stop_mirrors cluster2 2023-11-08T05:29:30.609 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2 2023-11-08T05:29:30.609 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local sig= 2023-11-08T05:29:30.609 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:++ seq 0 3 2023-11-08T05:29:30.609 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T05:29:30.610 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ stop_mirror cluster2:0 '' 2023-11-08T05:29:30.610 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2:0 2023-11-08T05:29:30.618 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local sig= 2023-11-08T05:29:30.619 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test -n 1 2023-11-08T05:29:30.619 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 2023-11-08T05:29:30.619 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T05:29:30.619 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ stop_mirror cluster2:1 '' 2023-11-08T05:29:30.620 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2:1 2023-11-08T05:29:30.620 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local sig= 2023-11-08T05:29:30.620 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test -n 1 2023-11-08T05:29:30.620 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 2023-11-08T05:29:30.620 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T05:29:30.621 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ stop_mirror cluster2:2 '' 2023-11-08T05:29:30.621 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2:2 2023-11-08T05:29:30.621 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local sig= 2023-11-08T05:29:30.621 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test -n 1 2023-11-08T05:29:30.622 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 2023-11-08T05:29:30.622 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-11-08T05:29:30.622 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ stop_mirror cluster2:3 '' 2023-11-08T05:29:30.622 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local cluster=cluster2:3 2023-11-08T05:29:30.622 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ local sig= 2023-11-08T05:29:30.623 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test -n 1 2023-11-08T05:29:30.623 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ return 2023-11-08T05:29:30.623 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' -z 1 ']' 2023-11-08T05:29:30.623 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ test '' = /tmp/tmp.Ludk8ikCAB 2023-11-08T05:29:30.624 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ rm -Rf /tmp/tmp.Ludk8ikCAB 2023-11-08T05:29:30.624 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ '[' 0 -eq 0 ']' 2023-11-08T05:29:30.624 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ echo OK 2023-11-08T05:29:30.625 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stdout:OK 2023-11-08T05:29:30.626 INFO:tasks.workunit.cluster1.client.mirror.smithi087.stderr:+ exit 0 2023-11-08T05:29:30.626 INFO:teuthology.orchestra.run:Running command with timeout 3600 2023-11-08T05:29:30.626 DEBUG:teuthology.orchestra.run.smithi087:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp 2023-11-08T05:29:30.696 INFO:tasks.workunit:Stopping ['rbd/rbd_mirror_stress.sh'] on cluster1.client.mirror... 2023-11-08T05:29:30.696 DEBUG:teuthology.orchestra.run.smithi087:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.cluster1.client.mirror /home/ubuntu/cephtest/clone.cluster1.client.mirror 2023-11-08T05:29:30.983 DEBUG:teuthology.parallel:result is None 2023-11-08T05:29:30.983 DEBUG:teuthology.orchestra.run.smithi087:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror 2023-11-08T05:29:31.010 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror 2023-11-08T05:29:31.010 DEBUG:teuthology.orchestra.run.smithi087:> rmdir -- /home/ubuntu/cephtest/mnt.cluster1.mirror 2023-11-08T05:29:31.167 INFO:tasks.workunit:Deleted artificial mount point /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror 2023-11-08T05:29:31.167 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror-thrash 2023-11-08T05:29:31.181 INFO:tasks.rbd_mirror_thrash:joining rbd_mirror_thrash 2023-11-08T05:29:31.182 INFO:tasks.rbd_mirror_thrash:stat['kill'] = 14 2023-11-08T05:29:31.182 INFO:tasks.rbd_mirror_thrash:done joining 2023-11-08T05:29:31.182 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror-thrash 2023-11-08T05:29:31.215 INFO:tasks.rbd_mirror_thrash:joining rbd_mirror_thrash 2023-11-08T05:29:31.215 INFO:tasks.rbd_mirror_thrash:stat['kill'] = 18 2023-11-08T05:29:31.215 INFO:tasks.rbd_mirror_thrash:done joining 2023-11-08T05:29:31.216 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror 2023-11-08T05:29:31.237 DEBUG:tasks.rbd_mirror.cluster2.client.mirror.3:waiting for process to exit 2023-11-08T05:29:31.238 INFO:teuthology.orchestra.run:waiting for 300 2023-11-08T05:29:31.239 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi087.stderr:2023-11-08T05:29:31.237+0000 7f548ab84700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3 (PID: 151159) UID: 1000 2023-11-08T05:29:31.239 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi087.stderr:2023-11-08T05:29:31.237+0000 7f5489b82700 -1 rbd::mirror::LeaderWatcher: 0x55813d300f00 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2023-11-08T05:29:31.239 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi087.stderr:2023-11-08T05:29:31.237+0000 7f5489b82700 -1 rbd::mirror::InstanceWatcher: 0x55813a301180 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6237 object: (2) No such file or directory 2023-11-08T05:29:31.241 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi087.stderr:2023-11-08T05:29:31.238+0000 7f5483b76700 -1 rbd::mirror::InstanceWatcher: 0x55813a300e00 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2023-11-08T05:29:31.241 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi087.stderr:2023-11-08T05:29:31.238+0000 7f5489b82700 -1 rbd::mirror::InstanceWatcher: 0x55813a301500 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6240 object: (2) No such file or directory 2023-11-08T05:29:31.241 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi087.stderr:2023-11-08T05:29:31.238+0000 7f5483375700 -1 rbd::mirror::InstanceWatcher: 0x55813a301180 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2023-11-08T05:29:31.241 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi087.stderr:2023-11-08T05:29:31.238+0000 7f5475b5a700 -1 rbd::mirror::InstanceWatcher: 0x55813a301500 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2023-11-08T05:29:31.242 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi087.stderr:2023-11-08T05:29:31.238+0000 7f547d369700 -1 rbd::mirror::MirrorStatusUpdater 0x55813bb1a480 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-11-08T05:29:31.242 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi087.stderr:2023-11-08T05:29:31.238+0000 7f547d369700 -1 rbd::mirror::MirrorStatusUpdater 0x55813c5f8360 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-11-08T05:29:31.242 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi087.stderr:2023-11-08T05:29:31.238+0000 7f548a383700 -1 rbd::mirror::NamespaceReplayer: 0x55813bb301a0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2023-11-08T05:29:31.242 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi087.stderr:2023-11-08T05:29:31.238+0000 7f548a383700 -1 rbd::mirror::NamespaceReplayer: 0x55813bb30340 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2023-11-08T05:29:31.242 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi087.stderr:2023-11-08T05:29:31.238+0000 7f546f34d700 -1 rbd::mirror::MirrorStatusUpdater 0x55813c7a86c0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-11-08T05:29:31.243 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi087.stderr:2023-11-08T05:29:31.238+0000 7f5489b82700 -1 rbd::mirror::NamespaceReplayer: 0x55813d2f0820 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2023-11-08T05:29:31.243 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi087.stderr:2023-11-08T05:29:31.238+0000 7f5483b76700 -1 rbd::mirror::MirrorStatusUpdater 0x55813c5f8120 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-11-08T05:29:31.243 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi087.stderr:2023-11-08T05:29:31.238+0000 7f548a383700 -1 rbd::mirror::NamespaceReplayer: 0x55813bb30340 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2023-11-08T05:29:31.243 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi087.stderr:2023-11-08T05:29:31.238+0000 7f547635b700 -1 rbd::mirror::MirrorStatusUpdater 0x55813c7446c0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-11-08T05:29:31.243 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi087.stderr:2023-11-08T05:29:31.238+0000 7f5489b82700 -1 rbd::mirror::NamespaceReplayer: 0x55813d2f0820 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2023-11-08T05:29:31.244 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi087.stderr:2023-11-08T05:29:31.240+0000 7f548a383700 -1 rbd::mirror::LeaderWatcher: 0x55813ba04c00 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2023-11-08T05:29:31.244 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi087.stderr:2023-11-08T05:29:31.240+0000 7f5483375700 -1 rbd::mirror::InstanceWatcher: 0x55813a300a80 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2023-11-08T05:29:31.245 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi087.stderr:2023-11-08T05:29:31.241+0000 7f547cb68700 -1 rbd::mirror::MirrorStatusUpdater 0x55813bb1a240 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-11-08T05:29:31.245 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi087.stderr:2023-11-08T05:29:31.241+0000 7f5489b82700 -1 rbd::mirror::NamespaceReplayer: 0x55813ba684e0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2023-11-08T05:29:31.245 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi087.stderr:2023-11-08T05:29:31.241+0000 7f5483375700 -1 rbd::mirror::MirrorStatusUpdater 0x55813af27680 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-11-08T05:29:31.245 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi087.stderr:2023-11-08T05:29:31.241+0000 7f548a383700 -1 rbd::mirror::NamespaceReplayer: 0x55813ba684e0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2023-11-08T05:29:31.247 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi087.stderr:daemon-helper: command failed with exit status 1 2023-11-08T05:29:31.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-08T05:29:31.252 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi087 with status 1: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3' 2023-11-08T05:29:31.253 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Stopped 2023-11-08T05:29:31.253 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror 2023-11-08T05:29:31.358 DEBUG:tasks.rbd_mirror.cluster2.client.mirror.2:waiting for process to exit 2023-11-08T05:29:31.358 INFO:teuthology.orchestra.run:waiting for 300 2023-11-08T05:29:31.361 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi087.stderr:2023-11-08T05:29:31.357+0000 7fc450e01700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 (PID: 134122) UID: 1000 2023-11-08T05:29:31.361 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi087.stderr:2023-11-08T05:29:31.357+0000 7fc43c5d8700 -1 rbd::mirror::image_deleter::TrashWatcher: 0x55eab3875d40 handle_unregister_watcher: error unregistering watcher for trash directory: (2) No such file or directory 2023-11-08T05:29:31.362 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi087.stderr:2023-11-08T05:29:31.357+0000 7fc4495f2700 -1 rbd::mirror::image_deleter::TrashWatcher: 0x55eab5d17440 handle_unregister_watcher: error unregistering watcher for trash directory: (2) No such file or directory 2023-11-08T05:29:31.362 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi087.stderr:2023-11-08T05:29:31.357+0000 7fc4495f2700 -1 rbd::mirror::image_deleter::TrashWatcher: 0x55eab5d178c0 handle_unregister_watcher: error unregistering watcher for trash directory: (2) No such file or directory 2023-11-08T05:29:31.362 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi087.stderr:2023-11-08T05:29:31.358+0000 7fc43bdd7700 -1 rbd::mirror::PoolWatcher: 0x55eab1e88f00 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2023-11-08T05:29:31.362 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi087.stderr:2023-11-08T05:29:31.358+0000 7fc449df3700 -1 rbd::mirror::PoolWatcher: 0x55eab1e88d20 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2023-11-08T05:29:31.363 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi087.stderr:2023-11-08T05:29:31.358+0000 7fc4495f2700 -1 rbd::mirror::PoolWatcher: 0x55eab1e88960 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2023-11-08T05:29:31.363 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi087.stderr:2023-11-08T05:29:31.358+0000 7fc4435e6700 -1 rbd::mirror::PoolWatcher: 0x55eab1e89c20 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2023-11-08T05:29:31.363 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi087.stderr:2023-11-08T05:29:31.358+0000 7fc4435e6700 -1 rbd::mirror::PoolWatcher: 0x55eab5f6a1e0 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2023-11-08T05:29:31.363 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi087.stderr:2023-11-08T05:29:31.358+0000 7fc4355ca700 -1 rbd::mirror::PoolWatcher: 0x55eab1e894a0 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2023-11-08T05:29:31.364 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi087.stderr:2023-11-08T05:29:31.358+0000 7fc450600700 -1 rbd::mirror::InstanceWatcher: 0x55eab2c34e00 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.5112 object: (2) No such file or directory 2023-11-08T05:29:31.364 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi087.stderr:2023-11-08T05:29:31.358+0000 7fc450600700 -1 rbd::mirror::InstanceWatcher: 0x55eab2c35180 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.5112 object: (2) No such file or directory 2023-11-08T05:29:31.364 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi087.stderr:2023-11-08T05:29:31.358+0000 7fc44fdff700 -1 rbd::mirror::LeaderWatcher: 0x55eab5c9e900 handle_notify_lock_released: error notifying leader lock released: (2) No such file or directory 2023-11-08T05:29:31.364 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi087.stderr:2023-11-08T05:29:31.358+0000 7fc44fdff700 -1 rbd::mirror::LeaderWatcher: 0x55eab5c9e900 handle_shut_down_leader_lock: error shutting down leader lock: (2) No such file or directory 2023-11-08T05:29:31.365 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi087.stderr:2023-11-08T05:29:31.358+0000 7fc450600700 -1 rbd::mirror::LeaderWatcher: 0x55eab5c9e900 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2023-11-08T05:29:31.365 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi087.stderr:2023-11-08T05:29:31.358+0000 7fc449df3700 -1 rbd::mirror::InstanceWatcher: 0x55eab2c34e00 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2023-11-08T05:29:31.365 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi087.stderr:2023-11-08T05:29:31.358+0000 7fc4495f2700 -1 rbd::mirror::InstanceWatcher: 0x55eab2c35180 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2023-11-08T05:29:31.365 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi087.stderr:2023-11-08T05:29:31.359+0000 7fc450600700 -1 rbd::mirror::InstanceWatcher: 0x55eab2c35880 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.5115 object: (2) No such file or directory 2023-11-08T05:29:31.366 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi087.stderr:2023-11-08T05:29:31.359+0000 7fc442de5700 -1 rbd::mirror::MirrorStatusUpdater 0x55eab43e46c0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-11-08T05:29:31.366 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi087.stderr:2023-11-08T05:29:31.359+0000 7fc442de5700 -1 rbd::mirror::MirrorStatusUpdater 0x55eab4f46360 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-11-08T05:29:31.366 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi087.stderr:2023-11-08T05:29:31.359+0000 7fc44fdff700 -1 rbd::mirror::NamespaceReplayer: 0x55eab4f44000 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2023-11-08T05:29:31.366 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi087.stderr:2023-11-08T05:29:31.359+0000 7fc44fdff700 -1 rbd::mirror::NamespaceReplayer: 0x55eab4f441a0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2023-11-08T05:29:31.366 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi087.stderr:2023-11-08T05:29:31.359+0000 7fc4495f2700 -1 rbd::mirror::MirrorStatusUpdater 0x55eab4f46000 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-11-08T05:29:31.367 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi087.stderr:2023-11-08T05:29:31.359+0000 7fc4495f2700 -1 rbd::mirror::MirrorStatusUpdater 0x55eab4f46120 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-11-08T05:29:31.367 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi087.stderr:2023-11-08T05:29:31.359+0000 7fc43bdd7700 -1 rbd::mirror::InstanceWatcher: 0x55eab2c35880 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2023-11-08T05:29:31.367 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi087.stderr:2023-11-08T05:29:31.359+0000 7fc450600700 -1 rbd::mirror::NamespaceReplayer: 0x55eab4f44000 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2023-11-08T05:29:31.367 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi087.stderr:2023-11-08T05:29:31.359+0000 7fc450600700 -1 rbd::mirror::NamespaceReplayer: 0x55eab4f441a0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2023-11-08T05:29:31.368 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi087.stderr:2023-11-08T05:29:31.359+0000 7fc4355ca700 -1 rbd::mirror::MirrorStatusUpdater 0x55eab43e4900 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-11-08T05:29:31.368 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi087.stderr:2023-11-08T05:29:31.359+0000 7fc44fdff700 -1 rbd::mirror::NamespaceReplayer: 0x55eab43b24e0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2023-11-08T05:29:31.368 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi087.stderr:2023-11-08T05:29:31.360+0000 7fc43c5d8700 -1 rbd::mirror::MirrorStatusUpdater 0x55eab509a480 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-11-08T05:29:31.368 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi087.stderr:2023-11-08T05:29:31.360+0000 7fc450600700 -1 rbd::mirror::NamespaceReplayer: 0x55eab43b24e0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2023-11-08T05:29:31.370 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi087.stderr:2023-11-08T05:29:31.361+0000 7fc449df3700 -1 rbd::mirror::image_deleter::TrashWatcher: 0x55eab5ba18c0 handle_unregister_watcher: error unregistering watcher for trash directory: (2) No such file or directory 2023-11-08T05:29:31.370 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi087.stderr:2023-11-08T05:29:31.362+0000 7fc449df3700 -1 rbd::mirror::PoolWatcher: 0x55eab1e890e0 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2023-11-08T05:29:31.370 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi087.stderr:2023-11-08T05:29:31.362+0000 7fc4435e6700 -1 rbd::mirror::PoolWatcher: 0x55eab5f6a000 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2023-11-08T05:29:31.371 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi087.stderr:2023-11-08T05:29:31.362+0000 7fc450600700 -1 rbd::mirror::LeaderWatcher: 0x55eab4340c00 handle_notify_lock_released: error notifying leader lock released: (2) No such file or directory 2023-11-08T05:29:31.371 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi087.stderr:2023-11-08T05:29:31.362+0000 7fc450600700 -1 rbd::mirror::LeaderWatcher: 0x55eab4340c00 handle_shut_down_leader_lock: error shutting down leader lock: (2) No such file or directory 2023-11-08T05:29:31.371 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi087.stderr:2023-11-08T05:29:31.362+0000 7fc44fdff700 -1 rbd::mirror::LeaderWatcher: 0x55eab4340c00 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2023-11-08T05:29:31.371 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi087.stderr:2023-11-08T05:29:31.363+0000 7fc44fdff700 -1 rbd::mirror::InstanceWatcher: 0x55eab2c34a80 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.5112 object: (2) No such file or directory 2023-11-08T05:29:31.371 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi087.stderr:2023-11-08T05:29:31.363+0000 7fc449df3700 -1 rbd::mirror::InstanceWatcher: 0x55eab2c34a80 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2023-11-08T05:29:31.372 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi087.stderr:2023-11-08T05:29:31.363+0000 7fc4435e6700 -1 rbd::mirror::MirrorStatusUpdater 0x55eab43e4480 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-11-08T05:29:31.372 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi087.stderr:2023-11-08T05:29:31.363+0000 7fc44fdff700 -1 rbd::mirror::NamespaceReplayer: 0x55eab43b2340 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2023-11-08T05:29:31.372 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi087.stderr:2023-11-08T05:29:31.363+0000 7fc449df3700 -1 rbd::mirror::MirrorStatusUpdater 0x55eab3817560 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-11-08T05:29:31.372 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi087.stderr:2023-11-08T05:29:31.363+0000 7fc450600700 -1 rbd::mirror::NamespaceReplayer: 0x55eab43b2340 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2023-11-08T05:29:31.373 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi087.stderr:daemon-helper: command failed with exit status 1 2023-11-08T05:29:31.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-08T05:29:31.377 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi087 with status 1: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2' 2023-11-08T05:29:31.377 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Stopped 2023-11-08T05:29:31.378 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror 2023-11-08T05:29:31.393 DEBUG:tasks.rbd_mirror.cluster2.client.mirror.1:waiting for process to exit 2023-11-08T05:29:31.393 INFO:teuthology.orchestra.run:waiting for 300 2023-11-08T05:29:31.418 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi087.stderr:2023-11-08T05:29:31.416+0000 7f9746f2a700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 (PID: 158224) UID: 1000 2023-11-08T05:29:31.420 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi087.stderr:2023-11-08T05:29:31.417+0000 7f9745f28700 -1 rbd::mirror::LeaderWatcher: 0x55f347c7ec00 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2023-11-08T05:29:31.420 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi087.stderr:2023-11-08T05:29:31.417+0000 7f9746729700 -1 rbd::mirror::InstanceWatcher: 0x55f344cb3180 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6584 object: (2) No such file or directory 2023-11-08T05:29:31.420 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi087.stderr:2023-11-08T05:29:31.417+0000 7f9745f28700 -1 rbd::mirror::InstanceWatcher: 0x55f344cb2e00 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6584 object: (2) No such file or directory 2023-11-08T05:29:31.421 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi087.stderr:2023-11-08T05:29:31.417+0000 7f9745f28700 -1 rbd::mirror::InstanceWatcher: 0x55f344cb3500 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6586 object: (2) No such file or directory 2023-11-08T05:29:31.421 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi087.stderr:2023-11-08T05:29:31.417+0000 7f973f71b700 -1 rbd::mirror::InstanceWatcher: 0x55f344cb3180 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2023-11-08T05:29:31.421 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi087.stderr:2023-11-08T05:29:31.417+0000 7f973f71b700 -1 rbd::mirror::InstanceWatcher: 0x55f344cb2e00 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2023-11-08T05:29:31.421 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi087.stderr:2023-11-08T05:29:31.417+0000 7f9732701700 -1 rbd::mirror::InstanceWatcher: 0x55f344cb3500 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2023-11-08T05:29:31.421 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi087.stderr:2023-11-08T05:29:31.418+0000 7f9738f0e700 -1 rbd::mirror::MirrorStatusUpdater 0x55f3464d0480 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-11-08T05:29:31.422 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi087.stderr:2023-11-08T05:29:31.418+0000 7f9746729700 -1 rbd::mirror::NamespaceReplayer: 0x55f346516000 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2023-11-08T05:29:31.422 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi087.stderr:2023-11-08T05:29:31.418+0000 7f9738f0e700 -1 rbd::mirror::MirrorStatusUpdater 0x55f34651a360 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-11-08T05:29:31.422 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi087.stderr:2023-11-08T05:29:31.418+0000 7f9745f28700 -1 rbd::mirror::NamespaceReplayer: 0x55f3465161a0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2023-11-08T05:29:31.422 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi087.stderr:2023-11-08T05:29:31.418+0000 7f972bef4700 -1 rbd::mirror::MirrorStatusUpdater 0x55f3471466c0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-11-08T05:29:31.423 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi087.stderr:2023-11-08T05:29:31.418+0000 7f9746729700 -1 rbd::mirror::NamespaceReplayer: 0x55f346516b60 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2023-11-08T05:29:31.423 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi087.stderr:2023-11-08T05:29:31.418+0000 7f973f71b700 -1 rbd::mirror::MirrorStatusUpdater 0x55f34651a000 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-11-08T05:29:31.423 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi087.stderr:2023-11-08T05:29:31.418+0000 7f9745f28700 -1 rbd::mirror::NamespaceReplayer: 0x55f346516000 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2023-11-08T05:29:31.423 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi087.stderr:2023-11-08T05:29:31.418+0000 7f973f71b700 -1 rbd::mirror::MirrorStatusUpdater 0x55f34651a120 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-11-08T05:29:31.423 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi087.stderr:2023-11-08T05:29:31.418+0000 7f9746729700 -1 rbd::mirror::NamespaceReplayer: 0x55f3465161a0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2023-11-08T05:29:31.424 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi087.stderr:2023-11-08T05:29:31.418+0000 7f9731f00700 -1 rbd::mirror::MirrorStatusUpdater 0x55f3470ee6c0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-11-08T05:29:31.424 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi087.stderr:2023-11-08T05:29:31.418+0000 7f9746729700 -1 rbd::mirror::NamespaceReplayer: 0x55f346516b60 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2023-11-08T05:29:31.425 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi087.stderr:2023-11-08T05:29:31.420+0000 7f9745f28700 -1 rbd::mirror::LeaderWatcher: 0x55f3463b6c00 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2023-11-08T05:29:31.425 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi087.stderr:2023-11-08T05:29:31.420+0000 7f9745f28700 -1 rbd::mirror::InstanceWatcher: 0x55f344cb2a80 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6584 object: (2) No such file or directory 2023-11-08T05:29:31.426 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi087.stderr:2023-11-08T05:29:31.420+0000 7f973ff1c700 -1 rbd::mirror::InstanceWatcher: 0x55f344cb2a80 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2023-11-08T05:29:31.426 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi087.stderr:2023-11-08T05:29:31.421+0000 7f9738f0e700 -1 rbd::mirror::MirrorStatusUpdater 0x55f3464d0240 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-11-08T05:29:31.426 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi087.stderr:2023-11-08T05:29:31.421+0000 7f9745f28700 -1 rbd::mirror::NamespaceReplayer: 0x55f3464244e0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2023-11-08T05:29:31.426 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi087.stderr:2023-11-08T05:29:31.421+0000 7f973f71b700 -1 rbd::mirror::MirrorStatusUpdater 0x55f34588b680 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-11-08T05:29:31.427 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi087.stderr:2023-11-08T05:29:31.421+0000 7f9746729700 -1 rbd::mirror::NamespaceReplayer: 0x55f3464244e0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2023-11-08T05:29:31.427 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi087.stderr:daemon-helper: command failed with exit status 1 2023-11-08T05:29:31.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-08T05:29:31.432 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi087 with status 1: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1' 2023-11-08T05:29:31.433 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Stopped 2023-11-08T05:29:31.433 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror 2023-11-08T05:29:31.446 DEBUG:tasks.rbd_mirror.cluster2.client.mirror.0:waiting for process to exit 2023-11-08T05:29:31.447 INFO:teuthology.orchestra.run:waiting for 300 2023-11-08T05:29:31.473 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi087.stderr:2023-11-08T05:29:31.472+0000 7f443d710700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 (PID: 135275) UID: 1000 2023-11-08T05:29:31.475 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi087.stderr:2023-11-08T05:29:31.472+0000 7f443cf0f700 -1 rbd::mirror::LeaderWatcher: 0x560b01462c00 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2023-11-08T05:29:31.476 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi087.stderr:2023-11-08T05:29:31.472+0000 7f443cf0f700 -1 rbd::mirror::InstanceWatcher: 0x560afe48f180 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.5233 object: (2) No such file or directory 2023-11-08T05:29:31.476 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi087.stderr:2023-11-08T05:29:31.472+0000 7f443cf0f700 -1 rbd::mirror::InstanceWatcher: 0x560afe48ee00 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.5233 object: (2) No such file or directory 2023-11-08T05:29:31.476 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi087.stderr:2023-11-08T05:29:31.473+0000 7f443cf0f700 -1 rbd::mirror::InstanceWatcher: 0x560afe48f500 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.5235 object: (2) No such file or directory 2023-11-08T05:29:31.476 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi087.stderr:2023-11-08T05:29:31.473+0000 7f4436702700 -1 rbd::mirror::InstanceWatcher: 0x560afe48f180 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2023-11-08T05:29:31.476 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi087.stderr:2023-11-08T05:29:31.473+0000 7f4435f01700 -1 rbd::mirror::InstanceWatcher: 0x560afe48ee00 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2023-11-08T05:29:31.477 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi087.stderr:2023-11-08T05:29:31.473+0000 7f44286e6700 -1 rbd::mirror::InstanceWatcher: 0x560afe48f500 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2023-11-08T05:29:31.477 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi087.stderr:2023-11-08T05:29:31.473+0000 7f442f6f4700 -1 rbd::mirror::MirrorStatusUpdater 0x560affc24480 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-11-08T05:29:31.477 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi087.stderr:2023-11-08T05:29:31.473+0000 7f443c70e700 -1 rbd::mirror::NamespaceReplayer: 0x560affcce820 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2023-11-08T05:29:31.477 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi087.stderr:2023-11-08T05:29:31.473+0000 7f442f6f4700 -1 rbd::mirror::MirrorStatusUpdater 0x560b00798360 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-11-08T05:29:31.478 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi087.stderr:2023-11-08T05:29:31.473+0000 7f443cf0f700 -1 rbd::mirror::NamespaceReplayer: 0x560affcce9c0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2023-11-08T05:29:31.478 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi087.stderr:2023-11-08T05:29:31.473+0000 7f44226da700 -1 rbd::mirror::MirrorStatusUpdater 0x560b009406c0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-11-08T05:29:31.478 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi087.stderr:2023-11-08T05:29:31.473+0000 7f443c70e700 -1 rbd::mirror::NamespaceReplayer: 0x560b0145a4e0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2023-11-08T05:29:31.478 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi087.stderr:2023-11-08T05:29:31.473+0000 7f4436702700 -1 rbd::mirror::MirrorStatusUpdater 0x560b00798000 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-11-08T05:29:31.479 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi087.stderr:2023-11-08T05:29:31.473+0000 7f443cf0f700 -1 rbd::mirror::NamespaceReplayer: 0x560affcce820 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2023-11-08T05:29:31.479 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi087.stderr:2023-11-08T05:29:31.473+0000 7f4436702700 -1 rbd::mirror::MirrorStatusUpdater 0x560b00798120 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-11-08T05:29:31.479 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi087.stderr:2023-11-08T05:29:31.473+0000 7f443c70e700 -1 rbd::mirror::NamespaceReplayer: 0x560affcce9c0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2023-11-08T05:29:31.479 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi087.stderr:2023-11-08T05:29:31.473+0000 7f4428ee7700 -1 rbd::mirror::MirrorStatusUpdater 0x560b008da6c0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-11-08T05:29:31.479 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi087.stderr:2023-11-08T05:29:31.473+0000 7f443cf0f700 -1 rbd::mirror::NamespaceReplayer: 0x560b0145a4e0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2023-11-08T05:29:31.480 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi087.stderr:2023-11-08T05:29:31.475+0000 7f443c70e700 -1 rbd::mirror::LeaderWatcher: 0x560affba6c00 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2023-11-08T05:29:31.481 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi087.stderr:2023-11-08T05:29:31.476+0000 7f443c70e700 -1 rbd::mirror::InstanceWatcher: 0x560afe48ea80 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.5233 object: (2) No such file or directory 2023-11-08T05:29:31.481 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi087.stderr:2023-11-08T05:29:31.476+0000 7f4435f01700 -1 rbd::mirror::InstanceWatcher: 0x560afe48ea80 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2023-11-08T05:29:31.481 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi087.stderr:2023-11-08T05:29:31.476+0000 7f442fef5700 -1 rbd::mirror::MirrorStatusUpdater 0x560affc24240 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-11-08T05:29:31.481 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi087.stderr:2023-11-08T05:29:31.476+0000 7f443c70e700 -1 rbd::mirror::NamespaceReplayer: 0x560affcce000 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2023-11-08T05:29:31.482 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi087.stderr:2023-11-08T05:29:31.476+0000 7f4436702700 -1 rbd::mirror::MirrorStatusUpdater 0x560aff053680 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-11-08T05:29:31.482 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi087.stderr:2023-11-08T05:29:31.476+0000 7f443cf0f700 -1 rbd::mirror::NamespaceReplayer: 0x560affcce000 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2023-11-08T05:29:31.484 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi087.stderr:daemon-helper: command failed with exit status 1 2023-11-08T05:29:31.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-08T05:29:31.489 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi087 with status 1: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0' 2023-11-08T05:29:31.490 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Stopped 2023-11-08T05:29:31.490 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror 2023-11-08T05:29:31.503 DEBUG:tasks.rbd_mirror.cluster1.client.mirror.3:waiting for process to exit 2023-11-08T05:29:31.504 INFO:teuthology.orchestra.run:waiting for 300 2023-11-08T05:29:31.530 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi087.stderr:2023-11-08T05:29:31.529+0000 7fc166a83700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.3 (PID: 145792) UID: 1000 2023-11-08T05:29:31.531 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi087.stderr:2023-11-08T05:29:31.529+0000 7fc166282700 -1 rbd::mirror::LeaderWatcher: 0x5625908e9b00 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2023-11-08T05:29:31.531 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi087.stderr:2023-11-08T05:29:31.530+0000 7fc166282700 -1 rbd::mirror::InstanceWatcher: 0x56258e55d500 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.5475 object: (2) No such file or directory 2023-11-08T05:29:31.532 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi087.stderr:2023-11-08T05:29:31.530+0000 7fc15225a700 -1 rbd::mirror::InstanceWatcher: 0x56258e55d500 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2023-11-08T05:29:31.533 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi087.stderr:2023-11-08T05:29:31.531+0000 7fc14ba4d700 -1 rbd::mirror::MirrorStatusUpdater 0x562590a066c0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-11-08T05:29:31.533 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi087.stderr:2023-11-08T05:29:31.531+0000 7fc166282700 -1 rbd::mirror::NamespaceReplayer: 0x5625915264e0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2023-11-08T05:29:31.533 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi087.stderr:2023-11-08T05:29:31.531+0000 7fc15225a700 -1 rbd::mirror::MirrorStatusUpdater 0x5625909b2240 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-11-08T05:29:31.533 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi087.stderr:2023-11-08T05:29:31.531+0000 7fc165a81700 -1 rbd::mirror::NamespaceReplayer: 0x5625915264e0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2023-11-08T05:29:31.538 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi087.stderr:daemon-helper: command failed with exit status 1 2023-11-08T05:29:31.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-08T05:29:31.543 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi087 with status 1: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.3' 2023-11-08T05:29:31.543 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Stopped 2023-11-08T05:29:31.544 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror 2023-11-08T05:29:31.555 DEBUG:tasks.rbd_mirror.cluster1.client.mirror.2:waiting for process to exit 2023-11-08T05:29:31.556 INFO:teuthology.orchestra.run:waiting for 300 2023-11-08T05:29:31.584 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi087.stderr:2023-11-08T05:29:31.583+0000 7fd8d01bc700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 (PID: 149729) UID: 1000 2023-11-08T05:29:31.585 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi087.stderr:2023-11-08T05:29:31.584+0000 7fd8bb192700 -1 rbd::mirror::image_deleter::TrashWatcher: 0x5630729f1b00 handle_unregister_watcher: error unregistering watcher for trash directory: (2) No such file or directory 2023-11-08T05:29:31.587 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi087.stderr:2023-11-08T05:29:31.584+0000 7fd8cf9bb700 -1 rbd::mirror::InstanceWatcher: 0x56306f983180 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.5685 object: (2) No such file or directory 2023-11-08T05:29:31.587 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi087.stderr:2023-11-08T05:29:31.584+0000 7fd8bb993700 -1 rbd::mirror::PoolWatcher: 0x563072ec01e0 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2023-11-08T05:29:31.588 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi087.stderr:2023-11-08T05:29:31.584+0000 7fd8b4985700 -1 rbd::mirror::PoolWatcher: 0x563072ec0000 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2023-11-08T05:29:31.588 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi087.stderr:2023-11-08T05:29:31.584+0000 7fd8cf1ba700 -1 rbd::mirror::InstanceWatcher: 0x56306f983500 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.5685 object: (2) No such file or directory 2023-11-08T05:29:31.588 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi087.stderr:2023-11-08T05:29:31.584+0000 7fd8c89ad700 -1 rbd::mirror::InstanceWatcher: 0x56306f983180 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2023-11-08T05:29:31.588 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi087.stderr:2023-11-08T05:29:31.584+0000 7fd8c91ae700 -1 rbd::mirror::InstanceWatcher: 0x56306f983500 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2023-11-08T05:29:31.588 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi087.stderr:2023-11-08T05:29:31.584+0000 7fd8cf1ba700 -1 rbd::mirror::LeaderWatcher: 0x563072946300 handle_notify_lock_released: error notifying leader lock released: (2) No such file or directory 2023-11-08T05:29:31.589 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi087.stderr:2023-11-08T05:29:31.584+0000 7fd8cf1ba700 -1 rbd::mirror::LeaderWatcher: 0x563072946300 handle_shut_down_leader_lock: error shutting down leader lock: (2) No such file or directory 2023-11-08T05:29:31.589 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi087.stderr:2023-11-08T05:29:31.584+0000 7fd8c21a0700 -1 rbd::mirror::MirrorStatusUpdater 0x56307120c360 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-11-08T05:29:31.589 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi087.stderr:2023-11-08T05:29:31.584+0000 7fd8c21a0700 -1 rbd::mirror::MirrorStatusUpdater 0x5630711ae240 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-11-08T05:29:31.589 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi087.stderr:2023-11-08T05:29:31.584+0000 7fd8cf9bb700 -1 rbd::mirror::NamespaceReplayer: 0x56307120a000 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2023-11-08T05:29:31.589 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi087.stderr:2023-11-08T05:29:31.585+0000 7fd8cf9bb700 -1 rbd::mirror::NamespaceReplayer: 0x56307120a1a0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2023-11-08T05:29:31.590 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi087.stderr:2023-11-08T05:29:31.585+0000 7fd8cf9bb700 -1 rbd::mirror::LeaderWatcher: 0x563072946300 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2023-11-08T05:29:31.590 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi087.stderr:2023-11-08T05:29:31.585+0000 7fd8c89ad700 -1 rbd::mirror::MirrorStatusUpdater 0x56307120c000 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-11-08T05:29:31.590 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi087.stderr:2023-11-08T05:29:31.585+0000 7fd8c89ad700 -1 rbd::mirror::MirrorStatusUpdater 0x56307120c120 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-11-08T05:29:31.590 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi087.stderr:2023-11-08T05:29:31.585+0000 7fd8cf1ba700 -1 rbd::mirror::NamespaceReplayer: 0x56307120a000 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2023-11-08T05:29:31.590 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi087.stderr:2023-11-08T05:29:31.585+0000 7fd8cf1ba700 -1 rbd::mirror::NamespaceReplayer: 0x56307120a1a0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2023-11-08T05:29:31.591 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi087.stderr:2023-11-08T05:29:31.585+0000 7fd8cf1ba700 -1 rbd::mirror::InstanceWatcher: 0x56306f983880 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.5688 object: (2) No such file or directory 2023-11-08T05:29:31.591 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi087.stderr:2023-11-08T05:29:31.585+0000 7fd8bb993700 -1 rbd::mirror::InstanceWatcher: 0x56306f983880 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2023-11-08T05:29:31.591 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi087.stderr:2023-11-08T05:29:31.585+0000 7fd8b4985700 -1 rbd::mirror::MirrorStatusUpdater 0x5630711ae360 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-11-08T05:29:31.591 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi087.stderr:2023-11-08T05:29:31.585+0000 7fd8cf1ba700 -1 rbd::mirror::NamespaceReplayer: 0x5630729e4000 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2023-11-08T05:29:31.591 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi087.stderr:2023-11-08T05:29:31.586+0000 7fd8bb993700 -1 rbd::mirror::MirrorStatusUpdater 0x563071db46c0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-11-08T05:29:31.591 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi087.stderr:2023-11-08T05:29:31.586+0000 7fd8cf9bb700 -1 rbd::mirror::NamespaceReplayer: 0x5630729e4000 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2023-11-08T05:29:31.592 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi087.stderr:2023-11-08T05:29:31.588+0000 7fd8cf1ba700 -1 rbd::mirror::LeaderWatcher: 0x56307107ec00 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2023-11-08T05:29:31.593 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi087.stderr:2023-11-08T05:29:31.588+0000 7fd8cf1ba700 -1 rbd::mirror::InstanceWatcher: 0x56306f982a80 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.5685 object: (2) No such file or directory 2023-11-08T05:29:31.593 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi087.stderr:2023-11-08T05:29:31.588+0000 7fd8c91ae700 -1 rbd::mirror::InstanceWatcher: 0x56306f982a80 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2023-11-08T05:29:31.593 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi087.stderr:2023-11-08T05:29:31.588+0000 7fd8c21a0700 -1 rbd::mirror::MirrorStatusUpdater 0x5630711ae000 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-11-08T05:29:31.593 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi087.stderr:2023-11-08T05:29:31.588+0000 7fd8cf1ba700 -1 rbd::mirror::NamespaceReplayer: 0x5630710e24e0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2023-11-08T05:29:31.593 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi087.stderr:2023-11-08T05:29:31.589+0000 7fd8c89ad700 -1 rbd::mirror::MirrorStatusUpdater 0x563070563680 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-11-08T05:29:31.594 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi087.stderr:2023-11-08T05:29:31.589+0000 7fd8cf9bb700 -1 rbd::mirror::NamespaceReplayer: 0x5630710e24e0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2023-11-08T05:29:31.596 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi087.stderr:daemon-helper: command failed with exit status 1 2023-11-08T05:29:31.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-08T05:29:31.601 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi087 with status 1: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2' 2023-11-08T05:29:31.602 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Stopped 2023-11-08T05:29:31.602 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror 2023-11-08T05:29:31.614 DEBUG:tasks.rbd_mirror.cluster1.client.mirror.1:waiting for process to exit 2023-11-08T05:29:31.615 INFO:teuthology.orchestra.run:waiting for 300 2023-11-08T05:29:31.643 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi087.stderr:2023-11-08T05:29:31.641+0000 7f3fc4bb2700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 (PID: 159258) UID: 1000 2023-11-08T05:29:31.645 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi087.stderr:2023-11-08T05:29:31.642+0000 7f3fc43b1700 -1 rbd::mirror::LeaderWatcher: 0x56483f66cc00 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2023-11-08T05:29:31.645 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi087.stderr:2023-11-08T05:29:31.642+0000 7f3fc3bb0700 -1 rbd::mirror::InstanceWatcher: 0x56483c684e00 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6173 object: (2) No such file or directory 2023-11-08T05:29:31.645 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi087.stderr:2023-11-08T05:29:31.642+0000 7f3fc43b1700 -1 rbd::mirror::InstanceWatcher: 0x56483c685180 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6173 object: (2) No such file or directory 2023-11-08T05:29:31.646 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi087.stderr:2023-11-08T05:29:31.642+0000 7f3fc43b1700 -1 rbd::mirror::InstanceWatcher: 0x56483c685500 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6176 object: (2) No such file or directory 2023-11-08T05:29:31.646 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi087.stderr:2023-11-08T05:29:31.642+0000 7f3fbdba4700 -1 rbd::mirror::InstanceWatcher: 0x56483c684e00 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2023-11-08T05:29:31.646 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi087.stderr:2023-11-08T05:29:31.642+0000 7f3fbd3a3700 -1 rbd::mirror::InstanceWatcher: 0x56483c685180 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2023-11-08T05:29:31.646 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi087.stderr:2023-11-08T05:29:31.642+0000 7f3fb0389700 -1 rbd::mirror::InstanceWatcher: 0x56483c685500 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2023-11-08T05:29:31.647 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi087.stderr:2023-11-08T05:29:31.642+0000 7f3fb7397700 -1 rbd::mirror::MirrorStatusUpdater 0x56483dea8480 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-11-08T05:29:31.647 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi087.stderr:2023-11-08T05:29:31.642+0000 7f3fb7397700 -1 rbd::mirror::MirrorStatusUpdater 0x56483e986360 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-11-08T05:29:31.647 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi087.stderr:2023-11-08T05:29:31.642+0000 7f3fc43b1700 -1 rbd::mirror::NamespaceReplayer: 0x56483e984000 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2023-11-08T05:29:31.647 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi087.stderr:2023-11-08T05:29:31.642+0000 7f3fc43b1700 -1 rbd::mirror::NamespaceReplayer: 0x56483e9841a0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2023-11-08T05:29:31.647 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi087.stderr:2023-11-08T05:29:31.643+0000 7f3fbd3a3700 -1 rbd::mirror::MirrorStatusUpdater 0x56483e986000 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-11-08T05:29:31.648 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi087.stderr:2023-11-08T05:29:31.643+0000 7f3fbd3a3700 -1 rbd::mirror::MirrorStatusUpdater 0x56483e986120 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-11-08T05:29:31.648 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi087.stderr:2023-11-08T05:29:31.643+0000 7f3fc3bb0700 -1 rbd::mirror::NamespaceReplayer: 0x56483e984000 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2023-11-08T05:29:31.648 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi087.stderr:2023-11-08T05:29:31.643+0000 7f3fc3bb0700 -1 rbd::mirror::NamespaceReplayer: 0x56483e9841a0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2023-11-08T05:29:31.648 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi087.stderr:2023-11-08T05:29:31.643+0000 7f3fa937b700 -1 rbd::mirror::MirrorStatusUpdater 0x56483eb326c0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-11-08T05:29:31.648 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi087.stderr:2023-11-08T05:29:31.643+0000 7f3fc43b1700 -1 rbd::mirror::NamespaceReplayer: 0x56483f66a340 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2023-11-08T05:29:31.649 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi087.stderr:2023-11-08T05:29:31.643+0000 7f3fafb88700 -1 rbd::mirror::MirrorStatusUpdater 0x56483eaaa6c0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-11-08T05:29:31.649 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi087.stderr:2023-11-08T05:29:31.643+0000 7f3fc3bb0700 -1 rbd::mirror::NamespaceReplayer: 0x56483f66a340 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2023-11-08T05:29:31.650 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi087.stderr:2023-11-08T05:29:31.645+0000 7f3fc43b1700 -1 rbd::mirror::LeaderWatcher: 0x56483dd8ec00 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2023-11-08T05:29:31.650 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi087.stderr:2023-11-08T05:29:31.645+0000 7f3fc43b1700 -1 rbd::mirror::InstanceWatcher: 0x56483c684a80 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6173 object: (2) No such file or directory 2023-11-08T05:29:31.650 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi087.stderr:2023-11-08T05:29:31.645+0000 7f3fbd3a3700 -1 rbd::mirror::InstanceWatcher: 0x56483c684a80 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2023-11-08T05:29:31.650 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi087.stderr:2023-11-08T05:29:31.645+0000 7f3fb7397700 -1 rbd::mirror::MirrorStatusUpdater 0x56483dea8240 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-11-08T05:29:31.651 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi087.stderr:2023-11-08T05:29:31.646+0000 7f3fc43b1700 -1 rbd::mirror::NamespaceReplayer: 0x56483ddf64e0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2023-11-08T05:29:31.651 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi087.stderr:2023-11-08T05:29:31.646+0000 7f3fbd3a3700 -1 rbd::mirror::MirrorStatusUpdater 0x56483d24f680 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-11-08T05:29:31.651 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi087.stderr:2023-11-08T05:29:31.646+0000 7f3fc3bb0700 -1 rbd::mirror::NamespaceReplayer: 0x56483ddf64e0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2023-11-08T05:29:31.652 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi087.stderr:daemon-helper: command failed with exit status 1 2023-11-08T05:29:31.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-08T05:29:31.657 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi087 with status 1: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1' 2023-11-08T05:29:31.658 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Stopped 2023-11-08T05:29:31.658 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror 2023-11-08T05:29:31.670 DEBUG:tasks.rbd_mirror.cluster1.client.mirror.0:waiting for process to exit 2023-11-08T05:29:31.670 INFO:teuthology.orchestra.run:waiting for 300 2023-11-08T05:29:31.698 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi087.stderr:2023-11-08T05:29:31.697+0000 7f141cdc2700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 157852) UID: 1000 2023-11-08T05:29:31.701 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi087.stderr:2023-11-08T05:29:31.698+0000 7f141bdc0700 -1 rbd::mirror::LeaderWatcher: 0x5592637fbb00 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2023-11-08T05:29:31.701 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi087.stderr:2023-11-08T05:29:31.698+0000 7f141bdc0700 -1 rbd::mirror::InstanceWatcher: 0x559261468e00 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6111 object: (2) No such file or directory 2023-11-08T05:29:31.701 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi087.stderr:2023-11-08T05:29:31.698+0000 7f141bdc0700 -1 rbd::mirror::InstanceWatcher: 0x559261469180 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6111 object: (2) No such file or directory 2023-11-08T05:29:31.702 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi087.stderr:2023-11-08T05:29:31.698+0000 7f141c5c1700 -1 rbd::mirror::InstanceWatcher: 0x559261469500 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6114 object: (2) No such file or directory 2023-11-08T05:29:31.702 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi087.stderr:2023-11-08T05:29:31.698+0000 7f1415db4700 -1 rbd::mirror::InstanceWatcher: 0x559261468e00 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2023-11-08T05:29:31.702 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi087.stderr:2023-11-08T05:29:31.698+0000 7f14155b3700 -1 rbd::mirror::InstanceWatcher: 0x559261469180 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2023-11-08T05:29:31.702 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi087.stderr:2023-11-08T05:29:31.698+0000 7f1407d98700 -1 rbd::mirror::InstanceWatcher: 0x559261469500 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2023-11-08T05:29:31.703 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi087.stderr:2023-11-08T05:29:31.698+0000 7f140eda6700 -1 rbd::mirror::MirrorStatusUpdater 0x559262ca8240 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-11-08T05:29:31.703 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi087.stderr:2023-11-08T05:29:31.698+0000 7f140eda6700 -1 rbd::mirror::MirrorStatusUpdater 0x559262cfc360 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-11-08T05:29:31.703 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi087.stderr:2023-11-08T05:29:31.698+0000 7f141bdc0700 -1 rbd::mirror::NamespaceReplayer: 0x559262cfa000 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2023-11-08T05:29:31.703 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi087.stderr:2023-11-08T05:29:31.698+0000 7f141bdc0700 -1 rbd::mirror::NamespaceReplayer: 0x559262cfa1a0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2023-11-08T05:29:31.703 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi087.stderr:2023-11-08T05:29:31.699+0000 7f1415db4700 -1 rbd::mirror::MirrorStatusUpdater 0x559262cfc000 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-11-08T05:29:31.704 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi087.stderr:2023-11-08T05:29:31.699+0000 7f1415db4700 -1 rbd::mirror::MirrorStatusUpdater 0x559262cfc120 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-11-08T05:29:31.704 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi087.stderr:2023-11-08T05:29:31.699+0000 7f141c5c1700 -1 rbd::mirror::NamespaceReplayer: 0x559262cfa000 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2023-11-08T05:29:31.704 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi087.stderr:2023-11-08T05:29:31.699+0000 7f1401d8c700 -1 rbd::mirror::MirrorStatusUpdater 0x5592639126c0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-11-08T05:29:31.704 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi087.stderr:2023-11-08T05:29:31.699+0000 7f141c5c1700 -1 rbd::mirror::NamespaceReplayer: 0x559262cfa1a0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2023-11-08T05:29:31.704 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi087.stderr:2023-11-08T05:29:31.699+0000 7f141c5c1700 -1 rbd::mirror::NamespaceReplayer: 0x5592644e4000 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2023-11-08T05:29:31.705 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi087.stderr:2023-11-08T05:29:31.699+0000 7f1408599700 -1 rbd::mirror::MirrorStatusUpdater 0x5592638925a0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-11-08T05:29:31.705 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi087.stderr:2023-11-08T05:29:31.699+0000 7f141bdc0700 -1 rbd::mirror::NamespaceReplayer: 0x5592644e4000 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2023-11-08T05:29:31.706 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi087.stderr:2023-11-08T05:29:31.701+0000 7f141c5c1700 -1 rbd::mirror::LeaderWatcher: 0x559262b86c00 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2023-11-08T05:29:31.706 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi087.stderr:2023-11-08T05:29:31.701+0000 7f141c5c1700 -1 rbd::mirror::InstanceWatcher: 0x559261468a80 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6111 object: (2) No such file or directory 2023-11-08T05:29:31.706 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi087.stderr:2023-11-08T05:29:31.701+0000 7f14155b3700 -1 rbd::mirror::InstanceWatcher: 0x559261468a80 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2023-11-08T05:29:31.706 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi087.stderr:2023-11-08T05:29:31.702+0000 7f140eda6700 -1 rbd::mirror::MirrorStatusUpdater 0x559262ca8000 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-11-08T05:29:31.707 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi087.stderr:2023-11-08T05:29:31.702+0000 7f141c5c1700 -1 rbd::mirror::NamespaceReplayer: 0x559261ff8680 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2023-11-08T05:29:31.707 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi087.stderr:2023-11-08T05:29:31.702+0000 7f1415db4700 -1 rbd::mirror::MirrorStatusUpdater 0x55926204f560 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-11-08T05:29:31.707 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi087.stderr:2023-11-08T05:29:31.702+0000 7f141bdc0700 -1 rbd::mirror::NamespaceReplayer: 0x559261ff8680 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2023-11-08T05:29:31.708 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi087.stderr:daemon-helper: command failed with exit status 1 2023-11-08T05:29:31.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-08T05:29:31.713 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi087 with status 1: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0' 2023-11-08T05:29:31.714 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Stopped 2023-11-08T05:29:31.714 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2023-11-08T05:29:31.726 INFO:tasks.ceph.ceph_manager.cluster2:Canceling any pending splits or merges... 2023-11-08T05:29:31.726 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd dump --format=json 2023-11-08T05:29:32.040 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T05:29:32.041 INFO:teuthology.orchestra.run.smithi087.stdout:{"epoch":453,"fsid":"ab4cbc06-449d-497d-b513-220d9db2b590","created":"2023-11-08T04:52:37.369406+0000","modified":"2023-11-08T05:29:30.463086+0000","last_up_change":"2023-11-08T05:28:12.529293+0000","last_in_change":"2023-11-08T04:52:38.299675+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2023-11-08T04:52:44.563937+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2023-11-08T04:52:45.947193+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"871ecae4-6fa4-41b3-beab-e6d861330b33","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":23,"down_at":445,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6800","nonce":116186},{"type":"v1","addr":"172.21.15.87:6801","nonce":116186}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6802","nonce":116186},{"type":"v1","addr":"172.21.15.87:6803","nonce":116186}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6806","nonce":116186},{"type":"v1","addr":"172.21.15.87:6807","nonce":116186}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6804","nonce":116186},{"type":"v1","addr":"172.21.15.87:6805","nonce":116186}]},"public_addr":"172.21.15.87:6801/116186","cluster_addr":"172.21.15.87:6803/116186","heartbeat_back_addr":"172.21.15.87:6807/116186","heartbeat_front_addr":"172.21.15.87:6805/116186","state":["exists"]},{"osd":1,"uuid":"6146f4ae-0032-4ec1-b1af-f785a8265290","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":445,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6808","nonce":116187},{"type":"v1","addr":"172.21.15.87:6809","nonce":116187}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6810","nonce":116187},{"type":"v1","addr":"172.21.15.87:6811","nonce":116187}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6814","nonce":116187},{"type":"v1","addr":"172.21.15.87:6815","nonce":116187}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6812","nonce":116187},{"type":"v1","addr":"172.21.15.87:6813","nonce":116187}]},"public_addr":"172.21.15.87:6809/116187","cluster_addr":"172.21.15.87:6811/116187","heartbeat_back_addr":"172.21.15.87:6815/116187","heartbeat_front_addr":"172.21.15.87:6813/116187","state":["exists","up"]},{"osd":2,"uuid":"f6939e19-a2a3-4f8d-a25b-41f149cf05b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":445,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6816","nonce":116188},{"type":"v1","addr":"172.21.15.87:6817","nonce":116188}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6818","nonce":116188},{"type":"v1","addr":"172.21.15.87:6819","nonce":116188}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6822","nonce":116188},{"type":"v1","addr":"172.21.15.87:6823","nonce":116188}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.87:6820","nonce":116188},{"type":"v1","addr":"172.21.15.87:6821","nonce":116188}]},"public_addr":"172.21.15.87:6817/116188","cluster_addr":"172.21.15.87:6819/116188","heartbeat_back_addr":"172.21.15.87:6823/116188","heartbeat_front_addr":"172.21.15.87:6821/116188","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T04:52:43.283591+0000","dead_epoch":445},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T04:52:43.416477+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T04:52:43.365275+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.87:0/2403640120":"2023-11-08T06:27:23.738431+0000","172.21.15.87:0/859559289":"2023-11-08T06:24:37.438106+0000","172.21.15.87:0/2651482489":"2023-11-08T06:23:24.368574+0000","172.21.15.87:0/251247415":"2023-11-08T06:20:51.601004+0000","172.21.15.87:0/3028165803":"2023-11-08T06:20:22.510557+0000","172.21.15.87:0/1303213215":"2023-11-08T06:20:00.032938+0000","172.21.15.87:0/732906698":"2023-11-08T06:19:30.183731+0000","172.21.15.87:0/1240673961":"2023-11-08T06:19:21.497589+0000","172.21.15.87:0/2500566473":"2023-11-08T06:19:15.180712+0000","172.21.15.87:0/1987160530":"2023-11-08T06:19:06.473608+0000","172.21.15.87:0/3251714005":"2023-11-08T06:18:59.499116+0000","172.21.15.87:0/546747707":"2023-11-08T06:19:51.817587+0000","172.21.15.87:0/1512888499":"2023-11-08T06:18:52.455120+0000","172.21.15.87:0/1430976065":"2023-11-08T06:18:45.447490+0000","172.21.15.87:0/3108972168":"2023-11-08T06:20:15.325042+0000","172.21.15.87:0/3293517566":"2023-11-08T06:18:38.392608+0000","172.21.15.87:0/2051597440":"2023-11-08T06:18:32.032771+0000","172.21.15.87:0/2103155055":"2023-11-08T06:17:50.594440+0000","172.21.15.87:0/3890739541":"2023-11-08T06:17:43.763001+0000","172.21.15.87:0/879698242":"2023-11-08T06:28:00.105013+0000","172.21.15.87:0/455651219":"2023-11-08T06:17:36.502158+0000","172.21.15.87:0/895097808":"2023-11-08T06:14:58.543315+0000","172.21.15.87:0/2891262357":"2023-11-08T06:14:37.692042+0000","172.21.15.87:0/888194921":"2023-11-08T05:59:43.094616+0000","172.21.15.87:0/999669531":"2023-11-08T06:16:15.379129+0000","172.21.15.87:0/1043640082":"2023-11-08T06:03:07.666685+0000","172.21.15.87:0/3454458493":"2023-11-08T06:02:13.259846+0000","172.21.15.87:0/2090674419":"2023-11-08T06:15:59.383050+0000","172.21.15.87:0/2542169731":"2023-11-08T06:15:19.631347+0000","172.21.15.87:0/3264435225":"2023-11-08T06:20:29.517280+0000","172.21.15.87:0/4037269304":"2023-11-08T06:16:54.391882+0000","172.21.15.87:0/1312616359":"2023-11-08T06:01:21.524108+0000","172.21.15.87:0/2730817470":"2023-11-08T06:17:29.531444+0000","172.21.15.87:0/3133274566":"2023-11-08T06:23:55.547868+0000","172.21.15.87:0/4073996674":"2023-11-08T06:14:05.432067+0000","172.21.15.87:0/4082020968":"2023-11-08T06:29:25.724162+0000","172.21.15.87:0/2007822815":"2023-11-08T06:28:43.577352+0000","172.21.15.87:0/2266395813":"2023-11-08T06:00:32.987506+0000","172.21.15.87:0/1253780808":"2023-11-08T06:14:51.513638+0000","172.21.15.87:0/1168446212":"2023-11-08T05:58:56.501971+0000","172.21.15.87:0/3172164993":"2023-11-08T05:55:55.758164+0000","172.21.15.87:0/3949520212":"2023-11-08T05:55:09.877157+0000","172.21.15.87:0/4128316838":"2023-11-08T06:20:38.186459+0000","172.21.15.87:0/1137769085":"2023-11-08T06:15:26.610349+0000","172.21.15.87:0/3114772437":"2023-11-08T05:57:33.780993+0000","172.21.15.87:0/4098714213":"2023-11-08T06:13:58.897229+0000","172.21.15.87:0/175792286":"2023-11-08T06:19:36.564522+0000","172.21.15.87:0/356277520":"2023-11-08T06:17:22.531135+0000","172.21.15.87:0/3978840133":"2023-11-08T06:16:40.076323+0000","172.21.15.87:0/289808995":"2023-11-08T06:26:41.893408+0000","172.21.15.87:0/2298018880":"2023-11-08T06:15:05.557890+0000","172.21.15.87:0/2565405428":"2023-11-08T06:16:07.385972+0000","172.21.15.87:0/1946152431":"2023-11-08T06:15:33.660757+0000","172.21.15.87:0/3151848987":"2023-11-08T06:15:42.322905+0000","172.21.15.87:0/3804114673":"2023-11-08T06:17:15.487748+0000","172.21.15.87:0/2332155426":"2023-11-08T06:19:45.224146+0000","172.21.15.87:0/1448782546":"2023-11-08T06:15:50.530558+0000","172.21.15.87:0/3261401019":"2023-11-08T06:20:06.464727+0000","172.21.15.87:0/471974763":"2023-11-08T06:17:57.840946+0000","172.21.15.87:0/857948210":"2023-11-08T06:17:08.472285+0000","172.21.15.87:0/2843844067":"2023-11-08T06:15:12.589596+0000","172.21.15.87:0/1044293180":"2023-11-08T05:56:43.430521+0000","172.21.15.87:0/3931384271":"2023-11-08T06:16:23.424334+0000","172.21.15.87:0/1232839753":"2023-11-08T06:26:03.503161+0000","172.21.15.87:0/1477552120":"2023-11-08T06:16:31.985962+0000","172.21.15.87:0/304087051":"2023-11-08T06:25:20.116635+0000","172.21.15.87:0/3126516831":"2023-11-08T06:20:44.989450+0000","172.21.15.87:0/2526392444":"2023-11-08T06:16:48.034887+0000","172.21.15.87:0/732022958":"2023-11-08T06:14:44.505090+0000","172.21.15.87:0/4070364010":"2023-11-08T06:17:01.447750+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":384,"length":1},{"begin":386,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-08T05:29:32.054 INFO:tasks.ceph.ceph_manager.cluster2:waiting for clean 2023-11-08T05:29:32.055 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-11-08T05:29:32.336 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T05:29:32.338 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2023-11-08T05:29:32.349 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":1554,"stamp":"2023-11-08T05:29:32.044474+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":274,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2634200,"kb_used_data":408844,"kb_used_omap":780,"kb_used_meta":2220403,"kb_avail":280481320,"statfs":{"total":289910292480,"available":287212871680,"internally_reserved":0,"allocated":418656256,"data_stored":410410161,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":799447,"internal_metadata":2273692969},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,4,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-536782902,"num_objects":-164,"num_object_clones":0,"num_object_copies":-328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-115,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-164,"num_whiteouts":0,"num_read":-122056,"num_read_kb":-39011942,"num_write":-413642,"num_write_kb":-151705339,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-8,"num_bytes_recovered":-16922624,"num_keys_recovered":-2,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.424817"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":453,"reported_epoch":452,"state":"active+undersized","last_fresh":"2023-11-08T05:29:28.539262+0000","last_change":"2023-11-08T05:28:13.557071+0000","last_active":"2023-11-08T05:29:28.539262+0000","last_peered":"2023-11-08T05:29:28.539262+0000","last_clean":"2023-11-08T05:28:01.458439+0000","last_became_active":"2023-11-08T05:28:13.557071+0000","last_became_peered":"2023-11-08T05:28:13.557071+0000","last_unstale":"2023-11-08T05:29:28.539262+0000","last_undegraded":"2023-11-08T05:29:28.539262+0000","last_fullsized":"2023-11-08T05:28:13.547025+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":15,"reported_epoch":452,"state":"active+undersized","last_fresh":"2023-11-08T05:29:28.550339+0000","last_change":"2023-11-08T05:28:13.559513+0000","last_active":"2023-11-08T05:29:28.550339+0000","last_peered":"2023-11-08T05:29:28.550339+0000","last_clean":"2023-11-08T04:52:46.531964+0000","last_became_active":"2023-11-08T05:28:13.559513+0000","last_became_peered":"2023-11-08T05:28:13.559513+0000","last_unstale":"2023-11-08T05:29:28.550339+0000","last_undegraded":"2023-11-08T05:29:28.550339+0000","last_fullsized":"2023-11-08T05:28:13.540852+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":449,"reported_epoch":451,"state":"active+clean","last_fresh":"2023-11-08T05:29:26.468926+0000","last_change":"2023-11-08T04:52:47.546100+0000","last_active":"2023-11-08T05:29:26.468926+0000","last_peered":"2023-11-08T05:29:26.468926+0000","last_clean":"2023-11-08T05:29:26.468926+0000","last_became_active":"2023-11-08T04:52:47.545840+0000","last_became_peered":"2023-11-08T04:52:47.545840+0000","last_unstale":"2023-11-08T05:29:26.468926+0000","last_undegraded":"2023-11-08T05:29:26.468926+0000","last_fullsized":"2023-11-08T05:29:26.468926+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":448,"reported_epoch":452,"state":"active+clean","last_fresh":"2023-11-08T05:29:28.534734+0000","last_change":"2023-11-08T04:52:47.548597+0000","last_active":"2023-11-08T05:29:28.534734+0000","last_peered":"2023-11-08T05:29:28.534734+0000","last_clean":"2023-11-08T05:29:28.534734+0000","last_became_active":"2023-11-08T04:52:47.548294+0000","last_became_peered":"2023-11-08T04:52:47.548294+0000","last_unstale":"2023-11-08T05:29:28.534734+0000","last_undegraded":"2023-11-08T05:29:28.534734+0000","last_fullsized":"2023-11-08T05:29:28.534734+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":454,"reported_epoch":452,"state":"active+undersized","last_fresh":"2023-11-08T05:29:28.532757+0000","last_change":"2023-11-08T05:28:13.545468+0000","last_active":"2023-11-08T05:29:28.532757+0000","last_peered":"2023-11-08T05:29:28.532757+0000","last_clean":"2023-11-08T05:28:01.456723+0000","last_became_active":"2023-11-08T05:28:13.545468+0000","last_became_peered":"2023-11-08T05:28:13.545468+0000","last_unstale":"2023-11-08T05:29:28.532757+0000","last_undegraded":"2023-11-08T05:29:28.532757+0000","last_fullsized":"2023-11-08T05:28:13.537207+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_clean_scrub_stamp":"2023-11-08T04:52:45.528796+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":449,"reported_epoch":451,"state":"active+clean","last_fresh":"2023-11-08T05:29:26.467396+0000","last_change":"2023-11-08T04:52:47.546022+0000","last_active":"2023-11-08T05:29:26.467396+0000","last_peered":"2023-11-08T05:29:26.467396+0000","last_clean":"2023-11-08T05:29:26.467396+0000","last_became_active":"2023-11-08T04:52:47.545767+0000","last_became_peered":"2023-11-08T04:52:47.545767+0000","last_unstale":"2023-11-08T05:29:26.467396+0000","last_undegraded":"2023-11-08T05:29:26.467396+0000","last_fullsized":"2023-11-08T05:29:26.467396+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":453,"reported_epoch":452,"state":"active+undersized","last_fresh":"2023-11-08T05:29:28.536706+0000","last_change":"2023-11-08T05:28:13.563357+0000","last_active":"2023-11-08T05:29:28.536706+0000","last_peered":"2023-11-08T05:29:28.536706+0000","last_clean":"2023-11-08T05:28:01.457691+0000","last_became_active":"2023-11-08T05:28:13.563357+0000","last_became_peered":"2023-11-08T05:28:13.563357+0000","last_unstale":"2023-11-08T05:29:28.536706+0000","last_undegraded":"2023-11-08T05:29:28.536706+0000","last_fullsized":"2023-11-08T05:28:13.541614+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":453,"reported_epoch":452,"state":"active+undersized","last_fresh":"2023-11-08T05:29:28.537348+0000","last_change":"2023-11-08T05:28:13.550672+0000","last_active":"2023-11-08T05:29:28.537348+0000","last_peered":"2023-11-08T05:29:28.537348+0000","last_clean":"2023-11-08T05:28:01.457904+0000","last_became_active":"2023-11-08T05:28:13.550672+0000","last_became_peered":"2023-11-08T05:28:13.550672+0000","last_unstale":"2023-11-08T05:29:28.537348+0000","last_undegraded":"2023-11-08T05:29:28.537348+0000","last_fullsized":"2023-11-08T05:28:13.539595+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":453,"reported_epoch":452,"state":"active+undersized","last_fresh":"2023-11-08T05:29:28.540077+0000","last_change":"2023-11-08T05:28:13.557968+0000","last_active":"2023-11-08T05:29:28.540077+0000","last_peered":"2023-11-08T05:29:28.540077+0000","last_clean":"2023-11-08T05:28:01.458433+0000","last_became_active":"2023-11-08T05:28:13.557968+0000","last_became_peered":"2023-11-08T05:28:13.557968+0000","last_unstale":"2023-11-08T05:29:28.540077+0000","last_undegraded":"2023-11-08T05:29:28.540077+0000","last_fullsized":"2023-11-08T05:28:13.542832+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":1,"acting":1,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738812,"num_pgs":80,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":644956,"kb_used_data":5268,"kb_used_omap":275,"kb_used_meta":639404,"kb_avail":93726884,"statfs":{"total":96636764160,"available":95976329216,"internally_reserved":0,"allocated":5394432,"data_stored":2652646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":281650,"internal_metadata":654750670},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,4,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:59 2023","interfaces":[{"interface":"back","average":{"1min":0.30499999999999999,"5min":0.439,"15min":0.36099999999999999},"min":{"1min":0.17399999999999999,"5min":0.14499999999999999,"15min":0.14499999999999999},"max":{"1min":0.60099999999999998,"5min":4.9379999999999997,"15min":4.9379999999999997},"last":0.379},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.49399999999999999,"15min":0.42399999999999999},"min":{"1min":0.221,"5min":0.19600000000000001,"15min":0.17000000000000001},"max":{"1min":0.63200000000000001,"5min":4.9859999999999998,"15min":4.9859999999999998},"last":0.40899999999999997}]},{"osd":1,"last update":"Wed Nov 8 05:29:01 2023","interfaces":[{"interface":"back","average":{"1min":0.30299999999999999,"5min":0.81899999999999995,"15min":0.47399999999999998},"min":{"1min":0.16,"5min":0.153,"15min":0.153},"max":{"1min":0.68200000000000005,"5min":23.242999999999999,"15min":23.242999999999999},"last":0.36399999999999999},{"interface":"front","average":{"1min":0.46500000000000002,"5min":0.72199999999999998,"15min":0.46100000000000002},"min":{"1min":0.17699999999999999,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":2.0419999999999998,"5min":23.251000000000001,"15min":23.251000000000001},"last":0.436}]}]},{"osd":0,"up_from":8,"seq":34359738798,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1339228,"kb_used_data":398304,"kb_used_omap":291,"kb_used_meta":936476,"kb_avail":93032612,"statfs":{"total":96636764160,"available":95265394688,"internally_reserved":0,"allocated":407863296,"data_stored":405103900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":298462,"internal_metadata":958951970},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.436,"15min":0.36299999999999999},"min":{"1min":0.19700000000000001,"5min":0.14699999999999999,"15min":0.13700000000000001},"max":{"1min":3.222,"5min":6.5499999999999998,"15min":6.9989999999999997},"last":0.245},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.46300000000000002,"15min":0.42299999999999999},"min":{"1min":0.17799999999999999,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":5.0030000000000001,"5min":6.0629999999999997,"15min":9.6869999999999994},"last":0.28599999999999998}]},{"osd":2,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.51600000000000001,"15min":0.47099999999999997},"min":{"1min":0.21099999999999999,"5min":0.153,"15min":0.153},"max":{"1min":5.7439999999999998,"5min":5.8280000000000003,"15min":9.7639999999999993},"last":0.252},{"interface":"front","average":{"1min":0.497,"5min":0.56200000000000006,"15min":0.42299999999999999},"min":{"1min":0.20399999999999999,"5min":0.187,"15min":0.16800000000000001},"max":{"1min":3.9319999999999999,"5min":8.8040000000000003,"15min":8.8040000000000003},"last":0.29899999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738814,"num_pgs":98,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":650016,"kb_used_data":5272,"kb_used_omap":214,"kb_used_meta":644521,"kb_avail":93721824,"statfs":{"total":96636764160,"available":95971147776,"internally_reserved":0,"allocated":5398528,"data_stored":2653615,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":219335,"internal_metadata":659990329},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:56 2023","interfaces":[{"interface":"back","average":{"1min":0.30399999999999999,"5min":0.34699999999999998,"15min":0.36499999999999999},"min":{"1min":0.23999999999999999,"5min":0.14199999999999999,"15min":0.14199999999999999},"max":{"1min":0.42399999999999999,"5min":1.595,"15min":7.4249999999999998},"last":0.17100000000000001},{"interface":"front","average":{"1min":0.35499999999999998,"5min":0.40899999999999997,"15min":0.376},"min":{"1min":0.23999999999999999,"5min":0.19600000000000001,"15min":0.183},"max":{"1min":0.46999999999999997,"5min":2.6859999999999999,"15min":2.6859999999999999},"last":0.23000000000000001}]},{"osd":2,"last update":"Wed Nov 8 05:28:56 2023","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.44400000000000001,"15min":0.376},"min":{"1min":0.187,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":2.2650000000000001,"5min":5.8209999999999997,"15min":5.8209999999999997},"last":0.29899999999999999},{"interface":"front","average":{"1min":0.316,"5min":0.40500000000000003,"15min":0.35999999999999999},"min":{"1min":0.19600000000000001,"5min":0.188,"15min":0.17599999999999999},"max":{"1min":1.137,"5min":4.2350000000000003,"15min":4.2350000000000003},"last":0.27100000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-08T05:29:32.351 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-11-08T05:29:32.632 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T05:29:32.634 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2023-11-08T05:29:32.645 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":1554,"stamp":"2023-11-08T05:29:32.044474+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":274,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2634200,"kb_used_data":408844,"kb_used_omap":780,"kb_used_meta":2220403,"kb_avail":280481320,"statfs":{"total":289910292480,"available":287212871680,"internally_reserved":0,"allocated":418656256,"data_stored":410410161,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":799447,"internal_metadata":2273692969},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,4,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-536782902,"num_objects":-164,"num_object_clones":0,"num_object_copies":-328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-115,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-164,"num_whiteouts":0,"num_read":-122056,"num_read_kb":-39011942,"num_write":-413642,"num_write_kb":-151705339,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-8,"num_bytes_recovered":-16922624,"num_keys_recovered":-2,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.424817"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":453,"reported_epoch":452,"state":"active+undersized","last_fresh":"2023-11-08T05:29:28.539262+0000","last_change":"2023-11-08T05:28:13.557071+0000","last_active":"2023-11-08T05:29:28.539262+0000","last_peered":"2023-11-08T05:29:28.539262+0000","last_clean":"2023-11-08T05:28:01.458439+0000","last_became_active":"2023-11-08T05:28:13.557071+0000","last_became_peered":"2023-11-08T05:28:13.557071+0000","last_unstale":"2023-11-08T05:29:28.539262+0000","last_undegraded":"2023-11-08T05:29:28.539262+0000","last_fullsized":"2023-11-08T05:28:13.547025+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":15,"reported_epoch":452,"state":"active+undersized","last_fresh":"2023-11-08T05:29:28.550339+0000","last_change":"2023-11-08T05:28:13.559513+0000","last_active":"2023-11-08T05:29:28.550339+0000","last_peered":"2023-11-08T05:29:28.550339+0000","last_clean":"2023-11-08T04:52:46.531964+0000","last_became_active":"2023-11-08T05:28:13.559513+0000","last_became_peered":"2023-11-08T05:28:13.559513+0000","last_unstale":"2023-11-08T05:29:28.550339+0000","last_undegraded":"2023-11-08T05:29:28.550339+0000","last_fullsized":"2023-11-08T05:28:13.540852+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":449,"reported_epoch":451,"state":"active+clean","last_fresh":"2023-11-08T05:29:26.468926+0000","last_change":"2023-11-08T04:52:47.546100+0000","last_active":"2023-11-08T05:29:26.468926+0000","last_peered":"2023-11-08T05:29:26.468926+0000","last_clean":"2023-11-08T05:29:26.468926+0000","last_became_active":"2023-11-08T04:52:47.545840+0000","last_became_peered":"2023-11-08T04:52:47.545840+0000","last_unstale":"2023-11-08T05:29:26.468926+0000","last_undegraded":"2023-11-08T05:29:26.468926+0000","last_fullsized":"2023-11-08T05:29:26.468926+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":448,"reported_epoch":452,"state":"active+clean","last_fresh":"2023-11-08T05:29:28.534734+0000","last_change":"2023-11-08T04:52:47.548597+0000","last_active":"2023-11-08T05:29:28.534734+0000","last_peered":"2023-11-08T05:29:28.534734+0000","last_clean":"2023-11-08T05:29:28.534734+0000","last_became_active":"2023-11-08T04:52:47.548294+0000","last_became_peered":"2023-11-08T04:52:47.548294+0000","last_unstale":"2023-11-08T05:29:28.534734+0000","last_undegraded":"2023-11-08T05:29:28.534734+0000","last_fullsized":"2023-11-08T05:29:28.534734+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":454,"reported_epoch":452,"state":"active+undersized","last_fresh":"2023-11-08T05:29:28.532757+0000","last_change":"2023-11-08T05:28:13.545468+0000","last_active":"2023-11-08T05:29:28.532757+0000","last_peered":"2023-11-08T05:29:28.532757+0000","last_clean":"2023-11-08T05:28:01.456723+0000","last_became_active":"2023-11-08T05:28:13.545468+0000","last_became_peered":"2023-11-08T05:28:13.545468+0000","last_unstale":"2023-11-08T05:29:28.532757+0000","last_undegraded":"2023-11-08T05:29:28.532757+0000","last_fullsized":"2023-11-08T05:28:13.537207+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_clean_scrub_stamp":"2023-11-08T04:52:45.528796+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":449,"reported_epoch":451,"state":"active+clean","last_fresh":"2023-11-08T05:29:26.467396+0000","last_change":"2023-11-08T04:52:47.546022+0000","last_active":"2023-11-08T05:29:26.467396+0000","last_peered":"2023-11-08T05:29:26.467396+0000","last_clean":"2023-11-08T05:29:26.467396+0000","last_became_active":"2023-11-08T04:52:47.545767+0000","last_became_peered":"2023-11-08T04:52:47.545767+0000","last_unstale":"2023-11-08T05:29:26.467396+0000","last_undegraded":"2023-11-08T05:29:26.467396+0000","last_fullsized":"2023-11-08T05:29:26.467396+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":453,"reported_epoch":452,"state":"active+undersized","last_fresh":"2023-11-08T05:29:28.536706+0000","last_change":"2023-11-08T05:28:13.563357+0000","last_active":"2023-11-08T05:29:28.536706+0000","last_peered":"2023-11-08T05:29:28.536706+0000","last_clean":"2023-11-08T05:28:01.457691+0000","last_became_active":"2023-11-08T05:28:13.563357+0000","last_became_peered":"2023-11-08T05:28:13.563357+0000","last_unstale":"2023-11-08T05:29:28.536706+0000","last_undegraded":"2023-11-08T05:29:28.536706+0000","last_fullsized":"2023-11-08T05:28:13.541614+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":453,"reported_epoch":452,"state":"active+undersized","last_fresh":"2023-11-08T05:29:28.537348+0000","last_change":"2023-11-08T05:28:13.550672+0000","last_active":"2023-11-08T05:29:28.537348+0000","last_peered":"2023-11-08T05:29:28.537348+0000","last_clean":"2023-11-08T05:28:01.457904+0000","last_became_active":"2023-11-08T05:28:13.550672+0000","last_became_peered":"2023-11-08T05:28:13.550672+0000","last_unstale":"2023-11-08T05:29:28.537348+0000","last_undegraded":"2023-11-08T05:29:28.537348+0000","last_fullsized":"2023-11-08T05:28:13.539595+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":453,"reported_epoch":452,"state":"active+undersized","last_fresh":"2023-11-08T05:29:28.540077+0000","last_change":"2023-11-08T05:28:13.557968+0000","last_active":"2023-11-08T05:29:28.540077+0000","last_peered":"2023-11-08T05:29:28.540077+0000","last_clean":"2023-11-08T05:28:01.458433+0000","last_became_active":"2023-11-08T05:28:13.557968+0000","last_became_peered":"2023-11-08T05:28:13.557968+0000","last_unstale":"2023-11-08T05:29:28.540077+0000","last_undegraded":"2023-11-08T05:29:28.540077+0000","last_fullsized":"2023-11-08T05:28:13.542832+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":1,"acting":1,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738812,"num_pgs":80,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":644956,"kb_used_data":5268,"kb_used_omap":275,"kb_used_meta":639404,"kb_avail":93726884,"statfs":{"total":96636764160,"available":95976329216,"internally_reserved":0,"allocated":5394432,"data_stored":2652646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":281650,"internal_metadata":654750670},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,4,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:59 2023","interfaces":[{"interface":"back","average":{"1min":0.30499999999999999,"5min":0.439,"15min":0.36099999999999999},"min":{"1min":0.17399999999999999,"5min":0.14499999999999999,"15min":0.14499999999999999},"max":{"1min":0.60099999999999998,"5min":4.9379999999999997,"15min":4.9379999999999997},"last":0.379},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.49399999999999999,"15min":0.42399999999999999},"min":{"1min":0.221,"5min":0.19600000000000001,"15min":0.17000000000000001},"max":{"1min":0.63200000000000001,"5min":4.9859999999999998,"15min":4.9859999999999998},"last":0.40899999999999997}]},{"osd":1,"last update":"Wed Nov 8 05:29:01 2023","interfaces":[{"interface":"back","average":{"1min":0.30299999999999999,"5min":0.81899999999999995,"15min":0.47399999999999998},"min":{"1min":0.16,"5min":0.153,"15min":0.153},"max":{"1min":0.68200000000000005,"5min":23.242999999999999,"15min":23.242999999999999},"last":0.36399999999999999},{"interface":"front","average":{"1min":0.46500000000000002,"5min":0.72199999999999998,"15min":0.46100000000000002},"min":{"1min":0.17699999999999999,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":2.0419999999999998,"5min":23.251000000000001,"15min":23.251000000000001},"last":0.436}]}]},{"osd":0,"up_from":8,"seq":34359738798,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1339228,"kb_used_data":398304,"kb_used_omap":291,"kb_used_meta":936476,"kb_avail":93032612,"statfs":{"total":96636764160,"available":95265394688,"internally_reserved":0,"allocated":407863296,"data_stored":405103900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":298462,"internal_metadata":958951970},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.436,"15min":0.36299999999999999},"min":{"1min":0.19700000000000001,"5min":0.14699999999999999,"15min":0.13700000000000001},"max":{"1min":3.222,"5min":6.5499999999999998,"15min":6.9989999999999997},"last":0.245},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.46300000000000002,"15min":0.42299999999999999},"min":{"1min":0.17799999999999999,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":5.0030000000000001,"5min":6.0629999999999997,"15min":9.6869999999999994},"last":0.28599999999999998}]},{"osd":2,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.51600000000000001,"15min":0.47099999999999997},"min":{"1min":0.21099999999999999,"5min":0.153,"15min":0.153},"max":{"1min":5.7439999999999998,"5min":5.8280000000000003,"15min":9.7639999999999993},"last":0.252},{"interface":"front","average":{"1min":0.497,"5min":0.56200000000000006,"15min":0.42299999999999999},"min":{"1min":0.20399999999999999,"5min":0.187,"15min":0.16800000000000001},"max":{"1min":3.9319999999999999,"5min":8.8040000000000003,"15min":8.8040000000000003},"last":0.29899999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738814,"num_pgs":98,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":650016,"kb_used_data":5272,"kb_used_omap":214,"kb_used_meta":644521,"kb_avail":93721824,"statfs":{"total":96636764160,"available":95971147776,"internally_reserved":0,"allocated":5398528,"data_stored":2653615,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":219335,"internal_metadata":659990329},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:56 2023","interfaces":[{"interface":"back","average":{"1min":0.30399999999999999,"5min":0.34699999999999998,"15min":0.36499999999999999},"min":{"1min":0.23999999999999999,"5min":0.14199999999999999,"15min":0.14199999999999999},"max":{"1min":0.42399999999999999,"5min":1.595,"15min":7.4249999999999998},"last":0.17100000000000001},{"interface":"front","average":{"1min":0.35499999999999998,"5min":0.40899999999999997,"15min":0.376},"min":{"1min":0.23999999999999999,"5min":0.19600000000000001,"15min":0.183},"max":{"1min":0.46999999999999997,"5min":2.6859999999999999,"15min":2.6859999999999999},"last":0.23000000000000001}]},{"osd":2,"last update":"Wed Nov 8 05:28:56 2023","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.44400000000000001,"15min":0.376},"min":{"1min":0.187,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":2.2650000000000001,"5min":5.8209999999999997,"15min":5.8209999999999997},"last":0.29899999999999999},{"interface":"front","average":{"1min":0.316,"5min":0.40500000000000003,"15min":0.35999999999999999},"min":{"1min":0.19600000000000001,"5min":0.188,"15min":0.17599999999999999},"max":{"1min":1.137,"5min":4.2350000000000003,"15min":4.2350000000000003},"last":0.27100000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-08T05:29:32.646 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-11-08T05:29:32.929 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T05:29:32.931 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2023-11-08T05:29:32.942 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":1554,"stamp":"2023-11-08T05:29:32.044474+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":274,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2634200,"kb_used_data":408844,"kb_used_omap":780,"kb_used_meta":2220403,"kb_avail":280481320,"statfs":{"total":289910292480,"available":287212871680,"internally_reserved":0,"allocated":418656256,"data_stored":410410161,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":799447,"internal_metadata":2273692969},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,4,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-536782902,"num_objects":-164,"num_object_clones":0,"num_object_copies":-328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-115,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-164,"num_whiteouts":0,"num_read":-122056,"num_read_kb":-39011942,"num_write":-413642,"num_write_kb":-151705339,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-8,"num_bytes_recovered":-16922624,"num_keys_recovered":-2,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.424817"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":453,"reported_epoch":452,"state":"active+undersized","last_fresh":"2023-11-08T05:29:28.539262+0000","last_change":"2023-11-08T05:28:13.557071+0000","last_active":"2023-11-08T05:29:28.539262+0000","last_peered":"2023-11-08T05:29:28.539262+0000","last_clean":"2023-11-08T05:28:01.458439+0000","last_became_active":"2023-11-08T05:28:13.557071+0000","last_became_peered":"2023-11-08T05:28:13.557071+0000","last_unstale":"2023-11-08T05:29:28.539262+0000","last_undegraded":"2023-11-08T05:29:28.539262+0000","last_fullsized":"2023-11-08T05:28:13.547025+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":15,"reported_epoch":452,"state":"active+undersized","last_fresh":"2023-11-08T05:29:28.550339+0000","last_change":"2023-11-08T05:28:13.559513+0000","last_active":"2023-11-08T05:29:28.550339+0000","last_peered":"2023-11-08T05:29:28.550339+0000","last_clean":"2023-11-08T04:52:46.531964+0000","last_became_active":"2023-11-08T05:28:13.559513+0000","last_became_peered":"2023-11-08T05:28:13.559513+0000","last_unstale":"2023-11-08T05:29:28.550339+0000","last_undegraded":"2023-11-08T05:29:28.550339+0000","last_fullsized":"2023-11-08T05:28:13.540852+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":449,"reported_epoch":451,"state":"active+clean","last_fresh":"2023-11-08T05:29:26.468926+0000","last_change":"2023-11-08T04:52:47.546100+0000","last_active":"2023-11-08T05:29:26.468926+0000","last_peered":"2023-11-08T05:29:26.468926+0000","last_clean":"2023-11-08T05:29:26.468926+0000","last_became_active":"2023-11-08T04:52:47.545840+0000","last_became_peered":"2023-11-08T04:52:47.545840+0000","last_unstale":"2023-11-08T05:29:26.468926+0000","last_undegraded":"2023-11-08T05:29:26.468926+0000","last_fullsized":"2023-11-08T05:29:26.468926+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":448,"reported_epoch":452,"state":"active+clean","last_fresh":"2023-11-08T05:29:28.534734+0000","last_change":"2023-11-08T04:52:47.548597+0000","last_active":"2023-11-08T05:29:28.534734+0000","last_peered":"2023-11-08T05:29:28.534734+0000","last_clean":"2023-11-08T05:29:28.534734+0000","last_became_active":"2023-11-08T04:52:47.548294+0000","last_became_peered":"2023-11-08T04:52:47.548294+0000","last_unstale":"2023-11-08T05:29:28.534734+0000","last_undegraded":"2023-11-08T05:29:28.534734+0000","last_fullsized":"2023-11-08T05:29:28.534734+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":454,"reported_epoch":452,"state":"active+undersized","last_fresh":"2023-11-08T05:29:28.532757+0000","last_change":"2023-11-08T05:28:13.545468+0000","last_active":"2023-11-08T05:29:28.532757+0000","last_peered":"2023-11-08T05:29:28.532757+0000","last_clean":"2023-11-08T05:28:01.456723+0000","last_became_active":"2023-11-08T05:28:13.545468+0000","last_became_peered":"2023-11-08T05:28:13.545468+0000","last_unstale":"2023-11-08T05:29:28.532757+0000","last_undegraded":"2023-11-08T05:29:28.532757+0000","last_fullsized":"2023-11-08T05:28:13.537207+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_clean_scrub_stamp":"2023-11-08T04:52:45.528796+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":449,"reported_epoch":451,"state":"active+clean","last_fresh":"2023-11-08T05:29:26.467396+0000","last_change":"2023-11-08T04:52:47.546022+0000","last_active":"2023-11-08T05:29:26.467396+0000","last_peered":"2023-11-08T05:29:26.467396+0000","last_clean":"2023-11-08T05:29:26.467396+0000","last_became_active":"2023-11-08T04:52:47.545767+0000","last_became_peered":"2023-11-08T04:52:47.545767+0000","last_unstale":"2023-11-08T05:29:26.467396+0000","last_undegraded":"2023-11-08T05:29:26.467396+0000","last_fullsized":"2023-11-08T05:29:26.467396+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":453,"reported_epoch":452,"state":"active+undersized","last_fresh":"2023-11-08T05:29:28.536706+0000","last_change":"2023-11-08T05:28:13.563357+0000","last_active":"2023-11-08T05:29:28.536706+0000","last_peered":"2023-11-08T05:29:28.536706+0000","last_clean":"2023-11-08T05:28:01.457691+0000","last_became_active":"2023-11-08T05:28:13.563357+0000","last_became_peered":"2023-11-08T05:28:13.563357+0000","last_unstale":"2023-11-08T05:29:28.536706+0000","last_undegraded":"2023-11-08T05:29:28.536706+0000","last_fullsized":"2023-11-08T05:28:13.541614+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":453,"reported_epoch":452,"state":"active+undersized","last_fresh":"2023-11-08T05:29:28.537348+0000","last_change":"2023-11-08T05:28:13.550672+0000","last_active":"2023-11-08T05:29:28.537348+0000","last_peered":"2023-11-08T05:29:28.537348+0000","last_clean":"2023-11-08T05:28:01.457904+0000","last_became_active":"2023-11-08T05:28:13.550672+0000","last_became_peered":"2023-11-08T05:28:13.550672+0000","last_unstale":"2023-11-08T05:29:28.537348+0000","last_undegraded":"2023-11-08T05:29:28.537348+0000","last_fullsized":"2023-11-08T05:28:13.539595+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":453,"reported_epoch":452,"state":"active+undersized","last_fresh":"2023-11-08T05:29:28.540077+0000","last_change":"2023-11-08T05:28:13.557968+0000","last_active":"2023-11-08T05:29:28.540077+0000","last_peered":"2023-11-08T05:29:28.540077+0000","last_clean":"2023-11-08T05:28:01.458433+0000","last_became_active":"2023-11-08T05:28:13.557968+0000","last_became_peered":"2023-11-08T05:28:13.557968+0000","last_unstale":"2023-11-08T05:29:28.540077+0000","last_undegraded":"2023-11-08T05:29:28.540077+0000","last_fullsized":"2023-11-08T05:28:13.542832+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":1,"acting":1,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738812,"num_pgs":80,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":644956,"kb_used_data":5268,"kb_used_omap":275,"kb_used_meta":639404,"kb_avail":93726884,"statfs":{"total":96636764160,"available":95976329216,"internally_reserved":0,"allocated":5394432,"data_stored":2652646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":281650,"internal_metadata":654750670},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,4,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:59 2023","interfaces":[{"interface":"back","average":{"1min":0.30499999999999999,"5min":0.439,"15min":0.36099999999999999},"min":{"1min":0.17399999999999999,"5min":0.14499999999999999,"15min":0.14499999999999999},"max":{"1min":0.60099999999999998,"5min":4.9379999999999997,"15min":4.9379999999999997},"last":0.379},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.49399999999999999,"15min":0.42399999999999999},"min":{"1min":0.221,"5min":0.19600000000000001,"15min":0.17000000000000001},"max":{"1min":0.63200000000000001,"5min":4.9859999999999998,"15min":4.9859999999999998},"last":0.40899999999999997}]},{"osd":1,"last update":"Wed Nov 8 05:29:01 2023","interfaces":[{"interface":"back","average":{"1min":0.30299999999999999,"5min":0.81899999999999995,"15min":0.47399999999999998},"min":{"1min":0.16,"5min":0.153,"15min":0.153},"max":{"1min":0.68200000000000005,"5min":23.242999999999999,"15min":23.242999999999999},"last":0.36399999999999999},{"interface":"front","average":{"1min":0.46500000000000002,"5min":0.72199999999999998,"15min":0.46100000000000002},"min":{"1min":0.17699999999999999,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":2.0419999999999998,"5min":23.251000000000001,"15min":23.251000000000001},"last":0.436}]}]},{"osd":0,"up_from":8,"seq":34359738798,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1339228,"kb_used_data":398304,"kb_used_omap":291,"kb_used_meta":936476,"kb_avail":93032612,"statfs":{"total":96636764160,"available":95265394688,"internally_reserved":0,"allocated":407863296,"data_stored":405103900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":298462,"internal_metadata":958951970},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.436,"15min":0.36299999999999999},"min":{"1min":0.19700000000000001,"5min":0.14699999999999999,"15min":0.13700000000000001},"max":{"1min":3.222,"5min":6.5499999999999998,"15min":6.9989999999999997},"last":0.245},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.46300000000000002,"15min":0.42299999999999999},"min":{"1min":0.17799999999999999,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":5.0030000000000001,"5min":6.0629999999999997,"15min":9.6869999999999994},"last":0.28599999999999998}]},{"osd":2,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.51600000000000001,"15min":0.47099999999999997},"min":{"1min":0.21099999999999999,"5min":0.153,"15min":0.153},"max":{"1min":5.7439999999999998,"5min":5.8280000000000003,"15min":9.7639999999999993},"last":0.252},{"interface":"front","average":{"1min":0.497,"5min":0.56200000000000006,"15min":0.42299999999999999},"min":{"1min":0.20399999999999999,"5min":0.187,"15min":0.16800000000000001},"max":{"1min":3.9319999999999999,"5min":8.8040000000000003,"15min":8.8040000000000003},"last":0.29899999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738814,"num_pgs":98,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":650016,"kb_used_data":5272,"kb_used_omap":214,"kb_used_meta":644521,"kb_avail":93721824,"statfs":{"total":96636764160,"available":95971147776,"internally_reserved":0,"allocated":5398528,"data_stored":2653615,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":219335,"internal_metadata":659990329},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:56 2023","interfaces":[{"interface":"back","average":{"1min":0.30399999999999999,"5min":0.34699999999999998,"15min":0.36499999999999999},"min":{"1min":0.23999999999999999,"5min":0.14199999999999999,"15min":0.14199999999999999},"max":{"1min":0.42399999999999999,"5min":1.595,"15min":7.4249999999999998},"last":0.17100000000000001},{"interface":"front","average":{"1min":0.35499999999999998,"5min":0.40899999999999997,"15min":0.376},"min":{"1min":0.23999999999999999,"5min":0.19600000000000001,"15min":0.183},"max":{"1min":0.46999999999999997,"5min":2.6859999999999999,"15min":2.6859999999999999},"last":0.23000000000000001}]},{"osd":2,"last update":"Wed Nov 8 05:28:56 2023","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.44400000000000001,"15min":0.376},"min":{"1min":0.187,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":2.2650000000000001,"5min":5.8209999999999997,"15min":5.8209999999999997},"last":0.29899999999999999},{"interface":"front","average":{"1min":0.316,"5min":0.40500000000000003,"15min":0.35999999999999999},"min":{"1min":0.19600000000000001,"5min":0.188,"15min":0.17599999999999999},"max":{"1min":1.137,"5min":4.2350000000000003,"15min":4.2350000000000003},"last":0.27100000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-08T05:29:32.943 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-11-08T05:29:32.944 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 453, 'reported_epoch': 452, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:29:28.539262+0000', 'last_change': '2023-11-08T05:28:13.557071+0000', 'last_active': '2023-11-08T05:29:28.539262+0000', 'last_peered': '2023-11-08T05:29:28.539262+0000', 'last_clean': '2023-11-08T05:28:01.458439+0000', 'last_became_active': '2023-11-08T05:28:13.557071+0000', 'last_became_peered': '2023-11-08T05:28:13.557071+0000', 'last_unstale': '2023-11-08T05:29:28.539262+0000', 'last_undegraded': '2023-11-08T05:29:28.539262+0000', 'last_fullsized': '2023-11-08T05:28:13.547025+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:29:32.944 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-11-08T05:29:32.944 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 15, 'reported_epoch': 452, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:29:28.550339+0000', 'last_change': '2023-11-08T05:28:13.559513+0000', 'last_active': '2023-11-08T05:29:28.550339+0000', 'last_peered': '2023-11-08T05:29:28.550339+0000', 'last_clean': '2023-11-08T04:52:46.531964+0000', 'last_became_active': '2023-11-08T05:28:13.559513+0000', 'last_became_peered': '2023-11-08T05:28:13.559513+0000', 'last_unstale': '2023-11-08T05:29:28.550339+0000', 'last_undegraded': '2023-11-08T05:29:28.550339+0000', 'last_fullsized': '2023-11-08T05:28:13.540852+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:29:32.944 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-11-08T05:29:32.945 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "0'0", 'reported_seq': 454, 'reported_epoch': 452, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:29:28.532757+0000', 'last_change': '2023-11-08T05:28:13.545468+0000', 'last_active': '2023-11-08T05:29:28.532757+0000', 'last_peered': '2023-11-08T05:29:28.532757+0000', 'last_clean': '2023-11-08T05:28:01.456723+0000', 'last_became_active': '2023-11-08T05:28:13.545468+0000', 'last_became_peered': '2023-11-08T05:28:13.545468+0000', 'last_unstale': '2023-11-08T05:29:28.532757+0000', 'last_undegraded': '2023-11-08T05:29:28.532757+0000', 'last_fullsized': '2023-11-08T05:28:13.537207+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 9, 'last_epoch_clean': 10, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:45.528796+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:45.528796+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:45.528796+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:29:32.945 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-11-08T05:29:32.946 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 453, 'reported_epoch': 452, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:29:28.536706+0000', 'last_change': '2023-11-08T05:28:13.563357+0000', 'last_active': '2023-11-08T05:29:28.536706+0000', 'last_peered': '2023-11-08T05:29:28.536706+0000', 'last_clean': '2023-11-08T05:28:01.457691+0000', 'last_became_active': '2023-11-08T05:28:13.563357+0000', 'last_became_peered': '2023-11-08T05:28:13.563357+0000', 'last_unstale': '2023-11-08T05:29:28.536706+0000', 'last_undegraded': '2023-11-08T05:29:28.536706+0000', 'last_fullsized': '2023-11-08T05:28:13.541614+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:29:32.946 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-11-08T05:29:32.946 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 453, 'reported_epoch': 452, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:29:28.537348+0000', 'last_change': '2023-11-08T05:28:13.550672+0000', 'last_active': '2023-11-08T05:29:28.537348+0000', 'last_peered': '2023-11-08T05:29:28.537348+0000', 'last_clean': '2023-11-08T05:28:01.457904+0000', 'last_became_active': '2023-11-08T05:28:13.550672+0000', 'last_became_peered': '2023-11-08T05:28:13.550672+0000', 'last_unstale': '2023-11-08T05:29:28.537348+0000', 'last_undegraded': '2023-11-08T05:29:28.537348+0000', 'last_fullsized': '2023-11-08T05:28:13.539595+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:29:32.947 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-11-08T05:29:32.947 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 453, 'reported_epoch': 452, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:29:28.540077+0000', 'last_change': '2023-11-08T05:28:13.557968+0000', 'last_active': '2023-11-08T05:29:28.540077+0000', 'last_peered': '2023-11-08T05:29:28.540077+0000', 'last_clean': '2023-11-08T05:28:01.458433+0000', 'last_became_active': '2023-11-08T05:28:13.557968+0000', 'last_became_peered': '2023-11-08T05:28:13.557968+0000', 'last_unstale': '2023-11-08T05:29:28.540077+0000', 'last_undegraded': '2023-11-08T05:29:28.540077+0000', 'last_fullsized': '2023-11-08T05:28:13.542832+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:29:32.948 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-11-08T05:29:33.292 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T05:29:33.293 INFO:teuthology.orchestra.run.smithi087.stdout:{"fsid":"ab4cbc06-449d-497d-b513-220d9db2b590","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 6 pgs undersized","count":6},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2215,"monmap":{"epoch":1,"min_mon_release_name":"pacific","num_mons":1},"osdmap":{"epoch":453,"num_osds":3,"num_up_osds":2,"osd_up_since":1699421292,"num_in_osds":3,"osd_in_since":1699419158,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":6},{"state_name":"active+clean","count":3}],"num_pgs":9,"num_pools":2,"num_objects":0,"data_bytes":0,"bytes_used":2697420800,"bytes_avail":287212871680,"bytes_total":289910292480},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":41,"modified":"2023-11-08T05:29:32.044544+0000","services":{}},"progress_events":{}} 2023-11-08T05:29:33.306 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-11-08T05:29:33.306 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-11-08T05:29:33.592 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T05:29:33.594 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2023-11-08T05:29:33.605 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":1554,"stamp":"2023-11-08T05:29:32.044474+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":274,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2634200,"kb_used_data":408844,"kb_used_omap":780,"kb_used_meta":2220403,"kb_avail":280481320,"statfs":{"total":289910292480,"available":287212871680,"internally_reserved":0,"allocated":418656256,"data_stored":410410161,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":799447,"internal_metadata":2273692969},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,4,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-536782902,"num_objects":-164,"num_object_clones":0,"num_object_copies":-328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-115,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-164,"num_whiteouts":0,"num_read":-122056,"num_read_kb":-39011942,"num_write":-413642,"num_write_kb":-151705339,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-8,"num_bytes_recovered":-16922624,"num_keys_recovered":-2,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.424817"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":453,"reported_epoch":452,"state":"active+undersized","last_fresh":"2023-11-08T05:29:28.539262+0000","last_change":"2023-11-08T05:28:13.557071+0000","last_active":"2023-11-08T05:29:28.539262+0000","last_peered":"2023-11-08T05:29:28.539262+0000","last_clean":"2023-11-08T05:28:01.458439+0000","last_became_active":"2023-11-08T05:28:13.557071+0000","last_became_peered":"2023-11-08T05:28:13.557071+0000","last_unstale":"2023-11-08T05:29:28.539262+0000","last_undegraded":"2023-11-08T05:29:28.539262+0000","last_fullsized":"2023-11-08T05:28:13.547025+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":15,"reported_epoch":452,"state":"active+undersized","last_fresh":"2023-11-08T05:29:28.550339+0000","last_change":"2023-11-08T05:28:13.559513+0000","last_active":"2023-11-08T05:29:28.550339+0000","last_peered":"2023-11-08T05:29:28.550339+0000","last_clean":"2023-11-08T04:52:46.531964+0000","last_became_active":"2023-11-08T05:28:13.559513+0000","last_became_peered":"2023-11-08T05:28:13.559513+0000","last_unstale":"2023-11-08T05:29:28.550339+0000","last_undegraded":"2023-11-08T05:29:28.550339+0000","last_fullsized":"2023-11-08T05:28:13.540852+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":449,"reported_epoch":451,"state":"active+clean","last_fresh":"2023-11-08T05:29:26.468926+0000","last_change":"2023-11-08T04:52:47.546100+0000","last_active":"2023-11-08T05:29:26.468926+0000","last_peered":"2023-11-08T05:29:26.468926+0000","last_clean":"2023-11-08T05:29:26.468926+0000","last_became_active":"2023-11-08T04:52:47.545840+0000","last_became_peered":"2023-11-08T04:52:47.545840+0000","last_unstale":"2023-11-08T05:29:26.468926+0000","last_undegraded":"2023-11-08T05:29:26.468926+0000","last_fullsized":"2023-11-08T05:29:26.468926+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":448,"reported_epoch":452,"state":"active+clean","last_fresh":"2023-11-08T05:29:28.534734+0000","last_change":"2023-11-08T04:52:47.548597+0000","last_active":"2023-11-08T05:29:28.534734+0000","last_peered":"2023-11-08T05:29:28.534734+0000","last_clean":"2023-11-08T05:29:28.534734+0000","last_became_active":"2023-11-08T04:52:47.548294+0000","last_became_peered":"2023-11-08T04:52:47.548294+0000","last_unstale":"2023-11-08T05:29:28.534734+0000","last_undegraded":"2023-11-08T05:29:28.534734+0000","last_fullsized":"2023-11-08T05:29:28.534734+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":454,"reported_epoch":452,"state":"active+undersized","last_fresh":"2023-11-08T05:29:28.532757+0000","last_change":"2023-11-08T05:28:13.545468+0000","last_active":"2023-11-08T05:29:28.532757+0000","last_peered":"2023-11-08T05:29:28.532757+0000","last_clean":"2023-11-08T05:28:01.456723+0000","last_became_active":"2023-11-08T05:28:13.545468+0000","last_became_peered":"2023-11-08T05:28:13.545468+0000","last_unstale":"2023-11-08T05:29:28.532757+0000","last_undegraded":"2023-11-08T05:29:28.532757+0000","last_fullsized":"2023-11-08T05:28:13.537207+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_clean_scrub_stamp":"2023-11-08T04:52:45.528796+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":449,"reported_epoch":451,"state":"active+clean","last_fresh":"2023-11-08T05:29:26.467396+0000","last_change":"2023-11-08T04:52:47.546022+0000","last_active":"2023-11-08T05:29:26.467396+0000","last_peered":"2023-11-08T05:29:26.467396+0000","last_clean":"2023-11-08T05:29:26.467396+0000","last_became_active":"2023-11-08T04:52:47.545767+0000","last_became_peered":"2023-11-08T04:52:47.545767+0000","last_unstale":"2023-11-08T05:29:26.467396+0000","last_undegraded":"2023-11-08T05:29:26.467396+0000","last_fullsized":"2023-11-08T05:29:26.467396+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":453,"reported_epoch":452,"state":"active+undersized","last_fresh":"2023-11-08T05:29:28.536706+0000","last_change":"2023-11-08T05:28:13.563357+0000","last_active":"2023-11-08T05:29:28.536706+0000","last_peered":"2023-11-08T05:29:28.536706+0000","last_clean":"2023-11-08T05:28:01.457691+0000","last_became_active":"2023-11-08T05:28:13.563357+0000","last_became_peered":"2023-11-08T05:28:13.563357+0000","last_unstale":"2023-11-08T05:29:28.536706+0000","last_undegraded":"2023-11-08T05:29:28.536706+0000","last_fullsized":"2023-11-08T05:28:13.541614+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":453,"reported_epoch":452,"state":"active+undersized","last_fresh":"2023-11-08T05:29:28.537348+0000","last_change":"2023-11-08T05:28:13.550672+0000","last_active":"2023-11-08T05:29:28.537348+0000","last_peered":"2023-11-08T05:29:28.537348+0000","last_clean":"2023-11-08T05:28:01.457904+0000","last_became_active":"2023-11-08T05:28:13.550672+0000","last_became_peered":"2023-11-08T05:28:13.550672+0000","last_unstale":"2023-11-08T05:29:28.537348+0000","last_undegraded":"2023-11-08T05:29:28.537348+0000","last_fullsized":"2023-11-08T05:28:13.539595+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":453,"reported_epoch":452,"state":"active+undersized","last_fresh":"2023-11-08T05:29:28.540077+0000","last_change":"2023-11-08T05:28:13.557968+0000","last_active":"2023-11-08T05:29:28.540077+0000","last_peered":"2023-11-08T05:29:28.540077+0000","last_clean":"2023-11-08T05:28:01.458433+0000","last_became_active":"2023-11-08T05:28:13.557968+0000","last_became_peered":"2023-11-08T05:28:13.557968+0000","last_unstale":"2023-11-08T05:29:28.540077+0000","last_undegraded":"2023-11-08T05:29:28.540077+0000","last_fullsized":"2023-11-08T05:28:13.542832+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":1,"acting":1,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738812,"num_pgs":80,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":644956,"kb_used_data":5268,"kb_used_omap":275,"kb_used_meta":639404,"kb_avail":93726884,"statfs":{"total":96636764160,"available":95976329216,"internally_reserved":0,"allocated":5394432,"data_stored":2652646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":281650,"internal_metadata":654750670},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,4,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:59 2023","interfaces":[{"interface":"back","average":{"1min":0.30499999999999999,"5min":0.439,"15min":0.36099999999999999},"min":{"1min":0.17399999999999999,"5min":0.14499999999999999,"15min":0.14499999999999999},"max":{"1min":0.60099999999999998,"5min":4.9379999999999997,"15min":4.9379999999999997},"last":0.379},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.49399999999999999,"15min":0.42399999999999999},"min":{"1min":0.221,"5min":0.19600000000000001,"15min":0.17000000000000001},"max":{"1min":0.63200000000000001,"5min":4.9859999999999998,"15min":4.9859999999999998},"last":0.40899999999999997}]},{"osd":1,"last update":"Wed Nov 8 05:29:01 2023","interfaces":[{"interface":"back","average":{"1min":0.30299999999999999,"5min":0.81899999999999995,"15min":0.47399999999999998},"min":{"1min":0.16,"5min":0.153,"15min":0.153},"max":{"1min":0.68200000000000005,"5min":23.242999999999999,"15min":23.242999999999999},"last":0.36399999999999999},{"interface":"front","average":{"1min":0.46500000000000002,"5min":0.72199999999999998,"15min":0.46100000000000002},"min":{"1min":0.17699999999999999,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":2.0419999999999998,"5min":23.251000000000001,"15min":23.251000000000001},"last":0.436}]}]},{"osd":0,"up_from":8,"seq":34359738798,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1339228,"kb_used_data":398304,"kb_used_omap":291,"kb_used_meta":936476,"kb_avail":93032612,"statfs":{"total":96636764160,"available":95265394688,"internally_reserved":0,"allocated":407863296,"data_stored":405103900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":298462,"internal_metadata":958951970},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.436,"15min":0.36299999999999999},"min":{"1min":0.19700000000000001,"5min":0.14699999999999999,"15min":0.13700000000000001},"max":{"1min":3.222,"5min":6.5499999999999998,"15min":6.9989999999999997},"last":0.245},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.46300000000000002,"15min":0.42299999999999999},"min":{"1min":0.17799999999999999,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":5.0030000000000001,"5min":6.0629999999999997,"15min":9.6869999999999994},"last":0.28599999999999998}]},{"osd":2,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.51600000000000001,"15min":0.47099999999999997},"min":{"1min":0.21099999999999999,"5min":0.153,"15min":0.153},"max":{"1min":5.7439999999999998,"5min":5.8280000000000003,"15min":9.7639999999999993},"last":0.252},{"interface":"front","average":{"1min":0.497,"5min":0.56200000000000006,"15min":0.42299999999999999},"min":{"1min":0.20399999999999999,"5min":0.187,"15min":0.16800000000000001},"max":{"1min":3.9319999999999999,"5min":8.8040000000000003,"15min":8.8040000000000003},"last":0.29899999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738814,"num_pgs":98,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":650016,"kb_used_data":5272,"kb_used_omap":214,"kb_used_meta":644521,"kb_avail":93721824,"statfs":{"total":96636764160,"available":95971147776,"internally_reserved":0,"allocated":5398528,"data_stored":2653615,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":219335,"internal_metadata":659990329},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:56 2023","interfaces":[{"interface":"back","average":{"1min":0.30399999999999999,"5min":0.34699999999999998,"15min":0.36499999999999999},"min":{"1min":0.23999999999999999,"5min":0.14199999999999999,"15min":0.14199999999999999},"max":{"1min":0.42399999999999999,"5min":1.595,"15min":7.4249999999999998},"last":0.17100000000000001},{"interface":"front","average":{"1min":0.35499999999999998,"5min":0.40899999999999997,"15min":0.376},"min":{"1min":0.23999999999999999,"5min":0.19600000000000001,"15min":0.183},"max":{"1min":0.46999999999999997,"5min":2.6859999999999999,"15min":2.6859999999999999},"last":0.23000000000000001}]},{"osd":2,"last update":"Wed Nov 8 05:28:56 2023","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.44400000000000001,"15min":0.376},"min":{"1min":0.187,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":2.2650000000000001,"5min":5.8209999999999997,"15min":5.8209999999999997},"last":0.29899999999999999},{"interface":"front","average":{"1min":0.316,"5min":0.40500000000000003,"15min":0.35999999999999999},"min":{"1min":0.19600000000000001,"5min":0.188,"15min":0.17599999999999999},"max":{"1min":1.137,"5min":4.2350000000000003,"15min":4.2350000000000003},"last":0.27100000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-08T05:29:33.703 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~76s 2023-11-08T05:29:36.608 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-11-08T05:29:36.905 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T05:29:36.907 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2023-11-08T05:29:36.918 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":1556,"stamp":"2023-11-08T05:29:36.045383+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2634224,"kb_used_data":408868,"kb_used_omap":780,"kb_used_meta":2220403,"kb_avail":280481296,"statfs":{"total":289910292480,"available":287212847104,"internally_reserved":0,"allocated":418680832,"data_stored":410423593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":799447,"internal_metadata":2273692969},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-536782902,"num_objects":-164,"num_object_clones":0,"num_object_copies":-328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-115,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-164,"num_whiteouts":0,"num_read":-122056,"num_read_kb":-39011942,"num_write":-413642,"num_write_kb":-151705339,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-8,"num_bytes_recovered":-16922624,"num_keys_recovered":-2,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.424817"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":453,"reported_epoch":452,"state":"active+undersized","last_fresh":"2023-11-08T05:29:28.539262+0000","last_change":"2023-11-08T05:28:13.557071+0000","last_active":"2023-11-08T05:29:28.539262+0000","last_peered":"2023-11-08T05:29:28.539262+0000","last_clean":"2023-11-08T05:28:01.458439+0000","last_became_active":"2023-11-08T05:28:13.557071+0000","last_became_peered":"2023-11-08T05:28:13.557071+0000","last_unstale":"2023-11-08T05:29:28.539262+0000","last_undegraded":"2023-11-08T05:29:28.539262+0000","last_fullsized":"2023-11-08T05:28:13.547025+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":15,"reported_epoch":452,"state":"active+undersized","last_fresh":"2023-11-08T05:29:28.550339+0000","last_change":"2023-11-08T05:28:13.559513+0000","last_active":"2023-11-08T05:29:28.550339+0000","last_peered":"2023-11-08T05:29:28.550339+0000","last_clean":"2023-11-08T04:52:46.531964+0000","last_became_active":"2023-11-08T05:28:13.559513+0000","last_became_peered":"2023-11-08T05:28:13.559513+0000","last_unstale":"2023-11-08T05:29:28.550339+0000","last_undegraded":"2023-11-08T05:29:28.550339+0000","last_fullsized":"2023-11-08T05:28:13.540852+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":450,"reported_epoch":452,"state":"active+clean","last_fresh":"2023-11-08T05:29:28.468933+0000","last_change":"2023-11-08T04:52:47.546100+0000","last_active":"2023-11-08T05:29:28.468933+0000","last_peered":"2023-11-08T05:29:28.468933+0000","last_clean":"2023-11-08T05:29:28.468933+0000","last_became_active":"2023-11-08T04:52:47.545840+0000","last_became_peered":"2023-11-08T04:52:47.545840+0000","last_unstale":"2023-11-08T05:29:28.468933+0000","last_undegraded":"2023-11-08T05:29:28.468933+0000","last_fullsized":"2023-11-08T05:29:28.468933+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":448,"reported_epoch":452,"state":"active+clean","last_fresh":"2023-11-08T05:29:28.534734+0000","last_change":"2023-11-08T04:52:47.548597+0000","last_active":"2023-11-08T05:29:28.534734+0000","last_peered":"2023-11-08T05:29:28.534734+0000","last_clean":"2023-11-08T05:29:28.534734+0000","last_became_active":"2023-11-08T04:52:47.548294+0000","last_became_peered":"2023-11-08T04:52:47.548294+0000","last_unstale":"2023-11-08T05:29:28.534734+0000","last_undegraded":"2023-11-08T05:29:28.534734+0000","last_fullsized":"2023-11-08T05:29:28.534734+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":454,"reported_epoch":452,"state":"active+undersized","last_fresh":"2023-11-08T05:29:28.532757+0000","last_change":"2023-11-08T05:28:13.545468+0000","last_active":"2023-11-08T05:29:28.532757+0000","last_peered":"2023-11-08T05:29:28.532757+0000","last_clean":"2023-11-08T05:28:01.456723+0000","last_became_active":"2023-11-08T05:28:13.545468+0000","last_became_peered":"2023-11-08T05:28:13.545468+0000","last_unstale":"2023-11-08T05:29:28.532757+0000","last_undegraded":"2023-11-08T05:29:28.532757+0000","last_fullsized":"2023-11-08T05:28:13.537207+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_clean_scrub_stamp":"2023-11-08T04:52:45.528796+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":450,"reported_epoch":452,"state":"active+clean","last_fresh":"2023-11-08T05:29:28.469973+0000","last_change":"2023-11-08T04:52:47.546022+0000","last_active":"2023-11-08T05:29:28.469973+0000","last_peered":"2023-11-08T05:29:28.469973+0000","last_clean":"2023-11-08T05:29:28.469973+0000","last_became_active":"2023-11-08T04:52:47.545767+0000","last_became_peered":"2023-11-08T04:52:47.545767+0000","last_unstale":"2023-11-08T05:29:28.469973+0000","last_undegraded":"2023-11-08T05:29:28.469973+0000","last_fullsized":"2023-11-08T05:29:28.469973+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":453,"reported_epoch":452,"state":"active+undersized","last_fresh":"2023-11-08T05:29:28.536706+0000","last_change":"2023-11-08T05:28:13.563357+0000","last_active":"2023-11-08T05:29:28.536706+0000","last_peered":"2023-11-08T05:29:28.536706+0000","last_clean":"2023-11-08T05:28:01.457691+0000","last_became_active":"2023-11-08T05:28:13.563357+0000","last_became_peered":"2023-11-08T05:28:13.563357+0000","last_unstale":"2023-11-08T05:29:28.536706+0000","last_undegraded":"2023-11-08T05:29:28.536706+0000","last_fullsized":"2023-11-08T05:28:13.541614+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":453,"reported_epoch":452,"state":"active+undersized","last_fresh":"2023-11-08T05:29:28.537348+0000","last_change":"2023-11-08T05:28:13.550672+0000","last_active":"2023-11-08T05:29:28.537348+0000","last_peered":"2023-11-08T05:29:28.537348+0000","last_clean":"2023-11-08T05:28:01.457904+0000","last_became_active":"2023-11-08T05:28:13.550672+0000","last_became_peered":"2023-11-08T05:28:13.550672+0000","last_unstale":"2023-11-08T05:29:28.537348+0000","last_undegraded":"2023-11-08T05:29:28.537348+0000","last_fullsized":"2023-11-08T05:28:13.539595+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":453,"reported_epoch":452,"state":"active+undersized","last_fresh":"2023-11-08T05:29:28.540077+0000","last_change":"2023-11-08T05:28:13.557968+0000","last_active":"2023-11-08T05:29:28.540077+0000","last_peered":"2023-11-08T05:29:28.540077+0000","last_clean":"2023-11-08T05:28:01.458433+0000","last_became_active":"2023-11-08T05:28:13.557968+0000","last_became_peered":"2023-11-08T05:28:13.557968+0000","last_unstale":"2023-11-08T05:29:28.540077+0000","last_undegraded":"2023-11-08T05:29:28.540077+0000","last_fullsized":"2023-11-08T05:28:13.542832+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":1,"acting":1,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738813,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":644968,"kb_used_data":5280,"kb_used_omap":275,"kb_used_meta":639404,"kb_avail":93726872,"statfs":{"total":96636764160,"available":95976316928,"internally_reserved":0,"allocated":5406720,"data_stored":2659362,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":281650,"internal_metadata":654750670},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:59 2023","interfaces":[{"interface":"back","average":{"1min":0.30499999999999999,"5min":0.439,"15min":0.36099999999999999},"min":{"1min":0.17399999999999999,"5min":0.14499999999999999,"15min":0.14499999999999999},"max":{"1min":0.60099999999999998,"5min":4.9379999999999997,"15min":4.9379999999999997},"last":0.379},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.49399999999999999,"15min":0.42399999999999999},"min":{"1min":0.221,"5min":0.19600000000000001,"15min":0.17000000000000001},"max":{"1min":0.63200000000000001,"5min":4.9859999999999998,"15min":4.9859999999999998},"last":0.40899999999999997}]},{"osd":1,"last update":"Wed Nov 8 05:29:01 2023","interfaces":[{"interface":"back","average":{"1min":0.30299999999999999,"5min":0.81899999999999995,"15min":0.47399999999999998},"min":{"1min":0.16,"5min":0.153,"15min":0.153},"max":{"1min":0.68200000000000005,"5min":23.242999999999999,"15min":23.242999999999999},"last":0.316},{"interface":"front","average":{"1min":0.46500000000000002,"5min":0.72199999999999998,"15min":0.46100000000000002},"min":{"1min":0.17699999999999999,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":2.0419999999999998,"5min":23.251000000000001,"15min":23.251000000000001},"last":0.33500000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738798,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1339228,"kb_used_data":398304,"kb_used_omap":291,"kb_used_meta":936476,"kb_avail":93032612,"statfs":{"total":96636764160,"available":95265394688,"internally_reserved":0,"allocated":407863296,"data_stored":405103900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":298462,"internal_metadata":958951970},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.436,"15min":0.36299999999999999},"min":{"1min":0.19700000000000001,"5min":0.14699999999999999,"15min":0.13700000000000001},"max":{"1min":3.222,"5min":6.5499999999999998,"15min":6.9989999999999997},"last":0.245},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.46300000000000002,"15min":0.42299999999999999},"min":{"1min":0.17799999999999999,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":5.0030000000000001,"5min":6.0629999999999997,"15min":9.6869999999999994},"last":0.28599999999999998}]},{"osd":2,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.51600000000000001,"15min":0.47099999999999997},"min":{"1min":0.21099999999999999,"5min":0.153,"15min":0.153},"max":{"1min":5.7439999999999998,"5min":5.8280000000000003,"15min":9.7639999999999993},"last":0.252},{"interface":"front","average":{"1min":0.497,"5min":0.56200000000000006,"15min":0.42299999999999999},"min":{"1min":0.20399999999999999,"5min":0.187,"15min":0.16800000000000001},"max":{"1min":3.9319999999999999,"5min":8.8040000000000003,"15min":8.8040000000000003},"last":0.29899999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738815,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":650028,"kb_used_data":5284,"kb_used_omap":214,"kb_used_meta":644521,"kb_avail":93721812,"statfs":{"total":96636764160,"available":95971135488,"internally_reserved":0,"allocated":5410816,"data_stored":2660331,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":219335,"internal_metadata":659990329},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:56 2023","interfaces":[{"interface":"back","average":{"1min":0.30399999999999999,"5min":0.34699999999999998,"15min":0.36499999999999999},"min":{"1min":0.23999999999999999,"5min":0.14199999999999999,"15min":0.14199999999999999},"max":{"1min":0.42399999999999999,"5min":1.595,"15min":7.4249999999999998},"last":0.17100000000000001},{"interface":"front","average":{"1min":0.35499999999999998,"5min":0.40899999999999997,"15min":0.376},"min":{"1min":0.23999999999999999,"5min":0.19600000000000001,"15min":0.183},"max":{"1min":0.46999999999999997,"5min":2.6859999999999999,"15min":2.6859999999999999},"last":0.23000000000000001}]},{"osd":2,"last update":"Wed Nov 8 05:28:56 2023","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.44400000000000001,"15min":0.376},"min":{"1min":0.187,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":2.2650000000000001,"5min":5.8209999999999997,"15min":5.8209999999999997},"last":0.23999999999999999},{"interface":"front","average":{"1min":0.316,"5min":0.40500000000000003,"15min":0.35999999999999999},"min":{"1min":0.19600000000000001,"5min":0.188,"15min":0.17599999999999999},"max":{"1min":1.137,"5min":4.2350000000000003,"15min":4.2350000000000003},"last":0.25}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-08T05:29:36.919 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-11-08T05:29:37.203 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T05:29:37.204 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2023-11-08T05:29:37.216 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":1556,"stamp":"2023-11-08T05:29:36.045383+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2634224,"kb_used_data":408868,"kb_used_omap":780,"kb_used_meta":2220403,"kb_avail":280481296,"statfs":{"total":289910292480,"available":287212847104,"internally_reserved":0,"allocated":418680832,"data_stored":410423593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":799447,"internal_metadata":2273692969},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-536782902,"num_objects":-164,"num_object_clones":0,"num_object_copies":-328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-115,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-164,"num_whiteouts":0,"num_read":-122056,"num_read_kb":-39011942,"num_write":-413642,"num_write_kb":-151705339,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-8,"num_bytes_recovered":-16922624,"num_keys_recovered":-2,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.424817"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":453,"reported_epoch":452,"state":"active+undersized","last_fresh":"2023-11-08T05:29:28.539262+0000","last_change":"2023-11-08T05:28:13.557071+0000","last_active":"2023-11-08T05:29:28.539262+0000","last_peered":"2023-11-08T05:29:28.539262+0000","last_clean":"2023-11-08T05:28:01.458439+0000","last_became_active":"2023-11-08T05:28:13.557071+0000","last_became_peered":"2023-11-08T05:28:13.557071+0000","last_unstale":"2023-11-08T05:29:28.539262+0000","last_undegraded":"2023-11-08T05:29:28.539262+0000","last_fullsized":"2023-11-08T05:28:13.547025+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":15,"reported_epoch":452,"state":"active+undersized","last_fresh":"2023-11-08T05:29:28.550339+0000","last_change":"2023-11-08T05:28:13.559513+0000","last_active":"2023-11-08T05:29:28.550339+0000","last_peered":"2023-11-08T05:29:28.550339+0000","last_clean":"2023-11-08T04:52:46.531964+0000","last_became_active":"2023-11-08T05:28:13.559513+0000","last_became_peered":"2023-11-08T05:28:13.559513+0000","last_unstale":"2023-11-08T05:29:28.550339+0000","last_undegraded":"2023-11-08T05:29:28.550339+0000","last_fullsized":"2023-11-08T05:28:13.540852+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":450,"reported_epoch":452,"state":"active+clean","last_fresh":"2023-11-08T05:29:28.468933+0000","last_change":"2023-11-08T04:52:47.546100+0000","last_active":"2023-11-08T05:29:28.468933+0000","last_peered":"2023-11-08T05:29:28.468933+0000","last_clean":"2023-11-08T05:29:28.468933+0000","last_became_active":"2023-11-08T04:52:47.545840+0000","last_became_peered":"2023-11-08T04:52:47.545840+0000","last_unstale":"2023-11-08T05:29:28.468933+0000","last_undegraded":"2023-11-08T05:29:28.468933+0000","last_fullsized":"2023-11-08T05:29:28.468933+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":448,"reported_epoch":452,"state":"active+clean","last_fresh":"2023-11-08T05:29:28.534734+0000","last_change":"2023-11-08T04:52:47.548597+0000","last_active":"2023-11-08T05:29:28.534734+0000","last_peered":"2023-11-08T05:29:28.534734+0000","last_clean":"2023-11-08T05:29:28.534734+0000","last_became_active":"2023-11-08T04:52:47.548294+0000","last_became_peered":"2023-11-08T04:52:47.548294+0000","last_unstale":"2023-11-08T05:29:28.534734+0000","last_undegraded":"2023-11-08T05:29:28.534734+0000","last_fullsized":"2023-11-08T05:29:28.534734+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":454,"reported_epoch":452,"state":"active+undersized","last_fresh":"2023-11-08T05:29:28.532757+0000","last_change":"2023-11-08T05:28:13.545468+0000","last_active":"2023-11-08T05:29:28.532757+0000","last_peered":"2023-11-08T05:29:28.532757+0000","last_clean":"2023-11-08T05:28:01.456723+0000","last_became_active":"2023-11-08T05:28:13.545468+0000","last_became_peered":"2023-11-08T05:28:13.545468+0000","last_unstale":"2023-11-08T05:29:28.532757+0000","last_undegraded":"2023-11-08T05:29:28.532757+0000","last_fullsized":"2023-11-08T05:28:13.537207+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_clean_scrub_stamp":"2023-11-08T04:52:45.528796+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":450,"reported_epoch":452,"state":"active+clean","last_fresh":"2023-11-08T05:29:28.469973+0000","last_change":"2023-11-08T04:52:47.546022+0000","last_active":"2023-11-08T05:29:28.469973+0000","last_peered":"2023-11-08T05:29:28.469973+0000","last_clean":"2023-11-08T05:29:28.469973+0000","last_became_active":"2023-11-08T04:52:47.545767+0000","last_became_peered":"2023-11-08T04:52:47.545767+0000","last_unstale":"2023-11-08T05:29:28.469973+0000","last_undegraded":"2023-11-08T05:29:28.469973+0000","last_fullsized":"2023-11-08T05:29:28.469973+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":453,"reported_epoch":452,"state":"active+undersized","last_fresh":"2023-11-08T05:29:28.536706+0000","last_change":"2023-11-08T05:28:13.563357+0000","last_active":"2023-11-08T05:29:28.536706+0000","last_peered":"2023-11-08T05:29:28.536706+0000","last_clean":"2023-11-08T05:28:01.457691+0000","last_became_active":"2023-11-08T05:28:13.563357+0000","last_became_peered":"2023-11-08T05:28:13.563357+0000","last_unstale":"2023-11-08T05:29:28.536706+0000","last_undegraded":"2023-11-08T05:29:28.536706+0000","last_fullsized":"2023-11-08T05:28:13.541614+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":453,"reported_epoch":452,"state":"active+undersized","last_fresh":"2023-11-08T05:29:28.537348+0000","last_change":"2023-11-08T05:28:13.550672+0000","last_active":"2023-11-08T05:29:28.537348+0000","last_peered":"2023-11-08T05:29:28.537348+0000","last_clean":"2023-11-08T05:28:01.457904+0000","last_became_active":"2023-11-08T05:28:13.550672+0000","last_became_peered":"2023-11-08T05:28:13.550672+0000","last_unstale":"2023-11-08T05:29:28.537348+0000","last_undegraded":"2023-11-08T05:29:28.537348+0000","last_fullsized":"2023-11-08T05:28:13.539595+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":453,"reported_epoch":452,"state":"active+undersized","last_fresh":"2023-11-08T05:29:28.540077+0000","last_change":"2023-11-08T05:28:13.557968+0000","last_active":"2023-11-08T05:29:28.540077+0000","last_peered":"2023-11-08T05:29:28.540077+0000","last_clean":"2023-11-08T05:28:01.458433+0000","last_became_active":"2023-11-08T05:28:13.557968+0000","last_became_peered":"2023-11-08T05:28:13.557968+0000","last_unstale":"2023-11-08T05:29:28.540077+0000","last_undegraded":"2023-11-08T05:29:28.540077+0000","last_fullsized":"2023-11-08T05:28:13.542832+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":1,"acting":1,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738813,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":644968,"kb_used_data":5280,"kb_used_omap":275,"kb_used_meta":639404,"kb_avail":93726872,"statfs":{"total":96636764160,"available":95976316928,"internally_reserved":0,"allocated":5406720,"data_stored":2659362,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":281650,"internal_metadata":654750670},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:59 2023","interfaces":[{"interface":"back","average":{"1min":0.30499999999999999,"5min":0.439,"15min":0.36099999999999999},"min":{"1min":0.17399999999999999,"5min":0.14499999999999999,"15min":0.14499999999999999},"max":{"1min":0.60099999999999998,"5min":4.9379999999999997,"15min":4.9379999999999997},"last":0.379},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.49399999999999999,"15min":0.42399999999999999},"min":{"1min":0.221,"5min":0.19600000000000001,"15min":0.17000000000000001},"max":{"1min":0.63200000000000001,"5min":4.9859999999999998,"15min":4.9859999999999998},"last":0.40899999999999997}]},{"osd":1,"last update":"Wed Nov 8 05:29:01 2023","interfaces":[{"interface":"back","average":{"1min":0.30299999999999999,"5min":0.81899999999999995,"15min":0.47399999999999998},"min":{"1min":0.16,"5min":0.153,"15min":0.153},"max":{"1min":0.68200000000000005,"5min":23.242999999999999,"15min":23.242999999999999},"last":0.316},{"interface":"front","average":{"1min":0.46500000000000002,"5min":0.72199999999999998,"15min":0.46100000000000002},"min":{"1min":0.17699999999999999,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":2.0419999999999998,"5min":23.251000000000001,"15min":23.251000000000001},"last":0.33500000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738798,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1339228,"kb_used_data":398304,"kb_used_omap":291,"kb_used_meta":936476,"kb_avail":93032612,"statfs":{"total":96636764160,"available":95265394688,"internally_reserved":0,"allocated":407863296,"data_stored":405103900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":298462,"internal_metadata":958951970},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.436,"15min":0.36299999999999999},"min":{"1min":0.19700000000000001,"5min":0.14699999999999999,"15min":0.13700000000000001},"max":{"1min":3.222,"5min":6.5499999999999998,"15min":6.9989999999999997},"last":0.245},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.46300000000000002,"15min":0.42299999999999999},"min":{"1min":0.17799999999999999,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":5.0030000000000001,"5min":6.0629999999999997,"15min":9.6869999999999994},"last":0.28599999999999998}]},{"osd":2,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.51600000000000001,"15min":0.47099999999999997},"min":{"1min":0.21099999999999999,"5min":0.153,"15min":0.153},"max":{"1min":5.7439999999999998,"5min":5.8280000000000003,"15min":9.7639999999999993},"last":0.252},{"interface":"front","average":{"1min":0.497,"5min":0.56200000000000006,"15min":0.42299999999999999},"min":{"1min":0.20399999999999999,"5min":0.187,"15min":0.16800000000000001},"max":{"1min":3.9319999999999999,"5min":8.8040000000000003,"15min":8.8040000000000003},"last":0.29899999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738815,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":650028,"kb_used_data":5284,"kb_used_omap":214,"kb_used_meta":644521,"kb_avail":93721812,"statfs":{"total":96636764160,"available":95971135488,"internally_reserved":0,"allocated":5410816,"data_stored":2660331,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":219335,"internal_metadata":659990329},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:56 2023","interfaces":[{"interface":"back","average":{"1min":0.30399999999999999,"5min":0.34699999999999998,"15min":0.36499999999999999},"min":{"1min":0.23999999999999999,"5min":0.14199999999999999,"15min":0.14199999999999999},"max":{"1min":0.42399999999999999,"5min":1.595,"15min":7.4249999999999998},"last":0.17100000000000001},{"interface":"front","average":{"1min":0.35499999999999998,"5min":0.40899999999999997,"15min":0.376},"min":{"1min":0.23999999999999999,"5min":0.19600000000000001,"15min":0.183},"max":{"1min":0.46999999999999997,"5min":2.6859999999999999,"15min":2.6859999999999999},"last":0.23000000000000001}]},{"osd":2,"last update":"Wed Nov 8 05:28:56 2023","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.44400000000000001,"15min":0.376},"min":{"1min":0.187,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":2.2650000000000001,"5min":5.8209999999999997,"15min":5.8209999999999997},"last":0.23999999999999999},{"interface":"front","average":{"1min":0.316,"5min":0.40500000000000003,"15min":0.35999999999999999},"min":{"1min":0.19600000000000001,"5min":0.188,"15min":0.17599999999999999},"max":{"1min":1.137,"5min":4.2350000000000003,"15min":4.2350000000000003},"last":0.25}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-08T05:29:37.218 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-11-08T05:29:37.218 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 453, 'reported_epoch': 452, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:29:28.539262+0000', 'last_change': '2023-11-08T05:28:13.557071+0000', 'last_active': '2023-11-08T05:29:28.539262+0000', 'last_peered': '2023-11-08T05:29:28.539262+0000', 'last_clean': '2023-11-08T05:28:01.458439+0000', 'last_became_active': '2023-11-08T05:28:13.557071+0000', 'last_became_peered': '2023-11-08T05:28:13.557071+0000', 'last_unstale': '2023-11-08T05:29:28.539262+0000', 'last_undegraded': '2023-11-08T05:29:28.539262+0000', 'last_fullsized': '2023-11-08T05:28:13.547025+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:29:37.219 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-11-08T05:29:37.220 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 15, 'reported_epoch': 452, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:29:28.550339+0000', 'last_change': '2023-11-08T05:28:13.559513+0000', 'last_active': '2023-11-08T05:29:28.550339+0000', 'last_peered': '2023-11-08T05:29:28.550339+0000', 'last_clean': '2023-11-08T04:52:46.531964+0000', 'last_became_active': '2023-11-08T05:28:13.559513+0000', 'last_became_peered': '2023-11-08T05:28:13.559513+0000', 'last_unstale': '2023-11-08T05:29:28.550339+0000', 'last_undegraded': '2023-11-08T05:29:28.550339+0000', 'last_fullsized': '2023-11-08T05:28:13.540852+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:29:37.221 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-11-08T05:29:37.221 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "0'0", 'reported_seq': 454, 'reported_epoch': 452, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:29:28.532757+0000', 'last_change': '2023-11-08T05:28:13.545468+0000', 'last_active': '2023-11-08T05:29:28.532757+0000', 'last_peered': '2023-11-08T05:29:28.532757+0000', 'last_clean': '2023-11-08T05:28:01.456723+0000', 'last_became_active': '2023-11-08T05:28:13.545468+0000', 'last_became_peered': '2023-11-08T05:28:13.545468+0000', 'last_unstale': '2023-11-08T05:29:28.532757+0000', 'last_undegraded': '2023-11-08T05:29:28.532757+0000', 'last_fullsized': '2023-11-08T05:28:13.537207+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 9, 'last_epoch_clean': 10, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:45.528796+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:45.528796+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:45.528796+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:29:37.222 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-11-08T05:29:37.222 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 453, 'reported_epoch': 452, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:29:28.536706+0000', 'last_change': '2023-11-08T05:28:13.563357+0000', 'last_active': '2023-11-08T05:29:28.536706+0000', 'last_peered': '2023-11-08T05:29:28.536706+0000', 'last_clean': '2023-11-08T05:28:01.457691+0000', 'last_became_active': '2023-11-08T05:28:13.563357+0000', 'last_became_peered': '2023-11-08T05:28:13.563357+0000', 'last_unstale': '2023-11-08T05:29:28.536706+0000', 'last_undegraded': '2023-11-08T05:29:28.536706+0000', 'last_fullsized': '2023-11-08T05:28:13.541614+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:29:37.223 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-11-08T05:29:37.223 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 453, 'reported_epoch': 452, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:29:28.537348+0000', 'last_change': '2023-11-08T05:28:13.550672+0000', 'last_active': '2023-11-08T05:29:28.537348+0000', 'last_peered': '2023-11-08T05:29:28.537348+0000', 'last_clean': '2023-11-08T05:28:01.457904+0000', 'last_became_active': '2023-11-08T05:28:13.550672+0000', 'last_became_peered': '2023-11-08T05:28:13.550672+0000', 'last_unstale': '2023-11-08T05:29:28.537348+0000', 'last_undegraded': '2023-11-08T05:29:28.537348+0000', 'last_fullsized': '2023-11-08T05:28:13.539595+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:29:37.224 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-11-08T05:29:37.224 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 453, 'reported_epoch': 452, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:29:28.540077+0000', 'last_change': '2023-11-08T05:28:13.557968+0000', 'last_active': '2023-11-08T05:29:28.540077+0000', 'last_peered': '2023-11-08T05:29:28.540077+0000', 'last_clean': '2023-11-08T05:28:01.458433+0000', 'last_became_active': '2023-11-08T05:28:13.557968+0000', 'last_became_peered': '2023-11-08T05:28:13.557968+0000', 'last_unstale': '2023-11-08T05:29:28.540077+0000', 'last_undegraded': '2023-11-08T05:29:28.540077+0000', 'last_fullsized': '2023-11-08T05:28:13.542832+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:29:37.225 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-11-08T05:29:37.575 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T05:29:37.575 INFO:teuthology.orchestra.run.smithi087.stdout:{"fsid":"ab4cbc06-449d-497d-b513-220d9db2b590","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 6 pgs undersized","count":6},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2220,"monmap":{"epoch":1,"min_mon_release_name":"pacific","num_mons":1},"osdmap":{"epoch":453,"num_osds":3,"num_up_osds":2,"osd_up_since":1699421292,"num_in_osds":3,"osd_in_since":1699419158,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":6},{"state_name":"active+clean","count":3}],"num_pgs":9,"num_pools":2,"num_objects":0,"data_bytes":0,"bytes_used":2697445376,"bytes_avail":287212847104,"bytes_total":289910292480},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":41,"modified":"2023-11-08T05:29:32.044544+0000","services":{}},"progress_events":{}} 2023-11-08T05:29:37.588 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-11-08T05:29:37.589 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-11-08T05:29:37.873 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T05:29:37.875 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2023-11-08T05:29:37.886 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":1556,"stamp":"2023-11-08T05:29:36.045383+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2634224,"kb_used_data":408868,"kb_used_omap":780,"kb_used_meta":2220403,"kb_avail":280481296,"statfs":{"total":289910292480,"available":287212847104,"internally_reserved":0,"allocated":418680832,"data_stored":410423593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":799447,"internal_metadata":2273692969},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-536782902,"num_objects":-164,"num_object_clones":0,"num_object_copies":-328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-115,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-164,"num_whiteouts":0,"num_read":-122056,"num_read_kb":-39011942,"num_write":-413642,"num_write_kb":-151705339,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-8,"num_bytes_recovered":-16922624,"num_keys_recovered":-2,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.424817"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":453,"reported_epoch":452,"state":"active+undersized","last_fresh":"2023-11-08T05:29:28.539262+0000","last_change":"2023-11-08T05:28:13.557071+0000","last_active":"2023-11-08T05:29:28.539262+0000","last_peered":"2023-11-08T05:29:28.539262+0000","last_clean":"2023-11-08T05:28:01.458439+0000","last_became_active":"2023-11-08T05:28:13.557071+0000","last_became_peered":"2023-11-08T05:28:13.557071+0000","last_unstale":"2023-11-08T05:29:28.539262+0000","last_undegraded":"2023-11-08T05:29:28.539262+0000","last_fullsized":"2023-11-08T05:28:13.547025+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":15,"reported_epoch":452,"state":"active+undersized","last_fresh":"2023-11-08T05:29:28.550339+0000","last_change":"2023-11-08T05:28:13.559513+0000","last_active":"2023-11-08T05:29:28.550339+0000","last_peered":"2023-11-08T05:29:28.550339+0000","last_clean":"2023-11-08T04:52:46.531964+0000","last_became_active":"2023-11-08T05:28:13.559513+0000","last_became_peered":"2023-11-08T05:28:13.559513+0000","last_unstale":"2023-11-08T05:29:28.550339+0000","last_undegraded":"2023-11-08T05:29:28.550339+0000","last_fullsized":"2023-11-08T05:28:13.540852+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":450,"reported_epoch":452,"state":"active+clean","last_fresh":"2023-11-08T05:29:28.468933+0000","last_change":"2023-11-08T04:52:47.546100+0000","last_active":"2023-11-08T05:29:28.468933+0000","last_peered":"2023-11-08T05:29:28.468933+0000","last_clean":"2023-11-08T05:29:28.468933+0000","last_became_active":"2023-11-08T04:52:47.545840+0000","last_became_peered":"2023-11-08T04:52:47.545840+0000","last_unstale":"2023-11-08T05:29:28.468933+0000","last_undegraded":"2023-11-08T05:29:28.468933+0000","last_fullsized":"2023-11-08T05:29:28.468933+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":448,"reported_epoch":452,"state":"active+clean","last_fresh":"2023-11-08T05:29:28.534734+0000","last_change":"2023-11-08T04:52:47.548597+0000","last_active":"2023-11-08T05:29:28.534734+0000","last_peered":"2023-11-08T05:29:28.534734+0000","last_clean":"2023-11-08T05:29:28.534734+0000","last_became_active":"2023-11-08T04:52:47.548294+0000","last_became_peered":"2023-11-08T04:52:47.548294+0000","last_unstale":"2023-11-08T05:29:28.534734+0000","last_undegraded":"2023-11-08T05:29:28.534734+0000","last_fullsized":"2023-11-08T05:29:28.534734+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":454,"reported_epoch":452,"state":"active+undersized","last_fresh":"2023-11-08T05:29:28.532757+0000","last_change":"2023-11-08T05:28:13.545468+0000","last_active":"2023-11-08T05:29:28.532757+0000","last_peered":"2023-11-08T05:29:28.532757+0000","last_clean":"2023-11-08T05:28:01.456723+0000","last_became_active":"2023-11-08T05:28:13.545468+0000","last_became_peered":"2023-11-08T05:28:13.545468+0000","last_unstale":"2023-11-08T05:29:28.532757+0000","last_undegraded":"2023-11-08T05:29:28.532757+0000","last_fullsized":"2023-11-08T05:28:13.537207+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_clean_scrub_stamp":"2023-11-08T04:52:45.528796+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":450,"reported_epoch":452,"state":"active+clean","last_fresh":"2023-11-08T05:29:28.469973+0000","last_change":"2023-11-08T04:52:47.546022+0000","last_active":"2023-11-08T05:29:28.469973+0000","last_peered":"2023-11-08T05:29:28.469973+0000","last_clean":"2023-11-08T05:29:28.469973+0000","last_became_active":"2023-11-08T04:52:47.545767+0000","last_became_peered":"2023-11-08T04:52:47.545767+0000","last_unstale":"2023-11-08T05:29:28.469973+0000","last_undegraded":"2023-11-08T05:29:28.469973+0000","last_fullsized":"2023-11-08T05:29:28.469973+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":453,"reported_epoch":452,"state":"active+undersized","last_fresh":"2023-11-08T05:29:28.536706+0000","last_change":"2023-11-08T05:28:13.563357+0000","last_active":"2023-11-08T05:29:28.536706+0000","last_peered":"2023-11-08T05:29:28.536706+0000","last_clean":"2023-11-08T05:28:01.457691+0000","last_became_active":"2023-11-08T05:28:13.563357+0000","last_became_peered":"2023-11-08T05:28:13.563357+0000","last_unstale":"2023-11-08T05:29:28.536706+0000","last_undegraded":"2023-11-08T05:29:28.536706+0000","last_fullsized":"2023-11-08T05:28:13.541614+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":453,"reported_epoch":452,"state":"active+undersized","last_fresh":"2023-11-08T05:29:28.537348+0000","last_change":"2023-11-08T05:28:13.550672+0000","last_active":"2023-11-08T05:29:28.537348+0000","last_peered":"2023-11-08T05:29:28.537348+0000","last_clean":"2023-11-08T05:28:01.457904+0000","last_became_active":"2023-11-08T05:28:13.550672+0000","last_became_peered":"2023-11-08T05:28:13.550672+0000","last_unstale":"2023-11-08T05:29:28.537348+0000","last_undegraded":"2023-11-08T05:29:28.537348+0000","last_fullsized":"2023-11-08T05:28:13.539595+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":453,"reported_epoch":452,"state":"active+undersized","last_fresh":"2023-11-08T05:29:28.540077+0000","last_change":"2023-11-08T05:28:13.557968+0000","last_active":"2023-11-08T05:29:28.540077+0000","last_peered":"2023-11-08T05:29:28.540077+0000","last_clean":"2023-11-08T05:28:01.458433+0000","last_became_active":"2023-11-08T05:28:13.557968+0000","last_became_peered":"2023-11-08T05:28:13.557968+0000","last_unstale":"2023-11-08T05:29:28.540077+0000","last_undegraded":"2023-11-08T05:29:28.540077+0000","last_fullsized":"2023-11-08T05:28:13.542832+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":1,"acting":1,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738813,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":644968,"kb_used_data":5280,"kb_used_omap":275,"kb_used_meta":639404,"kb_avail":93726872,"statfs":{"total":96636764160,"available":95976316928,"internally_reserved":0,"allocated":5406720,"data_stored":2659362,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":281650,"internal_metadata":654750670},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:59 2023","interfaces":[{"interface":"back","average":{"1min":0.30499999999999999,"5min":0.439,"15min":0.36099999999999999},"min":{"1min":0.17399999999999999,"5min":0.14499999999999999,"15min":0.14499999999999999},"max":{"1min":0.60099999999999998,"5min":4.9379999999999997,"15min":4.9379999999999997},"last":0.379},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.49399999999999999,"15min":0.42399999999999999},"min":{"1min":0.221,"5min":0.19600000000000001,"15min":0.17000000000000001},"max":{"1min":0.63200000000000001,"5min":4.9859999999999998,"15min":4.9859999999999998},"last":0.40899999999999997}]},{"osd":1,"last update":"Wed Nov 8 05:29:01 2023","interfaces":[{"interface":"back","average":{"1min":0.30299999999999999,"5min":0.81899999999999995,"15min":0.47399999999999998},"min":{"1min":0.16,"5min":0.153,"15min":0.153},"max":{"1min":0.68200000000000005,"5min":23.242999999999999,"15min":23.242999999999999},"last":0.316},{"interface":"front","average":{"1min":0.46500000000000002,"5min":0.72199999999999998,"15min":0.46100000000000002},"min":{"1min":0.17699999999999999,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":2.0419999999999998,"5min":23.251000000000001,"15min":23.251000000000001},"last":0.33500000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738798,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1339228,"kb_used_data":398304,"kb_used_omap":291,"kb_used_meta":936476,"kb_avail":93032612,"statfs":{"total":96636764160,"available":95265394688,"internally_reserved":0,"allocated":407863296,"data_stored":405103900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":298462,"internal_metadata":958951970},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.436,"15min":0.36299999999999999},"min":{"1min":0.19700000000000001,"5min":0.14699999999999999,"15min":0.13700000000000001},"max":{"1min":3.222,"5min":6.5499999999999998,"15min":6.9989999999999997},"last":0.245},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.46300000000000002,"15min":0.42299999999999999},"min":{"1min":0.17799999999999999,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":5.0030000000000001,"5min":6.0629999999999997,"15min":9.6869999999999994},"last":0.28599999999999998}]},{"osd":2,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.51600000000000001,"15min":0.47099999999999997},"min":{"1min":0.21099999999999999,"5min":0.153,"15min":0.153},"max":{"1min":5.7439999999999998,"5min":5.8280000000000003,"15min":9.7639999999999993},"last":0.252},{"interface":"front","average":{"1min":0.497,"5min":0.56200000000000006,"15min":0.42299999999999999},"min":{"1min":0.20399999999999999,"5min":0.187,"15min":0.16800000000000001},"max":{"1min":3.9319999999999999,"5min":8.8040000000000003,"15min":8.8040000000000003},"last":0.29899999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738815,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":650028,"kb_used_data":5284,"kb_used_omap":214,"kb_used_meta":644521,"kb_avail":93721812,"statfs":{"total":96636764160,"available":95971135488,"internally_reserved":0,"allocated":5410816,"data_stored":2660331,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":219335,"internal_metadata":659990329},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:56 2023","interfaces":[{"interface":"back","average":{"1min":0.30399999999999999,"5min":0.34699999999999998,"15min":0.36499999999999999},"min":{"1min":0.23999999999999999,"5min":0.14199999999999999,"15min":0.14199999999999999},"max":{"1min":0.42399999999999999,"5min":1.595,"15min":7.4249999999999998},"last":0.17100000000000001},{"interface":"front","average":{"1min":0.35499999999999998,"5min":0.40899999999999997,"15min":0.376},"min":{"1min":0.23999999999999999,"5min":0.19600000000000001,"15min":0.183},"max":{"1min":0.46999999999999997,"5min":2.6859999999999999,"15min":2.6859999999999999},"last":0.23000000000000001}]},{"osd":2,"last update":"Wed Nov 8 05:28:56 2023","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.44400000000000001,"15min":0.376},"min":{"1min":0.187,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":2.2650000000000001,"5min":5.8209999999999997,"15min":5.8209999999999997},"last":0.23999999999999999},{"interface":"front","average":{"1min":0.316,"5min":0.40500000000000003,"15min":0.35999999999999999},"min":{"1min":0.19600000000000001,"5min":0.188,"15min":0.17599999999999999},"max":{"1min":1.137,"5min":4.2350000000000003,"15min":4.2350000000000003},"last":0.25}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-08T05:29:39.107 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~81s 2023-11-08T05:29:40.888 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-11-08T05:29:41.185 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T05:29:41.187 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2023-11-08T05:29:41.198 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":1558,"stamp":"2023-11-08T05:29:40.046113+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2634224,"kb_used_data":408868,"kb_used_omap":780,"kb_used_meta":2220403,"kb_avail":280481296,"statfs":{"total":289910292480,"available":287212847104,"internally_reserved":0,"allocated":418680832,"data_stored":410423593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":799447,"internal_metadata":2273692969},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-536782902,"num_objects":-164,"num_object_clones":0,"num_object_copies":-328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-115,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-164,"num_whiteouts":0,"num_read":-122056,"num_read_kb":-39011942,"num_write":-413642,"num_write_kb":-151705339,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-8,"num_bytes_recovered":-16922624,"num_keys_recovered":-2,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.424817"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":453,"reported_epoch":452,"state":"active+undersized","last_fresh":"2023-11-08T05:29:28.539262+0000","last_change":"2023-11-08T05:28:13.557071+0000","last_active":"2023-11-08T05:29:28.539262+0000","last_peered":"2023-11-08T05:29:28.539262+0000","last_clean":"2023-11-08T05:28:01.458439+0000","last_became_active":"2023-11-08T05:28:13.557071+0000","last_became_peered":"2023-11-08T05:28:13.557071+0000","last_unstale":"2023-11-08T05:29:28.539262+0000","last_undegraded":"2023-11-08T05:29:28.539262+0000","last_fullsized":"2023-11-08T05:28:13.547025+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":15,"reported_epoch":452,"state":"active+undersized","last_fresh":"2023-11-08T05:29:28.550339+0000","last_change":"2023-11-08T05:28:13.559513+0000","last_active":"2023-11-08T05:29:28.550339+0000","last_peered":"2023-11-08T05:29:28.550339+0000","last_clean":"2023-11-08T04:52:46.531964+0000","last_became_active":"2023-11-08T05:28:13.559513+0000","last_became_peered":"2023-11-08T05:28:13.559513+0000","last_unstale":"2023-11-08T05:29:28.550339+0000","last_undegraded":"2023-11-08T05:29:28.550339+0000","last_fullsized":"2023-11-08T05:28:13.540852+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":450,"reported_epoch":452,"state":"active+clean","last_fresh":"2023-11-08T05:29:28.468933+0000","last_change":"2023-11-08T04:52:47.546100+0000","last_active":"2023-11-08T05:29:28.468933+0000","last_peered":"2023-11-08T05:29:28.468933+0000","last_clean":"2023-11-08T05:29:28.468933+0000","last_became_active":"2023-11-08T04:52:47.545840+0000","last_became_peered":"2023-11-08T04:52:47.545840+0000","last_unstale":"2023-11-08T05:29:28.468933+0000","last_undegraded":"2023-11-08T05:29:28.468933+0000","last_fullsized":"2023-11-08T05:29:28.468933+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":448,"reported_epoch":452,"state":"active+clean","last_fresh":"2023-11-08T05:29:28.534734+0000","last_change":"2023-11-08T04:52:47.548597+0000","last_active":"2023-11-08T05:29:28.534734+0000","last_peered":"2023-11-08T05:29:28.534734+0000","last_clean":"2023-11-08T05:29:28.534734+0000","last_became_active":"2023-11-08T04:52:47.548294+0000","last_became_peered":"2023-11-08T04:52:47.548294+0000","last_unstale":"2023-11-08T05:29:28.534734+0000","last_undegraded":"2023-11-08T05:29:28.534734+0000","last_fullsized":"2023-11-08T05:29:28.534734+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":454,"reported_epoch":452,"state":"active+undersized","last_fresh":"2023-11-08T05:29:28.532757+0000","last_change":"2023-11-08T05:28:13.545468+0000","last_active":"2023-11-08T05:29:28.532757+0000","last_peered":"2023-11-08T05:29:28.532757+0000","last_clean":"2023-11-08T05:28:01.456723+0000","last_became_active":"2023-11-08T05:28:13.545468+0000","last_became_peered":"2023-11-08T05:28:13.545468+0000","last_unstale":"2023-11-08T05:29:28.532757+0000","last_undegraded":"2023-11-08T05:29:28.532757+0000","last_fullsized":"2023-11-08T05:28:13.537207+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_clean_scrub_stamp":"2023-11-08T04:52:45.528796+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":450,"reported_epoch":452,"state":"active+clean","last_fresh":"2023-11-08T05:29:28.469973+0000","last_change":"2023-11-08T04:52:47.546022+0000","last_active":"2023-11-08T05:29:28.469973+0000","last_peered":"2023-11-08T05:29:28.469973+0000","last_clean":"2023-11-08T05:29:28.469973+0000","last_became_active":"2023-11-08T04:52:47.545767+0000","last_became_peered":"2023-11-08T04:52:47.545767+0000","last_unstale":"2023-11-08T05:29:28.469973+0000","last_undegraded":"2023-11-08T05:29:28.469973+0000","last_fullsized":"2023-11-08T05:29:28.469973+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":453,"reported_epoch":452,"state":"active+undersized","last_fresh":"2023-11-08T05:29:28.536706+0000","last_change":"2023-11-08T05:28:13.563357+0000","last_active":"2023-11-08T05:29:28.536706+0000","last_peered":"2023-11-08T05:29:28.536706+0000","last_clean":"2023-11-08T05:28:01.457691+0000","last_became_active":"2023-11-08T05:28:13.563357+0000","last_became_peered":"2023-11-08T05:28:13.563357+0000","last_unstale":"2023-11-08T05:29:28.536706+0000","last_undegraded":"2023-11-08T05:29:28.536706+0000","last_fullsized":"2023-11-08T05:28:13.541614+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":453,"reported_epoch":452,"state":"active+undersized","last_fresh":"2023-11-08T05:29:28.537348+0000","last_change":"2023-11-08T05:28:13.550672+0000","last_active":"2023-11-08T05:29:28.537348+0000","last_peered":"2023-11-08T05:29:28.537348+0000","last_clean":"2023-11-08T05:28:01.457904+0000","last_became_active":"2023-11-08T05:28:13.550672+0000","last_became_peered":"2023-11-08T05:28:13.550672+0000","last_unstale":"2023-11-08T05:29:28.537348+0000","last_undegraded":"2023-11-08T05:29:28.537348+0000","last_fullsized":"2023-11-08T05:28:13.539595+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":453,"reported_epoch":452,"state":"active+undersized","last_fresh":"2023-11-08T05:29:28.540077+0000","last_change":"2023-11-08T05:28:13.557968+0000","last_active":"2023-11-08T05:29:28.540077+0000","last_peered":"2023-11-08T05:29:28.540077+0000","last_clean":"2023-11-08T05:28:01.458433+0000","last_became_active":"2023-11-08T05:28:13.557968+0000","last_became_peered":"2023-11-08T05:28:13.557968+0000","last_unstale":"2023-11-08T05:29:28.540077+0000","last_undegraded":"2023-11-08T05:29:28.540077+0000","last_fullsized":"2023-11-08T05:28:13.542832+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":1,"acting":1,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738814,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":644968,"kb_used_data":5280,"kb_used_omap":275,"kb_used_meta":639404,"kb_avail":93726872,"statfs":{"total":96636764160,"available":95976316928,"internally_reserved":0,"allocated":5406720,"data_stored":2659362,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":281650,"internal_metadata":654750670},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:59 2023","interfaces":[{"interface":"back","average":{"1min":0.30499999999999999,"5min":0.439,"15min":0.36099999999999999},"min":{"1min":0.17399999999999999,"5min":0.14499999999999999,"15min":0.14499999999999999},"max":{"1min":0.60099999999999998,"5min":4.9379999999999997,"15min":4.9379999999999997},"last":0.379},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.49399999999999999,"15min":0.42399999999999999},"min":{"1min":0.221,"5min":0.19600000000000001,"15min":0.17000000000000001},"max":{"1min":0.63200000000000001,"5min":4.9859999999999998,"15min":4.9859999999999998},"last":0.40899999999999997}]},{"osd":1,"last update":"Wed Nov 8 05:29:01 2023","interfaces":[{"interface":"back","average":{"1min":0.30299999999999999,"5min":0.81899999999999995,"15min":0.47399999999999998},"min":{"1min":0.16,"5min":0.153,"15min":0.153},"max":{"1min":0.68200000000000005,"5min":23.242999999999999,"15min":23.242999999999999},"last":0.28000000000000003},{"interface":"front","average":{"1min":0.46500000000000002,"5min":0.72199999999999998,"15min":0.46100000000000002},"min":{"1min":0.17699999999999999,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":2.0419999999999998,"5min":23.251000000000001,"15min":23.251000000000001},"last":0.30299999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738798,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1339228,"kb_used_data":398304,"kb_used_omap":291,"kb_used_meta":936476,"kb_avail":93032612,"statfs":{"total":96636764160,"available":95265394688,"internally_reserved":0,"allocated":407863296,"data_stored":405103900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":298462,"internal_metadata":958951970},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.436,"15min":0.36299999999999999},"min":{"1min":0.19700000000000001,"5min":0.14699999999999999,"15min":0.13700000000000001},"max":{"1min":3.222,"5min":6.5499999999999998,"15min":6.9989999999999997},"last":0.245},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.46300000000000002,"15min":0.42299999999999999},"min":{"1min":0.17799999999999999,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":5.0030000000000001,"5min":6.0629999999999997,"15min":9.6869999999999994},"last":0.28599999999999998}]},{"osd":2,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.51600000000000001,"15min":0.47099999999999997},"min":{"1min":0.21099999999999999,"5min":0.153,"15min":0.153},"max":{"1min":5.7439999999999998,"5min":5.8280000000000003,"15min":9.7639999999999993},"last":0.252},{"interface":"front","average":{"1min":0.497,"5min":0.56200000000000006,"15min":0.42299999999999999},"min":{"1min":0.20399999999999999,"5min":0.187,"15min":0.16800000000000001},"max":{"1min":3.9319999999999999,"5min":8.8040000000000003,"15min":8.8040000000000003},"last":0.29899999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738815,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":650028,"kb_used_data":5284,"kb_used_omap":214,"kb_used_meta":644521,"kb_avail":93721812,"statfs":{"total":96636764160,"available":95971135488,"internally_reserved":0,"allocated":5410816,"data_stored":2660331,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":219335,"internal_metadata":659990329},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:56 2023","interfaces":[{"interface":"back","average":{"1min":0.30399999999999999,"5min":0.34699999999999998,"15min":0.36499999999999999},"min":{"1min":0.23999999999999999,"5min":0.14199999999999999,"15min":0.14199999999999999},"max":{"1min":0.42399999999999999,"5min":1.595,"15min":7.4249999999999998},"last":0.17100000000000001},{"interface":"front","average":{"1min":0.35499999999999998,"5min":0.40899999999999997,"15min":0.376},"min":{"1min":0.23999999999999999,"5min":0.19600000000000001,"15min":0.183},"max":{"1min":0.46999999999999997,"5min":2.6859999999999999,"15min":2.6859999999999999},"last":0.23000000000000001}]},{"osd":2,"last update":"Wed Nov 8 05:28:56 2023","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.44400000000000001,"15min":0.376},"min":{"1min":0.187,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":2.2650000000000001,"5min":5.8209999999999997,"15min":5.8209999999999997},"last":0.23999999999999999},{"interface":"front","average":{"1min":0.316,"5min":0.40500000000000003,"15min":0.35999999999999999},"min":{"1min":0.19600000000000001,"5min":0.188,"15min":0.17599999999999999},"max":{"1min":1.137,"5min":4.2350000000000003,"15min":4.2350000000000003},"last":0.25}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-08T05:29:41.199 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-11-08T05:29:41.481 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T05:29:41.483 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2023-11-08T05:29:41.494 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":1558,"stamp":"2023-11-08T05:29:40.046113+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2634224,"kb_used_data":408868,"kb_used_omap":780,"kb_used_meta":2220403,"kb_avail":280481296,"statfs":{"total":289910292480,"available":287212847104,"internally_reserved":0,"allocated":418680832,"data_stored":410423593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":799447,"internal_metadata":2273692969},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-536782902,"num_objects":-164,"num_object_clones":0,"num_object_copies":-328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-115,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-164,"num_whiteouts":0,"num_read":-122056,"num_read_kb":-39011942,"num_write":-413642,"num_write_kb":-151705339,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-8,"num_bytes_recovered":-16922624,"num_keys_recovered":-2,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.424817"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":453,"reported_epoch":452,"state":"active+undersized","last_fresh":"2023-11-08T05:29:28.539262+0000","last_change":"2023-11-08T05:28:13.557071+0000","last_active":"2023-11-08T05:29:28.539262+0000","last_peered":"2023-11-08T05:29:28.539262+0000","last_clean":"2023-11-08T05:28:01.458439+0000","last_became_active":"2023-11-08T05:28:13.557071+0000","last_became_peered":"2023-11-08T05:28:13.557071+0000","last_unstale":"2023-11-08T05:29:28.539262+0000","last_undegraded":"2023-11-08T05:29:28.539262+0000","last_fullsized":"2023-11-08T05:28:13.547025+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":15,"reported_epoch":452,"state":"active+undersized","last_fresh":"2023-11-08T05:29:28.550339+0000","last_change":"2023-11-08T05:28:13.559513+0000","last_active":"2023-11-08T05:29:28.550339+0000","last_peered":"2023-11-08T05:29:28.550339+0000","last_clean":"2023-11-08T04:52:46.531964+0000","last_became_active":"2023-11-08T05:28:13.559513+0000","last_became_peered":"2023-11-08T05:28:13.559513+0000","last_unstale":"2023-11-08T05:29:28.550339+0000","last_undegraded":"2023-11-08T05:29:28.550339+0000","last_fullsized":"2023-11-08T05:28:13.540852+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":450,"reported_epoch":452,"state":"active+clean","last_fresh":"2023-11-08T05:29:28.468933+0000","last_change":"2023-11-08T04:52:47.546100+0000","last_active":"2023-11-08T05:29:28.468933+0000","last_peered":"2023-11-08T05:29:28.468933+0000","last_clean":"2023-11-08T05:29:28.468933+0000","last_became_active":"2023-11-08T04:52:47.545840+0000","last_became_peered":"2023-11-08T04:52:47.545840+0000","last_unstale":"2023-11-08T05:29:28.468933+0000","last_undegraded":"2023-11-08T05:29:28.468933+0000","last_fullsized":"2023-11-08T05:29:28.468933+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":448,"reported_epoch":452,"state":"active+clean","last_fresh":"2023-11-08T05:29:28.534734+0000","last_change":"2023-11-08T04:52:47.548597+0000","last_active":"2023-11-08T05:29:28.534734+0000","last_peered":"2023-11-08T05:29:28.534734+0000","last_clean":"2023-11-08T05:29:28.534734+0000","last_became_active":"2023-11-08T04:52:47.548294+0000","last_became_peered":"2023-11-08T04:52:47.548294+0000","last_unstale":"2023-11-08T05:29:28.534734+0000","last_undegraded":"2023-11-08T05:29:28.534734+0000","last_fullsized":"2023-11-08T05:29:28.534734+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":454,"reported_epoch":452,"state":"active+undersized","last_fresh":"2023-11-08T05:29:28.532757+0000","last_change":"2023-11-08T05:28:13.545468+0000","last_active":"2023-11-08T05:29:28.532757+0000","last_peered":"2023-11-08T05:29:28.532757+0000","last_clean":"2023-11-08T05:28:01.456723+0000","last_became_active":"2023-11-08T05:28:13.545468+0000","last_became_peered":"2023-11-08T05:28:13.545468+0000","last_unstale":"2023-11-08T05:29:28.532757+0000","last_undegraded":"2023-11-08T05:29:28.532757+0000","last_fullsized":"2023-11-08T05:28:13.537207+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_clean_scrub_stamp":"2023-11-08T04:52:45.528796+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":450,"reported_epoch":452,"state":"active+clean","last_fresh":"2023-11-08T05:29:28.469973+0000","last_change":"2023-11-08T04:52:47.546022+0000","last_active":"2023-11-08T05:29:28.469973+0000","last_peered":"2023-11-08T05:29:28.469973+0000","last_clean":"2023-11-08T05:29:28.469973+0000","last_became_active":"2023-11-08T04:52:47.545767+0000","last_became_peered":"2023-11-08T04:52:47.545767+0000","last_unstale":"2023-11-08T05:29:28.469973+0000","last_undegraded":"2023-11-08T05:29:28.469973+0000","last_fullsized":"2023-11-08T05:29:28.469973+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":453,"reported_epoch":452,"state":"active+undersized","last_fresh":"2023-11-08T05:29:28.536706+0000","last_change":"2023-11-08T05:28:13.563357+0000","last_active":"2023-11-08T05:29:28.536706+0000","last_peered":"2023-11-08T05:29:28.536706+0000","last_clean":"2023-11-08T05:28:01.457691+0000","last_became_active":"2023-11-08T05:28:13.563357+0000","last_became_peered":"2023-11-08T05:28:13.563357+0000","last_unstale":"2023-11-08T05:29:28.536706+0000","last_undegraded":"2023-11-08T05:29:28.536706+0000","last_fullsized":"2023-11-08T05:28:13.541614+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":453,"reported_epoch":452,"state":"active+undersized","last_fresh":"2023-11-08T05:29:28.537348+0000","last_change":"2023-11-08T05:28:13.550672+0000","last_active":"2023-11-08T05:29:28.537348+0000","last_peered":"2023-11-08T05:29:28.537348+0000","last_clean":"2023-11-08T05:28:01.457904+0000","last_became_active":"2023-11-08T05:28:13.550672+0000","last_became_peered":"2023-11-08T05:28:13.550672+0000","last_unstale":"2023-11-08T05:29:28.537348+0000","last_undegraded":"2023-11-08T05:29:28.537348+0000","last_fullsized":"2023-11-08T05:28:13.539595+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":453,"reported_epoch":452,"state":"active+undersized","last_fresh":"2023-11-08T05:29:28.540077+0000","last_change":"2023-11-08T05:28:13.557968+0000","last_active":"2023-11-08T05:29:28.540077+0000","last_peered":"2023-11-08T05:29:28.540077+0000","last_clean":"2023-11-08T05:28:01.458433+0000","last_became_active":"2023-11-08T05:28:13.557968+0000","last_became_peered":"2023-11-08T05:28:13.557968+0000","last_unstale":"2023-11-08T05:29:28.540077+0000","last_undegraded":"2023-11-08T05:29:28.540077+0000","last_fullsized":"2023-11-08T05:28:13.542832+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":1,"acting":1,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738814,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":644968,"kb_used_data":5280,"kb_used_omap":275,"kb_used_meta":639404,"kb_avail":93726872,"statfs":{"total":96636764160,"available":95976316928,"internally_reserved":0,"allocated":5406720,"data_stored":2659362,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":281650,"internal_metadata":654750670},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:59 2023","interfaces":[{"interface":"back","average":{"1min":0.30499999999999999,"5min":0.439,"15min":0.36099999999999999},"min":{"1min":0.17399999999999999,"5min":0.14499999999999999,"15min":0.14499999999999999},"max":{"1min":0.60099999999999998,"5min":4.9379999999999997,"15min":4.9379999999999997},"last":0.379},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.49399999999999999,"15min":0.42399999999999999},"min":{"1min":0.221,"5min":0.19600000000000001,"15min":0.17000000000000001},"max":{"1min":0.63200000000000001,"5min":4.9859999999999998,"15min":4.9859999999999998},"last":0.40899999999999997}]},{"osd":1,"last update":"Wed Nov 8 05:29:01 2023","interfaces":[{"interface":"back","average":{"1min":0.30299999999999999,"5min":0.81899999999999995,"15min":0.47399999999999998},"min":{"1min":0.16,"5min":0.153,"15min":0.153},"max":{"1min":0.68200000000000005,"5min":23.242999999999999,"15min":23.242999999999999},"last":0.28000000000000003},{"interface":"front","average":{"1min":0.46500000000000002,"5min":0.72199999999999998,"15min":0.46100000000000002},"min":{"1min":0.17699999999999999,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":2.0419999999999998,"5min":23.251000000000001,"15min":23.251000000000001},"last":0.30299999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738798,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1339228,"kb_used_data":398304,"kb_used_omap":291,"kb_used_meta":936476,"kb_avail":93032612,"statfs":{"total":96636764160,"available":95265394688,"internally_reserved":0,"allocated":407863296,"data_stored":405103900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":298462,"internal_metadata":958951970},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.436,"15min":0.36299999999999999},"min":{"1min":0.19700000000000001,"5min":0.14699999999999999,"15min":0.13700000000000001},"max":{"1min":3.222,"5min":6.5499999999999998,"15min":6.9989999999999997},"last":0.245},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.46300000000000002,"15min":0.42299999999999999},"min":{"1min":0.17799999999999999,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":5.0030000000000001,"5min":6.0629999999999997,"15min":9.6869999999999994},"last":0.28599999999999998}]},{"osd":2,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.51600000000000001,"15min":0.47099999999999997},"min":{"1min":0.21099999999999999,"5min":0.153,"15min":0.153},"max":{"1min":5.7439999999999998,"5min":5.8280000000000003,"15min":9.7639999999999993},"last":0.252},{"interface":"front","average":{"1min":0.497,"5min":0.56200000000000006,"15min":0.42299999999999999},"min":{"1min":0.20399999999999999,"5min":0.187,"15min":0.16800000000000001},"max":{"1min":3.9319999999999999,"5min":8.8040000000000003,"15min":8.8040000000000003},"last":0.29899999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738815,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":650028,"kb_used_data":5284,"kb_used_omap":214,"kb_used_meta":644521,"kb_avail":93721812,"statfs":{"total":96636764160,"available":95971135488,"internally_reserved":0,"allocated":5410816,"data_stored":2660331,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":219335,"internal_metadata":659990329},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:56 2023","interfaces":[{"interface":"back","average":{"1min":0.30399999999999999,"5min":0.34699999999999998,"15min":0.36499999999999999},"min":{"1min":0.23999999999999999,"5min":0.14199999999999999,"15min":0.14199999999999999},"max":{"1min":0.42399999999999999,"5min":1.595,"15min":7.4249999999999998},"last":0.17100000000000001},{"interface":"front","average":{"1min":0.35499999999999998,"5min":0.40899999999999997,"15min":0.376},"min":{"1min":0.23999999999999999,"5min":0.19600000000000001,"15min":0.183},"max":{"1min":0.46999999999999997,"5min":2.6859999999999999,"15min":2.6859999999999999},"last":0.23000000000000001}]},{"osd":2,"last update":"Wed Nov 8 05:28:56 2023","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.44400000000000001,"15min":0.376},"min":{"1min":0.187,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":2.2650000000000001,"5min":5.8209999999999997,"15min":5.8209999999999997},"last":0.23999999999999999},{"interface":"front","average":{"1min":0.316,"5min":0.40500000000000003,"15min":0.35999999999999999},"min":{"1min":0.19600000000000001,"5min":0.188,"15min":0.17599999999999999},"max":{"1min":1.137,"5min":4.2350000000000003,"15min":4.2350000000000003},"last":0.25}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-08T05:29:41.495 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-11-08T05:29:41.495 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 453, 'reported_epoch': 452, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:29:28.539262+0000', 'last_change': '2023-11-08T05:28:13.557071+0000', 'last_active': '2023-11-08T05:29:28.539262+0000', 'last_peered': '2023-11-08T05:29:28.539262+0000', 'last_clean': '2023-11-08T05:28:01.458439+0000', 'last_became_active': '2023-11-08T05:28:13.557071+0000', 'last_became_peered': '2023-11-08T05:28:13.557071+0000', 'last_unstale': '2023-11-08T05:29:28.539262+0000', 'last_undegraded': '2023-11-08T05:29:28.539262+0000', 'last_fullsized': '2023-11-08T05:28:13.547025+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:29:41.496 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-11-08T05:29:41.496 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 15, 'reported_epoch': 452, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:29:28.550339+0000', 'last_change': '2023-11-08T05:28:13.559513+0000', 'last_active': '2023-11-08T05:29:28.550339+0000', 'last_peered': '2023-11-08T05:29:28.550339+0000', 'last_clean': '2023-11-08T04:52:46.531964+0000', 'last_became_active': '2023-11-08T05:28:13.559513+0000', 'last_became_peered': '2023-11-08T05:28:13.559513+0000', 'last_unstale': '2023-11-08T05:29:28.550339+0000', 'last_undegraded': '2023-11-08T05:29:28.550339+0000', 'last_fullsized': '2023-11-08T05:28:13.540852+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:29:41.496 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-11-08T05:29:41.497 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "0'0", 'reported_seq': 454, 'reported_epoch': 452, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:29:28.532757+0000', 'last_change': '2023-11-08T05:28:13.545468+0000', 'last_active': '2023-11-08T05:29:28.532757+0000', 'last_peered': '2023-11-08T05:29:28.532757+0000', 'last_clean': '2023-11-08T05:28:01.456723+0000', 'last_became_active': '2023-11-08T05:28:13.545468+0000', 'last_became_peered': '2023-11-08T05:28:13.545468+0000', 'last_unstale': '2023-11-08T05:29:28.532757+0000', 'last_undegraded': '2023-11-08T05:29:28.532757+0000', 'last_fullsized': '2023-11-08T05:28:13.537207+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 9, 'last_epoch_clean': 10, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:45.528796+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:45.528796+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:45.528796+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:29:41.497 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-11-08T05:29:41.497 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 453, 'reported_epoch': 452, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:29:28.536706+0000', 'last_change': '2023-11-08T05:28:13.563357+0000', 'last_active': '2023-11-08T05:29:28.536706+0000', 'last_peered': '2023-11-08T05:29:28.536706+0000', 'last_clean': '2023-11-08T05:28:01.457691+0000', 'last_became_active': '2023-11-08T05:28:13.563357+0000', 'last_became_peered': '2023-11-08T05:28:13.563357+0000', 'last_unstale': '2023-11-08T05:29:28.536706+0000', 'last_undegraded': '2023-11-08T05:29:28.536706+0000', 'last_fullsized': '2023-11-08T05:28:13.541614+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:29:41.498 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-11-08T05:29:41.498 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 453, 'reported_epoch': 452, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:29:28.537348+0000', 'last_change': '2023-11-08T05:28:13.550672+0000', 'last_active': '2023-11-08T05:29:28.537348+0000', 'last_peered': '2023-11-08T05:29:28.537348+0000', 'last_clean': '2023-11-08T05:28:01.457904+0000', 'last_became_active': '2023-11-08T05:28:13.550672+0000', 'last_became_peered': '2023-11-08T05:28:13.550672+0000', 'last_unstale': '2023-11-08T05:29:28.537348+0000', 'last_undegraded': '2023-11-08T05:29:28.537348+0000', 'last_fullsized': '2023-11-08T05:28:13.539595+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:29:41.498 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-11-08T05:29:41.499 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 453, 'reported_epoch': 452, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:29:28.540077+0000', 'last_change': '2023-11-08T05:28:13.557968+0000', 'last_active': '2023-11-08T05:29:28.540077+0000', 'last_peered': '2023-11-08T05:29:28.540077+0000', 'last_clean': '2023-11-08T05:28:01.458433+0000', 'last_became_active': '2023-11-08T05:28:13.557968+0000', 'last_became_peered': '2023-11-08T05:28:13.557968+0000', 'last_unstale': '2023-11-08T05:29:28.540077+0000', 'last_undegraded': '2023-11-08T05:29:28.540077+0000', 'last_fullsized': '2023-11-08T05:28:13.542832+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:29:41.499 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-11-08T05:29:41.617 INFO:tasks.ceph.cluster2.mgr.x.smithi077.stderr:2023-11-08T05:29:41.616+0000 7f16c17cf700 -1 calc_pg_upmaps abort due to max <= 0 2023-11-08T05:29:41.847 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T05:29:41.848 INFO:teuthology.orchestra.run.smithi087.stdout:{"fsid":"ab4cbc06-449d-497d-b513-220d9db2b590","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 6 pgs undersized","count":6},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2224,"monmap":{"epoch":1,"min_mon_release_name":"pacific","num_mons":1},"osdmap":{"epoch":453,"num_osds":3,"num_up_osds":2,"osd_up_since":1699421292,"num_in_osds":3,"osd_in_since":1699419158,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":6},{"state_name":"active+clean","count":3}],"num_pgs":9,"num_pools":2,"num_objects":0,"data_bytes":0,"bytes_used":2697445376,"bytes_avail":287212847104,"bytes_total":289910292480},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":41,"modified":"2023-11-08T05:29:32.044544+0000","services":{}},"progress_events":{}} 2023-11-08T05:29:41.861 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-11-08T05:29:41.862 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-11-08T05:29:42.144 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T05:29:42.146 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2023-11-08T05:29:42.157 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":1559,"stamp":"2023-11-08T05:29:42.046536+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2634224,"kb_used_data":408868,"kb_used_omap":780,"kb_used_meta":2220403,"kb_avail":280481296,"statfs":{"total":289910292480,"available":287212847104,"internally_reserved":0,"allocated":418680832,"data_stored":410423593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":799447,"internal_metadata":2273692969},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-536782902,"num_objects":-164,"num_object_clones":0,"num_object_copies":-328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-115,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-164,"num_whiteouts":0,"num_read":-122056,"num_read_kb":-39011942,"num_write":-413642,"num_write_kb":-151705339,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-8,"num_bytes_recovered":-16922624,"num_keys_recovered":-2,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.424817"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":453,"reported_epoch":452,"state":"active+undersized","last_fresh":"2023-11-08T05:29:28.539262+0000","last_change":"2023-11-08T05:28:13.557071+0000","last_active":"2023-11-08T05:29:28.539262+0000","last_peered":"2023-11-08T05:29:28.539262+0000","last_clean":"2023-11-08T05:28:01.458439+0000","last_became_active":"2023-11-08T05:28:13.557071+0000","last_became_peered":"2023-11-08T05:28:13.557071+0000","last_unstale":"2023-11-08T05:29:28.539262+0000","last_undegraded":"2023-11-08T05:29:28.539262+0000","last_fullsized":"2023-11-08T05:28:13.547025+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":15,"reported_epoch":452,"state":"active+undersized","last_fresh":"2023-11-08T05:29:28.550339+0000","last_change":"2023-11-08T05:28:13.559513+0000","last_active":"2023-11-08T05:29:28.550339+0000","last_peered":"2023-11-08T05:29:28.550339+0000","last_clean":"2023-11-08T04:52:46.531964+0000","last_became_active":"2023-11-08T05:28:13.559513+0000","last_became_peered":"2023-11-08T05:28:13.559513+0000","last_unstale":"2023-11-08T05:29:28.550339+0000","last_undegraded":"2023-11-08T05:29:28.550339+0000","last_fullsized":"2023-11-08T05:28:13.540852+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":450,"reported_epoch":452,"state":"active+clean","last_fresh":"2023-11-08T05:29:28.468933+0000","last_change":"2023-11-08T04:52:47.546100+0000","last_active":"2023-11-08T05:29:28.468933+0000","last_peered":"2023-11-08T05:29:28.468933+0000","last_clean":"2023-11-08T05:29:28.468933+0000","last_became_active":"2023-11-08T04:52:47.545840+0000","last_became_peered":"2023-11-08T04:52:47.545840+0000","last_unstale":"2023-11-08T05:29:28.468933+0000","last_undegraded":"2023-11-08T05:29:28.468933+0000","last_fullsized":"2023-11-08T05:29:28.468933+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":448,"reported_epoch":452,"state":"active+clean","last_fresh":"2023-11-08T05:29:28.534734+0000","last_change":"2023-11-08T04:52:47.548597+0000","last_active":"2023-11-08T05:29:28.534734+0000","last_peered":"2023-11-08T05:29:28.534734+0000","last_clean":"2023-11-08T05:29:28.534734+0000","last_became_active":"2023-11-08T04:52:47.548294+0000","last_became_peered":"2023-11-08T04:52:47.548294+0000","last_unstale":"2023-11-08T05:29:28.534734+0000","last_undegraded":"2023-11-08T05:29:28.534734+0000","last_fullsized":"2023-11-08T05:29:28.534734+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":454,"reported_epoch":452,"state":"active+undersized","last_fresh":"2023-11-08T05:29:28.532757+0000","last_change":"2023-11-08T05:28:13.545468+0000","last_active":"2023-11-08T05:29:28.532757+0000","last_peered":"2023-11-08T05:29:28.532757+0000","last_clean":"2023-11-08T05:28:01.456723+0000","last_became_active":"2023-11-08T05:28:13.545468+0000","last_became_peered":"2023-11-08T05:28:13.545468+0000","last_unstale":"2023-11-08T05:29:28.532757+0000","last_undegraded":"2023-11-08T05:29:28.532757+0000","last_fullsized":"2023-11-08T05:28:13.537207+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_clean_scrub_stamp":"2023-11-08T04:52:45.528796+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":450,"reported_epoch":452,"state":"active+clean","last_fresh":"2023-11-08T05:29:28.469973+0000","last_change":"2023-11-08T04:52:47.546022+0000","last_active":"2023-11-08T05:29:28.469973+0000","last_peered":"2023-11-08T05:29:28.469973+0000","last_clean":"2023-11-08T05:29:28.469973+0000","last_became_active":"2023-11-08T04:52:47.545767+0000","last_became_peered":"2023-11-08T04:52:47.545767+0000","last_unstale":"2023-11-08T05:29:28.469973+0000","last_undegraded":"2023-11-08T05:29:28.469973+0000","last_fullsized":"2023-11-08T05:29:28.469973+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":453,"reported_epoch":452,"state":"active+undersized","last_fresh":"2023-11-08T05:29:28.536706+0000","last_change":"2023-11-08T05:28:13.563357+0000","last_active":"2023-11-08T05:29:28.536706+0000","last_peered":"2023-11-08T05:29:28.536706+0000","last_clean":"2023-11-08T05:28:01.457691+0000","last_became_active":"2023-11-08T05:28:13.563357+0000","last_became_peered":"2023-11-08T05:28:13.563357+0000","last_unstale":"2023-11-08T05:29:28.536706+0000","last_undegraded":"2023-11-08T05:29:28.536706+0000","last_fullsized":"2023-11-08T05:28:13.541614+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":453,"reported_epoch":452,"state":"active+undersized","last_fresh":"2023-11-08T05:29:28.537348+0000","last_change":"2023-11-08T05:28:13.550672+0000","last_active":"2023-11-08T05:29:28.537348+0000","last_peered":"2023-11-08T05:29:28.537348+0000","last_clean":"2023-11-08T05:28:01.457904+0000","last_became_active":"2023-11-08T05:28:13.550672+0000","last_became_peered":"2023-11-08T05:28:13.550672+0000","last_unstale":"2023-11-08T05:29:28.537348+0000","last_undegraded":"2023-11-08T05:29:28.537348+0000","last_fullsized":"2023-11-08T05:28:13.539595+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":453,"reported_epoch":452,"state":"active+undersized","last_fresh":"2023-11-08T05:29:28.540077+0000","last_change":"2023-11-08T05:28:13.557968+0000","last_active":"2023-11-08T05:29:28.540077+0000","last_peered":"2023-11-08T05:29:28.540077+0000","last_clean":"2023-11-08T05:28:01.458433+0000","last_became_active":"2023-11-08T05:28:13.557968+0000","last_became_peered":"2023-11-08T05:28:13.557968+0000","last_unstale":"2023-11-08T05:29:28.540077+0000","last_undegraded":"2023-11-08T05:29:28.540077+0000","last_fullsized":"2023-11-08T05:28:13.542832+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":1,"acting":1,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738814,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":644968,"kb_used_data":5280,"kb_used_omap":275,"kb_used_meta":639404,"kb_avail":93726872,"statfs":{"total":96636764160,"available":95976316928,"internally_reserved":0,"allocated":5406720,"data_stored":2659362,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":281650,"internal_metadata":654750670},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:59 2023","interfaces":[{"interface":"back","average":{"1min":0.30499999999999999,"5min":0.439,"15min":0.36099999999999999},"min":{"1min":0.17399999999999999,"5min":0.14499999999999999,"15min":0.14499999999999999},"max":{"1min":0.60099999999999998,"5min":4.9379999999999997,"15min":4.9379999999999997},"last":0.379},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.49399999999999999,"15min":0.42399999999999999},"min":{"1min":0.221,"5min":0.19600000000000001,"15min":0.17000000000000001},"max":{"1min":0.63200000000000001,"5min":4.9859999999999998,"15min":4.9859999999999998},"last":0.40899999999999997}]},{"osd":1,"last update":"Wed Nov 8 05:29:01 2023","interfaces":[{"interface":"back","average":{"1min":0.30299999999999999,"5min":0.81899999999999995,"15min":0.47399999999999998},"min":{"1min":0.16,"5min":0.153,"15min":0.153},"max":{"1min":0.68200000000000005,"5min":23.242999999999999,"15min":23.242999999999999},"last":0.28000000000000003},{"interface":"front","average":{"1min":0.46500000000000002,"5min":0.72199999999999998,"15min":0.46100000000000002},"min":{"1min":0.17699999999999999,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":2.0419999999999998,"5min":23.251000000000001,"15min":23.251000000000001},"last":0.30299999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738798,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1339228,"kb_used_data":398304,"kb_used_omap":291,"kb_used_meta":936476,"kb_avail":93032612,"statfs":{"total":96636764160,"available":95265394688,"internally_reserved":0,"allocated":407863296,"data_stored":405103900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":298462,"internal_metadata":958951970},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.436,"15min":0.36299999999999999},"min":{"1min":0.19700000000000001,"5min":0.14699999999999999,"15min":0.13700000000000001},"max":{"1min":3.222,"5min":6.5499999999999998,"15min":6.9989999999999997},"last":0.245},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.46300000000000002,"15min":0.42299999999999999},"min":{"1min":0.17799999999999999,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":5.0030000000000001,"5min":6.0629999999999997,"15min":9.6869999999999994},"last":0.28599999999999998}]},{"osd":2,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.51600000000000001,"15min":0.47099999999999997},"min":{"1min":0.21099999999999999,"5min":0.153,"15min":0.153},"max":{"1min":5.7439999999999998,"5min":5.8280000000000003,"15min":9.7639999999999993},"last":0.252},{"interface":"front","average":{"1min":0.497,"5min":0.56200000000000006,"15min":0.42299999999999999},"min":{"1min":0.20399999999999999,"5min":0.187,"15min":0.16800000000000001},"max":{"1min":3.9319999999999999,"5min":8.8040000000000003,"15min":8.8040000000000003},"last":0.29899999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738816,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":650028,"kb_used_data":5284,"kb_used_omap":214,"kb_used_meta":644521,"kb_avail":93721812,"statfs":{"total":96636764160,"available":95971135488,"internally_reserved":0,"allocated":5410816,"data_stored":2660331,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":219335,"internal_metadata":659990329},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:56 2023","interfaces":[{"interface":"back","average":{"1min":0.30399999999999999,"5min":0.34699999999999998,"15min":0.36499999999999999},"min":{"1min":0.23999999999999999,"5min":0.14199999999999999,"15min":0.14199999999999999},"max":{"1min":0.42399999999999999,"5min":1.595,"15min":7.4249999999999998},"last":0.17100000000000001},{"interface":"front","average":{"1min":0.35499999999999998,"5min":0.40899999999999997,"15min":0.376},"min":{"1min":0.23999999999999999,"5min":0.19600000000000001,"15min":0.183},"max":{"1min":0.46999999999999997,"5min":2.6859999999999999,"15min":2.6859999999999999},"last":0.23000000000000001}]},{"osd":2,"last update":"Wed Nov 8 05:28:56 2023","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.44400000000000001,"15min":0.376},"min":{"1min":0.187,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":2.2650000000000001,"5min":5.8209999999999997,"15min":5.8209999999999997},"last":0.27200000000000002},{"interface":"front","average":{"1min":0.316,"5min":0.40500000000000003,"15min":0.35999999999999999},"min":{"1min":0.19600000000000001,"5min":0.188,"15min":0.17599999999999999},"max":{"1min":1.137,"5min":4.2350000000000003,"15min":4.2350000000000003},"last":0.245}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-08T05:29:44.511 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~86s 2023-11-08T05:29:45.159 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-11-08T05:29:45.457 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T05:29:45.459 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2023-11-08T05:29:45.470 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":1563,"stamp":"2023-11-08T05:29:44.391275+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2634224,"kb_used_data":408868,"kb_used_omap":780,"kb_used_meta":2220403,"kb_avail":280481296,"statfs":{"total":289910292480,"available":287212847104,"internally_reserved":0,"allocated":418680832,"data_stored":410423593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":799447,"internal_metadata":2273692969},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-536782902,"num_objects":-164,"num_object_clones":0,"num_object_copies":-328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-115,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-164,"num_whiteouts":0,"num_read":-122056,"num_read_kb":-39011942,"num_write":-413642,"num_write_kb":-151705339,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-8,"num_bytes_recovered":-16922624,"num_keys_recovered":-2,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.424817"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":453,"reported_epoch":452,"state":"active+undersized","last_fresh":"2023-11-08T05:29:28.539262+0000","last_change":"2023-11-08T05:28:13.557071+0000","last_active":"2023-11-08T05:29:28.539262+0000","last_peered":"2023-11-08T05:29:28.539262+0000","last_clean":"2023-11-08T05:28:01.458439+0000","last_became_active":"2023-11-08T05:28:13.557071+0000","last_became_peered":"2023-11-08T05:28:13.557071+0000","last_unstale":"2023-11-08T05:29:28.539262+0000","last_undegraded":"2023-11-08T05:29:28.539262+0000","last_fullsized":"2023-11-08T05:28:13.547025+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":15,"reported_epoch":452,"state":"active+undersized","last_fresh":"2023-11-08T05:29:28.550339+0000","last_change":"2023-11-08T05:28:13.559513+0000","last_active":"2023-11-08T05:29:28.550339+0000","last_peered":"2023-11-08T05:29:28.550339+0000","last_clean":"2023-11-08T04:52:46.531964+0000","last_became_active":"2023-11-08T05:28:13.559513+0000","last_became_peered":"2023-11-08T05:28:13.559513+0000","last_unstale":"2023-11-08T05:29:28.550339+0000","last_undegraded":"2023-11-08T05:29:28.550339+0000","last_fullsized":"2023-11-08T05:28:13.540852+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":450,"reported_epoch":452,"state":"active+clean","last_fresh":"2023-11-08T05:29:28.468933+0000","last_change":"2023-11-08T04:52:47.546100+0000","last_active":"2023-11-08T05:29:28.468933+0000","last_peered":"2023-11-08T05:29:28.468933+0000","last_clean":"2023-11-08T05:29:28.468933+0000","last_became_active":"2023-11-08T04:52:47.545840+0000","last_became_peered":"2023-11-08T04:52:47.545840+0000","last_unstale":"2023-11-08T05:29:28.468933+0000","last_undegraded":"2023-11-08T05:29:28.468933+0000","last_fullsized":"2023-11-08T05:29:28.468933+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":448,"reported_epoch":452,"state":"active+clean","last_fresh":"2023-11-08T05:29:28.534734+0000","last_change":"2023-11-08T04:52:47.548597+0000","last_active":"2023-11-08T05:29:28.534734+0000","last_peered":"2023-11-08T05:29:28.534734+0000","last_clean":"2023-11-08T05:29:28.534734+0000","last_became_active":"2023-11-08T04:52:47.548294+0000","last_became_peered":"2023-11-08T04:52:47.548294+0000","last_unstale":"2023-11-08T05:29:28.534734+0000","last_undegraded":"2023-11-08T05:29:28.534734+0000","last_fullsized":"2023-11-08T05:29:28.534734+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":454,"reported_epoch":452,"state":"active+undersized","last_fresh":"2023-11-08T05:29:28.532757+0000","last_change":"2023-11-08T05:28:13.545468+0000","last_active":"2023-11-08T05:29:28.532757+0000","last_peered":"2023-11-08T05:29:28.532757+0000","last_clean":"2023-11-08T05:28:01.456723+0000","last_became_active":"2023-11-08T05:28:13.545468+0000","last_became_peered":"2023-11-08T05:28:13.545468+0000","last_unstale":"2023-11-08T05:29:28.532757+0000","last_undegraded":"2023-11-08T05:29:28.532757+0000","last_fullsized":"2023-11-08T05:28:13.537207+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_clean_scrub_stamp":"2023-11-08T04:52:45.528796+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":450,"reported_epoch":452,"state":"active+clean","last_fresh":"2023-11-08T05:29:28.469973+0000","last_change":"2023-11-08T04:52:47.546022+0000","last_active":"2023-11-08T05:29:28.469973+0000","last_peered":"2023-11-08T05:29:28.469973+0000","last_clean":"2023-11-08T05:29:28.469973+0000","last_became_active":"2023-11-08T04:52:47.545767+0000","last_became_peered":"2023-11-08T04:52:47.545767+0000","last_unstale":"2023-11-08T05:29:28.469973+0000","last_undegraded":"2023-11-08T05:29:28.469973+0000","last_fullsized":"2023-11-08T05:29:28.469973+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":453,"reported_epoch":452,"state":"active+undersized","last_fresh":"2023-11-08T05:29:28.536706+0000","last_change":"2023-11-08T05:28:13.563357+0000","last_active":"2023-11-08T05:29:28.536706+0000","last_peered":"2023-11-08T05:29:28.536706+0000","last_clean":"2023-11-08T05:28:01.457691+0000","last_became_active":"2023-11-08T05:28:13.563357+0000","last_became_peered":"2023-11-08T05:28:13.563357+0000","last_unstale":"2023-11-08T05:29:28.536706+0000","last_undegraded":"2023-11-08T05:29:28.536706+0000","last_fullsized":"2023-11-08T05:28:13.541614+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":453,"reported_epoch":452,"state":"active+undersized","last_fresh":"2023-11-08T05:29:28.537348+0000","last_change":"2023-11-08T05:28:13.550672+0000","last_active":"2023-11-08T05:29:28.537348+0000","last_peered":"2023-11-08T05:29:28.537348+0000","last_clean":"2023-11-08T05:28:01.457904+0000","last_became_active":"2023-11-08T05:28:13.550672+0000","last_became_peered":"2023-11-08T05:28:13.550672+0000","last_unstale":"2023-11-08T05:29:28.537348+0000","last_undegraded":"2023-11-08T05:29:28.537348+0000","last_fullsized":"2023-11-08T05:28:13.539595+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":453,"reported_epoch":452,"state":"active+undersized","last_fresh":"2023-11-08T05:29:28.540077+0000","last_change":"2023-11-08T05:28:13.557968+0000","last_active":"2023-11-08T05:29:28.540077+0000","last_peered":"2023-11-08T05:29:28.540077+0000","last_clean":"2023-11-08T05:28:01.458433+0000","last_became_active":"2023-11-08T05:28:13.557968+0000","last_became_peered":"2023-11-08T05:28:13.557968+0000","last_unstale":"2023-11-08T05:29:28.540077+0000","last_undegraded":"2023-11-08T05:29:28.540077+0000","last_fullsized":"2023-11-08T05:28:13.542832+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":1,"acting":1,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738814,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":644968,"kb_used_data":5280,"kb_used_omap":275,"kb_used_meta":639404,"kb_avail":93726872,"statfs":{"total":96636764160,"available":95976316928,"internally_reserved":0,"allocated":5406720,"data_stored":2659362,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":281650,"internal_metadata":654750670},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:59 2023","interfaces":[{"interface":"back","average":{"1min":0.30499999999999999,"5min":0.439,"15min":0.36099999999999999},"min":{"1min":0.17399999999999999,"5min":0.14499999999999999,"15min":0.14499999999999999},"max":{"1min":0.60099999999999998,"5min":4.9379999999999997,"15min":4.9379999999999997},"last":0.379},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.49399999999999999,"15min":0.42399999999999999},"min":{"1min":0.221,"5min":0.19600000000000001,"15min":0.17000000000000001},"max":{"1min":0.63200000000000001,"5min":4.9859999999999998,"15min":4.9859999999999998},"last":0.40899999999999997}]},{"osd":1,"last update":"Wed Nov 8 05:29:01 2023","interfaces":[{"interface":"back","average":{"1min":0.30299999999999999,"5min":0.81899999999999995,"15min":0.47399999999999998},"min":{"1min":0.16,"5min":0.153,"15min":0.153},"max":{"1min":0.68200000000000005,"5min":23.242999999999999,"15min":23.242999999999999},"last":0.28000000000000003},{"interface":"front","average":{"1min":0.46500000000000002,"5min":0.72199999999999998,"15min":0.46100000000000002},"min":{"1min":0.17699999999999999,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":2.0419999999999998,"5min":23.251000000000001,"15min":23.251000000000001},"last":0.30299999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738798,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1339228,"kb_used_data":398304,"kb_used_omap":291,"kb_used_meta":936476,"kb_avail":93032612,"statfs":{"total":96636764160,"available":95265394688,"internally_reserved":0,"allocated":407863296,"data_stored":405103900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":298462,"internal_metadata":958951970},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.436,"15min":0.36299999999999999},"min":{"1min":0.19700000000000001,"5min":0.14699999999999999,"15min":0.13700000000000001},"max":{"1min":3.222,"5min":6.5499999999999998,"15min":6.9989999999999997},"last":0.245},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.46300000000000002,"15min":0.42299999999999999},"min":{"1min":0.17799999999999999,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":5.0030000000000001,"5min":6.0629999999999997,"15min":9.6869999999999994},"last":0.28599999999999998}]},{"osd":2,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.51600000000000001,"15min":0.47099999999999997},"min":{"1min":0.21099999999999999,"5min":0.153,"15min":0.153},"max":{"1min":5.7439999999999998,"5min":5.8280000000000003,"15min":9.7639999999999993},"last":0.252},{"interface":"front","average":{"1min":0.497,"5min":0.56200000000000006,"15min":0.42299999999999999},"min":{"1min":0.20399999999999999,"5min":0.187,"15min":0.16800000000000001},"max":{"1min":3.9319999999999999,"5min":8.8040000000000003,"15min":8.8040000000000003},"last":0.29899999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738816,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":650028,"kb_used_data":5284,"kb_used_omap":214,"kb_used_meta":644521,"kb_avail":93721812,"statfs":{"total":96636764160,"available":95971135488,"internally_reserved":0,"allocated":5410816,"data_stored":2660331,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":219335,"internal_metadata":659990329},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:56 2023","interfaces":[{"interface":"back","average":{"1min":0.30399999999999999,"5min":0.34699999999999998,"15min":0.36499999999999999},"min":{"1min":0.23999999999999999,"5min":0.14199999999999999,"15min":0.14199999999999999},"max":{"1min":0.42399999999999999,"5min":1.595,"15min":7.4249999999999998},"last":0.17100000000000001},{"interface":"front","average":{"1min":0.35499999999999998,"5min":0.40899999999999997,"15min":0.376},"min":{"1min":0.23999999999999999,"5min":0.19600000000000001,"15min":0.183},"max":{"1min":0.46999999999999997,"5min":2.6859999999999999,"15min":2.6859999999999999},"last":0.23000000000000001}]},{"osd":2,"last update":"Wed Nov 8 05:28:56 2023","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.44400000000000001,"15min":0.376},"min":{"1min":0.187,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":2.2650000000000001,"5min":5.8209999999999997,"15min":5.8209999999999997},"last":0.27200000000000002},{"interface":"front","average":{"1min":0.316,"5min":0.40500000000000003,"15min":0.35999999999999999},"min":{"1min":0.19600000000000001,"5min":0.188,"15min":0.17599999999999999},"max":{"1min":1.137,"5min":4.2350000000000003,"15min":4.2350000000000003},"last":0.245}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-08T05:29:45.472 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-11-08T05:29:45.756 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T05:29:45.758 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2023-11-08T05:29:45.769 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":1563,"stamp":"2023-11-08T05:29:44.391275+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2634224,"kb_used_data":408868,"kb_used_omap":780,"kb_used_meta":2220403,"kb_avail":280481296,"statfs":{"total":289910292480,"available":287212847104,"internally_reserved":0,"allocated":418680832,"data_stored":410423593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":799447,"internal_metadata":2273692969},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-536782902,"num_objects":-164,"num_object_clones":0,"num_object_copies":-328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-115,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-164,"num_whiteouts":0,"num_read":-122056,"num_read_kb":-39011942,"num_write":-413642,"num_write_kb":-151705339,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-8,"num_bytes_recovered":-16922624,"num_keys_recovered":-2,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.424817"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":453,"reported_epoch":452,"state":"active+undersized","last_fresh":"2023-11-08T05:29:28.539262+0000","last_change":"2023-11-08T05:28:13.557071+0000","last_active":"2023-11-08T05:29:28.539262+0000","last_peered":"2023-11-08T05:29:28.539262+0000","last_clean":"2023-11-08T05:28:01.458439+0000","last_became_active":"2023-11-08T05:28:13.557071+0000","last_became_peered":"2023-11-08T05:28:13.557071+0000","last_unstale":"2023-11-08T05:29:28.539262+0000","last_undegraded":"2023-11-08T05:29:28.539262+0000","last_fullsized":"2023-11-08T05:28:13.547025+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":15,"reported_epoch":452,"state":"active+undersized","last_fresh":"2023-11-08T05:29:28.550339+0000","last_change":"2023-11-08T05:28:13.559513+0000","last_active":"2023-11-08T05:29:28.550339+0000","last_peered":"2023-11-08T05:29:28.550339+0000","last_clean":"2023-11-08T04:52:46.531964+0000","last_became_active":"2023-11-08T05:28:13.559513+0000","last_became_peered":"2023-11-08T05:28:13.559513+0000","last_unstale":"2023-11-08T05:29:28.550339+0000","last_undegraded":"2023-11-08T05:29:28.550339+0000","last_fullsized":"2023-11-08T05:28:13.540852+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":450,"reported_epoch":452,"state":"active+clean","last_fresh":"2023-11-08T05:29:28.468933+0000","last_change":"2023-11-08T04:52:47.546100+0000","last_active":"2023-11-08T05:29:28.468933+0000","last_peered":"2023-11-08T05:29:28.468933+0000","last_clean":"2023-11-08T05:29:28.468933+0000","last_became_active":"2023-11-08T04:52:47.545840+0000","last_became_peered":"2023-11-08T04:52:47.545840+0000","last_unstale":"2023-11-08T05:29:28.468933+0000","last_undegraded":"2023-11-08T05:29:28.468933+0000","last_fullsized":"2023-11-08T05:29:28.468933+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":448,"reported_epoch":452,"state":"active+clean","last_fresh":"2023-11-08T05:29:28.534734+0000","last_change":"2023-11-08T04:52:47.548597+0000","last_active":"2023-11-08T05:29:28.534734+0000","last_peered":"2023-11-08T05:29:28.534734+0000","last_clean":"2023-11-08T05:29:28.534734+0000","last_became_active":"2023-11-08T04:52:47.548294+0000","last_became_peered":"2023-11-08T04:52:47.548294+0000","last_unstale":"2023-11-08T05:29:28.534734+0000","last_undegraded":"2023-11-08T05:29:28.534734+0000","last_fullsized":"2023-11-08T05:29:28.534734+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":454,"reported_epoch":452,"state":"active+undersized","last_fresh":"2023-11-08T05:29:28.532757+0000","last_change":"2023-11-08T05:28:13.545468+0000","last_active":"2023-11-08T05:29:28.532757+0000","last_peered":"2023-11-08T05:29:28.532757+0000","last_clean":"2023-11-08T05:28:01.456723+0000","last_became_active":"2023-11-08T05:28:13.545468+0000","last_became_peered":"2023-11-08T05:28:13.545468+0000","last_unstale":"2023-11-08T05:29:28.532757+0000","last_undegraded":"2023-11-08T05:29:28.532757+0000","last_fullsized":"2023-11-08T05:28:13.537207+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_clean_scrub_stamp":"2023-11-08T04:52:45.528796+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":450,"reported_epoch":452,"state":"active+clean","last_fresh":"2023-11-08T05:29:28.469973+0000","last_change":"2023-11-08T04:52:47.546022+0000","last_active":"2023-11-08T05:29:28.469973+0000","last_peered":"2023-11-08T05:29:28.469973+0000","last_clean":"2023-11-08T05:29:28.469973+0000","last_became_active":"2023-11-08T04:52:47.545767+0000","last_became_peered":"2023-11-08T04:52:47.545767+0000","last_unstale":"2023-11-08T05:29:28.469973+0000","last_undegraded":"2023-11-08T05:29:28.469973+0000","last_fullsized":"2023-11-08T05:29:28.469973+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":453,"reported_epoch":452,"state":"active+undersized","last_fresh":"2023-11-08T05:29:28.536706+0000","last_change":"2023-11-08T05:28:13.563357+0000","last_active":"2023-11-08T05:29:28.536706+0000","last_peered":"2023-11-08T05:29:28.536706+0000","last_clean":"2023-11-08T05:28:01.457691+0000","last_became_active":"2023-11-08T05:28:13.563357+0000","last_became_peered":"2023-11-08T05:28:13.563357+0000","last_unstale":"2023-11-08T05:29:28.536706+0000","last_undegraded":"2023-11-08T05:29:28.536706+0000","last_fullsized":"2023-11-08T05:28:13.541614+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":453,"reported_epoch":452,"state":"active+undersized","last_fresh":"2023-11-08T05:29:28.537348+0000","last_change":"2023-11-08T05:28:13.550672+0000","last_active":"2023-11-08T05:29:28.537348+0000","last_peered":"2023-11-08T05:29:28.537348+0000","last_clean":"2023-11-08T05:28:01.457904+0000","last_became_active":"2023-11-08T05:28:13.550672+0000","last_became_peered":"2023-11-08T05:28:13.550672+0000","last_unstale":"2023-11-08T05:29:28.537348+0000","last_undegraded":"2023-11-08T05:29:28.537348+0000","last_fullsized":"2023-11-08T05:28:13.539595+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":453,"reported_epoch":452,"state":"active+undersized","last_fresh":"2023-11-08T05:29:28.540077+0000","last_change":"2023-11-08T05:28:13.557968+0000","last_active":"2023-11-08T05:29:28.540077+0000","last_peered":"2023-11-08T05:29:28.540077+0000","last_clean":"2023-11-08T05:28:01.458433+0000","last_became_active":"2023-11-08T05:28:13.557968+0000","last_became_peered":"2023-11-08T05:28:13.557968+0000","last_unstale":"2023-11-08T05:29:28.540077+0000","last_undegraded":"2023-11-08T05:29:28.540077+0000","last_fullsized":"2023-11-08T05:28:13.542832+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":1,"acting":1,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738814,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":644968,"kb_used_data":5280,"kb_used_omap":275,"kb_used_meta":639404,"kb_avail":93726872,"statfs":{"total":96636764160,"available":95976316928,"internally_reserved":0,"allocated":5406720,"data_stored":2659362,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":281650,"internal_metadata":654750670},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:59 2023","interfaces":[{"interface":"back","average":{"1min":0.30499999999999999,"5min":0.439,"15min":0.36099999999999999},"min":{"1min":0.17399999999999999,"5min":0.14499999999999999,"15min":0.14499999999999999},"max":{"1min":0.60099999999999998,"5min":4.9379999999999997,"15min":4.9379999999999997},"last":0.379},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.49399999999999999,"15min":0.42399999999999999},"min":{"1min":0.221,"5min":0.19600000000000001,"15min":0.17000000000000001},"max":{"1min":0.63200000000000001,"5min":4.9859999999999998,"15min":4.9859999999999998},"last":0.40899999999999997}]},{"osd":1,"last update":"Wed Nov 8 05:29:01 2023","interfaces":[{"interface":"back","average":{"1min":0.30299999999999999,"5min":0.81899999999999995,"15min":0.47399999999999998},"min":{"1min":0.16,"5min":0.153,"15min":0.153},"max":{"1min":0.68200000000000005,"5min":23.242999999999999,"15min":23.242999999999999},"last":0.28000000000000003},{"interface":"front","average":{"1min":0.46500000000000002,"5min":0.72199999999999998,"15min":0.46100000000000002},"min":{"1min":0.17699999999999999,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":2.0419999999999998,"5min":23.251000000000001,"15min":23.251000000000001},"last":0.30299999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738798,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1339228,"kb_used_data":398304,"kb_used_omap":291,"kb_used_meta":936476,"kb_avail":93032612,"statfs":{"total":96636764160,"available":95265394688,"internally_reserved":0,"allocated":407863296,"data_stored":405103900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":298462,"internal_metadata":958951970},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.436,"15min":0.36299999999999999},"min":{"1min":0.19700000000000001,"5min":0.14699999999999999,"15min":0.13700000000000001},"max":{"1min":3.222,"5min":6.5499999999999998,"15min":6.9989999999999997},"last":0.245},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.46300000000000002,"15min":0.42299999999999999},"min":{"1min":0.17799999999999999,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":5.0030000000000001,"5min":6.0629999999999997,"15min":9.6869999999999994},"last":0.28599999999999998}]},{"osd":2,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.51600000000000001,"15min":0.47099999999999997},"min":{"1min":0.21099999999999999,"5min":0.153,"15min":0.153},"max":{"1min":5.7439999999999998,"5min":5.8280000000000003,"15min":9.7639999999999993},"last":0.252},{"interface":"front","average":{"1min":0.497,"5min":0.56200000000000006,"15min":0.42299999999999999},"min":{"1min":0.20399999999999999,"5min":0.187,"15min":0.16800000000000001},"max":{"1min":3.9319999999999999,"5min":8.8040000000000003,"15min":8.8040000000000003},"last":0.29899999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738816,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":650028,"kb_used_data":5284,"kb_used_omap":214,"kb_used_meta":644521,"kb_avail":93721812,"statfs":{"total":96636764160,"available":95971135488,"internally_reserved":0,"allocated":5410816,"data_stored":2660331,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":219335,"internal_metadata":659990329},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:56 2023","interfaces":[{"interface":"back","average":{"1min":0.30399999999999999,"5min":0.34699999999999998,"15min":0.36499999999999999},"min":{"1min":0.23999999999999999,"5min":0.14199999999999999,"15min":0.14199999999999999},"max":{"1min":0.42399999999999999,"5min":1.595,"15min":7.4249999999999998},"last":0.17100000000000001},{"interface":"front","average":{"1min":0.35499999999999998,"5min":0.40899999999999997,"15min":0.376},"min":{"1min":0.23999999999999999,"5min":0.19600000000000001,"15min":0.183},"max":{"1min":0.46999999999999997,"5min":2.6859999999999999,"15min":2.6859999999999999},"last":0.23000000000000001}]},{"osd":2,"last update":"Wed Nov 8 05:28:56 2023","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.44400000000000001,"15min":0.376},"min":{"1min":0.187,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":2.2650000000000001,"5min":5.8209999999999997,"15min":5.8209999999999997},"last":0.27200000000000002},{"interface":"front","average":{"1min":0.316,"5min":0.40500000000000003,"15min":0.35999999999999999},"min":{"1min":0.19600000000000001,"5min":0.188,"15min":0.17599999999999999},"max":{"1min":1.137,"5min":4.2350000000000003,"15min":4.2350000000000003},"last":0.245}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-08T05:29:45.770 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-11-08T05:29:45.770 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 453, 'reported_epoch': 452, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:29:28.539262+0000', 'last_change': '2023-11-08T05:28:13.557071+0000', 'last_active': '2023-11-08T05:29:28.539262+0000', 'last_peered': '2023-11-08T05:29:28.539262+0000', 'last_clean': '2023-11-08T05:28:01.458439+0000', 'last_became_active': '2023-11-08T05:28:13.557071+0000', 'last_became_peered': '2023-11-08T05:28:13.557071+0000', 'last_unstale': '2023-11-08T05:29:28.539262+0000', 'last_undegraded': '2023-11-08T05:29:28.539262+0000', 'last_fullsized': '2023-11-08T05:28:13.547025+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:29:45.771 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-11-08T05:29:45.771 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 15, 'reported_epoch': 452, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:29:28.550339+0000', 'last_change': '2023-11-08T05:28:13.559513+0000', 'last_active': '2023-11-08T05:29:28.550339+0000', 'last_peered': '2023-11-08T05:29:28.550339+0000', 'last_clean': '2023-11-08T04:52:46.531964+0000', 'last_became_active': '2023-11-08T05:28:13.559513+0000', 'last_became_peered': '2023-11-08T05:28:13.559513+0000', 'last_unstale': '2023-11-08T05:29:28.550339+0000', 'last_undegraded': '2023-11-08T05:29:28.550339+0000', 'last_fullsized': '2023-11-08T05:28:13.540852+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:29:45.771 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-11-08T05:29:45.772 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "0'0", 'reported_seq': 454, 'reported_epoch': 452, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:29:28.532757+0000', 'last_change': '2023-11-08T05:28:13.545468+0000', 'last_active': '2023-11-08T05:29:28.532757+0000', 'last_peered': '2023-11-08T05:29:28.532757+0000', 'last_clean': '2023-11-08T05:28:01.456723+0000', 'last_became_active': '2023-11-08T05:28:13.545468+0000', 'last_became_peered': '2023-11-08T05:28:13.545468+0000', 'last_unstale': '2023-11-08T05:29:28.532757+0000', 'last_undegraded': '2023-11-08T05:29:28.532757+0000', 'last_fullsized': '2023-11-08T05:28:13.537207+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 9, 'last_epoch_clean': 10, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:45.528796+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:45.528796+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:45.528796+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:29:45.772 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-11-08T05:29:45.772 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 453, 'reported_epoch': 452, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:29:28.536706+0000', 'last_change': '2023-11-08T05:28:13.563357+0000', 'last_active': '2023-11-08T05:29:28.536706+0000', 'last_peered': '2023-11-08T05:29:28.536706+0000', 'last_clean': '2023-11-08T05:28:01.457691+0000', 'last_became_active': '2023-11-08T05:28:13.563357+0000', 'last_became_peered': '2023-11-08T05:28:13.563357+0000', 'last_unstale': '2023-11-08T05:29:28.536706+0000', 'last_undegraded': '2023-11-08T05:29:28.536706+0000', 'last_fullsized': '2023-11-08T05:28:13.541614+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:29:45.773 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-11-08T05:29:45.773 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 453, 'reported_epoch': 452, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:29:28.537348+0000', 'last_change': '2023-11-08T05:28:13.550672+0000', 'last_active': '2023-11-08T05:29:28.537348+0000', 'last_peered': '2023-11-08T05:29:28.537348+0000', 'last_clean': '2023-11-08T05:28:01.457904+0000', 'last_became_active': '2023-11-08T05:28:13.550672+0000', 'last_became_peered': '2023-11-08T05:28:13.550672+0000', 'last_unstale': '2023-11-08T05:29:28.537348+0000', 'last_undegraded': '2023-11-08T05:29:28.537348+0000', 'last_fullsized': '2023-11-08T05:28:13.539595+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:29:45.773 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-11-08T05:29:45.774 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 453, 'reported_epoch': 452, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:29:28.540077+0000', 'last_change': '2023-11-08T05:28:13.557968+0000', 'last_active': '2023-11-08T05:29:28.540077+0000', 'last_peered': '2023-11-08T05:29:28.540077+0000', 'last_clean': '2023-11-08T05:28:01.458433+0000', 'last_became_active': '2023-11-08T05:28:13.557968+0000', 'last_became_peered': '2023-11-08T05:28:13.557968+0000', 'last_unstale': '2023-11-08T05:29:28.540077+0000', 'last_undegraded': '2023-11-08T05:29:28.540077+0000', 'last_fullsized': '2023-11-08T05:28:13.542832+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:29:45.774 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-11-08T05:29:46.121 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T05:29:46.122 INFO:teuthology.orchestra.run.smithi087.stdout:{"fsid":"ab4cbc06-449d-497d-b513-220d9db2b590","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 6 pgs undersized","count":6},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2228,"monmap":{"epoch":1,"min_mon_release_name":"pacific","num_mons":1},"osdmap":{"epoch":456,"num_osds":3,"num_up_osds":2,"osd_up_since":1699421292,"num_in_osds":3,"osd_in_since":1699419158,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":6},{"state_name":"active+clean","count":3}],"num_pgs":9,"num_pools":2,"num_objects":0,"data_bytes":0,"bytes_used":2697445376,"bytes_avail":287212847104,"bytes_total":289910292480},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":41,"modified":"2023-11-08T05:29:32.044544+0000","services":{}},"progress_events":{}} 2023-11-08T05:29:46.135 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-11-08T05:29:46.135 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-11-08T05:29:46.421 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T05:29:46.423 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2023-11-08T05:29:46.434 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":1564,"stamp":"2023-11-08T05:29:46.047412+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2634284,"kb_used_data":408928,"kb_used_omap":780,"kb_used_meta":2220403,"kb_avail":280481236,"statfs":{"total":289910292480,"available":287212785664,"internally_reserved":0,"allocated":418742272,"data_stored":410455190,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":799447,"internal_metadata":2273692969},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-536782902,"num_objects":-164,"num_object_clones":0,"num_object_copies":-328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-115,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-164,"num_whiteouts":0,"num_read":-122056,"num_read_kb":-39011942,"num_write":-413642,"num_write_kb":-151705339,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-8,"num_bytes_recovered":-16922624,"num_keys_recovered":-2,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.424817"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":456,"reported_epoch":455,"state":"active+undersized","last_fresh":"2023-11-08T05:29:43.389354+0000","last_change":"2023-11-08T05:28:13.557071+0000","last_active":"2023-11-08T05:29:43.389354+0000","last_peered":"2023-11-08T05:29:43.389354+0000","last_clean":"2023-11-08T05:28:01.458439+0000","last_became_active":"2023-11-08T05:28:13.557071+0000","last_became_peered":"2023-11-08T05:28:13.557071+0000","last_unstale":"2023-11-08T05:29:43.389354+0000","last_undegraded":"2023-11-08T05:29:43.389354+0000","last_fullsized":"2023-11-08T05:28:13.547025+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":18,"reported_epoch":455,"state":"active+undersized","last_fresh":"2023-11-08T05:29:43.389262+0000","last_change":"2023-11-08T05:28:13.559513+0000","last_active":"2023-11-08T05:29:43.389262+0000","last_peered":"2023-11-08T05:29:43.389262+0000","last_clean":"2023-11-08T04:52:46.531964+0000","last_became_active":"2023-11-08T05:28:13.559513+0000","last_became_peered":"2023-11-08T05:28:13.559513+0000","last_unstale":"2023-11-08T05:29:43.389262+0000","last_undegraded":"2023-11-08T05:29:43.389262+0000","last_fullsized":"2023-11-08T05:28:13.540852+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":453,"reported_epoch":455,"state":"active+clean","last_fresh":"2023-11-08T05:29:43.390991+0000","last_change":"2023-11-08T04:52:47.546100+0000","last_active":"2023-11-08T05:29:43.390991+0000","last_peered":"2023-11-08T05:29:43.390991+0000","last_clean":"2023-11-08T05:29:43.390991+0000","last_became_active":"2023-11-08T04:52:47.545840+0000","last_became_peered":"2023-11-08T04:52:47.545840+0000","last_unstale":"2023-11-08T05:29:43.390991+0000","last_undegraded":"2023-11-08T05:29:43.390991+0000","last_fullsized":"2023-11-08T05:29:43.390991+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":451,"reported_epoch":455,"state":"active+clean","last_fresh":"2023-11-08T05:29:43.389261+0000","last_change":"2023-11-08T04:52:47.548597+0000","last_active":"2023-11-08T05:29:43.389261+0000","last_peered":"2023-11-08T05:29:43.389261+0000","last_clean":"2023-11-08T05:29:43.389261+0000","last_became_active":"2023-11-08T04:52:47.548294+0000","last_became_peered":"2023-11-08T04:52:47.548294+0000","last_unstale":"2023-11-08T05:29:43.389261+0000","last_undegraded":"2023-11-08T05:29:43.389261+0000","last_fullsized":"2023-11-08T05:29:43.389261+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":457,"reported_epoch":455,"state":"active+undersized","last_fresh":"2023-11-08T05:29:43.389822+0000","last_change":"2023-11-08T05:28:13.545468+0000","last_active":"2023-11-08T05:29:43.389822+0000","last_peered":"2023-11-08T05:29:43.389822+0000","last_clean":"2023-11-08T05:28:01.456723+0000","last_became_active":"2023-11-08T05:28:13.545468+0000","last_became_peered":"2023-11-08T05:28:13.545468+0000","last_unstale":"2023-11-08T05:29:43.389822+0000","last_undegraded":"2023-11-08T05:29:43.389822+0000","last_fullsized":"2023-11-08T05:28:13.537207+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_clean_scrub_stamp":"2023-11-08T04:52:45.528796+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":462,"reported_epoch":456,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:29:44.395350+0000","last_change":"2023-11-08T05:29:44.395350+0000","last_active":"2023-11-08T05:29:44.395350+0000","last_peered":"2023-11-08T05:29:44.395350+0000","last_clean":"2023-11-08T05:29:44.395350+0000","last_became_active":"2023-11-08T05:29:44.395012+0000","last_became_peered":"2023-11-08T05:29:44.395012+0000","last_unstale":"2023-11-08T05:29:44.395350+0000","last_undegraded":"2023-11-08T05:29:44.395350+0000","last_fullsized":"2023-11-08T05:29:44.395350+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":456,"reported_epoch":455,"state":"active+undersized","last_fresh":"2023-11-08T05:29:43.389390+0000","last_change":"2023-11-08T05:28:13.563357+0000","last_active":"2023-11-08T05:29:43.389390+0000","last_peered":"2023-11-08T05:29:43.389390+0000","last_clean":"2023-11-08T05:28:01.457691+0000","last_became_active":"2023-11-08T05:28:13.563357+0000","last_became_peered":"2023-11-08T05:28:13.563357+0000","last_unstale":"2023-11-08T05:29:43.389390+0000","last_undegraded":"2023-11-08T05:29:43.389390+0000","last_fullsized":"2023-11-08T05:28:13.541614+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":456,"reported_epoch":455,"state":"active+undersized","last_fresh":"2023-11-08T05:29:43.389311+0000","last_change":"2023-11-08T05:28:13.550672+0000","last_active":"2023-11-08T05:29:43.389311+0000","last_peered":"2023-11-08T05:29:43.389311+0000","last_clean":"2023-11-08T05:28:01.457904+0000","last_became_active":"2023-11-08T05:28:13.550672+0000","last_became_peered":"2023-11-08T05:28:13.550672+0000","last_unstale":"2023-11-08T05:29:43.389311+0000","last_undegraded":"2023-11-08T05:29:43.389311+0000","last_fullsized":"2023-11-08T05:28:13.539595+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":456,"reported_epoch":455,"state":"active+undersized","last_fresh":"2023-11-08T05:29:43.389474+0000","last_change":"2023-11-08T05:28:13.557968+0000","last_active":"2023-11-08T05:29:43.389474+0000","last_peered":"2023-11-08T05:29:43.389474+0000","last_clean":"2023-11-08T05:28:01.458433+0000","last_became_active":"2023-11-08T05:28:13.557968+0000","last_became_peered":"2023-11-08T05:28:13.557968+0000","last_unstale":"2023-11-08T05:29:43.389474+0000","last_undegraded":"2023-11-08T05:29:43.389474+0000","last_fullsized":"2023-11-08T05:28:13.542832+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":1,"acting":1,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":10,"acting":11,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738815,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":644992,"kb_used_data":5304,"kb_used_omap":275,"kb_used_meta":639404,"kb_avail":93726848,"statfs":{"total":96636764160,"available":95976292352,"internally_reserved":0,"allocated":5431296,"data_stored":2672005,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":281650,"internal_metadata":654750670},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:59 2023","interfaces":[{"interface":"back","average":{"1min":0.30499999999999999,"5min":0.439,"15min":0.36099999999999999},"min":{"1min":0.17399999999999999,"5min":0.14499999999999999,"15min":0.14499999999999999},"max":{"1min":0.60099999999999998,"5min":4.9379999999999997,"15min":4.9379999999999997},"last":0.379},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.49399999999999999,"15min":0.42399999999999999},"min":{"1min":0.221,"5min":0.19600000000000001,"15min":0.17000000000000001},"max":{"1min":0.63200000000000001,"5min":4.9859999999999998,"15min":4.9859999999999998},"last":0.40899999999999997}]},{"osd":1,"last update":"Wed Nov 8 05:29:01 2023","interfaces":[{"interface":"back","average":{"1min":0.30299999999999999,"5min":0.81899999999999995,"15min":0.47399999999999998},"min":{"1min":0.16,"5min":0.153,"15min":0.153},"max":{"1min":0.68200000000000005,"5min":23.242999999999999,"15min":23.242999999999999},"last":0.27900000000000003},{"interface":"front","average":{"1min":0.46500000000000002,"5min":0.72199999999999998,"15min":0.46100000000000002},"min":{"1min":0.17699999999999999,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":2.0419999999999998,"5min":23.251000000000001,"15min":23.251000000000001},"last":0.29099999999999998}]}]},{"osd":0,"up_from":8,"seq":34359738798,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1339228,"kb_used_data":398304,"kb_used_omap":291,"kb_used_meta":936476,"kb_avail":93032612,"statfs":{"total":96636764160,"available":95265394688,"internally_reserved":0,"allocated":407863296,"data_stored":405103900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":298462,"internal_metadata":958951970},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.436,"15min":0.36299999999999999},"min":{"1min":0.19700000000000001,"5min":0.14699999999999999,"15min":0.13700000000000001},"max":{"1min":3.222,"5min":6.5499999999999998,"15min":6.9989999999999997},"last":0.245},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.46300000000000002,"15min":0.42299999999999999},"min":{"1min":0.17799999999999999,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":5.0030000000000001,"5min":6.0629999999999997,"15min":9.6869999999999994},"last":0.28599999999999998}]},{"osd":2,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.51600000000000001,"15min":0.47099999999999997},"min":{"1min":0.21099999999999999,"5min":0.153,"15min":0.153},"max":{"1min":5.7439999999999998,"5min":5.8280000000000003,"15min":9.7639999999999993},"last":0.252},{"interface":"front","average":{"1min":0.497,"5min":0.56200000000000006,"15min":0.42299999999999999},"min":{"1min":0.20399999999999999,"5min":0.187,"15min":0.16800000000000001},"max":{"1min":3.9319999999999999,"5min":8.8040000000000003,"15min":8.8040000000000003},"last":0.29899999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738817,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":650064,"kb_used_data":5320,"kb_used_omap":214,"kb_used_meta":644521,"kb_avail":93721776,"statfs":{"total":96636764160,"available":95971098624,"internally_reserved":0,"allocated":5447680,"data_stored":2679285,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":219335,"internal_metadata":659990329},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:56 2023","interfaces":[{"interface":"back","average":{"1min":0.30399999999999999,"5min":0.34699999999999998,"15min":0.36499999999999999},"min":{"1min":0.23999999999999999,"5min":0.14199999999999999,"15min":0.14199999999999999},"max":{"1min":0.42399999999999999,"5min":1.595,"15min":7.4249999999999998},"last":0.17100000000000001},{"interface":"front","average":{"1min":0.35499999999999998,"5min":0.40899999999999997,"15min":0.376},"min":{"1min":0.23999999999999999,"5min":0.19600000000000001,"15min":0.183},"max":{"1min":0.46999999999999997,"5min":2.6859999999999999,"15min":2.6859999999999999},"last":0.23000000000000001}]},{"osd":2,"last update":"Wed Nov 8 05:28:56 2023","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.44400000000000001,"15min":0.376},"min":{"1min":0.187,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":2.2650000000000001,"5min":5.8209999999999997,"15min":5.8209999999999997},"last":0.16300000000000001},{"interface":"front","average":{"1min":0.316,"5min":0.40500000000000003,"15min":0.35999999999999999},"min":{"1min":0.19600000000000001,"5min":0.188,"15min":0.17599999999999999},"max":{"1min":1.137,"5min":4.2350000000000003,"15min":4.2350000000000003},"last":0.17199999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-08T05:29:49.436 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-11-08T05:29:49.731 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T05:29:49.733 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2023-11-08T05:29:49.744 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":1565,"stamp":"2023-11-08T05:29:48.047805+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2634284,"kb_used_data":408928,"kb_used_omap":780,"kb_used_meta":2220403,"kb_avail":280481236,"statfs":{"total":289910292480,"available":287212785664,"internally_reserved":0,"allocated":418742272,"data_stored":410455190,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":799447,"internal_metadata":2273692969},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-536782902,"num_objects":-164,"num_object_clones":0,"num_object_copies":-328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-115,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-164,"num_whiteouts":0,"num_read":-122056,"num_read_kb":-39011942,"num_write":-413642,"num_write_kb":-151705339,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-8,"num_bytes_recovered":-16922624,"num_keys_recovered":-2,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.424817"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":456,"reported_epoch":455,"state":"active+undersized","last_fresh":"2023-11-08T05:29:43.389354+0000","last_change":"2023-11-08T05:28:13.557071+0000","last_active":"2023-11-08T05:29:43.389354+0000","last_peered":"2023-11-08T05:29:43.389354+0000","last_clean":"2023-11-08T05:28:01.458439+0000","last_became_active":"2023-11-08T05:28:13.557071+0000","last_became_peered":"2023-11-08T05:28:13.557071+0000","last_unstale":"2023-11-08T05:29:43.389354+0000","last_undegraded":"2023-11-08T05:29:43.389354+0000","last_fullsized":"2023-11-08T05:28:13.547025+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":18,"reported_epoch":455,"state":"active+undersized","last_fresh":"2023-11-08T05:29:43.389262+0000","last_change":"2023-11-08T05:28:13.559513+0000","last_active":"2023-11-08T05:29:43.389262+0000","last_peered":"2023-11-08T05:29:43.389262+0000","last_clean":"2023-11-08T04:52:46.531964+0000","last_became_active":"2023-11-08T05:28:13.559513+0000","last_became_peered":"2023-11-08T05:28:13.559513+0000","last_unstale":"2023-11-08T05:29:43.389262+0000","last_undegraded":"2023-11-08T05:29:43.389262+0000","last_fullsized":"2023-11-08T05:28:13.540852+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":453,"reported_epoch":455,"state":"active+clean","last_fresh":"2023-11-08T05:29:43.390991+0000","last_change":"2023-11-08T04:52:47.546100+0000","last_active":"2023-11-08T05:29:43.390991+0000","last_peered":"2023-11-08T05:29:43.390991+0000","last_clean":"2023-11-08T05:29:43.390991+0000","last_became_active":"2023-11-08T04:52:47.545840+0000","last_became_peered":"2023-11-08T04:52:47.545840+0000","last_unstale":"2023-11-08T05:29:43.390991+0000","last_undegraded":"2023-11-08T05:29:43.390991+0000","last_fullsized":"2023-11-08T05:29:43.390991+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":451,"reported_epoch":455,"state":"active+clean","last_fresh":"2023-11-08T05:29:43.389261+0000","last_change":"2023-11-08T04:52:47.548597+0000","last_active":"2023-11-08T05:29:43.389261+0000","last_peered":"2023-11-08T05:29:43.389261+0000","last_clean":"2023-11-08T05:29:43.389261+0000","last_became_active":"2023-11-08T04:52:47.548294+0000","last_became_peered":"2023-11-08T04:52:47.548294+0000","last_unstale":"2023-11-08T05:29:43.389261+0000","last_undegraded":"2023-11-08T05:29:43.389261+0000","last_fullsized":"2023-11-08T05:29:43.389261+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":457,"reported_epoch":455,"state":"active+undersized","last_fresh":"2023-11-08T05:29:43.389822+0000","last_change":"2023-11-08T05:28:13.545468+0000","last_active":"2023-11-08T05:29:43.389822+0000","last_peered":"2023-11-08T05:29:43.389822+0000","last_clean":"2023-11-08T05:28:01.456723+0000","last_became_active":"2023-11-08T05:28:13.545468+0000","last_became_peered":"2023-11-08T05:28:13.545468+0000","last_unstale":"2023-11-08T05:29:43.389822+0000","last_undegraded":"2023-11-08T05:29:43.389822+0000","last_fullsized":"2023-11-08T05:28:13.537207+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_clean_scrub_stamp":"2023-11-08T04:52:45.528796+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":462,"reported_epoch":456,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:29:44.395350+0000","last_change":"2023-11-08T05:29:44.395350+0000","last_active":"2023-11-08T05:29:44.395350+0000","last_peered":"2023-11-08T05:29:44.395350+0000","last_clean":"2023-11-08T05:29:44.395350+0000","last_became_active":"2023-11-08T05:29:44.395012+0000","last_became_peered":"2023-11-08T05:29:44.395012+0000","last_unstale":"2023-11-08T05:29:44.395350+0000","last_undegraded":"2023-11-08T05:29:44.395350+0000","last_fullsized":"2023-11-08T05:29:44.395350+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":456,"reported_epoch":455,"state":"active+undersized","last_fresh":"2023-11-08T05:29:43.389390+0000","last_change":"2023-11-08T05:28:13.563357+0000","last_active":"2023-11-08T05:29:43.389390+0000","last_peered":"2023-11-08T05:29:43.389390+0000","last_clean":"2023-11-08T05:28:01.457691+0000","last_became_active":"2023-11-08T05:28:13.563357+0000","last_became_peered":"2023-11-08T05:28:13.563357+0000","last_unstale":"2023-11-08T05:29:43.389390+0000","last_undegraded":"2023-11-08T05:29:43.389390+0000","last_fullsized":"2023-11-08T05:28:13.541614+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":456,"reported_epoch":455,"state":"active+undersized","last_fresh":"2023-11-08T05:29:43.389311+0000","last_change":"2023-11-08T05:28:13.550672+0000","last_active":"2023-11-08T05:29:43.389311+0000","last_peered":"2023-11-08T05:29:43.389311+0000","last_clean":"2023-11-08T05:28:01.457904+0000","last_became_active":"2023-11-08T05:28:13.550672+0000","last_became_peered":"2023-11-08T05:28:13.550672+0000","last_unstale":"2023-11-08T05:29:43.389311+0000","last_undegraded":"2023-11-08T05:29:43.389311+0000","last_fullsized":"2023-11-08T05:28:13.539595+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":456,"reported_epoch":455,"state":"active+undersized","last_fresh":"2023-11-08T05:29:43.389474+0000","last_change":"2023-11-08T05:28:13.557968+0000","last_active":"2023-11-08T05:29:43.389474+0000","last_peered":"2023-11-08T05:29:43.389474+0000","last_clean":"2023-11-08T05:28:01.458433+0000","last_became_active":"2023-11-08T05:28:13.557968+0000","last_became_peered":"2023-11-08T05:28:13.557968+0000","last_unstale":"2023-11-08T05:29:43.389474+0000","last_undegraded":"2023-11-08T05:29:43.389474+0000","last_fullsized":"2023-11-08T05:28:13.542832+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":1,"acting":1,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":10,"acting":11,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738815,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":644992,"kb_used_data":5304,"kb_used_omap":275,"kb_used_meta":639404,"kb_avail":93726848,"statfs":{"total":96636764160,"available":95976292352,"internally_reserved":0,"allocated":5431296,"data_stored":2672005,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":281650,"internal_metadata":654750670},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:59 2023","interfaces":[{"interface":"back","average":{"1min":0.30499999999999999,"5min":0.439,"15min":0.36099999999999999},"min":{"1min":0.17399999999999999,"5min":0.14499999999999999,"15min":0.14499999999999999},"max":{"1min":0.60099999999999998,"5min":4.9379999999999997,"15min":4.9379999999999997},"last":0.379},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.49399999999999999,"15min":0.42399999999999999},"min":{"1min":0.221,"5min":0.19600000000000001,"15min":0.17000000000000001},"max":{"1min":0.63200000000000001,"5min":4.9859999999999998,"15min":4.9859999999999998},"last":0.40899999999999997}]},{"osd":1,"last update":"Wed Nov 8 05:29:01 2023","interfaces":[{"interface":"back","average":{"1min":0.30299999999999999,"5min":0.81899999999999995,"15min":0.47399999999999998},"min":{"1min":0.16,"5min":0.153,"15min":0.153},"max":{"1min":0.68200000000000005,"5min":23.242999999999999,"15min":23.242999999999999},"last":0.27900000000000003},{"interface":"front","average":{"1min":0.46500000000000002,"5min":0.72199999999999998,"15min":0.46100000000000002},"min":{"1min":0.17699999999999999,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":2.0419999999999998,"5min":23.251000000000001,"15min":23.251000000000001},"last":0.29099999999999998}]}]},{"osd":0,"up_from":8,"seq":34359738798,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1339228,"kb_used_data":398304,"kb_used_omap":291,"kb_used_meta":936476,"kb_avail":93032612,"statfs":{"total":96636764160,"available":95265394688,"internally_reserved":0,"allocated":407863296,"data_stored":405103900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":298462,"internal_metadata":958951970},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.436,"15min":0.36299999999999999},"min":{"1min":0.19700000000000001,"5min":0.14699999999999999,"15min":0.13700000000000001},"max":{"1min":3.222,"5min":6.5499999999999998,"15min":6.9989999999999997},"last":0.245},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.46300000000000002,"15min":0.42299999999999999},"min":{"1min":0.17799999999999999,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":5.0030000000000001,"5min":6.0629999999999997,"15min":9.6869999999999994},"last":0.28599999999999998}]},{"osd":2,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.51600000000000001,"15min":0.47099999999999997},"min":{"1min":0.21099999999999999,"5min":0.153,"15min":0.153},"max":{"1min":5.7439999999999998,"5min":5.8280000000000003,"15min":9.7639999999999993},"last":0.252},{"interface":"front","average":{"1min":0.497,"5min":0.56200000000000006,"15min":0.42299999999999999},"min":{"1min":0.20399999999999999,"5min":0.187,"15min":0.16800000000000001},"max":{"1min":3.9319999999999999,"5min":8.8040000000000003,"15min":8.8040000000000003},"last":0.29899999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738817,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":650064,"kb_used_data":5320,"kb_used_omap":214,"kb_used_meta":644521,"kb_avail":93721776,"statfs":{"total":96636764160,"available":95971098624,"internally_reserved":0,"allocated":5447680,"data_stored":2679285,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":219335,"internal_metadata":659990329},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:56 2023","interfaces":[{"interface":"back","average":{"1min":0.30399999999999999,"5min":0.34699999999999998,"15min":0.36499999999999999},"min":{"1min":0.23999999999999999,"5min":0.14199999999999999,"15min":0.14199999999999999},"max":{"1min":0.42399999999999999,"5min":1.595,"15min":7.4249999999999998},"last":0.17100000000000001},{"interface":"front","average":{"1min":0.35499999999999998,"5min":0.40899999999999997,"15min":0.376},"min":{"1min":0.23999999999999999,"5min":0.19600000000000001,"15min":0.183},"max":{"1min":0.46999999999999997,"5min":2.6859999999999999,"15min":2.6859999999999999},"last":0.23000000000000001}]},{"osd":2,"last update":"Wed Nov 8 05:28:56 2023","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.44400000000000001,"15min":0.376},"min":{"1min":0.187,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":2.2650000000000001,"5min":5.8209999999999997,"15min":5.8209999999999997},"last":0.16300000000000001},{"interface":"front","average":{"1min":0.316,"5min":0.40500000000000003,"15min":0.35999999999999999},"min":{"1min":0.19600000000000001,"5min":0.188,"15min":0.17599999999999999},"max":{"1min":1.137,"5min":4.2350000000000003,"15min":4.2350000000000003},"last":0.17199999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-08T05:29:49.745 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-11-08T05:29:49.915 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~92s 2023-11-08T05:29:50.030 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T05:29:50.032 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2023-11-08T05:29:50.043 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":1565,"stamp":"2023-11-08T05:29:48.047805+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2634284,"kb_used_data":408928,"kb_used_omap":780,"kb_used_meta":2220403,"kb_avail":280481236,"statfs":{"total":289910292480,"available":287212785664,"internally_reserved":0,"allocated":418742272,"data_stored":410455190,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":799447,"internal_metadata":2273692969},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-536782902,"num_objects":-164,"num_object_clones":0,"num_object_copies":-328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-115,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-164,"num_whiteouts":0,"num_read":-122056,"num_read_kb":-39011942,"num_write":-413642,"num_write_kb":-151705339,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-8,"num_bytes_recovered":-16922624,"num_keys_recovered":-2,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.424817"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":456,"reported_epoch":455,"state":"active+undersized","last_fresh":"2023-11-08T05:29:43.389354+0000","last_change":"2023-11-08T05:28:13.557071+0000","last_active":"2023-11-08T05:29:43.389354+0000","last_peered":"2023-11-08T05:29:43.389354+0000","last_clean":"2023-11-08T05:28:01.458439+0000","last_became_active":"2023-11-08T05:28:13.557071+0000","last_became_peered":"2023-11-08T05:28:13.557071+0000","last_unstale":"2023-11-08T05:29:43.389354+0000","last_undegraded":"2023-11-08T05:29:43.389354+0000","last_fullsized":"2023-11-08T05:28:13.547025+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":18,"reported_epoch":455,"state":"active+undersized","last_fresh":"2023-11-08T05:29:43.389262+0000","last_change":"2023-11-08T05:28:13.559513+0000","last_active":"2023-11-08T05:29:43.389262+0000","last_peered":"2023-11-08T05:29:43.389262+0000","last_clean":"2023-11-08T04:52:46.531964+0000","last_became_active":"2023-11-08T05:28:13.559513+0000","last_became_peered":"2023-11-08T05:28:13.559513+0000","last_unstale":"2023-11-08T05:29:43.389262+0000","last_undegraded":"2023-11-08T05:29:43.389262+0000","last_fullsized":"2023-11-08T05:28:13.540852+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":453,"reported_epoch":455,"state":"active+clean","last_fresh":"2023-11-08T05:29:43.390991+0000","last_change":"2023-11-08T04:52:47.546100+0000","last_active":"2023-11-08T05:29:43.390991+0000","last_peered":"2023-11-08T05:29:43.390991+0000","last_clean":"2023-11-08T05:29:43.390991+0000","last_became_active":"2023-11-08T04:52:47.545840+0000","last_became_peered":"2023-11-08T04:52:47.545840+0000","last_unstale":"2023-11-08T05:29:43.390991+0000","last_undegraded":"2023-11-08T05:29:43.390991+0000","last_fullsized":"2023-11-08T05:29:43.390991+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":451,"reported_epoch":455,"state":"active+clean","last_fresh":"2023-11-08T05:29:43.389261+0000","last_change":"2023-11-08T04:52:47.548597+0000","last_active":"2023-11-08T05:29:43.389261+0000","last_peered":"2023-11-08T05:29:43.389261+0000","last_clean":"2023-11-08T05:29:43.389261+0000","last_became_active":"2023-11-08T04:52:47.548294+0000","last_became_peered":"2023-11-08T04:52:47.548294+0000","last_unstale":"2023-11-08T05:29:43.389261+0000","last_undegraded":"2023-11-08T05:29:43.389261+0000","last_fullsized":"2023-11-08T05:29:43.389261+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":457,"reported_epoch":455,"state":"active+undersized","last_fresh":"2023-11-08T05:29:43.389822+0000","last_change":"2023-11-08T05:28:13.545468+0000","last_active":"2023-11-08T05:29:43.389822+0000","last_peered":"2023-11-08T05:29:43.389822+0000","last_clean":"2023-11-08T05:28:01.456723+0000","last_became_active":"2023-11-08T05:28:13.545468+0000","last_became_peered":"2023-11-08T05:28:13.545468+0000","last_unstale":"2023-11-08T05:29:43.389822+0000","last_undegraded":"2023-11-08T05:29:43.389822+0000","last_fullsized":"2023-11-08T05:28:13.537207+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_clean_scrub_stamp":"2023-11-08T04:52:45.528796+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":462,"reported_epoch":456,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:29:44.395350+0000","last_change":"2023-11-08T05:29:44.395350+0000","last_active":"2023-11-08T05:29:44.395350+0000","last_peered":"2023-11-08T05:29:44.395350+0000","last_clean":"2023-11-08T05:29:44.395350+0000","last_became_active":"2023-11-08T05:29:44.395012+0000","last_became_peered":"2023-11-08T05:29:44.395012+0000","last_unstale":"2023-11-08T05:29:44.395350+0000","last_undegraded":"2023-11-08T05:29:44.395350+0000","last_fullsized":"2023-11-08T05:29:44.395350+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":456,"reported_epoch":455,"state":"active+undersized","last_fresh":"2023-11-08T05:29:43.389390+0000","last_change":"2023-11-08T05:28:13.563357+0000","last_active":"2023-11-08T05:29:43.389390+0000","last_peered":"2023-11-08T05:29:43.389390+0000","last_clean":"2023-11-08T05:28:01.457691+0000","last_became_active":"2023-11-08T05:28:13.563357+0000","last_became_peered":"2023-11-08T05:28:13.563357+0000","last_unstale":"2023-11-08T05:29:43.389390+0000","last_undegraded":"2023-11-08T05:29:43.389390+0000","last_fullsized":"2023-11-08T05:28:13.541614+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":456,"reported_epoch":455,"state":"active+undersized","last_fresh":"2023-11-08T05:29:43.389311+0000","last_change":"2023-11-08T05:28:13.550672+0000","last_active":"2023-11-08T05:29:43.389311+0000","last_peered":"2023-11-08T05:29:43.389311+0000","last_clean":"2023-11-08T05:28:01.457904+0000","last_became_active":"2023-11-08T05:28:13.550672+0000","last_became_peered":"2023-11-08T05:28:13.550672+0000","last_unstale":"2023-11-08T05:29:43.389311+0000","last_undegraded":"2023-11-08T05:29:43.389311+0000","last_fullsized":"2023-11-08T05:28:13.539595+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":456,"reported_epoch":455,"state":"active+undersized","last_fresh":"2023-11-08T05:29:43.389474+0000","last_change":"2023-11-08T05:28:13.557968+0000","last_active":"2023-11-08T05:29:43.389474+0000","last_peered":"2023-11-08T05:29:43.389474+0000","last_clean":"2023-11-08T05:28:01.458433+0000","last_became_active":"2023-11-08T05:28:13.557968+0000","last_became_peered":"2023-11-08T05:28:13.557968+0000","last_unstale":"2023-11-08T05:29:43.389474+0000","last_undegraded":"2023-11-08T05:29:43.389474+0000","last_fullsized":"2023-11-08T05:28:13.542832+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":1,"acting":1,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":10,"acting":11,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738815,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":644992,"kb_used_data":5304,"kb_used_omap":275,"kb_used_meta":639404,"kb_avail":93726848,"statfs":{"total":96636764160,"available":95976292352,"internally_reserved":0,"allocated":5431296,"data_stored":2672005,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":281650,"internal_metadata":654750670},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:59 2023","interfaces":[{"interface":"back","average":{"1min":0.30499999999999999,"5min":0.439,"15min":0.36099999999999999},"min":{"1min":0.17399999999999999,"5min":0.14499999999999999,"15min":0.14499999999999999},"max":{"1min":0.60099999999999998,"5min":4.9379999999999997,"15min":4.9379999999999997},"last":0.379},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.49399999999999999,"15min":0.42399999999999999},"min":{"1min":0.221,"5min":0.19600000000000001,"15min":0.17000000000000001},"max":{"1min":0.63200000000000001,"5min":4.9859999999999998,"15min":4.9859999999999998},"last":0.40899999999999997}]},{"osd":1,"last update":"Wed Nov 8 05:29:01 2023","interfaces":[{"interface":"back","average":{"1min":0.30299999999999999,"5min":0.81899999999999995,"15min":0.47399999999999998},"min":{"1min":0.16,"5min":0.153,"15min":0.153},"max":{"1min":0.68200000000000005,"5min":23.242999999999999,"15min":23.242999999999999},"last":0.27900000000000003},{"interface":"front","average":{"1min":0.46500000000000002,"5min":0.72199999999999998,"15min":0.46100000000000002},"min":{"1min":0.17699999999999999,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":2.0419999999999998,"5min":23.251000000000001,"15min":23.251000000000001},"last":0.29099999999999998}]}]},{"osd":0,"up_from":8,"seq":34359738798,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1339228,"kb_used_data":398304,"kb_used_omap":291,"kb_used_meta":936476,"kb_avail":93032612,"statfs":{"total":96636764160,"available":95265394688,"internally_reserved":0,"allocated":407863296,"data_stored":405103900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":298462,"internal_metadata":958951970},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.436,"15min":0.36299999999999999},"min":{"1min":0.19700000000000001,"5min":0.14699999999999999,"15min":0.13700000000000001},"max":{"1min":3.222,"5min":6.5499999999999998,"15min":6.9989999999999997},"last":0.245},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.46300000000000002,"15min":0.42299999999999999},"min":{"1min":0.17799999999999999,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":5.0030000000000001,"5min":6.0629999999999997,"15min":9.6869999999999994},"last":0.28599999999999998}]},{"osd":2,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.51600000000000001,"15min":0.47099999999999997},"min":{"1min":0.21099999999999999,"5min":0.153,"15min":0.153},"max":{"1min":5.7439999999999998,"5min":5.8280000000000003,"15min":9.7639999999999993},"last":0.252},{"interface":"front","average":{"1min":0.497,"5min":0.56200000000000006,"15min":0.42299999999999999},"min":{"1min":0.20399999999999999,"5min":0.187,"15min":0.16800000000000001},"max":{"1min":3.9319999999999999,"5min":8.8040000000000003,"15min":8.8040000000000003},"last":0.29899999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738817,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":650064,"kb_used_data":5320,"kb_used_omap":214,"kb_used_meta":644521,"kb_avail":93721776,"statfs":{"total":96636764160,"available":95971098624,"internally_reserved":0,"allocated":5447680,"data_stored":2679285,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":219335,"internal_metadata":659990329},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:56 2023","interfaces":[{"interface":"back","average":{"1min":0.30399999999999999,"5min":0.34699999999999998,"15min":0.36499999999999999},"min":{"1min":0.23999999999999999,"5min":0.14199999999999999,"15min":0.14199999999999999},"max":{"1min":0.42399999999999999,"5min":1.595,"15min":7.4249999999999998},"last":0.17100000000000001},{"interface":"front","average":{"1min":0.35499999999999998,"5min":0.40899999999999997,"15min":0.376},"min":{"1min":0.23999999999999999,"5min":0.19600000000000001,"15min":0.183},"max":{"1min":0.46999999999999997,"5min":2.6859999999999999,"15min":2.6859999999999999},"last":0.23000000000000001}]},{"osd":2,"last update":"Wed Nov 8 05:28:56 2023","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.44400000000000001,"15min":0.376},"min":{"1min":0.187,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":2.2650000000000001,"5min":5.8209999999999997,"15min":5.8209999999999997},"last":0.16300000000000001},{"interface":"front","average":{"1min":0.316,"5min":0.40500000000000003,"15min":0.35999999999999999},"min":{"1min":0.19600000000000001,"5min":0.188,"15min":0.17599999999999999},"max":{"1min":1.137,"5min":4.2350000000000003,"15min":4.2350000000000003},"last":0.17199999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-08T05:29:50.044 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-11-08T05:29:50.044 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 456, 'reported_epoch': 455, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:29:43.389354+0000', 'last_change': '2023-11-08T05:28:13.557071+0000', 'last_active': '2023-11-08T05:29:43.389354+0000', 'last_peered': '2023-11-08T05:29:43.389354+0000', 'last_clean': '2023-11-08T05:28:01.458439+0000', 'last_became_active': '2023-11-08T05:28:13.557071+0000', 'last_became_peered': '2023-11-08T05:28:13.557071+0000', 'last_unstale': '2023-11-08T05:29:43.389354+0000', 'last_undegraded': '2023-11-08T05:29:43.389354+0000', 'last_fullsized': '2023-11-08T05:28:13.547025+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:29:50.045 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-11-08T05:29:50.045 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 18, 'reported_epoch': 455, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:29:43.389262+0000', 'last_change': '2023-11-08T05:28:13.559513+0000', 'last_active': '2023-11-08T05:29:43.389262+0000', 'last_peered': '2023-11-08T05:29:43.389262+0000', 'last_clean': '2023-11-08T04:52:46.531964+0000', 'last_became_active': '2023-11-08T05:28:13.559513+0000', 'last_became_peered': '2023-11-08T05:28:13.559513+0000', 'last_unstale': '2023-11-08T05:29:43.389262+0000', 'last_undegraded': '2023-11-08T05:29:43.389262+0000', 'last_fullsized': '2023-11-08T05:28:13.540852+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:29:50.045 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-11-08T05:29:50.045 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "0'0", 'reported_seq': 457, 'reported_epoch': 455, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:29:43.389822+0000', 'last_change': '2023-11-08T05:28:13.545468+0000', 'last_active': '2023-11-08T05:29:43.389822+0000', 'last_peered': '2023-11-08T05:29:43.389822+0000', 'last_clean': '2023-11-08T05:28:01.456723+0000', 'last_became_active': '2023-11-08T05:28:13.545468+0000', 'last_became_peered': '2023-11-08T05:28:13.545468+0000', 'last_unstale': '2023-11-08T05:29:43.389822+0000', 'last_undegraded': '2023-11-08T05:29:43.389822+0000', 'last_fullsized': '2023-11-08T05:28:13.537207+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 9, 'last_epoch_clean': 10, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:45.528796+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:45.528796+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:45.528796+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:29:50.046 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.0 is not active+clean 2023-11-08T05:29:50.046 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.0', 'version': "0'0", 'reported_seq': 462, 'reported_epoch': 456, 'state': 'active+clean+remapped', 'last_fresh': '2023-11-08T05:29:44.395350+0000', 'last_change': '2023-11-08T05:29:44.395350+0000', 'last_active': '2023-11-08T05:29:44.395350+0000', 'last_peered': '2023-11-08T05:29:44.395350+0000', 'last_clean': '2023-11-08T05:29:44.395350+0000', 'last_became_active': '2023-11-08T05:29:44.395012+0000', 'last_became_peered': '2023-11-08T05:29:44.395012+0000', 'last_unstale': '2023-11-08T05:29:44.395350+0000', 'last_undegraded': '2023-11-08T05:29:44.395350+0000', 'last_fullsized': '2023-11-08T05:29:44.395350+0000', 'mapping_epoch': 455, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 456, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2], 'acting': [2, 1], 'avail_no_missing': ['2', '1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2023-11-08T05:29:50.046 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-11-08T05:29:50.047 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 456, 'reported_epoch': 455, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:29:43.389390+0000', 'last_change': '2023-11-08T05:28:13.563357+0000', 'last_active': '2023-11-08T05:29:43.389390+0000', 'last_peered': '2023-11-08T05:29:43.389390+0000', 'last_clean': '2023-11-08T05:28:01.457691+0000', 'last_became_active': '2023-11-08T05:28:13.563357+0000', 'last_became_peered': '2023-11-08T05:28:13.563357+0000', 'last_unstale': '2023-11-08T05:29:43.389390+0000', 'last_undegraded': '2023-11-08T05:29:43.389390+0000', 'last_fullsized': '2023-11-08T05:28:13.541614+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:29:50.047 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-11-08T05:29:50.047 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 456, 'reported_epoch': 455, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:29:43.389311+0000', 'last_change': '2023-11-08T05:28:13.550672+0000', 'last_active': '2023-11-08T05:29:43.389311+0000', 'last_peered': '2023-11-08T05:29:43.389311+0000', 'last_clean': '2023-11-08T05:28:01.457904+0000', 'last_became_active': '2023-11-08T05:28:13.550672+0000', 'last_became_peered': '2023-11-08T05:28:13.550672+0000', 'last_unstale': '2023-11-08T05:29:43.389311+0000', 'last_undegraded': '2023-11-08T05:29:43.389311+0000', 'last_fullsized': '2023-11-08T05:28:13.539595+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:29:50.048 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-11-08T05:29:50.048 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 456, 'reported_epoch': 455, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:29:43.389474+0000', 'last_change': '2023-11-08T05:28:13.557968+0000', 'last_active': '2023-11-08T05:29:43.389474+0000', 'last_peered': '2023-11-08T05:29:43.389474+0000', 'last_clean': '2023-11-08T05:28:01.458433+0000', 'last_became_active': '2023-11-08T05:28:13.557968+0000', 'last_became_peered': '2023-11-08T05:28:13.557968+0000', 'last_unstale': '2023-11-08T05:29:43.389474+0000', 'last_undegraded': '2023-11-08T05:29:43.389474+0000', 'last_fullsized': '2023-11-08T05:28:13.542832+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:29:50.048 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-11-08T05:29:50.400 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T05:29:50.401 INFO:teuthology.orchestra.run.smithi087.stdout:{"fsid":"ab4cbc06-449d-497d-b513-220d9db2b590","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 6 pgs undersized","count":6},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2233,"monmap":{"epoch":1,"min_mon_release_name":"pacific","num_mons":1},"osdmap":{"epoch":456,"num_osds":3,"num_up_osds":2,"osd_up_since":1699421292,"num_in_osds":3,"osd_in_since":1699419158,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":6},{"state_name":"active+clean","count":2},{"state_name":"active+clean+remapped","count":1}],"num_pgs":9,"num_pools":2,"num_objects":0,"data_bytes":0,"bytes_used":2697519104,"bytes_avail":287212773376,"bytes_total":289910292480},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":41,"modified":"2023-11-08T05:29:32.044544+0000","services":{}},"progress_events":{}} 2023-11-08T05:29:50.414 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-11-08T05:29:50.414 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-11-08T05:29:50.697 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T05:29:50.699 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2023-11-08T05:29:50.710 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":1566,"stamp":"2023-11-08T05:29:50.048286+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2634296,"kb_used_data":408940,"kb_used_omap":780,"kb_used_meta":2220403,"kb_avail":280481224,"statfs":{"total":289910292480,"available":287212773376,"internally_reserved":0,"allocated":418754560,"data_stored":410461501,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":799447,"internal_metadata":2273692969},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-536782902,"num_objects":-164,"num_object_clones":0,"num_object_copies":-328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-115,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-164,"num_whiteouts":0,"num_read":-122056,"num_read_kb":-39011942,"num_write":-413642,"num_write_kb":-151705339,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-8,"num_bytes_recovered":-16922624,"num_keys_recovered":-2,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.424817"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":456,"reported_epoch":455,"state":"active+undersized","last_fresh":"2023-11-08T05:29:43.389354+0000","last_change":"2023-11-08T05:28:13.557071+0000","last_active":"2023-11-08T05:29:43.389354+0000","last_peered":"2023-11-08T05:29:43.389354+0000","last_clean":"2023-11-08T05:28:01.458439+0000","last_became_active":"2023-11-08T05:28:13.557071+0000","last_became_peered":"2023-11-08T05:28:13.557071+0000","last_unstale":"2023-11-08T05:29:43.389354+0000","last_undegraded":"2023-11-08T05:29:43.389354+0000","last_fullsized":"2023-11-08T05:28:13.547025+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":18,"reported_epoch":455,"state":"active+undersized","last_fresh":"2023-11-08T05:29:43.389262+0000","last_change":"2023-11-08T05:28:13.559513+0000","last_active":"2023-11-08T05:29:43.389262+0000","last_peered":"2023-11-08T05:29:43.389262+0000","last_clean":"2023-11-08T04:52:46.531964+0000","last_became_active":"2023-11-08T05:28:13.559513+0000","last_became_peered":"2023-11-08T05:28:13.559513+0000","last_unstale":"2023-11-08T05:29:43.389262+0000","last_undegraded":"2023-11-08T05:29:43.389262+0000","last_fullsized":"2023-11-08T05:28:13.540852+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":453,"reported_epoch":455,"state":"active+clean","last_fresh":"2023-11-08T05:29:43.390991+0000","last_change":"2023-11-08T04:52:47.546100+0000","last_active":"2023-11-08T05:29:43.390991+0000","last_peered":"2023-11-08T05:29:43.390991+0000","last_clean":"2023-11-08T05:29:43.390991+0000","last_became_active":"2023-11-08T04:52:47.545840+0000","last_became_peered":"2023-11-08T04:52:47.545840+0000","last_unstale":"2023-11-08T05:29:43.390991+0000","last_undegraded":"2023-11-08T05:29:43.390991+0000","last_fullsized":"2023-11-08T05:29:43.390991+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":451,"reported_epoch":455,"state":"active+clean","last_fresh":"2023-11-08T05:29:43.389261+0000","last_change":"2023-11-08T04:52:47.548597+0000","last_active":"2023-11-08T05:29:43.389261+0000","last_peered":"2023-11-08T05:29:43.389261+0000","last_clean":"2023-11-08T05:29:43.389261+0000","last_became_active":"2023-11-08T04:52:47.548294+0000","last_became_peered":"2023-11-08T04:52:47.548294+0000","last_unstale":"2023-11-08T05:29:43.389261+0000","last_undegraded":"2023-11-08T05:29:43.389261+0000","last_fullsized":"2023-11-08T05:29:43.389261+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":457,"reported_epoch":455,"state":"active+undersized","last_fresh":"2023-11-08T05:29:43.389822+0000","last_change":"2023-11-08T05:28:13.545468+0000","last_active":"2023-11-08T05:29:43.389822+0000","last_peered":"2023-11-08T05:29:43.389822+0000","last_clean":"2023-11-08T05:28:01.456723+0000","last_became_active":"2023-11-08T05:28:13.545468+0000","last_became_peered":"2023-11-08T05:28:13.545468+0000","last_unstale":"2023-11-08T05:29:43.389822+0000","last_undegraded":"2023-11-08T05:29:43.389822+0000","last_fullsized":"2023-11-08T05:28:13.537207+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_clean_scrub_stamp":"2023-11-08T04:52:45.528796+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":462,"reported_epoch":456,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:29:44.395350+0000","last_change":"2023-11-08T05:29:44.395350+0000","last_active":"2023-11-08T05:29:44.395350+0000","last_peered":"2023-11-08T05:29:44.395350+0000","last_clean":"2023-11-08T05:29:44.395350+0000","last_became_active":"2023-11-08T05:29:44.395012+0000","last_became_peered":"2023-11-08T05:29:44.395012+0000","last_unstale":"2023-11-08T05:29:44.395350+0000","last_undegraded":"2023-11-08T05:29:44.395350+0000","last_fullsized":"2023-11-08T05:29:44.395350+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":456,"reported_epoch":455,"state":"active+undersized","last_fresh":"2023-11-08T05:29:43.389390+0000","last_change":"2023-11-08T05:28:13.563357+0000","last_active":"2023-11-08T05:29:43.389390+0000","last_peered":"2023-11-08T05:29:43.389390+0000","last_clean":"2023-11-08T05:28:01.457691+0000","last_became_active":"2023-11-08T05:28:13.563357+0000","last_became_peered":"2023-11-08T05:28:13.563357+0000","last_unstale":"2023-11-08T05:29:43.389390+0000","last_undegraded":"2023-11-08T05:29:43.389390+0000","last_fullsized":"2023-11-08T05:28:13.541614+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":456,"reported_epoch":455,"state":"active+undersized","last_fresh":"2023-11-08T05:29:43.389311+0000","last_change":"2023-11-08T05:28:13.550672+0000","last_active":"2023-11-08T05:29:43.389311+0000","last_peered":"2023-11-08T05:29:43.389311+0000","last_clean":"2023-11-08T05:28:01.457904+0000","last_became_active":"2023-11-08T05:28:13.550672+0000","last_became_peered":"2023-11-08T05:28:13.550672+0000","last_unstale":"2023-11-08T05:29:43.389311+0000","last_undegraded":"2023-11-08T05:29:43.389311+0000","last_fullsized":"2023-11-08T05:28:13.539595+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":456,"reported_epoch":455,"state":"active+undersized","last_fresh":"2023-11-08T05:29:43.389474+0000","last_change":"2023-11-08T05:28:13.557968+0000","last_active":"2023-11-08T05:29:43.389474+0000","last_peered":"2023-11-08T05:29:43.389474+0000","last_clean":"2023-11-08T05:28:01.458433+0000","last_became_active":"2023-11-08T05:28:13.557968+0000","last_became_peered":"2023-11-08T05:28:13.557968+0000","last_unstale":"2023-11-08T05:29:43.389474+0000","last_undegraded":"2023-11-08T05:29:43.389474+0000","last_fullsized":"2023-11-08T05:28:13.542832+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":1,"acting":1,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":10,"acting":11,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738816,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":645004,"kb_used_data":5316,"kb_used_omap":275,"kb_used_meta":639404,"kb_avail":93726836,"statfs":{"total":96636764160,"available":95976280064,"internally_reserved":0,"allocated":5443584,"data_stored":2678316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":281650,"internal_metadata":654750670},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:59 2023","interfaces":[{"interface":"back","average":{"1min":0.30499999999999999,"5min":0.439,"15min":0.36099999999999999},"min":{"1min":0.17399999999999999,"5min":0.14499999999999999,"15min":0.14499999999999999},"max":{"1min":0.60099999999999998,"5min":4.9379999999999997,"15min":4.9379999999999997},"last":0.379},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.49399999999999999,"15min":0.42399999999999999},"min":{"1min":0.221,"5min":0.19600000000000001,"15min":0.17000000000000001},"max":{"1min":0.63200000000000001,"5min":4.9859999999999998,"15min":4.9859999999999998},"last":0.40899999999999997}]},{"osd":1,"last update":"Wed Nov 8 05:29:01 2023","interfaces":[{"interface":"back","average":{"1min":0.30299999999999999,"5min":0.81899999999999995,"15min":0.47399999999999998},"min":{"1min":0.16,"5min":0.153,"15min":0.153},"max":{"1min":0.68200000000000005,"5min":23.242999999999999,"15min":23.242999999999999},"last":0.309},{"interface":"front","average":{"1min":0.46500000000000002,"5min":0.72199999999999998,"15min":0.46100000000000002},"min":{"1min":0.17699999999999999,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":2.0419999999999998,"5min":23.251000000000001,"15min":23.251000000000001},"last":0.32300000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738798,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1339228,"kb_used_data":398304,"kb_used_omap":291,"kb_used_meta":936476,"kb_avail":93032612,"statfs":{"total":96636764160,"available":95265394688,"internally_reserved":0,"allocated":407863296,"data_stored":405103900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":298462,"internal_metadata":958951970},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.436,"15min":0.36299999999999999},"min":{"1min":0.19700000000000001,"5min":0.14699999999999999,"15min":0.13700000000000001},"max":{"1min":3.222,"5min":6.5499999999999998,"15min":6.9989999999999997},"last":0.245},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.46300000000000002,"15min":0.42299999999999999},"min":{"1min":0.17799999999999999,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":5.0030000000000001,"5min":6.0629999999999997,"15min":9.6869999999999994},"last":0.28599999999999998}]},{"osd":2,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.51600000000000001,"15min":0.47099999999999997},"min":{"1min":0.21099999999999999,"5min":0.153,"15min":0.153},"max":{"1min":5.7439999999999998,"5min":5.8280000000000003,"15min":9.7639999999999993},"last":0.252},{"interface":"front","average":{"1min":0.497,"5min":0.56200000000000006,"15min":0.42299999999999999},"min":{"1min":0.20399999999999999,"5min":0.187,"15min":0.16800000000000001},"max":{"1min":3.9319999999999999,"5min":8.8040000000000003,"15min":8.8040000000000003},"last":0.29899999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738817,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":650064,"kb_used_data":5320,"kb_used_omap":214,"kb_used_meta":644521,"kb_avail":93721776,"statfs":{"total":96636764160,"available":95971098624,"internally_reserved":0,"allocated":5447680,"data_stored":2679285,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":219335,"internal_metadata":659990329},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:56 2023","interfaces":[{"interface":"back","average":{"1min":0.30399999999999999,"5min":0.34699999999999998,"15min":0.36499999999999999},"min":{"1min":0.23999999999999999,"5min":0.14199999999999999,"15min":0.14199999999999999},"max":{"1min":0.42399999999999999,"5min":1.595,"15min":7.4249999999999998},"last":0.17100000000000001},{"interface":"front","average":{"1min":0.35499999999999998,"5min":0.40899999999999997,"15min":0.376},"min":{"1min":0.23999999999999999,"5min":0.19600000000000001,"15min":0.183},"max":{"1min":0.46999999999999997,"5min":2.6859999999999999,"15min":2.6859999999999999},"last":0.23000000000000001}]},{"osd":2,"last update":"Wed Nov 8 05:28:56 2023","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.44400000000000001,"15min":0.376},"min":{"1min":0.187,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":2.2650000000000001,"5min":5.8209999999999997,"15min":5.8209999999999997},"last":0.16300000000000001},{"interface":"front","average":{"1min":0.316,"5min":0.40500000000000003,"15min":0.35999999999999999},"min":{"1min":0.19600000000000001,"5min":0.188,"15min":0.17599999999999999},"max":{"1min":1.137,"5min":4.2350000000000003,"15min":4.2350000000000003},"last":0.17199999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-08T05:29:53.712 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-11-08T05:29:54.007 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T05:29:54.009 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2023-11-08T05:29:54.020 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":1567,"stamp":"2023-11-08T05:29:52.048824+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2634296,"kb_used_data":408940,"kb_used_omap":780,"kb_used_meta":2220403,"kb_avail":280481224,"statfs":{"total":289910292480,"available":287212773376,"internally_reserved":0,"allocated":418754560,"data_stored":410461501,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":799447,"internal_metadata":2273692969},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-536782902,"num_objects":-164,"num_object_clones":0,"num_object_copies":-328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-115,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-164,"num_whiteouts":0,"num_read":-122056,"num_read_kb":-39011942,"num_write":-413642,"num_write_kb":-151705339,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-8,"num_bytes_recovered":-16922624,"num_keys_recovered":-2,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.424817"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":456,"reported_epoch":455,"state":"active+undersized","last_fresh":"2023-11-08T05:29:43.389354+0000","last_change":"2023-11-08T05:28:13.557071+0000","last_active":"2023-11-08T05:29:43.389354+0000","last_peered":"2023-11-08T05:29:43.389354+0000","last_clean":"2023-11-08T05:28:01.458439+0000","last_became_active":"2023-11-08T05:28:13.557071+0000","last_became_peered":"2023-11-08T05:28:13.557071+0000","last_unstale":"2023-11-08T05:29:43.389354+0000","last_undegraded":"2023-11-08T05:29:43.389354+0000","last_fullsized":"2023-11-08T05:28:13.547025+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":18,"reported_epoch":455,"state":"active+undersized","last_fresh":"2023-11-08T05:29:43.389262+0000","last_change":"2023-11-08T05:28:13.559513+0000","last_active":"2023-11-08T05:29:43.389262+0000","last_peered":"2023-11-08T05:29:43.389262+0000","last_clean":"2023-11-08T04:52:46.531964+0000","last_became_active":"2023-11-08T05:28:13.559513+0000","last_became_peered":"2023-11-08T05:28:13.559513+0000","last_unstale":"2023-11-08T05:29:43.389262+0000","last_undegraded":"2023-11-08T05:29:43.389262+0000","last_fullsized":"2023-11-08T05:28:13.540852+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":453,"reported_epoch":455,"state":"active+clean","last_fresh":"2023-11-08T05:29:43.390991+0000","last_change":"2023-11-08T04:52:47.546100+0000","last_active":"2023-11-08T05:29:43.390991+0000","last_peered":"2023-11-08T05:29:43.390991+0000","last_clean":"2023-11-08T05:29:43.390991+0000","last_became_active":"2023-11-08T04:52:47.545840+0000","last_became_peered":"2023-11-08T04:52:47.545840+0000","last_unstale":"2023-11-08T05:29:43.390991+0000","last_undegraded":"2023-11-08T05:29:43.390991+0000","last_fullsized":"2023-11-08T05:29:43.390991+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":451,"reported_epoch":455,"state":"active+clean","last_fresh":"2023-11-08T05:29:43.389261+0000","last_change":"2023-11-08T04:52:47.548597+0000","last_active":"2023-11-08T05:29:43.389261+0000","last_peered":"2023-11-08T05:29:43.389261+0000","last_clean":"2023-11-08T05:29:43.389261+0000","last_became_active":"2023-11-08T04:52:47.548294+0000","last_became_peered":"2023-11-08T04:52:47.548294+0000","last_unstale":"2023-11-08T05:29:43.389261+0000","last_undegraded":"2023-11-08T05:29:43.389261+0000","last_fullsized":"2023-11-08T05:29:43.389261+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":457,"reported_epoch":455,"state":"active+undersized","last_fresh":"2023-11-08T05:29:43.389822+0000","last_change":"2023-11-08T05:28:13.545468+0000","last_active":"2023-11-08T05:29:43.389822+0000","last_peered":"2023-11-08T05:29:43.389822+0000","last_clean":"2023-11-08T05:28:01.456723+0000","last_became_active":"2023-11-08T05:28:13.545468+0000","last_became_peered":"2023-11-08T05:28:13.545468+0000","last_unstale":"2023-11-08T05:29:43.389822+0000","last_undegraded":"2023-11-08T05:29:43.389822+0000","last_fullsized":"2023-11-08T05:28:13.537207+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_clean_scrub_stamp":"2023-11-08T04:52:45.528796+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":462,"reported_epoch":456,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:29:44.395350+0000","last_change":"2023-11-08T05:29:44.395350+0000","last_active":"2023-11-08T05:29:44.395350+0000","last_peered":"2023-11-08T05:29:44.395350+0000","last_clean":"2023-11-08T05:29:44.395350+0000","last_became_active":"2023-11-08T05:29:44.395012+0000","last_became_peered":"2023-11-08T05:29:44.395012+0000","last_unstale":"2023-11-08T05:29:44.395350+0000","last_undegraded":"2023-11-08T05:29:44.395350+0000","last_fullsized":"2023-11-08T05:29:44.395350+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":456,"reported_epoch":455,"state":"active+undersized","last_fresh":"2023-11-08T05:29:43.389390+0000","last_change":"2023-11-08T05:28:13.563357+0000","last_active":"2023-11-08T05:29:43.389390+0000","last_peered":"2023-11-08T05:29:43.389390+0000","last_clean":"2023-11-08T05:28:01.457691+0000","last_became_active":"2023-11-08T05:28:13.563357+0000","last_became_peered":"2023-11-08T05:28:13.563357+0000","last_unstale":"2023-11-08T05:29:43.389390+0000","last_undegraded":"2023-11-08T05:29:43.389390+0000","last_fullsized":"2023-11-08T05:28:13.541614+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":456,"reported_epoch":455,"state":"active+undersized","last_fresh":"2023-11-08T05:29:43.389311+0000","last_change":"2023-11-08T05:28:13.550672+0000","last_active":"2023-11-08T05:29:43.389311+0000","last_peered":"2023-11-08T05:29:43.389311+0000","last_clean":"2023-11-08T05:28:01.457904+0000","last_became_active":"2023-11-08T05:28:13.550672+0000","last_became_peered":"2023-11-08T05:28:13.550672+0000","last_unstale":"2023-11-08T05:29:43.389311+0000","last_undegraded":"2023-11-08T05:29:43.389311+0000","last_fullsized":"2023-11-08T05:28:13.539595+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":456,"reported_epoch":455,"state":"active+undersized","last_fresh":"2023-11-08T05:29:43.389474+0000","last_change":"2023-11-08T05:28:13.557968+0000","last_active":"2023-11-08T05:29:43.389474+0000","last_peered":"2023-11-08T05:29:43.389474+0000","last_clean":"2023-11-08T05:28:01.458433+0000","last_became_active":"2023-11-08T05:28:13.557968+0000","last_became_peered":"2023-11-08T05:28:13.557968+0000","last_unstale":"2023-11-08T05:29:43.389474+0000","last_undegraded":"2023-11-08T05:29:43.389474+0000","last_fullsized":"2023-11-08T05:28:13.542832+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":1,"acting":1,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":10,"acting":11,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738816,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":645004,"kb_used_data":5316,"kb_used_omap":275,"kb_used_meta":639404,"kb_avail":93726836,"statfs":{"total":96636764160,"available":95976280064,"internally_reserved":0,"allocated":5443584,"data_stored":2678316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":281650,"internal_metadata":654750670},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:59 2023","interfaces":[{"interface":"back","average":{"1min":0.30499999999999999,"5min":0.439,"15min":0.36099999999999999},"min":{"1min":0.17399999999999999,"5min":0.14499999999999999,"15min":0.14499999999999999},"max":{"1min":0.60099999999999998,"5min":4.9379999999999997,"15min":4.9379999999999997},"last":0.379},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.49399999999999999,"15min":0.42399999999999999},"min":{"1min":0.221,"5min":0.19600000000000001,"15min":0.17000000000000001},"max":{"1min":0.63200000000000001,"5min":4.9859999999999998,"15min":4.9859999999999998},"last":0.40899999999999997}]},{"osd":1,"last update":"Wed Nov 8 05:29:01 2023","interfaces":[{"interface":"back","average":{"1min":0.30299999999999999,"5min":0.81899999999999995,"15min":0.47399999999999998},"min":{"1min":0.16,"5min":0.153,"15min":0.153},"max":{"1min":0.68200000000000005,"5min":23.242999999999999,"15min":23.242999999999999},"last":0.309},{"interface":"front","average":{"1min":0.46500000000000002,"5min":0.72199999999999998,"15min":0.46100000000000002},"min":{"1min":0.17699999999999999,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":2.0419999999999998,"5min":23.251000000000001,"15min":23.251000000000001},"last":0.32300000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738798,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1339228,"kb_used_data":398304,"kb_used_omap":291,"kb_used_meta":936476,"kb_avail":93032612,"statfs":{"total":96636764160,"available":95265394688,"internally_reserved":0,"allocated":407863296,"data_stored":405103900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":298462,"internal_metadata":958951970},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.436,"15min":0.36299999999999999},"min":{"1min":0.19700000000000001,"5min":0.14699999999999999,"15min":0.13700000000000001},"max":{"1min":3.222,"5min":6.5499999999999998,"15min":6.9989999999999997},"last":0.245},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.46300000000000002,"15min":0.42299999999999999},"min":{"1min":0.17799999999999999,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":5.0030000000000001,"5min":6.0629999999999997,"15min":9.6869999999999994},"last":0.28599999999999998}]},{"osd":2,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.51600000000000001,"15min":0.47099999999999997},"min":{"1min":0.21099999999999999,"5min":0.153,"15min":0.153},"max":{"1min":5.7439999999999998,"5min":5.8280000000000003,"15min":9.7639999999999993},"last":0.252},{"interface":"front","average":{"1min":0.497,"5min":0.56200000000000006,"15min":0.42299999999999999},"min":{"1min":0.20399999999999999,"5min":0.187,"15min":0.16800000000000001},"max":{"1min":3.9319999999999999,"5min":8.8040000000000003,"15min":8.8040000000000003},"last":0.29899999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738818,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":650064,"kb_used_data":5320,"kb_used_omap":214,"kb_used_meta":644521,"kb_avail":93721776,"statfs":{"total":96636764160,"available":95971098624,"internally_reserved":0,"allocated":5447680,"data_stored":2679285,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":219335,"internal_metadata":659990329},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:56 2023","interfaces":[{"interface":"back","average":{"1min":0.30399999999999999,"5min":0.34699999999999998,"15min":0.36499999999999999},"min":{"1min":0.23999999999999999,"5min":0.14199999999999999,"15min":0.14199999999999999},"max":{"1min":0.42399999999999999,"5min":1.595,"15min":7.4249999999999998},"last":0.17100000000000001},{"interface":"front","average":{"1min":0.35499999999999998,"5min":0.40899999999999997,"15min":0.376},"min":{"1min":0.23999999999999999,"5min":0.19600000000000001,"15min":0.183},"max":{"1min":0.46999999999999997,"5min":2.6859999999999999,"15min":2.6859999999999999},"last":0.23000000000000001}]},{"osd":2,"last update":"Wed Nov 8 05:28:56 2023","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.44400000000000001,"15min":0.376},"min":{"1min":0.187,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":2.2650000000000001,"5min":5.8209999999999997,"15min":5.8209999999999997},"last":0.316},{"interface":"front","average":{"1min":0.316,"5min":0.40500000000000003,"15min":0.35999999999999999},"min":{"1min":0.19600000000000001,"5min":0.188,"15min":0.17599999999999999},"max":{"1min":1.137,"5min":4.2350000000000003,"15min":4.2350000000000003},"last":0.33900000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-08T05:29:54.021 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-11-08T05:29:54.305 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T05:29:54.307 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2023-11-08T05:29:54.318 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":1568,"stamp":"2023-11-08T05:29:54.049324+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2634296,"kb_used_data":408940,"kb_used_omap":780,"kb_used_meta":2220403,"kb_avail":280481224,"statfs":{"total":289910292480,"available":287212773376,"internally_reserved":0,"allocated":418754560,"data_stored":410461501,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":799447,"internal_metadata":2273692969},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-536782902,"num_objects":-164,"num_object_clones":0,"num_object_copies":-328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-115,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-164,"num_whiteouts":0,"num_read":-122056,"num_read_kb":-39011942,"num_write":-413642,"num_write_kb":-151705339,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-8,"num_bytes_recovered":-16922624,"num_keys_recovered":-2,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.424817"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":456,"reported_epoch":455,"state":"active+undersized","last_fresh":"2023-11-08T05:29:43.389354+0000","last_change":"2023-11-08T05:28:13.557071+0000","last_active":"2023-11-08T05:29:43.389354+0000","last_peered":"2023-11-08T05:29:43.389354+0000","last_clean":"2023-11-08T05:28:01.458439+0000","last_became_active":"2023-11-08T05:28:13.557071+0000","last_became_peered":"2023-11-08T05:28:13.557071+0000","last_unstale":"2023-11-08T05:29:43.389354+0000","last_undegraded":"2023-11-08T05:29:43.389354+0000","last_fullsized":"2023-11-08T05:28:13.547025+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":18,"reported_epoch":455,"state":"active+undersized","last_fresh":"2023-11-08T05:29:43.389262+0000","last_change":"2023-11-08T05:28:13.559513+0000","last_active":"2023-11-08T05:29:43.389262+0000","last_peered":"2023-11-08T05:29:43.389262+0000","last_clean":"2023-11-08T04:52:46.531964+0000","last_became_active":"2023-11-08T05:28:13.559513+0000","last_became_peered":"2023-11-08T05:28:13.559513+0000","last_unstale":"2023-11-08T05:29:43.389262+0000","last_undegraded":"2023-11-08T05:29:43.389262+0000","last_fullsized":"2023-11-08T05:28:13.540852+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":453,"reported_epoch":455,"state":"active+clean","last_fresh":"2023-11-08T05:29:43.390991+0000","last_change":"2023-11-08T04:52:47.546100+0000","last_active":"2023-11-08T05:29:43.390991+0000","last_peered":"2023-11-08T05:29:43.390991+0000","last_clean":"2023-11-08T05:29:43.390991+0000","last_became_active":"2023-11-08T04:52:47.545840+0000","last_became_peered":"2023-11-08T04:52:47.545840+0000","last_unstale":"2023-11-08T05:29:43.390991+0000","last_undegraded":"2023-11-08T05:29:43.390991+0000","last_fullsized":"2023-11-08T05:29:43.390991+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":451,"reported_epoch":455,"state":"active+clean","last_fresh":"2023-11-08T05:29:43.389261+0000","last_change":"2023-11-08T04:52:47.548597+0000","last_active":"2023-11-08T05:29:43.389261+0000","last_peered":"2023-11-08T05:29:43.389261+0000","last_clean":"2023-11-08T05:29:43.389261+0000","last_became_active":"2023-11-08T04:52:47.548294+0000","last_became_peered":"2023-11-08T04:52:47.548294+0000","last_unstale":"2023-11-08T05:29:43.389261+0000","last_undegraded":"2023-11-08T05:29:43.389261+0000","last_fullsized":"2023-11-08T05:29:43.389261+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":457,"reported_epoch":455,"state":"active+undersized","last_fresh":"2023-11-08T05:29:43.389822+0000","last_change":"2023-11-08T05:28:13.545468+0000","last_active":"2023-11-08T05:29:43.389822+0000","last_peered":"2023-11-08T05:29:43.389822+0000","last_clean":"2023-11-08T05:28:01.456723+0000","last_became_active":"2023-11-08T05:28:13.545468+0000","last_became_peered":"2023-11-08T05:28:13.545468+0000","last_unstale":"2023-11-08T05:29:43.389822+0000","last_undegraded":"2023-11-08T05:29:43.389822+0000","last_fullsized":"2023-11-08T05:28:13.537207+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_clean_scrub_stamp":"2023-11-08T04:52:45.528796+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":462,"reported_epoch":456,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:29:44.395350+0000","last_change":"2023-11-08T05:29:44.395350+0000","last_active":"2023-11-08T05:29:44.395350+0000","last_peered":"2023-11-08T05:29:44.395350+0000","last_clean":"2023-11-08T05:29:44.395350+0000","last_became_active":"2023-11-08T05:29:44.395012+0000","last_became_peered":"2023-11-08T05:29:44.395012+0000","last_unstale":"2023-11-08T05:29:44.395350+0000","last_undegraded":"2023-11-08T05:29:44.395350+0000","last_fullsized":"2023-11-08T05:29:44.395350+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":456,"reported_epoch":455,"state":"active+undersized","last_fresh":"2023-11-08T05:29:43.389390+0000","last_change":"2023-11-08T05:28:13.563357+0000","last_active":"2023-11-08T05:29:43.389390+0000","last_peered":"2023-11-08T05:29:43.389390+0000","last_clean":"2023-11-08T05:28:01.457691+0000","last_became_active":"2023-11-08T05:28:13.563357+0000","last_became_peered":"2023-11-08T05:28:13.563357+0000","last_unstale":"2023-11-08T05:29:43.389390+0000","last_undegraded":"2023-11-08T05:29:43.389390+0000","last_fullsized":"2023-11-08T05:28:13.541614+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":456,"reported_epoch":455,"state":"active+undersized","last_fresh":"2023-11-08T05:29:43.389311+0000","last_change":"2023-11-08T05:28:13.550672+0000","last_active":"2023-11-08T05:29:43.389311+0000","last_peered":"2023-11-08T05:29:43.389311+0000","last_clean":"2023-11-08T05:28:01.457904+0000","last_became_active":"2023-11-08T05:28:13.550672+0000","last_became_peered":"2023-11-08T05:28:13.550672+0000","last_unstale":"2023-11-08T05:29:43.389311+0000","last_undegraded":"2023-11-08T05:29:43.389311+0000","last_fullsized":"2023-11-08T05:28:13.539595+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":456,"reported_epoch":455,"state":"active+undersized","last_fresh":"2023-11-08T05:29:43.389474+0000","last_change":"2023-11-08T05:28:13.557968+0000","last_active":"2023-11-08T05:29:43.389474+0000","last_peered":"2023-11-08T05:29:43.389474+0000","last_clean":"2023-11-08T05:28:01.458433+0000","last_became_active":"2023-11-08T05:28:13.557968+0000","last_became_peered":"2023-11-08T05:28:13.557968+0000","last_unstale":"2023-11-08T05:29:43.389474+0000","last_undegraded":"2023-11-08T05:29:43.389474+0000","last_fullsized":"2023-11-08T05:28:13.542832+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":1,"acting":1,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":10,"acting":11,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738816,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":645004,"kb_used_data":5316,"kb_used_omap":275,"kb_used_meta":639404,"kb_avail":93726836,"statfs":{"total":96636764160,"available":95976280064,"internally_reserved":0,"allocated":5443584,"data_stored":2678316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":281650,"internal_metadata":654750670},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:59 2023","interfaces":[{"interface":"back","average":{"1min":0.30499999999999999,"5min":0.439,"15min":0.36099999999999999},"min":{"1min":0.17399999999999999,"5min":0.14499999999999999,"15min":0.14499999999999999},"max":{"1min":0.60099999999999998,"5min":4.9379999999999997,"15min":4.9379999999999997},"last":0.379},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.49399999999999999,"15min":0.42399999999999999},"min":{"1min":0.221,"5min":0.19600000000000001,"15min":0.17000000000000001},"max":{"1min":0.63200000000000001,"5min":4.9859999999999998,"15min":4.9859999999999998},"last":0.40899999999999997}]},{"osd":1,"last update":"Wed Nov 8 05:29:01 2023","interfaces":[{"interface":"back","average":{"1min":0.30299999999999999,"5min":0.81899999999999995,"15min":0.47399999999999998},"min":{"1min":0.16,"5min":0.153,"15min":0.153},"max":{"1min":0.68200000000000005,"5min":23.242999999999999,"15min":23.242999999999999},"last":0.309},{"interface":"front","average":{"1min":0.46500000000000002,"5min":0.72199999999999998,"15min":0.46100000000000002},"min":{"1min":0.17699999999999999,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":2.0419999999999998,"5min":23.251000000000001,"15min":23.251000000000001},"last":0.32300000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738798,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1339228,"kb_used_data":398304,"kb_used_omap":291,"kb_used_meta":936476,"kb_avail":93032612,"statfs":{"total":96636764160,"available":95265394688,"internally_reserved":0,"allocated":407863296,"data_stored":405103900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":298462,"internal_metadata":958951970},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.436,"15min":0.36299999999999999},"min":{"1min":0.19700000000000001,"5min":0.14699999999999999,"15min":0.13700000000000001},"max":{"1min":3.222,"5min":6.5499999999999998,"15min":6.9989999999999997},"last":0.245},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.46300000000000002,"15min":0.42299999999999999},"min":{"1min":0.17799999999999999,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":5.0030000000000001,"5min":6.0629999999999997,"15min":9.6869999999999994},"last":0.28599999999999998}]},{"osd":2,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.51600000000000001,"15min":0.47099999999999997},"min":{"1min":0.21099999999999999,"5min":0.153,"15min":0.153},"max":{"1min":5.7439999999999998,"5min":5.8280000000000003,"15min":9.7639999999999993},"last":0.252},{"interface":"front","average":{"1min":0.497,"5min":0.56200000000000006,"15min":0.42299999999999999},"min":{"1min":0.20399999999999999,"5min":0.187,"15min":0.16800000000000001},"max":{"1min":3.9319999999999999,"5min":8.8040000000000003,"15min":8.8040000000000003},"last":0.29899999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738818,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":650064,"kb_used_data":5320,"kb_used_omap":214,"kb_used_meta":644521,"kb_avail":93721776,"statfs":{"total":96636764160,"available":95971098624,"internally_reserved":0,"allocated":5447680,"data_stored":2679285,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":219335,"internal_metadata":659990329},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:56 2023","interfaces":[{"interface":"back","average":{"1min":0.30399999999999999,"5min":0.34699999999999998,"15min":0.36499999999999999},"min":{"1min":0.23999999999999999,"5min":0.14199999999999999,"15min":0.14199999999999999},"max":{"1min":0.42399999999999999,"5min":1.595,"15min":7.4249999999999998},"last":0.17100000000000001},{"interface":"front","average":{"1min":0.35499999999999998,"5min":0.40899999999999997,"15min":0.376},"min":{"1min":0.23999999999999999,"5min":0.19600000000000001,"15min":0.183},"max":{"1min":0.46999999999999997,"5min":2.6859999999999999,"15min":2.6859999999999999},"last":0.23000000000000001}]},{"osd":2,"last update":"Wed Nov 8 05:28:56 2023","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.44400000000000001,"15min":0.376},"min":{"1min":0.187,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":2.2650000000000001,"5min":5.8209999999999997,"15min":5.8209999999999997},"last":0.316},{"interface":"front","average":{"1min":0.316,"5min":0.40500000000000003,"15min":0.35999999999999999},"min":{"1min":0.19600000000000001,"5min":0.188,"15min":0.17599999999999999},"max":{"1min":1.137,"5min":4.2350000000000003,"15min":4.2350000000000003},"last":0.33900000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-08T05:29:54.319 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-11-08T05:29:54.321 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 456, 'reported_epoch': 455, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:29:43.389354+0000', 'last_change': '2023-11-08T05:28:13.557071+0000', 'last_active': '2023-11-08T05:29:43.389354+0000', 'last_peered': '2023-11-08T05:29:43.389354+0000', 'last_clean': '2023-11-08T05:28:01.458439+0000', 'last_became_active': '2023-11-08T05:28:13.557071+0000', 'last_became_peered': '2023-11-08T05:28:13.557071+0000', 'last_unstale': '2023-11-08T05:29:43.389354+0000', 'last_undegraded': '2023-11-08T05:29:43.389354+0000', 'last_fullsized': '2023-11-08T05:28:13.547025+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:29:54.321 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-11-08T05:29:54.321 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 18, 'reported_epoch': 455, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:29:43.389262+0000', 'last_change': '2023-11-08T05:28:13.559513+0000', 'last_active': '2023-11-08T05:29:43.389262+0000', 'last_peered': '2023-11-08T05:29:43.389262+0000', 'last_clean': '2023-11-08T04:52:46.531964+0000', 'last_became_active': '2023-11-08T05:28:13.559513+0000', 'last_became_peered': '2023-11-08T05:28:13.559513+0000', 'last_unstale': '2023-11-08T05:29:43.389262+0000', 'last_undegraded': '2023-11-08T05:29:43.389262+0000', 'last_fullsized': '2023-11-08T05:28:13.540852+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:29:54.321 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-11-08T05:29:54.322 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "0'0", 'reported_seq': 457, 'reported_epoch': 455, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:29:43.389822+0000', 'last_change': '2023-11-08T05:28:13.545468+0000', 'last_active': '2023-11-08T05:29:43.389822+0000', 'last_peered': '2023-11-08T05:29:43.389822+0000', 'last_clean': '2023-11-08T05:28:01.456723+0000', 'last_became_active': '2023-11-08T05:28:13.545468+0000', 'last_became_peered': '2023-11-08T05:28:13.545468+0000', 'last_unstale': '2023-11-08T05:29:43.389822+0000', 'last_undegraded': '2023-11-08T05:29:43.389822+0000', 'last_fullsized': '2023-11-08T05:28:13.537207+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 9, 'last_epoch_clean': 10, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:45.528796+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:45.528796+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:45.528796+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:29:54.322 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.0 is not active+clean 2023-11-08T05:29:54.322 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.0', 'version': "0'0", 'reported_seq': 462, 'reported_epoch': 456, 'state': 'active+clean+remapped', 'last_fresh': '2023-11-08T05:29:44.395350+0000', 'last_change': '2023-11-08T05:29:44.395350+0000', 'last_active': '2023-11-08T05:29:44.395350+0000', 'last_peered': '2023-11-08T05:29:44.395350+0000', 'last_clean': '2023-11-08T05:29:44.395350+0000', 'last_became_active': '2023-11-08T05:29:44.395012+0000', 'last_became_peered': '2023-11-08T05:29:44.395012+0000', 'last_unstale': '2023-11-08T05:29:44.395350+0000', 'last_undegraded': '2023-11-08T05:29:44.395350+0000', 'last_fullsized': '2023-11-08T05:29:44.395350+0000', 'mapping_epoch': 455, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 456, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2], 'acting': [2, 1], 'avail_no_missing': ['2', '1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2023-11-08T05:29:54.323 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-11-08T05:29:54.323 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 456, 'reported_epoch': 455, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:29:43.389390+0000', 'last_change': '2023-11-08T05:28:13.563357+0000', 'last_active': '2023-11-08T05:29:43.389390+0000', 'last_peered': '2023-11-08T05:29:43.389390+0000', 'last_clean': '2023-11-08T05:28:01.457691+0000', 'last_became_active': '2023-11-08T05:28:13.563357+0000', 'last_became_peered': '2023-11-08T05:28:13.563357+0000', 'last_unstale': '2023-11-08T05:29:43.389390+0000', 'last_undegraded': '2023-11-08T05:29:43.389390+0000', 'last_fullsized': '2023-11-08T05:28:13.541614+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:29:54.323 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-11-08T05:29:54.324 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 456, 'reported_epoch': 455, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:29:43.389311+0000', 'last_change': '2023-11-08T05:28:13.550672+0000', 'last_active': '2023-11-08T05:29:43.389311+0000', 'last_peered': '2023-11-08T05:29:43.389311+0000', 'last_clean': '2023-11-08T05:28:01.457904+0000', 'last_became_active': '2023-11-08T05:28:13.550672+0000', 'last_became_peered': '2023-11-08T05:28:13.550672+0000', 'last_unstale': '2023-11-08T05:29:43.389311+0000', 'last_undegraded': '2023-11-08T05:29:43.389311+0000', 'last_fullsized': '2023-11-08T05:28:13.539595+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:29:54.324 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-11-08T05:29:54.324 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 456, 'reported_epoch': 455, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:29:43.389474+0000', 'last_change': '2023-11-08T05:28:13.557968+0000', 'last_active': '2023-11-08T05:29:43.389474+0000', 'last_peered': '2023-11-08T05:29:43.389474+0000', 'last_clean': '2023-11-08T05:28:01.458433+0000', 'last_became_active': '2023-11-08T05:28:13.557968+0000', 'last_became_peered': '2023-11-08T05:28:13.557968+0000', 'last_unstale': '2023-11-08T05:29:43.389474+0000', 'last_undegraded': '2023-11-08T05:29:43.389474+0000', 'last_fullsized': '2023-11-08T05:28:13.542832+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:29:54.325 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-11-08T05:29:54.670 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T05:29:54.670 INFO:teuthology.orchestra.run.smithi087.stdout:{"fsid":"ab4cbc06-449d-497d-b513-220d9db2b590","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 6 pgs undersized","count":6},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2237,"monmap":{"epoch":1,"min_mon_release_name":"pacific","num_mons":1},"osdmap":{"epoch":456,"num_osds":3,"num_up_osds":2,"osd_up_since":1699421292,"num_in_osds":3,"osd_in_since":1699419158,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":6},{"state_name":"active+clean","count":2},{"state_name":"active+clean+remapped","count":1}],"num_pgs":9,"num_pools":2,"num_objects":0,"data_bytes":0,"bytes_used":2697519104,"bytes_avail":287212773376,"bytes_total":289910292480},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":41,"modified":"2023-11-08T05:29:32.044544+0000","services":{}},"progress_events":{}} 2023-11-08T05:29:54.683 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-11-08T05:29:54.683 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-11-08T05:29:54.966 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T05:29:54.968 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2023-11-08T05:29:54.979 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":1568,"stamp":"2023-11-08T05:29:54.049324+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2634296,"kb_used_data":408940,"kb_used_omap":780,"kb_used_meta":2220403,"kb_avail":280481224,"statfs":{"total":289910292480,"available":287212773376,"internally_reserved":0,"allocated":418754560,"data_stored":410461501,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":799447,"internal_metadata":2273692969},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-536782902,"num_objects":-164,"num_object_clones":0,"num_object_copies":-328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-115,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-164,"num_whiteouts":0,"num_read":-122056,"num_read_kb":-39011942,"num_write":-413642,"num_write_kb":-151705339,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-8,"num_bytes_recovered":-16922624,"num_keys_recovered":-2,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.424817"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":456,"reported_epoch":455,"state":"active+undersized","last_fresh":"2023-11-08T05:29:43.389354+0000","last_change":"2023-11-08T05:28:13.557071+0000","last_active":"2023-11-08T05:29:43.389354+0000","last_peered":"2023-11-08T05:29:43.389354+0000","last_clean":"2023-11-08T05:28:01.458439+0000","last_became_active":"2023-11-08T05:28:13.557071+0000","last_became_peered":"2023-11-08T05:28:13.557071+0000","last_unstale":"2023-11-08T05:29:43.389354+0000","last_undegraded":"2023-11-08T05:29:43.389354+0000","last_fullsized":"2023-11-08T05:28:13.547025+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":18,"reported_epoch":455,"state":"active+undersized","last_fresh":"2023-11-08T05:29:43.389262+0000","last_change":"2023-11-08T05:28:13.559513+0000","last_active":"2023-11-08T05:29:43.389262+0000","last_peered":"2023-11-08T05:29:43.389262+0000","last_clean":"2023-11-08T04:52:46.531964+0000","last_became_active":"2023-11-08T05:28:13.559513+0000","last_became_peered":"2023-11-08T05:28:13.559513+0000","last_unstale":"2023-11-08T05:29:43.389262+0000","last_undegraded":"2023-11-08T05:29:43.389262+0000","last_fullsized":"2023-11-08T05:28:13.540852+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":453,"reported_epoch":455,"state":"active+clean","last_fresh":"2023-11-08T05:29:43.390991+0000","last_change":"2023-11-08T04:52:47.546100+0000","last_active":"2023-11-08T05:29:43.390991+0000","last_peered":"2023-11-08T05:29:43.390991+0000","last_clean":"2023-11-08T05:29:43.390991+0000","last_became_active":"2023-11-08T04:52:47.545840+0000","last_became_peered":"2023-11-08T04:52:47.545840+0000","last_unstale":"2023-11-08T05:29:43.390991+0000","last_undegraded":"2023-11-08T05:29:43.390991+0000","last_fullsized":"2023-11-08T05:29:43.390991+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":451,"reported_epoch":455,"state":"active+clean","last_fresh":"2023-11-08T05:29:43.389261+0000","last_change":"2023-11-08T04:52:47.548597+0000","last_active":"2023-11-08T05:29:43.389261+0000","last_peered":"2023-11-08T05:29:43.389261+0000","last_clean":"2023-11-08T05:29:43.389261+0000","last_became_active":"2023-11-08T04:52:47.548294+0000","last_became_peered":"2023-11-08T04:52:47.548294+0000","last_unstale":"2023-11-08T05:29:43.389261+0000","last_undegraded":"2023-11-08T05:29:43.389261+0000","last_fullsized":"2023-11-08T05:29:43.389261+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":457,"reported_epoch":455,"state":"active+undersized","last_fresh":"2023-11-08T05:29:43.389822+0000","last_change":"2023-11-08T05:28:13.545468+0000","last_active":"2023-11-08T05:29:43.389822+0000","last_peered":"2023-11-08T05:29:43.389822+0000","last_clean":"2023-11-08T05:28:01.456723+0000","last_became_active":"2023-11-08T05:28:13.545468+0000","last_became_peered":"2023-11-08T05:28:13.545468+0000","last_unstale":"2023-11-08T05:29:43.389822+0000","last_undegraded":"2023-11-08T05:29:43.389822+0000","last_fullsized":"2023-11-08T05:28:13.537207+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_clean_scrub_stamp":"2023-11-08T04:52:45.528796+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":462,"reported_epoch":456,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:29:44.395350+0000","last_change":"2023-11-08T05:29:44.395350+0000","last_active":"2023-11-08T05:29:44.395350+0000","last_peered":"2023-11-08T05:29:44.395350+0000","last_clean":"2023-11-08T05:29:44.395350+0000","last_became_active":"2023-11-08T05:29:44.395012+0000","last_became_peered":"2023-11-08T05:29:44.395012+0000","last_unstale":"2023-11-08T05:29:44.395350+0000","last_undegraded":"2023-11-08T05:29:44.395350+0000","last_fullsized":"2023-11-08T05:29:44.395350+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":456,"reported_epoch":455,"state":"active+undersized","last_fresh":"2023-11-08T05:29:43.389390+0000","last_change":"2023-11-08T05:28:13.563357+0000","last_active":"2023-11-08T05:29:43.389390+0000","last_peered":"2023-11-08T05:29:43.389390+0000","last_clean":"2023-11-08T05:28:01.457691+0000","last_became_active":"2023-11-08T05:28:13.563357+0000","last_became_peered":"2023-11-08T05:28:13.563357+0000","last_unstale":"2023-11-08T05:29:43.389390+0000","last_undegraded":"2023-11-08T05:29:43.389390+0000","last_fullsized":"2023-11-08T05:28:13.541614+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":456,"reported_epoch":455,"state":"active+undersized","last_fresh":"2023-11-08T05:29:43.389311+0000","last_change":"2023-11-08T05:28:13.550672+0000","last_active":"2023-11-08T05:29:43.389311+0000","last_peered":"2023-11-08T05:29:43.389311+0000","last_clean":"2023-11-08T05:28:01.457904+0000","last_became_active":"2023-11-08T05:28:13.550672+0000","last_became_peered":"2023-11-08T05:28:13.550672+0000","last_unstale":"2023-11-08T05:29:43.389311+0000","last_undegraded":"2023-11-08T05:29:43.389311+0000","last_fullsized":"2023-11-08T05:28:13.539595+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":456,"reported_epoch":455,"state":"active+undersized","last_fresh":"2023-11-08T05:29:43.389474+0000","last_change":"2023-11-08T05:28:13.557968+0000","last_active":"2023-11-08T05:29:43.389474+0000","last_peered":"2023-11-08T05:29:43.389474+0000","last_clean":"2023-11-08T05:28:01.458433+0000","last_became_active":"2023-11-08T05:28:13.557968+0000","last_became_peered":"2023-11-08T05:28:13.557968+0000","last_unstale":"2023-11-08T05:29:43.389474+0000","last_undegraded":"2023-11-08T05:29:43.389474+0000","last_fullsized":"2023-11-08T05:28:13.542832+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":1,"acting":1,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":10,"acting":11,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738816,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":645004,"kb_used_data":5316,"kb_used_omap":275,"kb_used_meta":639404,"kb_avail":93726836,"statfs":{"total":96636764160,"available":95976280064,"internally_reserved":0,"allocated":5443584,"data_stored":2678316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":281650,"internal_metadata":654750670},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:59 2023","interfaces":[{"interface":"back","average":{"1min":0.30499999999999999,"5min":0.439,"15min":0.36099999999999999},"min":{"1min":0.17399999999999999,"5min":0.14499999999999999,"15min":0.14499999999999999},"max":{"1min":0.60099999999999998,"5min":4.9379999999999997,"15min":4.9379999999999997},"last":0.379},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.49399999999999999,"15min":0.42399999999999999},"min":{"1min":0.221,"5min":0.19600000000000001,"15min":0.17000000000000001},"max":{"1min":0.63200000000000001,"5min":4.9859999999999998,"15min":4.9859999999999998},"last":0.40899999999999997}]},{"osd":1,"last update":"Wed Nov 8 05:29:01 2023","interfaces":[{"interface":"back","average":{"1min":0.30299999999999999,"5min":0.81899999999999995,"15min":0.47399999999999998},"min":{"1min":0.16,"5min":0.153,"15min":0.153},"max":{"1min":0.68200000000000005,"5min":23.242999999999999,"15min":23.242999999999999},"last":0.309},{"interface":"front","average":{"1min":0.46500000000000002,"5min":0.72199999999999998,"15min":0.46100000000000002},"min":{"1min":0.17699999999999999,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":2.0419999999999998,"5min":23.251000000000001,"15min":23.251000000000001},"last":0.32300000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738798,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1339228,"kb_used_data":398304,"kb_used_omap":291,"kb_used_meta":936476,"kb_avail":93032612,"statfs":{"total":96636764160,"available":95265394688,"internally_reserved":0,"allocated":407863296,"data_stored":405103900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":298462,"internal_metadata":958951970},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.436,"15min":0.36299999999999999},"min":{"1min":0.19700000000000001,"5min":0.14699999999999999,"15min":0.13700000000000001},"max":{"1min":3.222,"5min":6.5499999999999998,"15min":6.9989999999999997},"last":0.245},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.46300000000000002,"15min":0.42299999999999999},"min":{"1min":0.17799999999999999,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":5.0030000000000001,"5min":6.0629999999999997,"15min":9.6869999999999994},"last":0.28599999999999998}]},{"osd":2,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.51600000000000001,"15min":0.47099999999999997},"min":{"1min":0.21099999999999999,"5min":0.153,"15min":0.153},"max":{"1min":5.7439999999999998,"5min":5.8280000000000003,"15min":9.7639999999999993},"last":0.252},{"interface":"front","average":{"1min":0.497,"5min":0.56200000000000006,"15min":0.42299999999999999},"min":{"1min":0.20399999999999999,"5min":0.187,"15min":0.16800000000000001},"max":{"1min":3.9319999999999999,"5min":8.8040000000000003,"15min":8.8040000000000003},"last":0.29899999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738818,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":650064,"kb_used_data":5320,"kb_used_omap":214,"kb_used_meta":644521,"kb_avail":93721776,"statfs":{"total":96636764160,"available":95971098624,"internally_reserved":0,"allocated":5447680,"data_stored":2679285,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":219335,"internal_metadata":659990329},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:56 2023","interfaces":[{"interface":"back","average":{"1min":0.30399999999999999,"5min":0.34699999999999998,"15min":0.36499999999999999},"min":{"1min":0.23999999999999999,"5min":0.14199999999999999,"15min":0.14199999999999999},"max":{"1min":0.42399999999999999,"5min":1.595,"15min":7.4249999999999998},"last":0.17100000000000001},{"interface":"front","average":{"1min":0.35499999999999998,"5min":0.40899999999999997,"15min":0.376},"min":{"1min":0.23999999999999999,"5min":0.19600000000000001,"15min":0.183},"max":{"1min":0.46999999999999997,"5min":2.6859999999999999,"15min":2.6859999999999999},"last":0.23000000000000001}]},{"osd":2,"last update":"Wed Nov 8 05:28:56 2023","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.44400000000000001,"15min":0.376},"min":{"1min":0.187,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":2.2650000000000001,"5min":5.8209999999999997,"15min":5.8209999999999997},"last":0.316},{"interface":"front","average":{"1min":0.316,"5min":0.40500000000000003,"15min":0.35999999999999999},"min":{"1min":0.19600000000000001,"5min":0.188,"15min":0.17599999999999999},"max":{"1min":1.137,"5min":4.2350000000000003,"15min":4.2350000000000003},"last":0.33900000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-08T05:29:55.318 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~97s 2023-11-08T05:29:57.981 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-11-08T05:29:58.273 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T05:29:58.275 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2023-11-08T05:29:58.286 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":1570,"stamp":"2023-11-08T05:29:58.050154+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2634296,"kb_used_data":408940,"kb_used_omap":780,"kb_used_meta":2220403,"kb_avail":280481224,"statfs":{"total":289910292480,"available":287212773376,"internally_reserved":0,"allocated":418754560,"data_stored":410461501,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":799447,"internal_metadata":2273692969},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-536782902,"num_objects":-164,"num_object_clones":0,"num_object_copies":-328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-115,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-164,"num_whiteouts":0,"num_read":-122056,"num_read_kb":-39011942,"num_write":-413642,"num_write_kb":-151705339,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-8,"num_bytes_recovered":-16922624,"num_keys_recovered":-2,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.424817"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":456,"reported_epoch":455,"state":"active+undersized","last_fresh":"2023-11-08T05:29:43.389354+0000","last_change":"2023-11-08T05:28:13.557071+0000","last_active":"2023-11-08T05:29:43.389354+0000","last_peered":"2023-11-08T05:29:43.389354+0000","last_clean":"2023-11-08T05:28:01.458439+0000","last_became_active":"2023-11-08T05:28:13.557071+0000","last_became_peered":"2023-11-08T05:28:13.557071+0000","last_unstale":"2023-11-08T05:29:43.389354+0000","last_undegraded":"2023-11-08T05:29:43.389354+0000","last_fullsized":"2023-11-08T05:28:13.547025+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":18,"reported_epoch":455,"state":"active+undersized","last_fresh":"2023-11-08T05:29:43.389262+0000","last_change":"2023-11-08T05:28:13.559513+0000","last_active":"2023-11-08T05:29:43.389262+0000","last_peered":"2023-11-08T05:29:43.389262+0000","last_clean":"2023-11-08T04:52:46.531964+0000","last_became_active":"2023-11-08T05:28:13.559513+0000","last_became_peered":"2023-11-08T05:28:13.559513+0000","last_unstale":"2023-11-08T05:29:43.389262+0000","last_undegraded":"2023-11-08T05:29:43.389262+0000","last_fullsized":"2023-11-08T05:28:13.540852+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":453,"reported_epoch":455,"state":"active+clean","last_fresh":"2023-11-08T05:29:43.390991+0000","last_change":"2023-11-08T04:52:47.546100+0000","last_active":"2023-11-08T05:29:43.390991+0000","last_peered":"2023-11-08T05:29:43.390991+0000","last_clean":"2023-11-08T05:29:43.390991+0000","last_became_active":"2023-11-08T04:52:47.545840+0000","last_became_peered":"2023-11-08T04:52:47.545840+0000","last_unstale":"2023-11-08T05:29:43.390991+0000","last_undegraded":"2023-11-08T05:29:43.390991+0000","last_fullsized":"2023-11-08T05:29:43.390991+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":451,"reported_epoch":455,"state":"active+clean","last_fresh":"2023-11-08T05:29:43.389261+0000","last_change":"2023-11-08T04:52:47.548597+0000","last_active":"2023-11-08T05:29:43.389261+0000","last_peered":"2023-11-08T05:29:43.389261+0000","last_clean":"2023-11-08T05:29:43.389261+0000","last_became_active":"2023-11-08T04:52:47.548294+0000","last_became_peered":"2023-11-08T04:52:47.548294+0000","last_unstale":"2023-11-08T05:29:43.389261+0000","last_undegraded":"2023-11-08T05:29:43.389261+0000","last_fullsized":"2023-11-08T05:29:43.389261+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":457,"reported_epoch":455,"state":"active+undersized","last_fresh":"2023-11-08T05:29:43.389822+0000","last_change":"2023-11-08T05:28:13.545468+0000","last_active":"2023-11-08T05:29:43.389822+0000","last_peered":"2023-11-08T05:29:43.389822+0000","last_clean":"2023-11-08T05:28:01.456723+0000","last_became_active":"2023-11-08T05:28:13.545468+0000","last_became_peered":"2023-11-08T05:28:13.545468+0000","last_unstale":"2023-11-08T05:29:43.389822+0000","last_undegraded":"2023-11-08T05:29:43.389822+0000","last_fullsized":"2023-11-08T05:28:13.537207+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_clean_scrub_stamp":"2023-11-08T04:52:45.528796+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":462,"reported_epoch":456,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:29:44.395350+0000","last_change":"2023-11-08T05:29:44.395350+0000","last_active":"2023-11-08T05:29:44.395350+0000","last_peered":"2023-11-08T05:29:44.395350+0000","last_clean":"2023-11-08T05:29:44.395350+0000","last_became_active":"2023-11-08T05:29:44.395012+0000","last_became_peered":"2023-11-08T05:29:44.395012+0000","last_unstale":"2023-11-08T05:29:44.395350+0000","last_undegraded":"2023-11-08T05:29:44.395350+0000","last_fullsized":"2023-11-08T05:29:44.395350+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":456,"reported_epoch":455,"state":"active+undersized","last_fresh":"2023-11-08T05:29:43.389390+0000","last_change":"2023-11-08T05:28:13.563357+0000","last_active":"2023-11-08T05:29:43.389390+0000","last_peered":"2023-11-08T05:29:43.389390+0000","last_clean":"2023-11-08T05:28:01.457691+0000","last_became_active":"2023-11-08T05:28:13.563357+0000","last_became_peered":"2023-11-08T05:28:13.563357+0000","last_unstale":"2023-11-08T05:29:43.389390+0000","last_undegraded":"2023-11-08T05:29:43.389390+0000","last_fullsized":"2023-11-08T05:28:13.541614+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":456,"reported_epoch":455,"state":"active+undersized","last_fresh":"2023-11-08T05:29:43.389311+0000","last_change":"2023-11-08T05:28:13.550672+0000","last_active":"2023-11-08T05:29:43.389311+0000","last_peered":"2023-11-08T05:29:43.389311+0000","last_clean":"2023-11-08T05:28:01.457904+0000","last_became_active":"2023-11-08T05:28:13.550672+0000","last_became_peered":"2023-11-08T05:28:13.550672+0000","last_unstale":"2023-11-08T05:29:43.389311+0000","last_undegraded":"2023-11-08T05:29:43.389311+0000","last_fullsized":"2023-11-08T05:28:13.539595+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":456,"reported_epoch":455,"state":"active+undersized","last_fresh":"2023-11-08T05:29:43.389474+0000","last_change":"2023-11-08T05:28:13.557968+0000","last_active":"2023-11-08T05:29:43.389474+0000","last_peered":"2023-11-08T05:29:43.389474+0000","last_clean":"2023-11-08T05:28:01.458433+0000","last_became_active":"2023-11-08T05:28:13.557968+0000","last_became_peered":"2023-11-08T05:28:13.557968+0000","last_unstale":"2023-11-08T05:29:43.389474+0000","last_undegraded":"2023-11-08T05:29:43.389474+0000","last_fullsized":"2023-11-08T05:28:13.542832+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":1,"acting":1,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":10,"acting":11,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738817,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":645004,"kb_used_data":5316,"kb_used_omap":275,"kb_used_meta":639404,"kb_avail":93726836,"statfs":{"total":96636764160,"available":95976280064,"internally_reserved":0,"allocated":5443584,"data_stored":2678316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":281650,"internal_metadata":654750670},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:59 2023","interfaces":[{"interface":"back","average":{"1min":0.30499999999999999,"5min":0.439,"15min":0.36099999999999999},"min":{"1min":0.17399999999999999,"5min":0.14499999999999999,"15min":0.14499999999999999},"max":{"1min":0.60099999999999998,"5min":4.9379999999999997,"15min":4.9379999999999997},"last":0.379},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.49399999999999999,"15min":0.42399999999999999},"min":{"1min":0.221,"5min":0.19600000000000001,"15min":0.17000000000000001},"max":{"1min":0.63200000000000001,"5min":4.9859999999999998,"15min":4.9859999999999998},"last":0.40899999999999997}]},{"osd":1,"last update":"Wed Nov 8 05:29:01 2023","interfaces":[{"interface":"back","average":{"1min":0.30299999999999999,"5min":0.81899999999999995,"15min":0.47399999999999998},"min":{"1min":0.16,"5min":0.153,"15min":0.153},"max":{"1min":0.68200000000000005,"5min":23.242999999999999,"15min":23.242999999999999},"last":0.26600000000000001},{"interface":"front","average":{"1min":0.46500000000000002,"5min":0.72199999999999998,"15min":0.46100000000000002},"min":{"1min":0.17699999999999999,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":2.0419999999999998,"5min":23.251000000000001,"15min":23.251000000000001},"last":0.248}]}]},{"osd":0,"up_from":8,"seq":34359738798,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1339228,"kb_used_data":398304,"kb_used_omap":291,"kb_used_meta":936476,"kb_avail":93032612,"statfs":{"total":96636764160,"available":95265394688,"internally_reserved":0,"allocated":407863296,"data_stored":405103900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":298462,"internal_metadata":958951970},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.436,"15min":0.36299999999999999},"min":{"1min":0.19700000000000001,"5min":0.14699999999999999,"15min":0.13700000000000001},"max":{"1min":3.222,"5min":6.5499999999999998,"15min":6.9989999999999997},"last":0.245},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.46300000000000002,"15min":0.42299999999999999},"min":{"1min":0.17799999999999999,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":5.0030000000000001,"5min":6.0629999999999997,"15min":9.6869999999999994},"last":0.28599999999999998}]},{"osd":2,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.51600000000000001,"15min":0.47099999999999997},"min":{"1min":0.21099999999999999,"5min":0.153,"15min":0.153},"max":{"1min":5.7439999999999998,"5min":5.8280000000000003,"15min":9.7639999999999993},"last":0.252},{"interface":"front","average":{"1min":0.497,"5min":0.56200000000000006,"15min":0.42299999999999999},"min":{"1min":0.20399999999999999,"5min":0.187,"15min":0.16800000000000001},"max":{"1min":3.9319999999999999,"5min":8.8040000000000003,"15min":8.8040000000000003},"last":0.29899999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738819,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":650064,"kb_used_data":5320,"kb_used_omap":214,"kb_used_meta":644521,"kb_avail":93721776,"statfs":{"total":96636764160,"available":95971098624,"internally_reserved":0,"allocated":5447680,"data_stored":2679285,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":219335,"internal_metadata":659990329},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:56 2023","interfaces":[{"interface":"back","average":{"1min":0.30399999999999999,"5min":0.34699999999999998,"15min":0.36499999999999999},"min":{"1min":0.23999999999999999,"5min":0.14199999999999999,"15min":0.14199999999999999},"max":{"1min":0.42399999999999999,"5min":1.595,"15min":7.4249999999999998},"last":0.17100000000000001},{"interface":"front","average":{"1min":0.35499999999999998,"5min":0.40899999999999997,"15min":0.376},"min":{"1min":0.23999999999999999,"5min":0.19600000000000001,"15min":0.183},"max":{"1min":0.46999999999999997,"5min":2.6859999999999999,"15min":2.6859999999999999},"last":0.23000000000000001}]},{"osd":2,"last update":"Wed Nov 8 05:28:56 2023","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.44400000000000001,"15min":0.376},"min":{"1min":0.187,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":2.2650000000000001,"5min":5.8209999999999997,"15min":5.8209999999999997},"last":0.25700000000000001},{"interface":"front","average":{"1min":0.316,"5min":0.40500000000000003,"15min":0.35999999999999999},"min":{"1min":0.19600000000000001,"5min":0.188,"15min":0.17599999999999999},"max":{"1min":1.137,"5min":4.2350000000000003,"15min":4.2350000000000003},"last":0.27500000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-08T05:29:58.288 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-11-08T05:29:58.572 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T05:29:58.574 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2023-11-08T05:29:58.585 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":1570,"stamp":"2023-11-08T05:29:58.050154+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2634296,"kb_used_data":408940,"kb_used_omap":780,"kb_used_meta":2220403,"kb_avail":280481224,"statfs":{"total":289910292480,"available":287212773376,"internally_reserved":0,"allocated":418754560,"data_stored":410461501,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":799447,"internal_metadata":2273692969},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-536782902,"num_objects":-164,"num_object_clones":0,"num_object_copies":-328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-115,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-164,"num_whiteouts":0,"num_read":-122056,"num_read_kb":-39011942,"num_write":-413642,"num_write_kb":-151705339,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-8,"num_bytes_recovered":-16922624,"num_keys_recovered":-2,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.424817"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":456,"reported_epoch":455,"state":"active+undersized","last_fresh":"2023-11-08T05:29:43.389354+0000","last_change":"2023-11-08T05:28:13.557071+0000","last_active":"2023-11-08T05:29:43.389354+0000","last_peered":"2023-11-08T05:29:43.389354+0000","last_clean":"2023-11-08T05:28:01.458439+0000","last_became_active":"2023-11-08T05:28:13.557071+0000","last_became_peered":"2023-11-08T05:28:13.557071+0000","last_unstale":"2023-11-08T05:29:43.389354+0000","last_undegraded":"2023-11-08T05:29:43.389354+0000","last_fullsized":"2023-11-08T05:28:13.547025+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":18,"reported_epoch":455,"state":"active+undersized","last_fresh":"2023-11-08T05:29:43.389262+0000","last_change":"2023-11-08T05:28:13.559513+0000","last_active":"2023-11-08T05:29:43.389262+0000","last_peered":"2023-11-08T05:29:43.389262+0000","last_clean":"2023-11-08T04:52:46.531964+0000","last_became_active":"2023-11-08T05:28:13.559513+0000","last_became_peered":"2023-11-08T05:28:13.559513+0000","last_unstale":"2023-11-08T05:29:43.389262+0000","last_undegraded":"2023-11-08T05:29:43.389262+0000","last_fullsized":"2023-11-08T05:28:13.540852+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":453,"reported_epoch":455,"state":"active+clean","last_fresh":"2023-11-08T05:29:43.390991+0000","last_change":"2023-11-08T04:52:47.546100+0000","last_active":"2023-11-08T05:29:43.390991+0000","last_peered":"2023-11-08T05:29:43.390991+0000","last_clean":"2023-11-08T05:29:43.390991+0000","last_became_active":"2023-11-08T04:52:47.545840+0000","last_became_peered":"2023-11-08T04:52:47.545840+0000","last_unstale":"2023-11-08T05:29:43.390991+0000","last_undegraded":"2023-11-08T05:29:43.390991+0000","last_fullsized":"2023-11-08T05:29:43.390991+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":451,"reported_epoch":455,"state":"active+clean","last_fresh":"2023-11-08T05:29:43.389261+0000","last_change":"2023-11-08T04:52:47.548597+0000","last_active":"2023-11-08T05:29:43.389261+0000","last_peered":"2023-11-08T05:29:43.389261+0000","last_clean":"2023-11-08T05:29:43.389261+0000","last_became_active":"2023-11-08T04:52:47.548294+0000","last_became_peered":"2023-11-08T04:52:47.548294+0000","last_unstale":"2023-11-08T05:29:43.389261+0000","last_undegraded":"2023-11-08T05:29:43.389261+0000","last_fullsized":"2023-11-08T05:29:43.389261+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":457,"reported_epoch":455,"state":"active+undersized","last_fresh":"2023-11-08T05:29:43.389822+0000","last_change":"2023-11-08T05:28:13.545468+0000","last_active":"2023-11-08T05:29:43.389822+0000","last_peered":"2023-11-08T05:29:43.389822+0000","last_clean":"2023-11-08T05:28:01.456723+0000","last_became_active":"2023-11-08T05:28:13.545468+0000","last_became_peered":"2023-11-08T05:28:13.545468+0000","last_unstale":"2023-11-08T05:29:43.389822+0000","last_undegraded":"2023-11-08T05:29:43.389822+0000","last_fullsized":"2023-11-08T05:28:13.537207+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_clean_scrub_stamp":"2023-11-08T04:52:45.528796+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":462,"reported_epoch":456,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:29:44.395350+0000","last_change":"2023-11-08T05:29:44.395350+0000","last_active":"2023-11-08T05:29:44.395350+0000","last_peered":"2023-11-08T05:29:44.395350+0000","last_clean":"2023-11-08T05:29:44.395350+0000","last_became_active":"2023-11-08T05:29:44.395012+0000","last_became_peered":"2023-11-08T05:29:44.395012+0000","last_unstale":"2023-11-08T05:29:44.395350+0000","last_undegraded":"2023-11-08T05:29:44.395350+0000","last_fullsized":"2023-11-08T05:29:44.395350+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":456,"reported_epoch":455,"state":"active+undersized","last_fresh":"2023-11-08T05:29:43.389390+0000","last_change":"2023-11-08T05:28:13.563357+0000","last_active":"2023-11-08T05:29:43.389390+0000","last_peered":"2023-11-08T05:29:43.389390+0000","last_clean":"2023-11-08T05:28:01.457691+0000","last_became_active":"2023-11-08T05:28:13.563357+0000","last_became_peered":"2023-11-08T05:28:13.563357+0000","last_unstale":"2023-11-08T05:29:43.389390+0000","last_undegraded":"2023-11-08T05:29:43.389390+0000","last_fullsized":"2023-11-08T05:28:13.541614+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":456,"reported_epoch":455,"state":"active+undersized","last_fresh":"2023-11-08T05:29:43.389311+0000","last_change":"2023-11-08T05:28:13.550672+0000","last_active":"2023-11-08T05:29:43.389311+0000","last_peered":"2023-11-08T05:29:43.389311+0000","last_clean":"2023-11-08T05:28:01.457904+0000","last_became_active":"2023-11-08T05:28:13.550672+0000","last_became_peered":"2023-11-08T05:28:13.550672+0000","last_unstale":"2023-11-08T05:29:43.389311+0000","last_undegraded":"2023-11-08T05:29:43.389311+0000","last_fullsized":"2023-11-08T05:28:13.539595+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":456,"reported_epoch":455,"state":"active+undersized","last_fresh":"2023-11-08T05:29:43.389474+0000","last_change":"2023-11-08T05:28:13.557968+0000","last_active":"2023-11-08T05:29:43.389474+0000","last_peered":"2023-11-08T05:29:43.389474+0000","last_clean":"2023-11-08T05:28:01.458433+0000","last_became_active":"2023-11-08T05:28:13.557968+0000","last_became_peered":"2023-11-08T05:28:13.557968+0000","last_unstale":"2023-11-08T05:29:43.389474+0000","last_undegraded":"2023-11-08T05:29:43.389474+0000","last_fullsized":"2023-11-08T05:28:13.542832+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":1,"acting":1,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":10,"acting":11,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738817,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":645004,"kb_used_data":5316,"kb_used_omap":275,"kb_used_meta":639404,"kb_avail":93726836,"statfs":{"total":96636764160,"available":95976280064,"internally_reserved":0,"allocated":5443584,"data_stored":2678316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":281650,"internal_metadata":654750670},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:59 2023","interfaces":[{"interface":"back","average":{"1min":0.30499999999999999,"5min":0.439,"15min":0.36099999999999999},"min":{"1min":0.17399999999999999,"5min":0.14499999999999999,"15min":0.14499999999999999},"max":{"1min":0.60099999999999998,"5min":4.9379999999999997,"15min":4.9379999999999997},"last":0.379},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.49399999999999999,"15min":0.42399999999999999},"min":{"1min":0.221,"5min":0.19600000000000001,"15min":0.17000000000000001},"max":{"1min":0.63200000000000001,"5min":4.9859999999999998,"15min":4.9859999999999998},"last":0.40899999999999997}]},{"osd":1,"last update":"Wed Nov 8 05:29:01 2023","interfaces":[{"interface":"back","average":{"1min":0.30299999999999999,"5min":0.81899999999999995,"15min":0.47399999999999998},"min":{"1min":0.16,"5min":0.153,"15min":0.153},"max":{"1min":0.68200000000000005,"5min":23.242999999999999,"15min":23.242999999999999},"last":0.26600000000000001},{"interface":"front","average":{"1min":0.46500000000000002,"5min":0.72199999999999998,"15min":0.46100000000000002},"min":{"1min":0.17699999999999999,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":2.0419999999999998,"5min":23.251000000000001,"15min":23.251000000000001},"last":0.248}]}]},{"osd":0,"up_from":8,"seq":34359738798,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1339228,"kb_used_data":398304,"kb_used_omap":291,"kb_used_meta":936476,"kb_avail":93032612,"statfs":{"total":96636764160,"available":95265394688,"internally_reserved":0,"allocated":407863296,"data_stored":405103900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":298462,"internal_metadata":958951970},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.436,"15min":0.36299999999999999},"min":{"1min":0.19700000000000001,"5min":0.14699999999999999,"15min":0.13700000000000001},"max":{"1min":3.222,"5min":6.5499999999999998,"15min":6.9989999999999997},"last":0.245},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.46300000000000002,"15min":0.42299999999999999},"min":{"1min":0.17799999999999999,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":5.0030000000000001,"5min":6.0629999999999997,"15min":9.6869999999999994},"last":0.28599999999999998}]},{"osd":2,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.51600000000000001,"15min":0.47099999999999997},"min":{"1min":0.21099999999999999,"5min":0.153,"15min":0.153},"max":{"1min":5.7439999999999998,"5min":5.8280000000000003,"15min":9.7639999999999993},"last":0.252},{"interface":"front","average":{"1min":0.497,"5min":0.56200000000000006,"15min":0.42299999999999999},"min":{"1min":0.20399999999999999,"5min":0.187,"15min":0.16800000000000001},"max":{"1min":3.9319999999999999,"5min":8.8040000000000003,"15min":8.8040000000000003},"last":0.29899999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738819,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":650064,"kb_used_data":5320,"kb_used_omap":214,"kb_used_meta":644521,"kb_avail":93721776,"statfs":{"total":96636764160,"available":95971098624,"internally_reserved":0,"allocated":5447680,"data_stored":2679285,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":219335,"internal_metadata":659990329},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:56 2023","interfaces":[{"interface":"back","average":{"1min":0.30399999999999999,"5min":0.34699999999999998,"15min":0.36499999999999999},"min":{"1min":0.23999999999999999,"5min":0.14199999999999999,"15min":0.14199999999999999},"max":{"1min":0.42399999999999999,"5min":1.595,"15min":7.4249999999999998},"last":0.17100000000000001},{"interface":"front","average":{"1min":0.35499999999999998,"5min":0.40899999999999997,"15min":0.376},"min":{"1min":0.23999999999999999,"5min":0.19600000000000001,"15min":0.183},"max":{"1min":0.46999999999999997,"5min":2.6859999999999999,"15min":2.6859999999999999},"last":0.23000000000000001}]},{"osd":2,"last update":"Wed Nov 8 05:28:56 2023","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.44400000000000001,"15min":0.376},"min":{"1min":0.187,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":2.2650000000000001,"5min":5.8209999999999997,"15min":5.8209999999999997},"last":0.25700000000000001},{"interface":"front","average":{"1min":0.316,"5min":0.40500000000000003,"15min":0.35999999999999999},"min":{"1min":0.19600000000000001,"5min":0.188,"15min":0.17599999999999999},"max":{"1min":1.137,"5min":4.2350000000000003,"15min":4.2350000000000003},"last":0.27500000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-08T05:29:58.586 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-11-08T05:29:58.586 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 456, 'reported_epoch': 455, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:29:43.389354+0000', 'last_change': '2023-11-08T05:28:13.557071+0000', 'last_active': '2023-11-08T05:29:43.389354+0000', 'last_peered': '2023-11-08T05:29:43.389354+0000', 'last_clean': '2023-11-08T05:28:01.458439+0000', 'last_became_active': '2023-11-08T05:28:13.557071+0000', 'last_became_peered': '2023-11-08T05:28:13.557071+0000', 'last_unstale': '2023-11-08T05:29:43.389354+0000', 'last_undegraded': '2023-11-08T05:29:43.389354+0000', 'last_fullsized': '2023-11-08T05:28:13.547025+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:29:58.587 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-11-08T05:29:58.587 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 18, 'reported_epoch': 455, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:29:43.389262+0000', 'last_change': '2023-11-08T05:28:13.559513+0000', 'last_active': '2023-11-08T05:29:43.389262+0000', 'last_peered': '2023-11-08T05:29:43.389262+0000', 'last_clean': '2023-11-08T04:52:46.531964+0000', 'last_became_active': '2023-11-08T05:28:13.559513+0000', 'last_became_peered': '2023-11-08T05:28:13.559513+0000', 'last_unstale': '2023-11-08T05:29:43.389262+0000', 'last_undegraded': '2023-11-08T05:29:43.389262+0000', 'last_fullsized': '2023-11-08T05:28:13.540852+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:29:58.587 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-11-08T05:29:58.588 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "0'0", 'reported_seq': 457, 'reported_epoch': 455, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:29:43.389822+0000', 'last_change': '2023-11-08T05:28:13.545468+0000', 'last_active': '2023-11-08T05:29:43.389822+0000', 'last_peered': '2023-11-08T05:29:43.389822+0000', 'last_clean': '2023-11-08T05:28:01.456723+0000', 'last_became_active': '2023-11-08T05:28:13.545468+0000', 'last_became_peered': '2023-11-08T05:28:13.545468+0000', 'last_unstale': '2023-11-08T05:29:43.389822+0000', 'last_undegraded': '2023-11-08T05:29:43.389822+0000', 'last_fullsized': '2023-11-08T05:28:13.537207+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 9, 'last_epoch_clean': 10, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:45.528796+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:45.528796+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:45.528796+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:29:58.588 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.0 is not active+clean 2023-11-08T05:29:58.588 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.0', 'version': "0'0", 'reported_seq': 462, 'reported_epoch': 456, 'state': 'active+clean+remapped', 'last_fresh': '2023-11-08T05:29:44.395350+0000', 'last_change': '2023-11-08T05:29:44.395350+0000', 'last_active': '2023-11-08T05:29:44.395350+0000', 'last_peered': '2023-11-08T05:29:44.395350+0000', 'last_clean': '2023-11-08T05:29:44.395350+0000', 'last_became_active': '2023-11-08T05:29:44.395012+0000', 'last_became_peered': '2023-11-08T05:29:44.395012+0000', 'last_unstale': '2023-11-08T05:29:44.395350+0000', 'last_undegraded': '2023-11-08T05:29:44.395350+0000', 'last_fullsized': '2023-11-08T05:29:44.395350+0000', 'mapping_epoch': 455, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 456, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2], 'acting': [2, 1], 'avail_no_missing': ['2', '1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2023-11-08T05:29:58.588 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-11-08T05:29:58.589 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 456, 'reported_epoch': 455, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:29:43.389390+0000', 'last_change': '2023-11-08T05:28:13.563357+0000', 'last_active': '2023-11-08T05:29:43.389390+0000', 'last_peered': '2023-11-08T05:29:43.389390+0000', 'last_clean': '2023-11-08T05:28:01.457691+0000', 'last_became_active': '2023-11-08T05:28:13.563357+0000', 'last_became_peered': '2023-11-08T05:28:13.563357+0000', 'last_unstale': '2023-11-08T05:29:43.389390+0000', 'last_undegraded': '2023-11-08T05:29:43.389390+0000', 'last_fullsized': '2023-11-08T05:28:13.541614+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:29:58.589 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-11-08T05:29:58.589 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 456, 'reported_epoch': 455, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:29:43.389311+0000', 'last_change': '2023-11-08T05:28:13.550672+0000', 'last_active': '2023-11-08T05:29:43.389311+0000', 'last_peered': '2023-11-08T05:29:43.389311+0000', 'last_clean': '2023-11-08T05:28:01.457904+0000', 'last_became_active': '2023-11-08T05:28:13.550672+0000', 'last_became_peered': '2023-11-08T05:28:13.550672+0000', 'last_unstale': '2023-11-08T05:29:43.389311+0000', 'last_undegraded': '2023-11-08T05:29:43.389311+0000', 'last_fullsized': '2023-11-08T05:28:13.539595+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:29:58.590 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-11-08T05:29:58.590 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 456, 'reported_epoch': 455, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:29:43.389474+0000', 'last_change': '2023-11-08T05:28:13.557968+0000', 'last_active': '2023-11-08T05:29:43.389474+0000', 'last_peered': '2023-11-08T05:29:43.389474+0000', 'last_clean': '2023-11-08T05:28:01.458433+0000', 'last_became_active': '2023-11-08T05:28:13.557968+0000', 'last_became_peered': '2023-11-08T05:28:13.557968+0000', 'last_unstale': '2023-11-08T05:29:43.389474+0000', 'last_undegraded': '2023-11-08T05:29:43.389474+0000', 'last_fullsized': '2023-11-08T05:28:13.542832+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:29:58.591 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-11-08T05:29:58.936 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T05:29:58.937 INFO:teuthology.orchestra.run.smithi087.stdout:{"fsid":"ab4cbc06-449d-497d-b513-220d9db2b590","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 6 pgs undersized","count":6},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2241,"monmap":{"epoch":1,"min_mon_release_name":"pacific","num_mons":1},"osdmap":{"epoch":456,"num_osds":3,"num_up_osds":2,"osd_up_since":1699421292,"num_in_osds":3,"osd_in_since":1699419158,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":6},{"state_name":"active+clean","count":2},{"state_name":"active+clean+remapped","count":1}],"num_pgs":9,"num_pools":2,"num_objects":0,"data_bytes":0,"bytes_used":2697519104,"bytes_avail":287212773376,"bytes_total":289910292480},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":41,"modified":"2023-11-08T05:29:32.044544+0000","services":{}},"progress_events":{}} 2023-11-08T05:29:58.950 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-11-08T05:29:58.950 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-11-08T05:29:59.232 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T05:29:59.233 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2023-11-08T05:29:59.244 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":1570,"stamp":"2023-11-08T05:29:58.050154+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2634296,"kb_used_data":408940,"kb_used_omap":780,"kb_used_meta":2220403,"kb_avail":280481224,"statfs":{"total":289910292480,"available":287212773376,"internally_reserved":0,"allocated":418754560,"data_stored":410461501,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":799447,"internal_metadata":2273692969},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-536782902,"num_objects":-164,"num_object_clones":0,"num_object_copies":-328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-115,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-164,"num_whiteouts":0,"num_read":-122056,"num_read_kb":-39011942,"num_write":-413642,"num_write_kb":-151705339,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-8,"num_bytes_recovered":-16922624,"num_keys_recovered":-2,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.424817"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":456,"reported_epoch":455,"state":"active+undersized","last_fresh":"2023-11-08T05:29:43.389354+0000","last_change":"2023-11-08T05:28:13.557071+0000","last_active":"2023-11-08T05:29:43.389354+0000","last_peered":"2023-11-08T05:29:43.389354+0000","last_clean":"2023-11-08T05:28:01.458439+0000","last_became_active":"2023-11-08T05:28:13.557071+0000","last_became_peered":"2023-11-08T05:28:13.557071+0000","last_unstale":"2023-11-08T05:29:43.389354+0000","last_undegraded":"2023-11-08T05:29:43.389354+0000","last_fullsized":"2023-11-08T05:28:13.547025+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":18,"reported_epoch":455,"state":"active+undersized","last_fresh":"2023-11-08T05:29:43.389262+0000","last_change":"2023-11-08T05:28:13.559513+0000","last_active":"2023-11-08T05:29:43.389262+0000","last_peered":"2023-11-08T05:29:43.389262+0000","last_clean":"2023-11-08T04:52:46.531964+0000","last_became_active":"2023-11-08T05:28:13.559513+0000","last_became_peered":"2023-11-08T05:28:13.559513+0000","last_unstale":"2023-11-08T05:29:43.389262+0000","last_undegraded":"2023-11-08T05:29:43.389262+0000","last_fullsized":"2023-11-08T05:28:13.540852+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":453,"reported_epoch":455,"state":"active+clean","last_fresh":"2023-11-08T05:29:43.390991+0000","last_change":"2023-11-08T04:52:47.546100+0000","last_active":"2023-11-08T05:29:43.390991+0000","last_peered":"2023-11-08T05:29:43.390991+0000","last_clean":"2023-11-08T05:29:43.390991+0000","last_became_active":"2023-11-08T04:52:47.545840+0000","last_became_peered":"2023-11-08T04:52:47.545840+0000","last_unstale":"2023-11-08T05:29:43.390991+0000","last_undegraded":"2023-11-08T05:29:43.390991+0000","last_fullsized":"2023-11-08T05:29:43.390991+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":451,"reported_epoch":455,"state":"active+clean","last_fresh":"2023-11-08T05:29:43.389261+0000","last_change":"2023-11-08T04:52:47.548597+0000","last_active":"2023-11-08T05:29:43.389261+0000","last_peered":"2023-11-08T05:29:43.389261+0000","last_clean":"2023-11-08T05:29:43.389261+0000","last_became_active":"2023-11-08T04:52:47.548294+0000","last_became_peered":"2023-11-08T04:52:47.548294+0000","last_unstale":"2023-11-08T05:29:43.389261+0000","last_undegraded":"2023-11-08T05:29:43.389261+0000","last_fullsized":"2023-11-08T05:29:43.389261+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":457,"reported_epoch":455,"state":"active+undersized","last_fresh":"2023-11-08T05:29:43.389822+0000","last_change":"2023-11-08T05:28:13.545468+0000","last_active":"2023-11-08T05:29:43.389822+0000","last_peered":"2023-11-08T05:29:43.389822+0000","last_clean":"2023-11-08T05:28:01.456723+0000","last_became_active":"2023-11-08T05:28:13.545468+0000","last_became_peered":"2023-11-08T05:28:13.545468+0000","last_unstale":"2023-11-08T05:29:43.389822+0000","last_undegraded":"2023-11-08T05:29:43.389822+0000","last_fullsized":"2023-11-08T05:28:13.537207+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_clean_scrub_stamp":"2023-11-08T04:52:45.528796+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":462,"reported_epoch":456,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:29:44.395350+0000","last_change":"2023-11-08T05:29:44.395350+0000","last_active":"2023-11-08T05:29:44.395350+0000","last_peered":"2023-11-08T05:29:44.395350+0000","last_clean":"2023-11-08T05:29:44.395350+0000","last_became_active":"2023-11-08T05:29:44.395012+0000","last_became_peered":"2023-11-08T05:29:44.395012+0000","last_unstale":"2023-11-08T05:29:44.395350+0000","last_undegraded":"2023-11-08T05:29:44.395350+0000","last_fullsized":"2023-11-08T05:29:44.395350+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":456,"reported_epoch":455,"state":"active+undersized","last_fresh":"2023-11-08T05:29:43.389390+0000","last_change":"2023-11-08T05:28:13.563357+0000","last_active":"2023-11-08T05:29:43.389390+0000","last_peered":"2023-11-08T05:29:43.389390+0000","last_clean":"2023-11-08T05:28:01.457691+0000","last_became_active":"2023-11-08T05:28:13.563357+0000","last_became_peered":"2023-11-08T05:28:13.563357+0000","last_unstale":"2023-11-08T05:29:43.389390+0000","last_undegraded":"2023-11-08T05:29:43.389390+0000","last_fullsized":"2023-11-08T05:28:13.541614+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":456,"reported_epoch":455,"state":"active+undersized","last_fresh":"2023-11-08T05:29:43.389311+0000","last_change":"2023-11-08T05:28:13.550672+0000","last_active":"2023-11-08T05:29:43.389311+0000","last_peered":"2023-11-08T05:29:43.389311+0000","last_clean":"2023-11-08T05:28:01.457904+0000","last_became_active":"2023-11-08T05:28:13.550672+0000","last_became_peered":"2023-11-08T05:28:13.550672+0000","last_unstale":"2023-11-08T05:29:43.389311+0000","last_undegraded":"2023-11-08T05:29:43.389311+0000","last_fullsized":"2023-11-08T05:28:13.539595+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":456,"reported_epoch":455,"state":"active+undersized","last_fresh":"2023-11-08T05:29:43.389474+0000","last_change":"2023-11-08T05:28:13.557968+0000","last_active":"2023-11-08T05:29:43.389474+0000","last_peered":"2023-11-08T05:29:43.389474+0000","last_clean":"2023-11-08T05:28:01.458433+0000","last_became_active":"2023-11-08T05:28:13.557968+0000","last_became_peered":"2023-11-08T05:28:13.557968+0000","last_unstale":"2023-11-08T05:29:43.389474+0000","last_undegraded":"2023-11-08T05:29:43.389474+0000","last_fullsized":"2023-11-08T05:28:13.542832+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":1,"acting":1,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":10,"acting":11,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738817,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":645004,"kb_used_data":5316,"kb_used_omap":275,"kb_used_meta":639404,"kb_avail":93726836,"statfs":{"total":96636764160,"available":95976280064,"internally_reserved":0,"allocated":5443584,"data_stored":2678316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":281650,"internal_metadata":654750670},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:59 2023","interfaces":[{"interface":"back","average":{"1min":0.30499999999999999,"5min":0.439,"15min":0.36099999999999999},"min":{"1min":0.17399999999999999,"5min":0.14499999999999999,"15min":0.14499999999999999},"max":{"1min":0.60099999999999998,"5min":4.9379999999999997,"15min":4.9379999999999997},"last":0.379},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.49399999999999999,"15min":0.42399999999999999},"min":{"1min":0.221,"5min":0.19600000000000001,"15min":0.17000000000000001},"max":{"1min":0.63200000000000001,"5min":4.9859999999999998,"15min":4.9859999999999998},"last":0.40899999999999997}]},{"osd":1,"last update":"Wed Nov 8 05:29:01 2023","interfaces":[{"interface":"back","average":{"1min":0.30299999999999999,"5min":0.81899999999999995,"15min":0.47399999999999998},"min":{"1min":0.16,"5min":0.153,"15min":0.153},"max":{"1min":0.68200000000000005,"5min":23.242999999999999,"15min":23.242999999999999},"last":0.26600000000000001},{"interface":"front","average":{"1min":0.46500000000000002,"5min":0.72199999999999998,"15min":0.46100000000000002},"min":{"1min":0.17699999999999999,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":2.0419999999999998,"5min":23.251000000000001,"15min":23.251000000000001},"last":0.248}]}]},{"osd":0,"up_from":8,"seq":34359738798,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1339228,"kb_used_data":398304,"kb_used_omap":291,"kb_used_meta":936476,"kb_avail":93032612,"statfs":{"total":96636764160,"available":95265394688,"internally_reserved":0,"allocated":407863296,"data_stored":405103900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":298462,"internal_metadata":958951970},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.436,"15min":0.36299999999999999},"min":{"1min":0.19700000000000001,"5min":0.14699999999999999,"15min":0.13700000000000001},"max":{"1min":3.222,"5min":6.5499999999999998,"15min":6.9989999999999997},"last":0.245},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.46300000000000002,"15min":0.42299999999999999},"min":{"1min":0.17799999999999999,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":5.0030000000000001,"5min":6.0629999999999997,"15min":9.6869999999999994},"last":0.28599999999999998}]},{"osd":2,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.51600000000000001,"15min":0.47099999999999997},"min":{"1min":0.21099999999999999,"5min":0.153,"15min":0.153},"max":{"1min":5.7439999999999998,"5min":5.8280000000000003,"15min":9.7639999999999993},"last":0.252},{"interface":"front","average":{"1min":0.497,"5min":0.56200000000000006,"15min":0.42299999999999999},"min":{"1min":0.20399999999999999,"5min":0.187,"15min":0.16800000000000001},"max":{"1min":3.9319999999999999,"5min":8.8040000000000003,"15min":8.8040000000000003},"last":0.29899999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738819,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":650064,"kb_used_data":5320,"kb_used_omap":214,"kb_used_meta":644521,"kb_avail":93721776,"statfs":{"total":96636764160,"available":95971098624,"internally_reserved":0,"allocated":5447680,"data_stored":2679285,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":219335,"internal_metadata":659990329},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:56 2023","interfaces":[{"interface":"back","average":{"1min":0.30399999999999999,"5min":0.34699999999999998,"15min":0.36499999999999999},"min":{"1min":0.23999999999999999,"5min":0.14199999999999999,"15min":0.14199999999999999},"max":{"1min":0.42399999999999999,"5min":1.595,"15min":7.4249999999999998},"last":0.17100000000000001},{"interface":"front","average":{"1min":0.35499999999999998,"5min":0.40899999999999997,"15min":0.376},"min":{"1min":0.23999999999999999,"5min":0.19600000000000001,"15min":0.183},"max":{"1min":0.46999999999999997,"5min":2.6859999999999999,"15min":2.6859999999999999},"last":0.23000000000000001}]},{"osd":2,"last update":"Wed Nov 8 05:28:56 2023","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.44400000000000001,"15min":0.376},"min":{"1min":0.187,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":2.2650000000000001,"5min":5.8209999999999997,"15min":5.8209999999999997},"last":0.25700000000000001},{"interface":"front","average":{"1min":0.316,"5min":0.40500000000000003,"15min":0.35999999999999999},"min":{"1min":0.19600000000000001,"5min":0.188,"15min":0.17599999999999999},"max":{"1min":1.137,"5min":4.2350000000000003,"15min":4.2350000000000003},"last":0.27500000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-08T05:30:00.722 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~103s 2023-11-08T05:30:02.246 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-11-08T05:30:02.545 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T05:30:02.546 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2023-11-08T05:30:02.557 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":1572,"stamp":"2023-11-08T05:30:02.051022+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2634296,"kb_used_data":408940,"kb_used_omap":780,"kb_used_meta":2220403,"kb_avail":280481224,"statfs":{"total":289910292480,"available":287212773376,"internally_reserved":0,"allocated":418754560,"data_stored":410461501,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":799447,"internal_metadata":2273692969},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-536782902,"num_objects":-164,"num_object_clones":0,"num_object_copies":-328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-115,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-164,"num_whiteouts":0,"num_read":-122056,"num_read_kb":-39011942,"num_write":-413642,"num_write_kb":-151705339,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-8,"num_bytes_recovered":-16922624,"num_keys_recovered":-2,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.424817"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":456,"reported_epoch":455,"state":"active+undersized","last_fresh":"2023-11-08T05:29:43.389354+0000","last_change":"2023-11-08T05:28:13.557071+0000","last_active":"2023-11-08T05:29:43.389354+0000","last_peered":"2023-11-08T05:29:43.389354+0000","last_clean":"2023-11-08T05:28:01.458439+0000","last_became_active":"2023-11-08T05:28:13.557071+0000","last_became_peered":"2023-11-08T05:28:13.557071+0000","last_unstale":"2023-11-08T05:29:43.389354+0000","last_undegraded":"2023-11-08T05:29:43.389354+0000","last_fullsized":"2023-11-08T05:28:13.547025+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":18,"reported_epoch":455,"state":"active+undersized","last_fresh":"2023-11-08T05:29:43.389262+0000","last_change":"2023-11-08T05:28:13.559513+0000","last_active":"2023-11-08T05:29:43.389262+0000","last_peered":"2023-11-08T05:29:43.389262+0000","last_clean":"2023-11-08T04:52:46.531964+0000","last_became_active":"2023-11-08T05:28:13.559513+0000","last_became_peered":"2023-11-08T05:28:13.559513+0000","last_unstale":"2023-11-08T05:29:43.389262+0000","last_undegraded":"2023-11-08T05:29:43.389262+0000","last_fullsized":"2023-11-08T05:28:13.540852+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":453,"reported_epoch":455,"state":"active+clean","last_fresh":"2023-11-08T05:29:43.390991+0000","last_change":"2023-11-08T04:52:47.546100+0000","last_active":"2023-11-08T05:29:43.390991+0000","last_peered":"2023-11-08T05:29:43.390991+0000","last_clean":"2023-11-08T05:29:43.390991+0000","last_became_active":"2023-11-08T04:52:47.545840+0000","last_became_peered":"2023-11-08T04:52:47.545840+0000","last_unstale":"2023-11-08T05:29:43.390991+0000","last_undegraded":"2023-11-08T05:29:43.390991+0000","last_fullsized":"2023-11-08T05:29:43.390991+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":451,"reported_epoch":455,"state":"active+clean","last_fresh":"2023-11-08T05:29:43.389261+0000","last_change":"2023-11-08T04:52:47.548597+0000","last_active":"2023-11-08T05:29:43.389261+0000","last_peered":"2023-11-08T05:29:43.389261+0000","last_clean":"2023-11-08T05:29:43.389261+0000","last_became_active":"2023-11-08T04:52:47.548294+0000","last_became_peered":"2023-11-08T04:52:47.548294+0000","last_unstale":"2023-11-08T05:29:43.389261+0000","last_undegraded":"2023-11-08T05:29:43.389261+0000","last_fullsized":"2023-11-08T05:29:43.389261+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":457,"reported_epoch":455,"state":"active+undersized","last_fresh":"2023-11-08T05:29:43.389822+0000","last_change":"2023-11-08T05:28:13.545468+0000","last_active":"2023-11-08T05:29:43.389822+0000","last_peered":"2023-11-08T05:29:43.389822+0000","last_clean":"2023-11-08T05:28:01.456723+0000","last_became_active":"2023-11-08T05:28:13.545468+0000","last_became_peered":"2023-11-08T05:28:13.545468+0000","last_unstale":"2023-11-08T05:29:43.389822+0000","last_undegraded":"2023-11-08T05:29:43.389822+0000","last_fullsized":"2023-11-08T05:28:13.537207+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_clean_scrub_stamp":"2023-11-08T04:52:45.528796+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":462,"reported_epoch":456,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:29:44.395350+0000","last_change":"2023-11-08T05:29:44.395350+0000","last_active":"2023-11-08T05:29:44.395350+0000","last_peered":"2023-11-08T05:29:44.395350+0000","last_clean":"2023-11-08T05:29:44.395350+0000","last_became_active":"2023-11-08T05:29:44.395012+0000","last_became_peered":"2023-11-08T05:29:44.395012+0000","last_unstale":"2023-11-08T05:29:44.395350+0000","last_undegraded":"2023-11-08T05:29:44.395350+0000","last_fullsized":"2023-11-08T05:29:44.395350+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":456,"reported_epoch":455,"state":"active+undersized","last_fresh":"2023-11-08T05:29:43.389390+0000","last_change":"2023-11-08T05:28:13.563357+0000","last_active":"2023-11-08T05:29:43.389390+0000","last_peered":"2023-11-08T05:29:43.389390+0000","last_clean":"2023-11-08T05:28:01.457691+0000","last_became_active":"2023-11-08T05:28:13.563357+0000","last_became_peered":"2023-11-08T05:28:13.563357+0000","last_unstale":"2023-11-08T05:29:43.389390+0000","last_undegraded":"2023-11-08T05:29:43.389390+0000","last_fullsized":"2023-11-08T05:28:13.541614+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":456,"reported_epoch":455,"state":"active+undersized","last_fresh":"2023-11-08T05:29:43.389311+0000","last_change":"2023-11-08T05:28:13.550672+0000","last_active":"2023-11-08T05:29:43.389311+0000","last_peered":"2023-11-08T05:29:43.389311+0000","last_clean":"2023-11-08T05:28:01.457904+0000","last_became_active":"2023-11-08T05:28:13.550672+0000","last_became_peered":"2023-11-08T05:28:13.550672+0000","last_unstale":"2023-11-08T05:29:43.389311+0000","last_undegraded":"2023-11-08T05:29:43.389311+0000","last_fullsized":"2023-11-08T05:28:13.539595+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":456,"reported_epoch":455,"state":"active+undersized","last_fresh":"2023-11-08T05:29:43.389474+0000","last_change":"2023-11-08T05:28:13.557968+0000","last_active":"2023-11-08T05:29:43.389474+0000","last_peered":"2023-11-08T05:29:43.389474+0000","last_clean":"2023-11-08T05:28:01.458433+0000","last_became_active":"2023-11-08T05:28:13.557968+0000","last_became_peered":"2023-11-08T05:28:13.557968+0000","last_unstale":"2023-11-08T05:29:43.389474+0000","last_undegraded":"2023-11-08T05:29:43.389474+0000","last_fullsized":"2023-11-08T05:28:13.542832+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":1,"acting":1,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":10,"acting":11,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738818,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":645004,"kb_used_data":5316,"kb_used_omap":275,"kb_used_meta":639404,"kb_avail":93726836,"statfs":{"total":96636764160,"available":95976280064,"internally_reserved":0,"allocated":5443584,"data_stored":2678316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":281650,"internal_metadata":654750670},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:59 2023","interfaces":[{"interface":"back","average":{"1min":0.30499999999999999,"5min":0.439,"15min":0.36099999999999999},"min":{"1min":0.17399999999999999,"5min":0.14499999999999999,"15min":0.14499999999999999},"max":{"1min":0.60099999999999998,"5min":4.9379999999999997,"15min":4.9379999999999997},"last":0.379},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.49399999999999999,"15min":0.42399999999999999},"min":{"1min":0.221,"5min":0.19600000000000001,"15min":0.17000000000000001},"max":{"1min":0.63200000000000001,"5min":4.9859999999999998,"15min":4.9859999999999998},"last":0.40899999999999997}]},{"osd":1,"last update":"Wed Nov 8 05:29:01 2023","interfaces":[{"interface":"back","average":{"1min":0.30299999999999999,"5min":0.81899999999999995,"15min":0.47399999999999998},"min":{"1min":0.16,"5min":0.153,"15min":0.153},"max":{"1min":0.68200000000000005,"5min":23.242999999999999,"15min":23.242999999999999},"last":0.26900000000000002},{"interface":"front","average":{"1min":0.46500000000000002,"5min":0.72199999999999998,"15min":0.46100000000000002},"min":{"1min":0.17699999999999999,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":2.0419999999999998,"5min":23.251000000000001,"15min":23.251000000000001},"last":0.23599999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738798,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1339228,"kb_used_data":398304,"kb_used_omap":291,"kb_used_meta":936476,"kb_avail":93032612,"statfs":{"total":96636764160,"available":95265394688,"internally_reserved":0,"allocated":407863296,"data_stored":405103900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":298462,"internal_metadata":958951970},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.436,"15min":0.36299999999999999},"min":{"1min":0.19700000000000001,"5min":0.14699999999999999,"15min":0.13700000000000001},"max":{"1min":3.222,"5min":6.5499999999999998,"15min":6.9989999999999997},"last":0.245},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.46300000000000002,"15min":0.42299999999999999},"min":{"1min":0.17799999999999999,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":5.0030000000000001,"5min":6.0629999999999997,"15min":9.6869999999999994},"last":0.28599999999999998}]},{"osd":2,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.51600000000000001,"15min":0.47099999999999997},"min":{"1min":0.21099999999999999,"5min":0.153,"15min":0.153},"max":{"1min":5.7439999999999998,"5min":5.8280000000000003,"15min":9.7639999999999993},"last":0.252},{"interface":"front","average":{"1min":0.497,"5min":0.56200000000000006,"15min":0.42299999999999999},"min":{"1min":0.20399999999999999,"5min":0.187,"15min":0.16800000000000001},"max":{"1min":3.9319999999999999,"5min":8.8040000000000003,"15min":8.8040000000000003},"last":0.29899999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738820,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":650064,"kb_used_data":5320,"kb_used_omap":214,"kb_used_meta":644521,"kb_avail":93721776,"statfs":{"total":96636764160,"available":95971098624,"internally_reserved":0,"allocated":5447680,"data_stored":2679285,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":219335,"internal_metadata":659990329},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:56 2023","interfaces":[{"interface":"back","average":{"1min":0.30399999999999999,"5min":0.34699999999999998,"15min":0.36499999999999999},"min":{"1min":0.23999999999999999,"5min":0.14199999999999999,"15min":0.14199999999999999},"max":{"1min":0.42399999999999999,"5min":1.595,"15min":7.4249999999999998},"last":0.17100000000000001},{"interface":"front","average":{"1min":0.35499999999999998,"5min":0.40899999999999997,"15min":0.376},"min":{"1min":0.23999999999999999,"5min":0.19600000000000001,"15min":0.183},"max":{"1min":0.46999999999999997,"5min":2.6859999999999999,"15min":2.6859999999999999},"last":0.23000000000000001}]},{"osd":2,"last update":"Wed Nov 8 05:29:57 2023","interfaces":[{"interface":"back","average":{"1min":0.28199999999999997,"5min":0.36099999999999999,"15min":0.373},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":0.34399999999999997,"5min":2.2650000000000001,"15min":5.8209999999999997},"last":0.32300000000000001},{"interface":"front","average":{"1min":0.28599999999999998,"5min":0.31,"15min":0.35799999999999998},"min":{"1min":0.17199999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.35699999999999998,"5min":1.137,"15min":4.2350000000000003},"last":0.33500000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-08T05:30:02.559 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-11-08T05:30:02.841 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T05:30:02.842 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2023-11-08T05:30:02.853 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":1572,"stamp":"2023-11-08T05:30:02.051022+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2634296,"kb_used_data":408940,"kb_used_omap":780,"kb_used_meta":2220403,"kb_avail":280481224,"statfs":{"total":289910292480,"available":287212773376,"internally_reserved":0,"allocated":418754560,"data_stored":410461501,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":799447,"internal_metadata":2273692969},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-536782902,"num_objects":-164,"num_object_clones":0,"num_object_copies":-328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-115,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-164,"num_whiteouts":0,"num_read":-122056,"num_read_kb":-39011942,"num_write":-413642,"num_write_kb":-151705339,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-8,"num_bytes_recovered":-16922624,"num_keys_recovered":-2,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.424817"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":456,"reported_epoch":455,"state":"active+undersized","last_fresh":"2023-11-08T05:29:43.389354+0000","last_change":"2023-11-08T05:28:13.557071+0000","last_active":"2023-11-08T05:29:43.389354+0000","last_peered":"2023-11-08T05:29:43.389354+0000","last_clean":"2023-11-08T05:28:01.458439+0000","last_became_active":"2023-11-08T05:28:13.557071+0000","last_became_peered":"2023-11-08T05:28:13.557071+0000","last_unstale":"2023-11-08T05:29:43.389354+0000","last_undegraded":"2023-11-08T05:29:43.389354+0000","last_fullsized":"2023-11-08T05:28:13.547025+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":18,"reported_epoch":455,"state":"active+undersized","last_fresh":"2023-11-08T05:29:43.389262+0000","last_change":"2023-11-08T05:28:13.559513+0000","last_active":"2023-11-08T05:29:43.389262+0000","last_peered":"2023-11-08T05:29:43.389262+0000","last_clean":"2023-11-08T04:52:46.531964+0000","last_became_active":"2023-11-08T05:28:13.559513+0000","last_became_peered":"2023-11-08T05:28:13.559513+0000","last_unstale":"2023-11-08T05:29:43.389262+0000","last_undegraded":"2023-11-08T05:29:43.389262+0000","last_fullsized":"2023-11-08T05:28:13.540852+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":453,"reported_epoch":455,"state":"active+clean","last_fresh":"2023-11-08T05:29:43.390991+0000","last_change":"2023-11-08T04:52:47.546100+0000","last_active":"2023-11-08T05:29:43.390991+0000","last_peered":"2023-11-08T05:29:43.390991+0000","last_clean":"2023-11-08T05:29:43.390991+0000","last_became_active":"2023-11-08T04:52:47.545840+0000","last_became_peered":"2023-11-08T04:52:47.545840+0000","last_unstale":"2023-11-08T05:29:43.390991+0000","last_undegraded":"2023-11-08T05:29:43.390991+0000","last_fullsized":"2023-11-08T05:29:43.390991+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":451,"reported_epoch":455,"state":"active+clean","last_fresh":"2023-11-08T05:29:43.389261+0000","last_change":"2023-11-08T04:52:47.548597+0000","last_active":"2023-11-08T05:29:43.389261+0000","last_peered":"2023-11-08T05:29:43.389261+0000","last_clean":"2023-11-08T05:29:43.389261+0000","last_became_active":"2023-11-08T04:52:47.548294+0000","last_became_peered":"2023-11-08T04:52:47.548294+0000","last_unstale":"2023-11-08T05:29:43.389261+0000","last_undegraded":"2023-11-08T05:29:43.389261+0000","last_fullsized":"2023-11-08T05:29:43.389261+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":457,"reported_epoch":455,"state":"active+undersized","last_fresh":"2023-11-08T05:29:43.389822+0000","last_change":"2023-11-08T05:28:13.545468+0000","last_active":"2023-11-08T05:29:43.389822+0000","last_peered":"2023-11-08T05:29:43.389822+0000","last_clean":"2023-11-08T05:28:01.456723+0000","last_became_active":"2023-11-08T05:28:13.545468+0000","last_became_peered":"2023-11-08T05:28:13.545468+0000","last_unstale":"2023-11-08T05:29:43.389822+0000","last_undegraded":"2023-11-08T05:29:43.389822+0000","last_fullsized":"2023-11-08T05:28:13.537207+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_clean_scrub_stamp":"2023-11-08T04:52:45.528796+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":462,"reported_epoch":456,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:29:44.395350+0000","last_change":"2023-11-08T05:29:44.395350+0000","last_active":"2023-11-08T05:29:44.395350+0000","last_peered":"2023-11-08T05:29:44.395350+0000","last_clean":"2023-11-08T05:29:44.395350+0000","last_became_active":"2023-11-08T05:29:44.395012+0000","last_became_peered":"2023-11-08T05:29:44.395012+0000","last_unstale":"2023-11-08T05:29:44.395350+0000","last_undegraded":"2023-11-08T05:29:44.395350+0000","last_fullsized":"2023-11-08T05:29:44.395350+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":456,"reported_epoch":455,"state":"active+undersized","last_fresh":"2023-11-08T05:29:43.389390+0000","last_change":"2023-11-08T05:28:13.563357+0000","last_active":"2023-11-08T05:29:43.389390+0000","last_peered":"2023-11-08T05:29:43.389390+0000","last_clean":"2023-11-08T05:28:01.457691+0000","last_became_active":"2023-11-08T05:28:13.563357+0000","last_became_peered":"2023-11-08T05:28:13.563357+0000","last_unstale":"2023-11-08T05:29:43.389390+0000","last_undegraded":"2023-11-08T05:29:43.389390+0000","last_fullsized":"2023-11-08T05:28:13.541614+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":456,"reported_epoch":455,"state":"active+undersized","last_fresh":"2023-11-08T05:29:43.389311+0000","last_change":"2023-11-08T05:28:13.550672+0000","last_active":"2023-11-08T05:29:43.389311+0000","last_peered":"2023-11-08T05:29:43.389311+0000","last_clean":"2023-11-08T05:28:01.457904+0000","last_became_active":"2023-11-08T05:28:13.550672+0000","last_became_peered":"2023-11-08T05:28:13.550672+0000","last_unstale":"2023-11-08T05:29:43.389311+0000","last_undegraded":"2023-11-08T05:29:43.389311+0000","last_fullsized":"2023-11-08T05:28:13.539595+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":456,"reported_epoch":455,"state":"active+undersized","last_fresh":"2023-11-08T05:29:43.389474+0000","last_change":"2023-11-08T05:28:13.557968+0000","last_active":"2023-11-08T05:29:43.389474+0000","last_peered":"2023-11-08T05:29:43.389474+0000","last_clean":"2023-11-08T05:28:01.458433+0000","last_became_active":"2023-11-08T05:28:13.557968+0000","last_became_peered":"2023-11-08T05:28:13.557968+0000","last_unstale":"2023-11-08T05:29:43.389474+0000","last_undegraded":"2023-11-08T05:29:43.389474+0000","last_fullsized":"2023-11-08T05:28:13.542832+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":1,"acting":1,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":10,"acting":11,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738818,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":645004,"kb_used_data":5316,"kb_used_omap":275,"kb_used_meta":639404,"kb_avail":93726836,"statfs":{"total":96636764160,"available":95976280064,"internally_reserved":0,"allocated":5443584,"data_stored":2678316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":281650,"internal_metadata":654750670},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:59 2023","interfaces":[{"interface":"back","average":{"1min":0.30499999999999999,"5min":0.439,"15min":0.36099999999999999},"min":{"1min":0.17399999999999999,"5min":0.14499999999999999,"15min":0.14499999999999999},"max":{"1min":0.60099999999999998,"5min":4.9379999999999997,"15min":4.9379999999999997},"last":0.379},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.49399999999999999,"15min":0.42399999999999999},"min":{"1min":0.221,"5min":0.19600000000000001,"15min":0.17000000000000001},"max":{"1min":0.63200000000000001,"5min":4.9859999999999998,"15min":4.9859999999999998},"last":0.40899999999999997}]},{"osd":1,"last update":"Wed Nov 8 05:29:01 2023","interfaces":[{"interface":"back","average":{"1min":0.30299999999999999,"5min":0.81899999999999995,"15min":0.47399999999999998},"min":{"1min":0.16,"5min":0.153,"15min":0.153},"max":{"1min":0.68200000000000005,"5min":23.242999999999999,"15min":23.242999999999999},"last":0.26900000000000002},{"interface":"front","average":{"1min":0.46500000000000002,"5min":0.72199999999999998,"15min":0.46100000000000002},"min":{"1min":0.17699999999999999,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":2.0419999999999998,"5min":23.251000000000001,"15min":23.251000000000001},"last":0.23599999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738798,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1339228,"kb_used_data":398304,"kb_used_omap":291,"kb_used_meta":936476,"kb_avail":93032612,"statfs":{"total":96636764160,"available":95265394688,"internally_reserved":0,"allocated":407863296,"data_stored":405103900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":298462,"internal_metadata":958951970},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.436,"15min":0.36299999999999999},"min":{"1min":0.19700000000000001,"5min":0.14699999999999999,"15min":0.13700000000000001},"max":{"1min":3.222,"5min":6.5499999999999998,"15min":6.9989999999999997},"last":0.245},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.46300000000000002,"15min":0.42299999999999999},"min":{"1min":0.17799999999999999,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":5.0030000000000001,"5min":6.0629999999999997,"15min":9.6869999999999994},"last":0.28599999999999998}]},{"osd":2,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.51600000000000001,"15min":0.47099999999999997},"min":{"1min":0.21099999999999999,"5min":0.153,"15min":0.153},"max":{"1min":5.7439999999999998,"5min":5.8280000000000003,"15min":9.7639999999999993},"last":0.252},{"interface":"front","average":{"1min":0.497,"5min":0.56200000000000006,"15min":0.42299999999999999},"min":{"1min":0.20399999999999999,"5min":0.187,"15min":0.16800000000000001},"max":{"1min":3.9319999999999999,"5min":8.8040000000000003,"15min":8.8040000000000003},"last":0.29899999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738820,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":650064,"kb_used_data":5320,"kb_used_omap":214,"kb_used_meta":644521,"kb_avail":93721776,"statfs":{"total":96636764160,"available":95971098624,"internally_reserved":0,"allocated":5447680,"data_stored":2679285,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":219335,"internal_metadata":659990329},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:56 2023","interfaces":[{"interface":"back","average":{"1min":0.30399999999999999,"5min":0.34699999999999998,"15min":0.36499999999999999},"min":{"1min":0.23999999999999999,"5min":0.14199999999999999,"15min":0.14199999999999999},"max":{"1min":0.42399999999999999,"5min":1.595,"15min":7.4249999999999998},"last":0.17100000000000001},{"interface":"front","average":{"1min":0.35499999999999998,"5min":0.40899999999999997,"15min":0.376},"min":{"1min":0.23999999999999999,"5min":0.19600000000000001,"15min":0.183},"max":{"1min":0.46999999999999997,"5min":2.6859999999999999,"15min":2.6859999999999999},"last":0.23000000000000001}]},{"osd":2,"last update":"Wed Nov 8 05:29:57 2023","interfaces":[{"interface":"back","average":{"1min":0.28199999999999997,"5min":0.36099999999999999,"15min":0.373},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":0.34399999999999997,"5min":2.2650000000000001,"15min":5.8209999999999997},"last":0.32300000000000001},{"interface":"front","average":{"1min":0.28599999999999998,"5min":0.31,"15min":0.35799999999999998},"min":{"1min":0.17199999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.35699999999999998,"5min":1.137,"15min":4.2350000000000003},"last":0.33500000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-08T05:30:02.855 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-11-08T05:30:02.855 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 456, 'reported_epoch': 455, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:29:43.389354+0000', 'last_change': '2023-11-08T05:28:13.557071+0000', 'last_active': '2023-11-08T05:29:43.389354+0000', 'last_peered': '2023-11-08T05:29:43.389354+0000', 'last_clean': '2023-11-08T05:28:01.458439+0000', 'last_became_active': '2023-11-08T05:28:13.557071+0000', 'last_became_peered': '2023-11-08T05:28:13.557071+0000', 'last_unstale': '2023-11-08T05:29:43.389354+0000', 'last_undegraded': '2023-11-08T05:29:43.389354+0000', 'last_fullsized': '2023-11-08T05:28:13.547025+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:30:02.855 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-11-08T05:30:02.856 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 18, 'reported_epoch': 455, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:29:43.389262+0000', 'last_change': '2023-11-08T05:28:13.559513+0000', 'last_active': '2023-11-08T05:29:43.389262+0000', 'last_peered': '2023-11-08T05:29:43.389262+0000', 'last_clean': '2023-11-08T04:52:46.531964+0000', 'last_became_active': '2023-11-08T05:28:13.559513+0000', 'last_became_peered': '2023-11-08T05:28:13.559513+0000', 'last_unstale': '2023-11-08T05:29:43.389262+0000', 'last_undegraded': '2023-11-08T05:29:43.389262+0000', 'last_fullsized': '2023-11-08T05:28:13.540852+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:30:02.856 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-11-08T05:30:02.856 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "0'0", 'reported_seq': 457, 'reported_epoch': 455, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:29:43.389822+0000', 'last_change': '2023-11-08T05:28:13.545468+0000', 'last_active': '2023-11-08T05:29:43.389822+0000', 'last_peered': '2023-11-08T05:29:43.389822+0000', 'last_clean': '2023-11-08T05:28:01.456723+0000', 'last_became_active': '2023-11-08T05:28:13.545468+0000', 'last_became_peered': '2023-11-08T05:28:13.545468+0000', 'last_unstale': '2023-11-08T05:29:43.389822+0000', 'last_undegraded': '2023-11-08T05:29:43.389822+0000', 'last_fullsized': '2023-11-08T05:28:13.537207+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 9, 'last_epoch_clean': 10, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:45.528796+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:45.528796+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:45.528796+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:30:02.857 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.0 is not active+clean 2023-11-08T05:30:02.857 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.0', 'version': "0'0", 'reported_seq': 462, 'reported_epoch': 456, 'state': 'active+clean+remapped', 'last_fresh': '2023-11-08T05:29:44.395350+0000', 'last_change': '2023-11-08T05:29:44.395350+0000', 'last_active': '2023-11-08T05:29:44.395350+0000', 'last_peered': '2023-11-08T05:29:44.395350+0000', 'last_clean': '2023-11-08T05:29:44.395350+0000', 'last_became_active': '2023-11-08T05:29:44.395012+0000', 'last_became_peered': '2023-11-08T05:29:44.395012+0000', 'last_unstale': '2023-11-08T05:29:44.395350+0000', 'last_undegraded': '2023-11-08T05:29:44.395350+0000', 'last_fullsized': '2023-11-08T05:29:44.395350+0000', 'mapping_epoch': 455, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 456, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2], 'acting': [2, 1], 'avail_no_missing': ['2', '1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2023-11-08T05:30:02.857 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-11-08T05:30:02.858 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 456, 'reported_epoch': 455, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:29:43.389390+0000', 'last_change': '2023-11-08T05:28:13.563357+0000', 'last_active': '2023-11-08T05:29:43.389390+0000', 'last_peered': '2023-11-08T05:29:43.389390+0000', 'last_clean': '2023-11-08T05:28:01.457691+0000', 'last_became_active': '2023-11-08T05:28:13.563357+0000', 'last_became_peered': '2023-11-08T05:28:13.563357+0000', 'last_unstale': '2023-11-08T05:29:43.389390+0000', 'last_undegraded': '2023-11-08T05:29:43.389390+0000', 'last_fullsized': '2023-11-08T05:28:13.541614+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:30:02.858 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-11-08T05:30:02.858 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 456, 'reported_epoch': 455, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:29:43.389311+0000', 'last_change': '2023-11-08T05:28:13.550672+0000', 'last_active': '2023-11-08T05:29:43.389311+0000', 'last_peered': '2023-11-08T05:29:43.389311+0000', 'last_clean': '2023-11-08T05:28:01.457904+0000', 'last_became_active': '2023-11-08T05:28:13.550672+0000', 'last_became_peered': '2023-11-08T05:28:13.550672+0000', 'last_unstale': '2023-11-08T05:29:43.389311+0000', 'last_undegraded': '2023-11-08T05:29:43.389311+0000', 'last_fullsized': '2023-11-08T05:28:13.539595+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:30:02.859 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-11-08T05:30:02.859 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 456, 'reported_epoch': 455, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:29:43.389474+0000', 'last_change': '2023-11-08T05:28:13.557968+0000', 'last_active': '2023-11-08T05:29:43.389474+0000', 'last_peered': '2023-11-08T05:29:43.389474+0000', 'last_clean': '2023-11-08T05:28:01.458433+0000', 'last_became_active': '2023-11-08T05:28:13.557968+0000', 'last_became_peered': '2023-11-08T05:28:13.557968+0000', 'last_unstale': '2023-11-08T05:29:43.389474+0000', 'last_undegraded': '2023-11-08T05:29:43.389474+0000', 'last_fullsized': '2023-11-08T05:28:13.542832+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:30:02.860 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-11-08T05:30:03.204 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T05:30:03.205 INFO:teuthology.orchestra.run.smithi087.stdout:{"fsid":"ab4cbc06-449d-497d-b513-220d9db2b590","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 6 pgs undersized","count":6},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2245,"monmap":{"epoch":1,"min_mon_release_name":"pacific","num_mons":1},"osdmap":{"epoch":456,"num_osds":3,"num_up_osds":2,"osd_up_since":1699421292,"num_in_osds":3,"osd_in_since":1699419158,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":6},{"state_name":"active+clean","count":2},{"state_name":"active+clean+remapped","count":1}],"num_pgs":9,"num_pools":2,"num_objects":0,"data_bytes":0,"bytes_used":2697519104,"bytes_avail":287212773376,"bytes_total":289910292480},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":41,"modified":"2023-11-08T05:29:32.044544+0000","services":{}},"progress_events":{}} 2023-11-08T05:30:03.218 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-11-08T05:30:03.218 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-11-08T05:30:03.500 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T05:30:03.502 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2023-11-08T05:30:03.513 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":1572,"stamp":"2023-11-08T05:30:02.051022+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2634296,"kb_used_data":408940,"kb_used_omap":780,"kb_used_meta":2220403,"kb_avail":280481224,"statfs":{"total":289910292480,"available":287212773376,"internally_reserved":0,"allocated":418754560,"data_stored":410461501,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":799447,"internal_metadata":2273692969},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-536782902,"num_objects":-164,"num_object_clones":0,"num_object_copies":-328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-115,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-164,"num_whiteouts":0,"num_read":-122056,"num_read_kb":-39011942,"num_write":-413642,"num_write_kb":-151705339,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-8,"num_bytes_recovered":-16922624,"num_keys_recovered":-2,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.424817"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":456,"reported_epoch":455,"state":"active+undersized","last_fresh":"2023-11-08T05:29:43.389354+0000","last_change":"2023-11-08T05:28:13.557071+0000","last_active":"2023-11-08T05:29:43.389354+0000","last_peered":"2023-11-08T05:29:43.389354+0000","last_clean":"2023-11-08T05:28:01.458439+0000","last_became_active":"2023-11-08T05:28:13.557071+0000","last_became_peered":"2023-11-08T05:28:13.557071+0000","last_unstale":"2023-11-08T05:29:43.389354+0000","last_undegraded":"2023-11-08T05:29:43.389354+0000","last_fullsized":"2023-11-08T05:28:13.547025+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":18,"reported_epoch":455,"state":"active+undersized","last_fresh":"2023-11-08T05:29:43.389262+0000","last_change":"2023-11-08T05:28:13.559513+0000","last_active":"2023-11-08T05:29:43.389262+0000","last_peered":"2023-11-08T05:29:43.389262+0000","last_clean":"2023-11-08T04:52:46.531964+0000","last_became_active":"2023-11-08T05:28:13.559513+0000","last_became_peered":"2023-11-08T05:28:13.559513+0000","last_unstale":"2023-11-08T05:29:43.389262+0000","last_undegraded":"2023-11-08T05:29:43.389262+0000","last_fullsized":"2023-11-08T05:28:13.540852+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":453,"reported_epoch":455,"state":"active+clean","last_fresh":"2023-11-08T05:29:43.390991+0000","last_change":"2023-11-08T04:52:47.546100+0000","last_active":"2023-11-08T05:29:43.390991+0000","last_peered":"2023-11-08T05:29:43.390991+0000","last_clean":"2023-11-08T05:29:43.390991+0000","last_became_active":"2023-11-08T04:52:47.545840+0000","last_became_peered":"2023-11-08T04:52:47.545840+0000","last_unstale":"2023-11-08T05:29:43.390991+0000","last_undegraded":"2023-11-08T05:29:43.390991+0000","last_fullsized":"2023-11-08T05:29:43.390991+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":451,"reported_epoch":455,"state":"active+clean","last_fresh":"2023-11-08T05:29:43.389261+0000","last_change":"2023-11-08T04:52:47.548597+0000","last_active":"2023-11-08T05:29:43.389261+0000","last_peered":"2023-11-08T05:29:43.389261+0000","last_clean":"2023-11-08T05:29:43.389261+0000","last_became_active":"2023-11-08T04:52:47.548294+0000","last_became_peered":"2023-11-08T04:52:47.548294+0000","last_unstale":"2023-11-08T05:29:43.389261+0000","last_undegraded":"2023-11-08T05:29:43.389261+0000","last_fullsized":"2023-11-08T05:29:43.389261+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":457,"reported_epoch":455,"state":"active+undersized","last_fresh":"2023-11-08T05:29:43.389822+0000","last_change":"2023-11-08T05:28:13.545468+0000","last_active":"2023-11-08T05:29:43.389822+0000","last_peered":"2023-11-08T05:29:43.389822+0000","last_clean":"2023-11-08T05:28:01.456723+0000","last_became_active":"2023-11-08T05:28:13.545468+0000","last_became_peered":"2023-11-08T05:28:13.545468+0000","last_unstale":"2023-11-08T05:29:43.389822+0000","last_undegraded":"2023-11-08T05:29:43.389822+0000","last_fullsized":"2023-11-08T05:28:13.537207+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_clean_scrub_stamp":"2023-11-08T04:52:45.528796+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":462,"reported_epoch":456,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:29:44.395350+0000","last_change":"2023-11-08T05:29:44.395350+0000","last_active":"2023-11-08T05:29:44.395350+0000","last_peered":"2023-11-08T05:29:44.395350+0000","last_clean":"2023-11-08T05:29:44.395350+0000","last_became_active":"2023-11-08T05:29:44.395012+0000","last_became_peered":"2023-11-08T05:29:44.395012+0000","last_unstale":"2023-11-08T05:29:44.395350+0000","last_undegraded":"2023-11-08T05:29:44.395350+0000","last_fullsized":"2023-11-08T05:29:44.395350+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":456,"reported_epoch":455,"state":"active+undersized","last_fresh":"2023-11-08T05:29:43.389390+0000","last_change":"2023-11-08T05:28:13.563357+0000","last_active":"2023-11-08T05:29:43.389390+0000","last_peered":"2023-11-08T05:29:43.389390+0000","last_clean":"2023-11-08T05:28:01.457691+0000","last_became_active":"2023-11-08T05:28:13.563357+0000","last_became_peered":"2023-11-08T05:28:13.563357+0000","last_unstale":"2023-11-08T05:29:43.389390+0000","last_undegraded":"2023-11-08T05:29:43.389390+0000","last_fullsized":"2023-11-08T05:28:13.541614+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":456,"reported_epoch":455,"state":"active+undersized","last_fresh":"2023-11-08T05:29:43.389311+0000","last_change":"2023-11-08T05:28:13.550672+0000","last_active":"2023-11-08T05:29:43.389311+0000","last_peered":"2023-11-08T05:29:43.389311+0000","last_clean":"2023-11-08T05:28:01.457904+0000","last_became_active":"2023-11-08T05:28:13.550672+0000","last_became_peered":"2023-11-08T05:28:13.550672+0000","last_unstale":"2023-11-08T05:29:43.389311+0000","last_undegraded":"2023-11-08T05:29:43.389311+0000","last_fullsized":"2023-11-08T05:28:13.539595+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":456,"reported_epoch":455,"state":"active+undersized","last_fresh":"2023-11-08T05:29:43.389474+0000","last_change":"2023-11-08T05:28:13.557968+0000","last_active":"2023-11-08T05:29:43.389474+0000","last_peered":"2023-11-08T05:29:43.389474+0000","last_clean":"2023-11-08T05:28:01.458433+0000","last_became_active":"2023-11-08T05:28:13.557968+0000","last_became_peered":"2023-11-08T05:28:13.557968+0000","last_unstale":"2023-11-08T05:29:43.389474+0000","last_undegraded":"2023-11-08T05:29:43.389474+0000","last_fullsized":"2023-11-08T05:28:13.542832+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":1,"acting":1,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":10,"acting":11,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738818,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":645004,"kb_used_data":5316,"kb_used_omap":275,"kb_used_meta":639404,"kb_avail":93726836,"statfs":{"total":96636764160,"available":95976280064,"internally_reserved":0,"allocated":5443584,"data_stored":2678316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":281650,"internal_metadata":654750670},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:59 2023","interfaces":[{"interface":"back","average":{"1min":0.30499999999999999,"5min":0.439,"15min":0.36099999999999999},"min":{"1min":0.17399999999999999,"5min":0.14499999999999999,"15min":0.14499999999999999},"max":{"1min":0.60099999999999998,"5min":4.9379999999999997,"15min":4.9379999999999997},"last":0.379},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.49399999999999999,"15min":0.42399999999999999},"min":{"1min":0.221,"5min":0.19600000000000001,"15min":0.17000000000000001},"max":{"1min":0.63200000000000001,"5min":4.9859999999999998,"15min":4.9859999999999998},"last":0.40899999999999997}]},{"osd":1,"last update":"Wed Nov 8 05:29:01 2023","interfaces":[{"interface":"back","average":{"1min":0.30299999999999999,"5min":0.81899999999999995,"15min":0.47399999999999998},"min":{"1min":0.16,"5min":0.153,"15min":0.153},"max":{"1min":0.68200000000000005,"5min":23.242999999999999,"15min":23.242999999999999},"last":0.26900000000000002},{"interface":"front","average":{"1min":0.46500000000000002,"5min":0.72199999999999998,"15min":0.46100000000000002},"min":{"1min":0.17699999999999999,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":2.0419999999999998,"5min":23.251000000000001,"15min":23.251000000000001},"last":0.23599999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738798,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1339228,"kb_used_data":398304,"kb_used_omap":291,"kb_used_meta":936476,"kb_avail":93032612,"statfs":{"total":96636764160,"available":95265394688,"internally_reserved":0,"allocated":407863296,"data_stored":405103900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":298462,"internal_metadata":958951970},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.436,"15min":0.36299999999999999},"min":{"1min":0.19700000000000001,"5min":0.14699999999999999,"15min":0.13700000000000001},"max":{"1min":3.222,"5min":6.5499999999999998,"15min":6.9989999999999997},"last":0.245},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.46300000000000002,"15min":0.42299999999999999},"min":{"1min":0.17799999999999999,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":5.0030000000000001,"5min":6.0629999999999997,"15min":9.6869999999999994},"last":0.28599999999999998}]},{"osd":2,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.51600000000000001,"15min":0.47099999999999997},"min":{"1min":0.21099999999999999,"5min":0.153,"15min":0.153},"max":{"1min":5.7439999999999998,"5min":5.8280000000000003,"15min":9.7639999999999993},"last":0.252},{"interface":"front","average":{"1min":0.497,"5min":0.56200000000000006,"15min":0.42299999999999999},"min":{"1min":0.20399999999999999,"5min":0.187,"15min":0.16800000000000001},"max":{"1min":3.9319999999999999,"5min":8.8040000000000003,"15min":8.8040000000000003},"last":0.29899999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738820,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":650064,"kb_used_data":5320,"kb_used_omap":214,"kb_used_meta":644521,"kb_avail":93721776,"statfs":{"total":96636764160,"available":95971098624,"internally_reserved":0,"allocated":5447680,"data_stored":2679285,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":219335,"internal_metadata":659990329},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:56 2023","interfaces":[{"interface":"back","average":{"1min":0.30399999999999999,"5min":0.34699999999999998,"15min":0.36499999999999999},"min":{"1min":0.23999999999999999,"5min":0.14199999999999999,"15min":0.14199999999999999},"max":{"1min":0.42399999999999999,"5min":1.595,"15min":7.4249999999999998},"last":0.17100000000000001},{"interface":"front","average":{"1min":0.35499999999999998,"5min":0.40899999999999997,"15min":0.376},"min":{"1min":0.23999999999999999,"5min":0.19600000000000001,"15min":0.183},"max":{"1min":0.46999999999999997,"5min":2.6859999999999999,"15min":2.6859999999999999},"last":0.23000000000000001}]},{"osd":2,"last update":"Wed Nov 8 05:29:57 2023","interfaces":[{"interface":"back","average":{"1min":0.28199999999999997,"5min":0.36099999999999999,"15min":0.373},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":0.34399999999999997,"5min":2.2650000000000001,"15min":5.8209999999999997},"last":0.32300000000000001},{"interface":"front","average":{"1min":0.28599999999999998,"5min":0.31,"15min":0.35799999999999998},"min":{"1min":0.17199999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.35699999999999998,"5min":1.137,"15min":4.2350000000000003},"last":0.33500000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-08T05:30:06.126 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~108s 2023-11-08T05:30:06.514 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-11-08T05:30:06.811 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T05:30:06.813 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2023-11-08T05:30:06.824 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":1574,"stamp":"2023-11-08T05:30:06.051777+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2634296,"kb_used_data":408940,"kb_used_omap":780,"kb_used_meta":2220403,"kb_avail":280481224,"statfs":{"total":289910292480,"available":287212773376,"internally_reserved":0,"allocated":418754560,"data_stored":410461501,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":799447,"internal_metadata":2273692969},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-536782902,"num_objects":-164,"num_object_clones":0,"num_object_copies":-328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-115,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-164,"num_whiteouts":0,"num_read":-122056,"num_read_kb":-39011942,"num_write":-413642,"num_write_kb":-151705339,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-8,"num_bytes_recovered":-16922624,"num_keys_recovered":-2,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.424817"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":456,"reported_epoch":455,"state":"active+undersized","last_fresh":"2023-11-08T05:29:43.389354+0000","last_change":"2023-11-08T05:28:13.557071+0000","last_active":"2023-11-08T05:29:43.389354+0000","last_peered":"2023-11-08T05:29:43.389354+0000","last_clean":"2023-11-08T05:28:01.458439+0000","last_became_active":"2023-11-08T05:28:13.557071+0000","last_became_peered":"2023-11-08T05:28:13.557071+0000","last_unstale":"2023-11-08T05:29:43.389354+0000","last_undegraded":"2023-11-08T05:29:43.389354+0000","last_fullsized":"2023-11-08T05:28:13.547025+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":18,"reported_epoch":455,"state":"active+undersized","last_fresh":"2023-11-08T05:29:43.389262+0000","last_change":"2023-11-08T05:28:13.559513+0000","last_active":"2023-11-08T05:29:43.389262+0000","last_peered":"2023-11-08T05:29:43.389262+0000","last_clean":"2023-11-08T04:52:46.531964+0000","last_became_active":"2023-11-08T05:28:13.559513+0000","last_became_peered":"2023-11-08T05:28:13.559513+0000","last_unstale":"2023-11-08T05:29:43.389262+0000","last_undegraded":"2023-11-08T05:29:43.389262+0000","last_fullsized":"2023-11-08T05:28:13.540852+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":453,"reported_epoch":455,"state":"active+clean","last_fresh":"2023-11-08T05:29:43.390991+0000","last_change":"2023-11-08T04:52:47.546100+0000","last_active":"2023-11-08T05:29:43.390991+0000","last_peered":"2023-11-08T05:29:43.390991+0000","last_clean":"2023-11-08T05:29:43.390991+0000","last_became_active":"2023-11-08T04:52:47.545840+0000","last_became_peered":"2023-11-08T04:52:47.545840+0000","last_unstale":"2023-11-08T05:29:43.390991+0000","last_undegraded":"2023-11-08T05:29:43.390991+0000","last_fullsized":"2023-11-08T05:29:43.390991+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":451,"reported_epoch":455,"state":"active+clean","last_fresh":"2023-11-08T05:29:43.389261+0000","last_change":"2023-11-08T04:52:47.548597+0000","last_active":"2023-11-08T05:29:43.389261+0000","last_peered":"2023-11-08T05:29:43.389261+0000","last_clean":"2023-11-08T05:29:43.389261+0000","last_became_active":"2023-11-08T04:52:47.548294+0000","last_became_peered":"2023-11-08T04:52:47.548294+0000","last_unstale":"2023-11-08T05:29:43.389261+0000","last_undegraded":"2023-11-08T05:29:43.389261+0000","last_fullsized":"2023-11-08T05:29:43.389261+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":457,"reported_epoch":455,"state":"active+undersized","last_fresh":"2023-11-08T05:29:43.389822+0000","last_change":"2023-11-08T05:28:13.545468+0000","last_active":"2023-11-08T05:29:43.389822+0000","last_peered":"2023-11-08T05:29:43.389822+0000","last_clean":"2023-11-08T05:28:01.456723+0000","last_became_active":"2023-11-08T05:28:13.545468+0000","last_became_peered":"2023-11-08T05:28:13.545468+0000","last_unstale":"2023-11-08T05:29:43.389822+0000","last_undegraded":"2023-11-08T05:29:43.389822+0000","last_fullsized":"2023-11-08T05:28:13.537207+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_clean_scrub_stamp":"2023-11-08T04:52:45.528796+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":462,"reported_epoch":456,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:29:44.395350+0000","last_change":"2023-11-08T05:29:44.395350+0000","last_active":"2023-11-08T05:29:44.395350+0000","last_peered":"2023-11-08T05:29:44.395350+0000","last_clean":"2023-11-08T05:29:44.395350+0000","last_became_active":"2023-11-08T05:29:44.395012+0000","last_became_peered":"2023-11-08T05:29:44.395012+0000","last_unstale":"2023-11-08T05:29:44.395350+0000","last_undegraded":"2023-11-08T05:29:44.395350+0000","last_fullsized":"2023-11-08T05:29:44.395350+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":456,"reported_epoch":455,"state":"active+undersized","last_fresh":"2023-11-08T05:29:43.389390+0000","last_change":"2023-11-08T05:28:13.563357+0000","last_active":"2023-11-08T05:29:43.389390+0000","last_peered":"2023-11-08T05:29:43.389390+0000","last_clean":"2023-11-08T05:28:01.457691+0000","last_became_active":"2023-11-08T05:28:13.563357+0000","last_became_peered":"2023-11-08T05:28:13.563357+0000","last_unstale":"2023-11-08T05:29:43.389390+0000","last_undegraded":"2023-11-08T05:29:43.389390+0000","last_fullsized":"2023-11-08T05:28:13.541614+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":456,"reported_epoch":455,"state":"active+undersized","last_fresh":"2023-11-08T05:29:43.389311+0000","last_change":"2023-11-08T05:28:13.550672+0000","last_active":"2023-11-08T05:29:43.389311+0000","last_peered":"2023-11-08T05:29:43.389311+0000","last_clean":"2023-11-08T05:28:01.457904+0000","last_became_active":"2023-11-08T05:28:13.550672+0000","last_became_peered":"2023-11-08T05:28:13.550672+0000","last_unstale":"2023-11-08T05:29:43.389311+0000","last_undegraded":"2023-11-08T05:29:43.389311+0000","last_fullsized":"2023-11-08T05:28:13.539595+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":456,"reported_epoch":455,"state":"active+undersized","last_fresh":"2023-11-08T05:29:43.389474+0000","last_change":"2023-11-08T05:28:13.557968+0000","last_active":"2023-11-08T05:29:43.389474+0000","last_peered":"2023-11-08T05:29:43.389474+0000","last_clean":"2023-11-08T05:28:01.458433+0000","last_became_active":"2023-11-08T05:28:13.557968+0000","last_became_peered":"2023-11-08T05:28:13.557968+0000","last_unstale":"2023-11-08T05:29:43.389474+0000","last_undegraded":"2023-11-08T05:29:43.389474+0000","last_fullsized":"2023-11-08T05:28:13.542832+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":1,"acting":1,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":10,"acting":11,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738819,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":645004,"kb_used_data":5316,"kb_used_omap":275,"kb_used_meta":639404,"kb_avail":93726836,"statfs":{"total":96636764160,"available":95976280064,"internally_reserved":0,"allocated":5443584,"data_stored":2678316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":281650,"internal_metadata":654750670},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:59 2023","interfaces":[{"interface":"back","average":{"1min":0.30499999999999999,"5min":0.439,"15min":0.36099999999999999},"min":{"1min":0.17399999999999999,"5min":0.14499999999999999,"15min":0.14499999999999999},"max":{"1min":0.60099999999999998,"5min":4.9379999999999997,"15min":4.9379999999999997},"last":0.379},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.49399999999999999,"15min":0.42399999999999999},"min":{"1min":0.221,"5min":0.19600000000000001,"15min":0.17000000000000001},"max":{"1min":0.63200000000000001,"5min":4.9859999999999998,"15min":4.9859999999999998},"last":0.40899999999999997}]},{"osd":1,"last update":"Wed Nov 8 05:30:03 2023","interfaces":[{"interface":"back","average":{"1min":0.28499999999999998,"5min":0.65100000000000002,"15min":0.47099999999999997},"min":{"1min":0.20100000000000001,"5min":0.153,"15min":0.153},"max":{"1min":0.39800000000000002,"5min":23.242999999999999,"15min":23.242999999999999},"last":0.20100000000000001},{"interface":"front","average":{"1min":0.29499999999999998,"5min":0.63100000000000001,"15min":0.45900000000000002},"min":{"1min":0.224,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":0.436,"5min":23.251000000000001,"15min":23.251000000000001},"last":0.224}]}]},{"osd":0,"up_from":8,"seq":34359738798,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1339228,"kb_used_data":398304,"kb_used_omap":291,"kb_used_meta":936476,"kb_avail":93032612,"statfs":{"total":96636764160,"available":95265394688,"internally_reserved":0,"allocated":407863296,"data_stored":405103900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":298462,"internal_metadata":958951970},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.436,"15min":0.36299999999999999},"min":{"1min":0.19700000000000001,"5min":0.14699999999999999,"15min":0.13700000000000001},"max":{"1min":3.222,"5min":6.5499999999999998,"15min":6.9989999999999997},"last":0.245},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.46300000000000002,"15min":0.42299999999999999},"min":{"1min":0.17799999999999999,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":5.0030000000000001,"5min":6.0629999999999997,"15min":9.6869999999999994},"last":0.28599999999999998}]},{"osd":2,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.51600000000000001,"15min":0.47099999999999997},"min":{"1min":0.21099999999999999,"5min":0.153,"15min":0.153},"max":{"1min":5.7439999999999998,"5min":5.8280000000000003,"15min":9.7639999999999993},"last":0.252},{"interface":"front","average":{"1min":0.497,"5min":0.56200000000000006,"15min":0.42299999999999999},"min":{"1min":0.20399999999999999,"5min":0.187,"15min":0.16800000000000001},"max":{"1min":3.9319999999999999,"5min":8.8040000000000003,"15min":8.8040000000000003},"last":0.29899999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738821,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":650064,"kb_used_data":5320,"kb_used_omap":214,"kb_used_meta":644521,"kb_avail":93721776,"statfs":{"total":96636764160,"available":95971098624,"internally_reserved":0,"allocated":5447680,"data_stored":2679285,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":219335,"internal_metadata":659990329},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:56 2023","interfaces":[{"interface":"back","average":{"1min":0.30399999999999999,"5min":0.34699999999999998,"15min":0.36499999999999999},"min":{"1min":0.23999999999999999,"5min":0.14199999999999999,"15min":0.14199999999999999},"max":{"1min":0.42399999999999999,"5min":1.595,"15min":7.4249999999999998},"last":0.17100000000000001},{"interface":"front","average":{"1min":0.35499999999999998,"5min":0.40899999999999997,"15min":0.376},"min":{"1min":0.23999999999999999,"5min":0.19600000000000001,"15min":0.183},"max":{"1min":0.46999999999999997,"5min":2.6859999999999999,"15min":2.6859999999999999},"last":0.23000000000000001}]},{"osd":2,"last update":"Wed Nov 8 05:29:57 2023","interfaces":[{"interface":"back","average":{"1min":0.28199999999999997,"5min":0.36099999999999999,"15min":0.373},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":0.34399999999999997,"5min":2.2650000000000001,"15min":5.8209999999999997},"last":0.248},{"interface":"front","average":{"1min":0.28599999999999998,"5min":0.31,"15min":0.35799999999999998},"min":{"1min":0.17199999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.35699999999999998,"5min":1.137,"15min":4.2350000000000003},"last":0.28399999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-08T05:30:06.825 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-11-08T05:30:07.107 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T05:30:07.108 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2023-11-08T05:30:07.119 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":1574,"stamp":"2023-11-08T05:30:06.051777+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2634296,"kb_used_data":408940,"kb_used_omap":780,"kb_used_meta":2220403,"kb_avail":280481224,"statfs":{"total":289910292480,"available":287212773376,"internally_reserved":0,"allocated":418754560,"data_stored":410461501,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":799447,"internal_metadata":2273692969},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-536782902,"num_objects":-164,"num_object_clones":0,"num_object_copies":-328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-115,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-164,"num_whiteouts":0,"num_read":-122056,"num_read_kb":-39011942,"num_write":-413642,"num_write_kb":-151705339,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-8,"num_bytes_recovered":-16922624,"num_keys_recovered":-2,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.424817"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":456,"reported_epoch":455,"state":"active+undersized","last_fresh":"2023-11-08T05:29:43.389354+0000","last_change":"2023-11-08T05:28:13.557071+0000","last_active":"2023-11-08T05:29:43.389354+0000","last_peered":"2023-11-08T05:29:43.389354+0000","last_clean":"2023-11-08T05:28:01.458439+0000","last_became_active":"2023-11-08T05:28:13.557071+0000","last_became_peered":"2023-11-08T05:28:13.557071+0000","last_unstale":"2023-11-08T05:29:43.389354+0000","last_undegraded":"2023-11-08T05:29:43.389354+0000","last_fullsized":"2023-11-08T05:28:13.547025+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":18,"reported_epoch":455,"state":"active+undersized","last_fresh":"2023-11-08T05:29:43.389262+0000","last_change":"2023-11-08T05:28:13.559513+0000","last_active":"2023-11-08T05:29:43.389262+0000","last_peered":"2023-11-08T05:29:43.389262+0000","last_clean":"2023-11-08T04:52:46.531964+0000","last_became_active":"2023-11-08T05:28:13.559513+0000","last_became_peered":"2023-11-08T05:28:13.559513+0000","last_unstale":"2023-11-08T05:29:43.389262+0000","last_undegraded":"2023-11-08T05:29:43.389262+0000","last_fullsized":"2023-11-08T05:28:13.540852+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":453,"reported_epoch":455,"state":"active+clean","last_fresh":"2023-11-08T05:29:43.390991+0000","last_change":"2023-11-08T04:52:47.546100+0000","last_active":"2023-11-08T05:29:43.390991+0000","last_peered":"2023-11-08T05:29:43.390991+0000","last_clean":"2023-11-08T05:29:43.390991+0000","last_became_active":"2023-11-08T04:52:47.545840+0000","last_became_peered":"2023-11-08T04:52:47.545840+0000","last_unstale":"2023-11-08T05:29:43.390991+0000","last_undegraded":"2023-11-08T05:29:43.390991+0000","last_fullsized":"2023-11-08T05:29:43.390991+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":451,"reported_epoch":455,"state":"active+clean","last_fresh":"2023-11-08T05:29:43.389261+0000","last_change":"2023-11-08T04:52:47.548597+0000","last_active":"2023-11-08T05:29:43.389261+0000","last_peered":"2023-11-08T05:29:43.389261+0000","last_clean":"2023-11-08T05:29:43.389261+0000","last_became_active":"2023-11-08T04:52:47.548294+0000","last_became_peered":"2023-11-08T04:52:47.548294+0000","last_unstale":"2023-11-08T05:29:43.389261+0000","last_undegraded":"2023-11-08T05:29:43.389261+0000","last_fullsized":"2023-11-08T05:29:43.389261+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":457,"reported_epoch":455,"state":"active+undersized","last_fresh":"2023-11-08T05:29:43.389822+0000","last_change":"2023-11-08T05:28:13.545468+0000","last_active":"2023-11-08T05:29:43.389822+0000","last_peered":"2023-11-08T05:29:43.389822+0000","last_clean":"2023-11-08T05:28:01.456723+0000","last_became_active":"2023-11-08T05:28:13.545468+0000","last_became_peered":"2023-11-08T05:28:13.545468+0000","last_unstale":"2023-11-08T05:29:43.389822+0000","last_undegraded":"2023-11-08T05:29:43.389822+0000","last_fullsized":"2023-11-08T05:28:13.537207+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_clean_scrub_stamp":"2023-11-08T04:52:45.528796+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":462,"reported_epoch":456,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:29:44.395350+0000","last_change":"2023-11-08T05:29:44.395350+0000","last_active":"2023-11-08T05:29:44.395350+0000","last_peered":"2023-11-08T05:29:44.395350+0000","last_clean":"2023-11-08T05:29:44.395350+0000","last_became_active":"2023-11-08T05:29:44.395012+0000","last_became_peered":"2023-11-08T05:29:44.395012+0000","last_unstale":"2023-11-08T05:29:44.395350+0000","last_undegraded":"2023-11-08T05:29:44.395350+0000","last_fullsized":"2023-11-08T05:29:44.395350+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":456,"reported_epoch":455,"state":"active+undersized","last_fresh":"2023-11-08T05:29:43.389390+0000","last_change":"2023-11-08T05:28:13.563357+0000","last_active":"2023-11-08T05:29:43.389390+0000","last_peered":"2023-11-08T05:29:43.389390+0000","last_clean":"2023-11-08T05:28:01.457691+0000","last_became_active":"2023-11-08T05:28:13.563357+0000","last_became_peered":"2023-11-08T05:28:13.563357+0000","last_unstale":"2023-11-08T05:29:43.389390+0000","last_undegraded":"2023-11-08T05:29:43.389390+0000","last_fullsized":"2023-11-08T05:28:13.541614+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":456,"reported_epoch":455,"state":"active+undersized","last_fresh":"2023-11-08T05:29:43.389311+0000","last_change":"2023-11-08T05:28:13.550672+0000","last_active":"2023-11-08T05:29:43.389311+0000","last_peered":"2023-11-08T05:29:43.389311+0000","last_clean":"2023-11-08T05:28:01.457904+0000","last_became_active":"2023-11-08T05:28:13.550672+0000","last_became_peered":"2023-11-08T05:28:13.550672+0000","last_unstale":"2023-11-08T05:29:43.389311+0000","last_undegraded":"2023-11-08T05:29:43.389311+0000","last_fullsized":"2023-11-08T05:28:13.539595+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":456,"reported_epoch":455,"state":"active+undersized","last_fresh":"2023-11-08T05:29:43.389474+0000","last_change":"2023-11-08T05:28:13.557968+0000","last_active":"2023-11-08T05:29:43.389474+0000","last_peered":"2023-11-08T05:29:43.389474+0000","last_clean":"2023-11-08T05:28:01.458433+0000","last_became_active":"2023-11-08T05:28:13.557968+0000","last_became_peered":"2023-11-08T05:28:13.557968+0000","last_unstale":"2023-11-08T05:29:43.389474+0000","last_undegraded":"2023-11-08T05:29:43.389474+0000","last_fullsized":"2023-11-08T05:28:13.542832+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":1,"acting":1,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":10,"acting":11,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738819,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":645004,"kb_used_data":5316,"kb_used_omap":275,"kb_used_meta":639404,"kb_avail":93726836,"statfs":{"total":96636764160,"available":95976280064,"internally_reserved":0,"allocated":5443584,"data_stored":2678316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":281650,"internal_metadata":654750670},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:59 2023","interfaces":[{"interface":"back","average":{"1min":0.30499999999999999,"5min":0.439,"15min":0.36099999999999999},"min":{"1min":0.17399999999999999,"5min":0.14499999999999999,"15min":0.14499999999999999},"max":{"1min":0.60099999999999998,"5min":4.9379999999999997,"15min":4.9379999999999997},"last":0.379},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.49399999999999999,"15min":0.42399999999999999},"min":{"1min":0.221,"5min":0.19600000000000001,"15min":0.17000000000000001},"max":{"1min":0.63200000000000001,"5min":4.9859999999999998,"15min":4.9859999999999998},"last":0.40899999999999997}]},{"osd":1,"last update":"Wed Nov 8 05:30:03 2023","interfaces":[{"interface":"back","average":{"1min":0.28499999999999998,"5min":0.65100000000000002,"15min":0.47099999999999997},"min":{"1min":0.20100000000000001,"5min":0.153,"15min":0.153},"max":{"1min":0.39800000000000002,"5min":23.242999999999999,"15min":23.242999999999999},"last":0.20100000000000001},{"interface":"front","average":{"1min":0.29499999999999998,"5min":0.63100000000000001,"15min":0.45900000000000002},"min":{"1min":0.224,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":0.436,"5min":23.251000000000001,"15min":23.251000000000001},"last":0.224}]}]},{"osd":0,"up_from":8,"seq":34359738798,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1339228,"kb_used_data":398304,"kb_used_omap":291,"kb_used_meta":936476,"kb_avail":93032612,"statfs":{"total":96636764160,"available":95265394688,"internally_reserved":0,"allocated":407863296,"data_stored":405103900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":298462,"internal_metadata":958951970},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.436,"15min":0.36299999999999999},"min":{"1min":0.19700000000000001,"5min":0.14699999999999999,"15min":0.13700000000000001},"max":{"1min":3.222,"5min":6.5499999999999998,"15min":6.9989999999999997},"last":0.245},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.46300000000000002,"15min":0.42299999999999999},"min":{"1min":0.17799999999999999,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":5.0030000000000001,"5min":6.0629999999999997,"15min":9.6869999999999994},"last":0.28599999999999998}]},{"osd":2,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.51600000000000001,"15min":0.47099999999999997},"min":{"1min":0.21099999999999999,"5min":0.153,"15min":0.153},"max":{"1min":5.7439999999999998,"5min":5.8280000000000003,"15min":9.7639999999999993},"last":0.252},{"interface":"front","average":{"1min":0.497,"5min":0.56200000000000006,"15min":0.42299999999999999},"min":{"1min":0.20399999999999999,"5min":0.187,"15min":0.16800000000000001},"max":{"1min":3.9319999999999999,"5min":8.8040000000000003,"15min":8.8040000000000003},"last":0.29899999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738821,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":650064,"kb_used_data":5320,"kb_used_omap":214,"kb_used_meta":644521,"kb_avail":93721776,"statfs":{"total":96636764160,"available":95971098624,"internally_reserved":0,"allocated":5447680,"data_stored":2679285,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":219335,"internal_metadata":659990329},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:56 2023","interfaces":[{"interface":"back","average":{"1min":0.30399999999999999,"5min":0.34699999999999998,"15min":0.36499999999999999},"min":{"1min":0.23999999999999999,"5min":0.14199999999999999,"15min":0.14199999999999999},"max":{"1min":0.42399999999999999,"5min":1.595,"15min":7.4249999999999998},"last":0.17100000000000001},{"interface":"front","average":{"1min":0.35499999999999998,"5min":0.40899999999999997,"15min":0.376},"min":{"1min":0.23999999999999999,"5min":0.19600000000000001,"15min":0.183},"max":{"1min":0.46999999999999997,"5min":2.6859999999999999,"15min":2.6859999999999999},"last":0.23000000000000001}]},{"osd":2,"last update":"Wed Nov 8 05:29:57 2023","interfaces":[{"interface":"back","average":{"1min":0.28199999999999997,"5min":0.36099999999999999,"15min":0.373},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":0.34399999999999997,"5min":2.2650000000000001,"15min":5.8209999999999997},"last":0.248},{"interface":"front","average":{"1min":0.28599999999999998,"5min":0.31,"15min":0.35799999999999998},"min":{"1min":0.17199999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.35699999999999998,"5min":1.137,"15min":4.2350000000000003},"last":0.28399999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-08T05:30:07.120 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-11-08T05:30:07.121 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 456, 'reported_epoch': 455, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:29:43.389354+0000', 'last_change': '2023-11-08T05:28:13.557071+0000', 'last_active': '2023-11-08T05:29:43.389354+0000', 'last_peered': '2023-11-08T05:29:43.389354+0000', 'last_clean': '2023-11-08T05:28:01.458439+0000', 'last_became_active': '2023-11-08T05:28:13.557071+0000', 'last_became_peered': '2023-11-08T05:28:13.557071+0000', 'last_unstale': '2023-11-08T05:29:43.389354+0000', 'last_undegraded': '2023-11-08T05:29:43.389354+0000', 'last_fullsized': '2023-11-08T05:28:13.547025+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:30:07.121 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-11-08T05:30:07.121 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 18, 'reported_epoch': 455, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:29:43.389262+0000', 'last_change': '2023-11-08T05:28:13.559513+0000', 'last_active': '2023-11-08T05:29:43.389262+0000', 'last_peered': '2023-11-08T05:29:43.389262+0000', 'last_clean': '2023-11-08T04:52:46.531964+0000', 'last_became_active': '2023-11-08T05:28:13.559513+0000', 'last_became_peered': '2023-11-08T05:28:13.559513+0000', 'last_unstale': '2023-11-08T05:29:43.389262+0000', 'last_undegraded': '2023-11-08T05:29:43.389262+0000', 'last_fullsized': '2023-11-08T05:28:13.540852+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:30:07.122 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-11-08T05:30:07.122 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "0'0", 'reported_seq': 457, 'reported_epoch': 455, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:29:43.389822+0000', 'last_change': '2023-11-08T05:28:13.545468+0000', 'last_active': '2023-11-08T05:29:43.389822+0000', 'last_peered': '2023-11-08T05:29:43.389822+0000', 'last_clean': '2023-11-08T05:28:01.456723+0000', 'last_became_active': '2023-11-08T05:28:13.545468+0000', 'last_became_peered': '2023-11-08T05:28:13.545468+0000', 'last_unstale': '2023-11-08T05:29:43.389822+0000', 'last_undegraded': '2023-11-08T05:29:43.389822+0000', 'last_fullsized': '2023-11-08T05:28:13.537207+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 9, 'last_epoch_clean': 10, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:45.528796+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:45.528796+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:45.528796+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:30:07.122 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.0 is not active+clean 2023-11-08T05:30:07.123 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.0', 'version': "0'0", 'reported_seq': 462, 'reported_epoch': 456, 'state': 'active+clean+remapped', 'last_fresh': '2023-11-08T05:29:44.395350+0000', 'last_change': '2023-11-08T05:29:44.395350+0000', 'last_active': '2023-11-08T05:29:44.395350+0000', 'last_peered': '2023-11-08T05:29:44.395350+0000', 'last_clean': '2023-11-08T05:29:44.395350+0000', 'last_became_active': '2023-11-08T05:29:44.395012+0000', 'last_became_peered': '2023-11-08T05:29:44.395012+0000', 'last_unstale': '2023-11-08T05:29:44.395350+0000', 'last_undegraded': '2023-11-08T05:29:44.395350+0000', 'last_fullsized': '2023-11-08T05:29:44.395350+0000', 'mapping_epoch': 455, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 456, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2], 'acting': [2, 1], 'avail_no_missing': ['2', '1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2023-11-08T05:30:07.123 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-11-08T05:30:07.123 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 456, 'reported_epoch': 455, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:29:43.389390+0000', 'last_change': '2023-11-08T05:28:13.563357+0000', 'last_active': '2023-11-08T05:29:43.389390+0000', 'last_peered': '2023-11-08T05:29:43.389390+0000', 'last_clean': '2023-11-08T05:28:01.457691+0000', 'last_became_active': '2023-11-08T05:28:13.563357+0000', 'last_became_peered': '2023-11-08T05:28:13.563357+0000', 'last_unstale': '2023-11-08T05:29:43.389390+0000', 'last_undegraded': '2023-11-08T05:29:43.389390+0000', 'last_fullsized': '2023-11-08T05:28:13.541614+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:30:07.124 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-11-08T05:30:07.124 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 456, 'reported_epoch': 455, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:29:43.389311+0000', 'last_change': '2023-11-08T05:28:13.550672+0000', 'last_active': '2023-11-08T05:29:43.389311+0000', 'last_peered': '2023-11-08T05:29:43.389311+0000', 'last_clean': '2023-11-08T05:28:01.457904+0000', 'last_became_active': '2023-11-08T05:28:13.550672+0000', 'last_became_peered': '2023-11-08T05:28:13.550672+0000', 'last_unstale': '2023-11-08T05:29:43.389311+0000', 'last_undegraded': '2023-11-08T05:29:43.389311+0000', 'last_fullsized': '2023-11-08T05:28:13.539595+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:30:07.124 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-11-08T05:30:07.125 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 456, 'reported_epoch': 455, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:29:43.389474+0000', 'last_change': '2023-11-08T05:28:13.557968+0000', 'last_active': '2023-11-08T05:29:43.389474+0000', 'last_peered': '2023-11-08T05:29:43.389474+0000', 'last_clean': '2023-11-08T05:28:01.458433+0000', 'last_became_active': '2023-11-08T05:28:13.557968+0000', 'last_became_peered': '2023-11-08T05:28:13.557968+0000', 'last_unstale': '2023-11-08T05:29:43.389474+0000', 'last_undegraded': '2023-11-08T05:29:43.389474+0000', 'last_fullsized': '2023-11-08T05:28:13.542832+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:30:07.125 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-11-08T05:30:07.472 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T05:30:07.472 INFO:teuthology.orchestra.run.smithi087.stdout:{"fsid":"ab4cbc06-449d-497d-b513-220d9db2b590","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 6 pgs undersized","count":6},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2250,"monmap":{"epoch":1,"min_mon_release_name":"pacific","num_mons":1},"osdmap":{"epoch":456,"num_osds":3,"num_up_osds":2,"osd_up_since":1699421292,"num_in_osds":3,"osd_in_since":1699419158,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":6},{"state_name":"active+clean","count":2},{"state_name":"active+clean+remapped","count":1}],"num_pgs":9,"num_pools":2,"num_objects":0,"data_bytes":0,"bytes_used":2697519104,"bytes_avail":287212773376,"bytes_total":289910292480},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":41,"modified":"2023-11-08T05:29:32.044544+0000","services":{}},"progress_events":{}} 2023-11-08T05:30:07.485 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-11-08T05:30:07.486 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-11-08T05:30:07.767 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T05:30:07.768 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2023-11-08T05:30:07.779 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":1574,"stamp":"2023-11-08T05:30:06.051777+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2634296,"kb_used_data":408940,"kb_used_omap":780,"kb_used_meta":2220403,"kb_avail":280481224,"statfs":{"total":289910292480,"available":287212773376,"internally_reserved":0,"allocated":418754560,"data_stored":410461501,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":799447,"internal_metadata":2273692969},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-536782902,"num_objects":-164,"num_object_clones":0,"num_object_copies":-328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-115,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-164,"num_whiteouts":0,"num_read":-122056,"num_read_kb":-39011942,"num_write":-413642,"num_write_kb":-151705339,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-8,"num_bytes_recovered":-16922624,"num_keys_recovered":-2,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.424817"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":456,"reported_epoch":455,"state":"active+undersized","last_fresh":"2023-11-08T05:29:43.389354+0000","last_change":"2023-11-08T05:28:13.557071+0000","last_active":"2023-11-08T05:29:43.389354+0000","last_peered":"2023-11-08T05:29:43.389354+0000","last_clean":"2023-11-08T05:28:01.458439+0000","last_became_active":"2023-11-08T05:28:13.557071+0000","last_became_peered":"2023-11-08T05:28:13.557071+0000","last_unstale":"2023-11-08T05:29:43.389354+0000","last_undegraded":"2023-11-08T05:29:43.389354+0000","last_fullsized":"2023-11-08T05:28:13.547025+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":18,"reported_epoch":455,"state":"active+undersized","last_fresh":"2023-11-08T05:29:43.389262+0000","last_change":"2023-11-08T05:28:13.559513+0000","last_active":"2023-11-08T05:29:43.389262+0000","last_peered":"2023-11-08T05:29:43.389262+0000","last_clean":"2023-11-08T04:52:46.531964+0000","last_became_active":"2023-11-08T05:28:13.559513+0000","last_became_peered":"2023-11-08T05:28:13.559513+0000","last_unstale":"2023-11-08T05:29:43.389262+0000","last_undegraded":"2023-11-08T05:29:43.389262+0000","last_fullsized":"2023-11-08T05:28:13.540852+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":453,"reported_epoch":455,"state":"active+clean","last_fresh":"2023-11-08T05:29:43.390991+0000","last_change":"2023-11-08T04:52:47.546100+0000","last_active":"2023-11-08T05:29:43.390991+0000","last_peered":"2023-11-08T05:29:43.390991+0000","last_clean":"2023-11-08T05:29:43.390991+0000","last_became_active":"2023-11-08T04:52:47.545840+0000","last_became_peered":"2023-11-08T04:52:47.545840+0000","last_unstale":"2023-11-08T05:29:43.390991+0000","last_undegraded":"2023-11-08T05:29:43.390991+0000","last_fullsized":"2023-11-08T05:29:43.390991+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":451,"reported_epoch":455,"state":"active+clean","last_fresh":"2023-11-08T05:29:43.389261+0000","last_change":"2023-11-08T04:52:47.548597+0000","last_active":"2023-11-08T05:29:43.389261+0000","last_peered":"2023-11-08T05:29:43.389261+0000","last_clean":"2023-11-08T05:29:43.389261+0000","last_became_active":"2023-11-08T04:52:47.548294+0000","last_became_peered":"2023-11-08T04:52:47.548294+0000","last_unstale":"2023-11-08T05:29:43.389261+0000","last_undegraded":"2023-11-08T05:29:43.389261+0000","last_fullsized":"2023-11-08T05:29:43.389261+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":457,"reported_epoch":455,"state":"active+undersized","last_fresh":"2023-11-08T05:29:43.389822+0000","last_change":"2023-11-08T05:28:13.545468+0000","last_active":"2023-11-08T05:29:43.389822+0000","last_peered":"2023-11-08T05:29:43.389822+0000","last_clean":"2023-11-08T05:28:01.456723+0000","last_became_active":"2023-11-08T05:28:13.545468+0000","last_became_peered":"2023-11-08T05:28:13.545468+0000","last_unstale":"2023-11-08T05:29:43.389822+0000","last_undegraded":"2023-11-08T05:29:43.389822+0000","last_fullsized":"2023-11-08T05:28:13.537207+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_clean_scrub_stamp":"2023-11-08T04:52:45.528796+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":462,"reported_epoch":456,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:29:44.395350+0000","last_change":"2023-11-08T05:29:44.395350+0000","last_active":"2023-11-08T05:29:44.395350+0000","last_peered":"2023-11-08T05:29:44.395350+0000","last_clean":"2023-11-08T05:29:44.395350+0000","last_became_active":"2023-11-08T05:29:44.395012+0000","last_became_peered":"2023-11-08T05:29:44.395012+0000","last_unstale":"2023-11-08T05:29:44.395350+0000","last_undegraded":"2023-11-08T05:29:44.395350+0000","last_fullsized":"2023-11-08T05:29:44.395350+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":456,"reported_epoch":455,"state":"active+undersized","last_fresh":"2023-11-08T05:29:43.389390+0000","last_change":"2023-11-08T05:28:13.563357+0000","last_active":"2023-11-08T05:29:43.389390+0000","last_peered":"2023-11-08T05:29:43.389390+0000","last_clean":"2023-11-08T05:28:01.457691+0000","last_became_active":"2023-11-08T05:28:13.563357+0000","last_became_peered":"2023-11-08T05:28:13.563357+0000","last_unstale":"2023-11-08T05:29:43.389390+0000","last_undegraded":"2023-11-08T05:29:43.389390+0000","last_fullsized":"2023-11-08T05:28:13.541614+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":456,"reported_epoch":455,"state":"active+undersized","last_fresh":"2023-11-08T05:29:43.389311+0000","last_change":"2023-11-08T05:28:13.550672+0000","last_active":"2023-11-08T05:29:43.389311+0000","last_peered":"2023-11-08T05:29:43.389311+0000","last_clean":"2023-11-08T05:28:01.457904+0000","last_became_active":"2023-11-08T05:28:13.550672+0000","last_became_peered":"2023-11-08T05:28:13.550672+0000","last_unstale":"2023-11-08T05:29:43.389311+0000","last_undegraded":"2023-11-08T05:29:43.389311+0000","last_fullsized":"2023-11-08T05:28:13.539595+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":456,"reported_epoch":455,"state":"active+undersized","last_fresh":"2023-11-08T05:29:43.389474+0000","last_change":"2023-11-08T05:28:13.557968+0000","last_active":"2023-11-08T05:29:43.389474+0000","last_peered":"2023-11-08T05:29:43.389474+0000","last_clean":"2023-11-08T05:28:01.458433+0000","last_became_active":"2023-11-08T05:28:13.557968+0000","last_became_peered":"2023-11-08T05:28:13.557968+0000","last_unstale":"2023-11-08T05:29:43.389474+0000","last_undegraded":"2023-11-08T05:29:43.389474+0000","last_fullsized":"2023-11-08T05:28:13.542832+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":1,"acting":1,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":10,"acting":11,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738819,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":645004,"kb_used_data":5316,"kb_used_omap":275,"kb_used_meta":639404,"kb_avail":93726836,"statfs":{"total":96636764160,"available":95976280064,"internally_reserved":0,"allocated":5443584,"data_stored":2678316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":281650,"internal_metadata":654750670},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:59 2023","interfaces":[{"interface":"back","average":{"1min":0.30499999999999999,"5min":0.439,"15min":0.36099999999999999},"min":{"1min":0.17399999999999999,"5min":0.14499999999999999,"15min":0.14499999999999999},"max":{"1min":0.60099999999999998,"5min":4.9379999999999997,"15min":4.9379999999999997},"last":0.379},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.49399999999999999,"15min":0.42399999999999999},"min":{"1min":0.221,"5min":0.19600000000000001,"15min":0.17000000000000001},"max":{"1min":0.63200000000000001,"5min":4.9859999999999998,"15min":4.9859999999999998},"last":0.40899999999999997}]},{"osd":1,"last update":"Wed Nov 8 05:30:03 2023","interfaces":[{"interface":"back","average":{"1min":0.28499999999999998,"5min":0.65100000000000002,"15min":0.47099999999999997},"min":{"1min":0.20100000000000001,"5min":0.153,"15min":0.153},"max":{"1min":0.39800000000000002,"5min":23.242999999999999,"15min":23.242999999999999},"last":0.20100000000000001},{"interface":"front","average":{"1min":0.29499999999999998,"5min":0.63100000000000001,"15min":0.45900000000000002},"min":{"1min":0.224,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":0.436,"5min":23.251000000000001,"15min":23.251000000000001},"last":0.224}]}]},{"osd":0,"up_from":8,"seq":34359738798,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1339228,"kb_used_data":398304,"kb_used_omap":291,"kb_used_meta":936476,"kb_avail":93032612,"statfs":{"total":96636764160,"available":95265394688,"internally_reserved":0,"allocated":407863296,"data_stored":405103900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":298462,"internal_metadata":958951970},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.436,"15min":0.36299999999999999},"min":{"1min":0.19700000000000001,"5min":0.14699999999999999,"15min":0.13700000000000001},"max":{"1min":3.222,"5min":6.5499999999999998,"15min":6.9989999999999997},"last":0.245},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.46300000000000002,"15min":0.42299999999999999},"min":{"1min":0.17799999999999999,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":5.0030000000000001,"5min":6.0629999999999997,"15min":9.6869999999999994},"last":0.28599999999999998}]},{"osd":2,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.51600000000000001,"15min":0.47099999999999997},"min":{"1min":0.21099999999999999,"5min":0.153,"15min":0.153},"max":{"1min":5.7439999999999998,"5min":5.8280000000000003,"15min":9.7639999999999993},"last":0.252},{"interface":"front","average":{"1min":0.497,"5min":0.56200000000000006,"15min":0.42299999999999999},"min":{"1min":0.20399999999999999,"5min":0.187,"15min":0.16800000000000001},"max":{"1min":3.9319999999999999,"5min":8.8040000000000003,"15min":8.8040000000000003},"last":0.29899999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738821,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":650064,"kb_used_data":5320,"kb_used_omap":214,"kb_used_meta":644521,"kb_avail":93721776,"statfs":{"total":96636764160,"available":95971098624,"internally_reserved":0,"allocated":5447680,"data_stored":2679285,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":219335,"internal_metadata":659990329},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:56 2023","interfaces":[{"interface":"back","average":{"1min":0.30399999999999999,"5min":0.34699999999999998,"15min":0.36499999999999999},"min":{"1min":0.23999999999999999,"5min":0.14199999999999999,"15min":0.14199999999999999},"max":{"1min":0.42399999999999999,"5min":1.595,"15min":7.4249999999999998},"last":0.17100000000000001},{"interface":"front","average":{"1min":0.35499999999999998,"5min":0.40899999999999997,"15min":0.376},"min":{"1min":0.23999999999999999,"5min":0.19600000000000001,"15min":0.183},"max":{"1min":0.46999999999999997,"5min":2.6859999999999999,"15min":2.6859999999999999},"last":0.23000000000000001}]},{"osd":2,"last update":"Wed Nov 8 05:29:57 2023","interfaces":[{"interface":"back","average":{"1min":0.28199999999999997,"5min":0.36099999999999999,"15min":0.373},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":0.34399999999999997,"5min":2.2650000000000001,"15min":5.8209999999999997},"last":0.248},{"interface":"front","average":{"1min":0.28599999999999998,"5min":0.31,"15min":0.35799999999999998},"min":{"1min":0.17199999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.35699999999999998,"5min":1.137,"15min":4.2350000000000003},"last":0.28399999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-08T05:30:10.781 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-11-08T05:30:11.078 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T05:30:11.080 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2023-11-08T05:30:11.091 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":1576,"stamp":"2023-11-08T05:30:10.052795+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2634296,"kb_used_data":408940,"kb_used_omap":780,"kb_used_meta":2220403,"kb_avail":280481224,"statfs":{"total":289910292480,"available":287212773376,"internally_reserved":0,"allocated":418754560,"data_stored":410461501,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":799447,"internal_metadata":2273692969},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-536782902,"num_objects":-164,"num_object_clones":0,"num_object_copies":-328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-115,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-164,"num_whiteouts":0,"num_read":-122056,"num_read_kb":-39011942,"num_write":-413642,"num_write_kb":-151705339,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-8,"num_bytes_recovered":-16922624,"num_keys_recovered":-2,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.424817"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":456,"reported_epoch":455,"state":"active+undersized","last_fresh":"2023-11-08T05:29:43.389354+0000","last_change":"2023-11-08T05:28:13.557071+0000","last_active":"2023-11-08T05:29:43.389354+0000","last_peered":"2023-11-08T05:29:43.389354+0000","last_clean":"2023-11-08T05:28:01.458439+0000","last_became_active":"2023-11-08T05:28:13.557071+0000","last_became_peered":"2023-11-08T05:28:13.557071+0000","last_unstale":"2023-11-08T05:29:43.389354+0000","last_undegraded":"2023-11-08T05:29:43.389354+0000","last_fullsized":"2023-11-08T05:28:13.547025+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":18,"reported_epoch":455,"state":"active+undersized","last_fresh":"2023-11-08T05:29:43.389262+0000","last_change":"2023-11-08T05:28:13.559513+0000","last_active":"2023-11-08T05:29:43.389262+0000","last_peered":"2023-11-08T05:29:43.389262+0000","last_clean":"2023-11-08T04:52:46.531964+0000","last_became_active":"2023-11-08T05:28:13.559513+0000","last_became_peered":"2023-11-08T05:28:13.559513+0000","last_unstale":"2023-11-08T05:29:43.389262+0000","last_undegraded":"2023-11-08T05:29:43.389262+0000","last_fullsized":"2023-11-08T05:28:13.540852+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":453,"reported_epoch":455,"state":"active+clean","last_fresh":"2023-11-08T05:29:43.390991+0000","last_change":"2023-11-08T04:52:47.546100+0000","last_active":"2023-11-08T05:29:43.390991+0000","last_peered":"2023-11-08T05:29:43.390991+0000","last_clean":"2023-11-08T05:29:43.390991+0000","last_became_active":"2023-11-08T04:52:47.545840+0000","last_became_peered":"2023-11-08T04:52:47.545840+0000","last_unstale":"2023-11-08T05:29:43.390991+0000","last_undegraded":"2023-11-08T05:29:43.390991+0000","last_fullsized":"2023-11-08T05:29:43.390991+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":451,"reported_epoch":455,"state":"active+clean","last_fresh":"2023-11-08T05:29:43.389261+0000","last_change":"2023-11-08T04:52:47.548597+0000","last_active":"2023-11-08T05:29:43.389261+0000","last_peered":"2023-11-08T05:29:43.389261+0000","last_clean":"2023-11-08T05:29:43.389261+0000","last_became_active":"2023-11-08T04:52:47.548294+0000","last_became_peered":"2023-11-08T04:52:47.548294+0000","last_unstale":"2023-11-08T05:29:43.389261+0000","last_undegraded":"2023-11-08T05:29:43.389261+0000","last_fullsized":"2023-11-08T05:29:43.389261+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":457,"reported_epoch":455,"state":"active+undersized","last_fresh":"2023-11-08T05:29:43.389822+0000","last_change":"2023-11-08T05:28:13.545468+0000","last_active":"2023-11-08T05:29:43.389822+0000","last_peered":"2023-11-08T05:29:43.389822+0000","last_clean":"2023-11-08T05:28:01.456723+0000","last_became_active":"2023-11-08T05:28:13.545468+0000","last_became_peered":"2023-11-08T05:28:13.545468+0000","last_unstale":"2023-11-08T05:29:43.389822+0000","last_undegraded":"2023-11-08T05:29:43.389822+0000","last_fullsized":"2023-11-08T05:28:13.537207+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_clean_scrub_stamp":"2023-11-08T04:52:45.528796+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":462,"reported_epoch":456,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:29:44.395350+0000","last_change":"2023-11-08T05:29:44.395350+0000","last_active":"2023-11-08T05:29:44.395350+0000","last_peered":"2023-11-08T05:29:44.395350+0000","last_clean":"2023-11-08T05:29:44.395350+0000","last_became_active":"2023-11-08T05:29:44.395012+0000","last_became_peered":"2023-11-08T05:29:44.395012+0000","last_unstale":"2023-11-08T05:29:44.395350+0000","last_undegraded":"2023-11-08T05:29:44.395350+0000","last_fullsized":"2023-11-08T05:29:44.395350+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":456,"reported_epoch":455,"state":"active+undersized","last_fresh":"2023-11-08T05:29:43.389390+0000","last_change":"2023-11-08T05:28:13.563357+0000","last_active":"2023-11-08T05:29:43.389390+0000","last_peered":"2023-11-08T05:29:43.389390+0000","last_clean":"2023-11-08T05:28:01.457691+0000","last_became_active":"2023-11-08T05:28:13.563357+0000","last_became_peered":"2023-11-08T05:28:13.563357+0000","last_unstale":"2023-11-08T05:29:43.389390+0000","last_undegraded":"2023-11-08T05:29:43.389390+0000","last_fullsized":"2023-11-08T05:28:13.541614+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":456,"reported_epoch":455,"state":"active+undersized","last_fresh":"2023-11-08T05:29:43.389311+0000","last_change":"2023-11-08T05:28:13.550672+0000","last_active":"2023-11-08T05:29:43.389311+0000","last_peered":"2023-11-08T05:29:43.389311+0000","last_clean":"2023-11-08T05:28:01.457904+0000","last_became_active":"2023-11-08T05:28:13.550672+0000","last_became_peered":"2023-11-08T05:28:13.550672+0000","last_unstale":"2023-11-08T05:29:43.389311+0000","last_undegraded":"2023-11-08T05:29:43.389311+0000","last_fullsized":"2023-11-08T05:28:13.539595+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":456,"reported_epoch":455,"state":"active+undersized","last_fresh":"2023-11-08T05:29:43.389474+0000","last_change":"2023-11-08T05:28:13.557968+0000","last_active":"2023-11-08T05:29:43.389474+0000","last_peered":"2023-11-08T05:29:43.389474+0000","last_clean":"2023-11-08T05:28:01.458433+0000","last_became_active":"2023-11-08T05:28:13.557968+0000","last_became_peered":"2023-11-08T05:28:13.557968+0000","last_unstale":"2023-11-08T05:29:43.389474+0000","last_undegraded":"2023-11-08T05:29:43.389474+0000","last_fullsized":"2023-11-08T05:28:13.542832+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":1,"acting":1,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":10,"acting":11,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738820,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":645004,"kb_used_data":5316,"kb_used_omap":275,"kb_used_meta":639404,"kb_avail":93726836,"statfs":{"total":96636764160,"available":95976280064,"internally_reserved":0,"allocated":5443584,"data_stored":2678316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":281650,"internal_metadata":654750670},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:59 2023","interfaces":[{"interface":"back","average":{"1min":0.30499999999999999,"5min":0.439,"15min":0.36099999999999999},"min":{"1min":0.17399999999999999,"5min":0.14499999999999999,"15min":0.14499999999999999},"max":{"1min":0.60099999999999998,"5min":4.9379999999999997,"15min":4.9379999999999997},"last":0.379},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.49399999999999999,"15min":0.42399999999999999},"min":{"1min":0.221,"5min":0.19600000000000001,"15min":0.17000000000000001},"max":{"1min":0.63200000000000001,"5min":4.9859999999999998,"15min":4.9859999999999998},"last":0.40899999999999997}]},{"osd":1,"last update":"Wed Nov 8 05:30:03 2023","interfaces":[{"interface":"back","average":{"1min":0.28499999999999998,"5min":0.65100000000000002,"15min":0.47099999999999997},"min":{"1min":0.20100000000000001,"5min":0.153,"15min":0.153},"max":{"1min":0.39800000000000002,"5min":23.242999999999999,"15min":23.242999999999999},"last":0.41099999999999998},{"interface":"front","average":{"1min":0.29499999999999998,"5min":0.63100000000000001,"15min":0.45900000000000002},"min":{"1min":0.224,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":0.436,"5min":23.251000000000001,"15min":23.251000000000001},"last":0.45300000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738798,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1339228,"kb_used_data":398304,"kb_used_omap":291,"kb_used_meta":936476,"kb_avail":93032612,"statfs":{"total":96636764160,"available":95265394688,"internally_reserved":0,"allocated":407863296,"data_stored":405103900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":298462,"internal_metadata":958951970},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.436,"15min":0.36299999999999999},"min":{"1min":0.19700000000000001,"5min":0.14699999999999999,"15min":0.13700000000000001},"max":{"1min":3.222,"5min":6.5499999999999998,"15min":6.9989999999999997},"last":0.245},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.46300000000000002,"15min":0.42299999999999999},"min":{"1min":0.17799999999999999,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":5.0030000000000001,"5min":6.0629999999999997,"15min":9.6869999999999994},"last":0.28599999999999998}]},{"osd":2,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.51600000000000001,"15min":0.47099999999999997},"min":{"1min":0.21099999999999999,"5min":0.153,"15min":0.153},"max":{"1min":5.7439999999999998,"5min":5.8280000000000003,"15min":9.7639999999999993},"last":0.252},{"interface":"front","average":{"1min":0.497,"5min":0.56200000000000006,"15min":0.42299999999999999},"min":{"1min":0.20399999999999999,"5min":0.187,"15min":0.16800000000000001},"max":{"1min":3.9319999999999999,"5min":8.8040000000000003,"15min":8.8040000000000003},"last":0.29899999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738821,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":650064,"kb_used_data":5320,"kb_used_omap":214,"kb_used_meta":644521,"kb_avail":93721776,"statfs":{"total":96636764160,"available":95971098624,"internally_reserved":0,"allocated":5447680,"data_stored":2679285,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":219335,"internal_metadata":659990329},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:56 2023","interfaces":[{"interface":"back","average":{"1min":0.30399999999999999,"5min":0.34699999999999998,"15min":0.36499999999999999},"min":{"1min":0.23999999999999999,"5min":0.14199999999999999,"15min":0.14199999999999999},"max":{"1min":0.42399999999999999,"5min":1.595,"15min":7.4249999999999998},"last":0.17100000000000001},{"interface":"front","average":{"1min":0.35499999999999998,"5min":0.40899999999999997,"15min":0.376},"min":{"1min":0.23999999999999999,"5min":0.19600000000000001,"15min":0.183},"max":{"1min":0.46999999999999997,"5min":2.6859999999999999,"15min":2.6859999999999999},"last":0.23000000000000001}]},{"osd":2,"last update":"Wed Nov 8 05:29:57 2023","interfaces":[{"interface":"back","average":{"1min":0.28199999999999997,"5min":0.36099999999999999,"15min":0.373},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":0.34399999999999997,"5min":2.2650000000000001,"15min":5.8209999999999997},"last":0.248},{"interface":"front","average":{"1min":0.28599999999999998,"5min":0.31,"15min":0.35799999999999998},"min":{"1min":0.17199999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.35699999999999998,"5min":1.137,"15min":4.2350000000000003},"last":0.28399999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-08T05:30:11.092 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-11-08T05:30:11.375 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T05:30:11.376 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2023-11-08T05:30:11.388 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":1576,"stamp":"2023-11-08T05:30:10.052795+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2634296,"kb_used_data":408940,"kb_used_omap":780,"kb_used_meta":2220403,"kb_avail":280481224,"statfs":{"total":289910292480,"available":287212773376,"internally_reserved":0,"allocated":418754560,"data_stored":410461501,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":799447,"internal_metadata":2273692969},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-536782902,"num_objects":-164,"num_object_clones":0,"num_object_copies":-328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-115,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-164,"num_whiteouts":0,"num_read":-122056,"num_read_kb":-39011942,"num_write":-413642,"num_write_kb":-151705339,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-8,"num_bytes_recovered":-16922624,"num_keys_recovered":-2,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.424817"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":456,"reported_epoch":455,"state":"active+undersized","last_fresh":"2023-11-08T05:29:43.389354+0000","last_change":"2023-11-08T05:28:13.557071+0000","last_active":"2023-11-08T05:29:43.389354+0000","last_peered":"2023-11-08T05:29:43.389354+0000","last_clean":"2023-11-08T05:28:01.458439+0000","last_became_active":"2023-11-08T05:28:13.557071+0000","last_became_peered":"2023-11-08T05:28:13.557071+0000","last_unstale":"2023-11-08T05:29:43.389354+0000","last_undegraded":"2023-11-08T05:29:43.389354+0000","last_fullsized":"2023-11-08T05:28:13.547025+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":18,"reported_epoch":455,"state":"active+undersized","last_fresh":"2023-11-08T05:29:43.389262+0000","last_change":"2023-11-08T05:28:13.559513+0000","last_active":"2023-11-08T05:29:43.389262+0000","last_peered":"2023-11-08T05:29:43.389262+0000","last_clean":"2023-11-08T04:52:46.531964+0000","last_became_active":"2023-11-08T05:28:13.559513+0000","last_became_peered":"2023-11-08T05:28:13.559513+0000","last_unstale":"2023-11-08T05:29:43.389262+0000","last_undegraded":"2023-11-08T05:29:43.389262+0000","last_fullsized":"2023-11-08T05:28:13.540852+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":453,"reported_epoch":455,"state":"active+clean","last_fresh":"2023-11-08T05:29:43.390991+0000","last_change":"2023-11-08T04:52:47.546100+0000","last_active":"2023-11-08T05:29:43.390991+0000","last_peered":"2023-11-08T05:29:43.390991+0000","last_clean":"2023-11-08T05:29:43.390991+0000","last_became_active":"2023-11-08T04:52:47.545840+0000","last_became_peered":"2023-11-08T04:52:47.545840+0000","last_unstale":"2023-11-08T05:29:43.390991+0000","last_undegraded":"2023-11-08T05:29:43.390991+0000","last_fullsized":"2023-11-08T05:29:43.390991+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":451,"reported_epoch":455,"state":"active+clean","last_fresh":"2023-11-08T05:29:43.389261+0000","last_change":"2023-11-08T04:52:47.548597+0000","last_active":"2023-11-08T05:29:43.389261+0000","last_peered":"2023-11-08T05:29:43.389261+0000","last_clean":"2023-11-08T05:29:43.389261+0000","last_became_active":"2023-11-08T04:52:47.548294+0000","last_became_peered":"2023-11-08T04:52:47.548294+0000","last_unstale":"2023-11-08T05:29:43.389261+0000","last_undegraded":"2023-11-08T05:29:43.389261+0000","last_fullsized":"2023-11-08T05:29:43.389261+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":457,"reported_epoch":455,"state":"active+undersized","last_fresh":"2023-11-08T05:29:43.389822+0000","last_change":"2023-11-08T05:28:13.545468+0000","last_active":"2023-11-08T05:29:43.389822+0000","last_peered":"2023-11-08T05:29:43.389822+0000","last_clean":"2023-11-08T05:28:01.456723+0000","last_became_active":"2023-11-08T05:28:13.545468+0000","last_became_peered":"2023-11-08T05:28:13.545468+0000","last_unstale":"2023-11-08T05:29:43.389822+0000","last_undegraded":"2023-11-08T05:29:43.389822+0000","last_fullsized":"2023-11-08T05:28:13.537207+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_clean_scrub_stamp":"2023-11-08T04:52:45.528796+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":462,"reported_epoch":456,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:29:44.395350+0000","last_change":"2023-11-08T05:29:44.395350+0000","last_active":"2023-11-08T05:29:44.395350+0000","last_peered":"2023-11-08T05:29:44.395350+0000","last_clean":"2023-11-08T05:29:44.395350+0000","last_became_active":"2023-11-08T05:29:44.395012+0000","last_became_peered":"2023-11-08T05:29:44.395012+0000","last_unstale":"2023-11-08T05:29:44.395350+0000","last_undegraded":"2023-11-08T05:29:44.395350+0000","last_fullsized":"2023-11-08T05:29:44.395350+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":456,"reported_epoch":455,"state":"active+undersized","last_fresh":"2023-11-08T05:29:43.389390+0000","last_change":"2023-11-08T05:28:13.563357+0000","last_active":"2023-11-08T05:29:43.389390+0000","last_peered":"2023-11-08T05:29:43.389390+0000","last_clean":"2023-11-08T05:28:01.457691+0000","last_became_active":"2023-11-08T05:28:13.563357+0000","last_became_peered":"2023-11-08T05:28:13.563357+0000","last_unstale":"2023-11-08T05:29:43.389390+0000","last_undegraded":"2023-11-08T05:29:43.389390+0000","last_fullsized":"2023-11-08T05:28:13.541614+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":456,"reported_epoch":455,"state":"active+undersized","last_fresh":"2023-11-08T05:29:43.389311+0000","last_change":"2023-11-08T05:28:13.550672+0000","last_active":"2023-11-08T05:29:43.389311+0000","last_peered":"2023-11-08T05:29:43.389311+0000","last_clean":"2023-11-08T05:28:01.457904+0000","last_became_active":"2023-11-08T05:28:13.550672+0000","last_became_peered":"2023-11-08T05:28:13.550672+0000","last_unstale":"2023-11-08T05:29:43.389311+0000","last_undegraded":"2023-11-08T05:29:43.389311+0000","last_fullsized":"2023-11-08T05:28:13.539595+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":456,"reported_epoch":455,"state":"active+undersized","last_fresh":"2023-11-08T05:29:43.389474+0000","last_change":"2023-11-08T05:28:13.557968+0000","last_active":"2023-11-08T05:29:43.389474+0000","last_peered":"2023-11-08T05:29:43.389474+0000","last_clean":"2023-11-08T05:28:01.458433+0000","last_became_active":"2023-11-08T05:28:13.557968+0000","last_became_peered":"2023-11-08T05:28:13.557968+0000","last_unstale":"2023-11-08T05:29:43.389474+0000","last_undegraded":"2023-11-08T05:29:43.389474+0000","last_fullsized":"2023-11-08T05:28:13.542832+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":1,"acting":1,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":10,"acting":11,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738820,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":645004,"kb_used_data":5316,"kb_used_omap":275,"kb_used_meta":639404,"kb_avail":93726836,"statfs":{"total":96636764160,"available":95976280064,"internally_reserved":0,"allocated":5443584,"data_stored":2678316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":281650,"internal_metadata":654750670},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:59 2023","interfaces":[{"interface":"back","average":{"1min":0.30499999999999999,"5min":0.439,"15min":0.36099999999999999},"min":{"1min":0.17399999999999999,"5min":0.14499999999999999,"15min":0.14499999999999999},"max":{"1min":0.60099999999999998,"5min":4.9379999999999997,"15min":4.9379999999999997},"last":0.379},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.49399999999999999,"15min":0.42399999999999999},"min":{"1min":0.221,"5min":0.19600000000000001,"15min":0.17000000000000001},"max":{"1min":0.63200000000000001,"5min":4.9859999999999998,"15min":4.9859999999999998},"last":0.40899999999999997}]},{"osd":1,"last update":"Wed Nov 8 05:30:03 2023","interfaces":[{"interface":"back","average":{"1min":0.28499999999999998,"5min":0.65100000000000002,"15min":0.47099999999999997},"min":{"1min":0.20100000000000001,"5min":0.153,"15min":0.153},"max":{"1min":0.39800000000000002,"5min":23.242999999999999,"15min":23.242999999999999},"last":0.41099999999999998},{"interface":"front","average":{"1min":0.29499999999999998,"5min":0.63100000000000001,"15min":0.45900000000000002},"min":{"1min":0.224,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":0.436,"5min":23.251000000000001,"15min":23.251000000000001},"last":0.45300000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738798,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1339228,"kb_used_data":398304,"kb_used_omap":291,"kb_used_meta":936476,"kb_avail":93032612,"statfs":{"total":96636764160,"available":95265394688,"internally_reserved":0,"allocated":407863296,"data_stored":405103900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":298462,"internal_metadata":958951970},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.436,"15min":0.36299999999999999},"min":{"1min":0.19700000000000001,"5min":0.14699999999999999,"15min":0.13700000000000001},"max":{"1min":3.222,"5min":6.5499999999999998,"15min":6.9989999999999997},"last":0.245},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.46300000000000002,"15min":0.42299999999999999},"min":{"1min":0.17799999999999999,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":5.0030000000000001,"5min":6.0629999999999997,"15min":9.6869999999999994},"last":0.28599999999999998}]},{"osd":2,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.51600000000000001,"15min":0.47099999999999997},"min":{"1min":0.21099999999999999,"5min":0.153,"15min":0.153},"max":{"1min":5.7439999999999998,"5min":5.8280000000000003,"15min":9.7639999999999993},"last":0.252},{"interface":"front","average":{"1min":0.497,"5min":0.56200000000000006,"15min":0.42299999999999999},"min":{"1min":0.20399999999999999,"5min":0.187,"15min":0.16800000000000001},"max":{"1min":3.9319999999999999,"5min":8.8040000000000003,"15min":8.8040000000000003},"last":0.29899999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738821,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":650064,"kb_used_data":5320,"kb_used_omap":214,"kb_used_meta":644521,"kb_avail":93721776,"statfs":{"total":96636764160,"available":95971098624,"internally_reserved":0,"allocated":5447680,"data_stored":2679285,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":219335,"internal_metadata":659990329},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:56 2023","interfaces":[{"interface":"back","average":{"1min":0.30399999999999999,"5min":0.34699999999999998,"15min":0.36499999999999999},"min":{"1min":0.23999999999999999,"5min":0.14199999999999999,"15min":0.14199999999999999},"max":{"1min":0.42399999999999999,"5min":1.595,"15min":7.4249999999999998},"last":0.17100000000000001},{"interface":"front","average":{"1min":0.35499999999999998,"5min":0.40899999999999997,"15min":0.376},"min":{"1min":0.23999999999999999,"5min":0.19600000000000001,"15min":0.183},"max":{"1min":0.46999999999999997,"5min":2.6859999999999999,"15min":2.6859999999999999},"last":0.23000000000000001}]},{"osd":2,"last update":"Wed Nov 8 05:29:57 2023","interfaces":[{"interface":"back","average":{"1min":0.28199999999999997,"5min":0.36099999999999999,"15min":0.373},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":0.34399999999999997,"5min":2.2650000000000001,"15min":5.8209999999999997},"last":0.248},{"interface":"front","average":{"1min":0.28599999999999998,"5min":0.31,"15min":0.35799999999999998},"min":{"1min":0.17199999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.35699999999999998,"5min":1.137,"15min":4.2350000000000003},"last":0.28399999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-08T05:30:11.389 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-11-08T05:30:11.389 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 456, 'reported_epoch': 455, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:29:43.389354+0000', 'last_change': '2023-11-08T05:28:13.557071+0000', 'last_active': '2023-11-08T05:29:43.389354+0000', 'last_peered': '2023-11-08T05:29:43.389354+0000', 'last_clean': '2023-11-08T05:28:01.458439+0000', 'last_became_active': '2023-11-08T05:28:13.557071+0000', 'last_became_peered': '2023-11-08T05:28:13.557071+0000', 'last_unstale': '2023-11-08T05:29:43.389354+0000', 'last_undegraded': '2023-11-08T05:29:43.389354+0000', 'last_fullsized': '2023-11-08T05:28:13.547025+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:30:11.389 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-11-08T05:30:11.390 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 18, 'reported_epoch': 455, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:29:43.389262+0000', 'last_change': '2023-11-08T05:28:13.559513+0000', 'last_active': '2023-11-08T05:29:43.389262+0000', 'last_peered': '2023-11-08T05:29:43.389262+0000', 'last_clean': '2023-11-08T04:52:46.531964+0000', 'last_became_active': '2023-11-08T05:28:13.559513+0000', 'last_became_peered': '2023-11-08T05:28:13.559513+0000', 'last_unstale': '2023-11-08T05:29:43.389262+0000', 'last_undegraded': '2023-11-08T05:29:43.389262+0000', 'last_fullsized': '2023-11-08T05:28:13.540852+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:30:11.390 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-11-08T05:30:11.390 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "0'0", 'reported_seq': 457, 'reported_epoch': 455, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:29:43.389822+0000', 'last_change': '2023-11-08T05:28:13.545468+0000', 'last_active': '2023-11-08T05:29:43.389822+0000', 'last_peered': '2023-11-08T05:29:43.389822+0000', 'last_clean': '2023-11-08T05:28:01.456723+0000', 'last_became_active': '2023-11-08T05:28:13.545468+0000', 'last_became_peered': '2023-11-08T05:28:13.545468+0000', 'last_unstale': '2023-11-08T05:29:43.389822+0000', 'last_undegraded': '2023-11-08T05:29:43.389822+0000', 'last_fullsized': '2023-11-08T05:28:13.537207+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 9, 'last_epoch_clean': 10, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:45.528796+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:45.528796+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:45.528796+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:30:11.391 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.0 is not active+clean 2023-11-08T05:30:11.391 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.0', 'version': "0'0", 'reported_seq': 462, 'reported_epoch': 456, 'state': 'active+clean+remapped', 'last_fresh': '2023-11-08T05:29:44.395350+0000', 'last_change': '2023-11-08T05:29:44.395350+0000', 'last_active': '2023-11-08T05:29:44.395350+0000', 'last_peered': '2023-11-08T05:29:44.395350+0000', 'last_clean': '2023-11-08T05:29:44.395350+0000', 'last_became_active': '2023-11-08T05:29:44.395012+0000', 'last_became_peered': '2023-11-08T05:29:44.395012+0000', 'last_unstale': '2023-11-08T05:29:44.395350+0000', 'last_undegraded': '2023-11-08T05:29:44.395350+0000', 'last_fullsized': '2023-11-08T05:29:44.395350+0000', 'mapping_epoch': 455, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 456, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2], 'acting': [2, 1], 'avail_no_missing': ['2', '1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2023-11-08T05:30:11.391 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-11-08T05:30:11.391 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 456, 'reported_epoch': 455, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:29:43.389390+0000', 'last_change': '2023-11-08T05:28:13.563357+0000', 'last_active': '2023-11-08T05:29:43.389390+0000', 'last_peered': '2023-11-08T05:29:43.389390+0000', 'last_clean': '2023-11-08T05:28:01.457691+0000', 'last_became_active': '2023-11-08T05:28:13.563357+0000', 'last_became_peered': '2023-11-08T05:28:13.563357+0000', 'last_unstale': '2023-11-08T05:29:43.389390+0000', 'last_undegraded': '2023-11-08T05:29:43.389390+0000', 'last_fullsized': '2023-11-08T05:28:13.541614+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:30:11.392 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-11-08T05:30:11.392 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 456, 'reported_epoch': 455, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:29:43.389311+0000', 'last_change': '2023-11-08T05:28:13.550672+0000', 'last_active': '2023-11-08T05:29:43.389311+0000', 'last_peered': '2023-11-08T05:29:43.389311+0000', 'last_clean': '2023-11-08T05:28:01.457904+0000', 'last_became_active': '2023-11-08T05:28:13.550672+0000', 'last_became_peered': '2023-11-08T05:28:13.550672+0000', 'last_unstale': '2023-11-08T05:29:43.389311+0000', 'last_undegraded': '2023-11-08T05:29:43.389311+0000', 'last_fullsized': '2023-11-08T05:28:13.539595+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:30:11.392 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-11-08T05:30:11.393 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 456, 'reported_epoch': 455, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:29:43.389474+0000', 'last_change': '2023-11-08T05:28:13.557968+0000', 'last_active': '2023-11-08T05:29:43.389474+0000', 'last_peered': '2023-11-08T05:29:43.389474+0000', 'last_clean': '2023-11-08T05:28:01.458433+0000', 'last_became_active': '2023-11-08T05:28:13.557968+0000', 'last_became_peered': '2023-11-08T05:28:13.557968+0000', 'last_unstale': '2023-11-08T05:29:43.389474+0000', 'last_undegraded': '2023-11-08T05:29:43.389474+0000', 'last_fullsized': '2023-11-08T05:28:13.542832+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:30:11.393 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-11-08T05:30:11.530 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~113s 2023-11-08T05:30:11.740 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T05:30:11.740 INFO:teuthology.orchestra.run.smithi087.stdout:{"fsid":"ab4cbc06-449d-497d-b513-220d9db2b590","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 6 pgs undersized","count":6},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2254,"monmap":{"epoch":1,"min_mon_release_name":"pacific","num_mons":1},"osdmap":{"epoch":456,"num_osds":3,"num_up_osds":2,"osd_up_since":1699421292,"num_in_osds":3,"osd_in_since":1699419158,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":6},{"state_name":"active+clean","count":2},{"state_name":"active+clean+remapped","count":1}],"num_pgs":9,"num_pools":2,"num_objects":0,"data_bytes":0,"bytes_used":2697519104,"bytes_avail":287212773376,"bytes_total":289910292480},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":41,"modified":"2023-11-08T05:29:32.044544+0000","services":{}},"progress_events":{}} 2023-11-08T05:30:11.753 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-11-08T05:30:11.754 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-11-08T05:30:12.035 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T05:30:12.037 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2023-11-08T05:30:12.048 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":1576,"stamp":"2023-11-08T05:30:10.052795+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2634296,"kb_used_data":408940,"kb_used_omap":780,"kb_used_meta":2220403,"kb_avail":280481224,"statfs":{"total":289910292480,"available":287212773376,"internally_reserved":0,"allocated":418754560,"data_stored":410461501,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":799447,"internal_metadata":2273692969},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-536782902,"num_objects":-164,"num_object_clones":0,"num_object_copies":-328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-115,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-164,"num_whiteouts":0,"num_read":-122056,"num_read_kb":-39011942,"num_write":-413642,"num_write_kb":-151705339,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-8,"num_bytes_recovered":-16922624,"num_keys_recovered":-2,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.424817"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":456,"reported_epoch":455,"state":"active+undersized","last_fresh":"2023-11-08T05:29:43.389354+0000","last_change":"2023-11-08T05:28:13.557071+0000","last_active":"2023-11-08T05:29:43.389354+0000","last_peered":"2023-11-08T05:29:43.389354+0000","last_clean":"2023-11-08T05:28:01.458439+0000","last_became_active":"2023-11-08T05:28:13.557071+0000","last_became_peered":"2023-11-08T05:28:13.557071+0000","last_unstale":"2023-11-08T05:29:43.389354+0000","last_undegraded":"2023-11-08T05:29:43.389354+0000","last_fullsized":"2023-11-08T05:28:13.547025+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":18,"reported_epoch":455,"state":"active+undersized","last_fresh":"2023-11-08T05:29:43.389262+0000","last_change":"2023-11-08T05:28:13.559513+0000","last_active":"2023-11-08T05:29:43.389262+0000","last_peered":"2023-11-08T05:29:43.389262+0000","last_clean":"2023-11-08T04:52:46.531964+0000","last_became_active":"2023-11-08T05:28:13.559513+0000","last_became_peered":"2023-11-08T05:28:13.559513+0000","last_unstale":"2023-11-08T05:29:43.389262+0000","last_undegraded":"2023-11-08T05:29:43.389262+0000","last_fullsized":"2023-11-08T05:28:13.540852+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":453,"reported_epoch":455,"state":"active+clean","last_fresh":"2023-11-08T05:29:43.390991+0000","last_change":"2023-11-08T04:52:47.546100+0000","last_active":"2023-11-08T05:29:43.390991+0000","last_peered":"2023-11-08T05:29:43.390991+0000","last_clean":"2023-11-08T05:29:43.390991+0000","last_became_active":"2023-11-08T04:52:47.545840+0000","last_became_peered":"2023-11-08T04:52:47.545840+0000","last_unstale":"2023-11-08T05:29:43.390991+0000","last_undegraded":"2023-11-08T05:29:43.390991+0000","last_fullsized":"2023-11-08T05:29:43.390991+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":451,"reported_epoch":455,"state":"active+clean","last_fresh":"2023-11-08T05:29:43.389261+0000","last_change":"2023-11-08T04:52:47.548597+0000","last_active":"2023-11-08T05:29:43.389261+0000","last_peered":"2023-11-08T05:29:43.389261+0000","last_clean":"2023-11-08T05:29:43.389261+0000","last_became_active":"2023-11-08T04:52:47.548294+0000","last_became_peered":"2023-11-08T04:52:47.548294+0000","last_unstale":"2023-11-08T05:29:43.389261+0000","last_undegraded":"2023-11-08T05:29:43.389261+0000","last_fullsized":"2023-11-08T05:29:43.389261+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":457,"reported_epoch":455,"state":"active+undersized","last_fresh":"2023-11-08T05:29:43.389822+0000","last_change":"2023-11-08T05:28:13.545468+0000","last_active":"2023-11-08T05:29:43.389822+0000","last_peered":"2023-11-08T05:29:43.389822+0000","last_clean":"2023-11-08T05:28:01.456723+0000","last_became_active":"2023-11-08T05:28:13.545468+0000","last_became_peered":"2023-11-08T05:28:13.545468+0000","last_unstale":"2023-11-08T05:29:43.389822+0000","last_undegraded":"2023-11-08T05:29:43.389822+0000","last_fullsized":"2023-11-08T05:28:13.537207+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_clean_scrub_stamp":"2023-11-08T04:52:45.528796+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":462,"reported_epoch":456,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:29:44.395350+0000","last_change":"2023-11-08T05:29:44.395350+0000","last_active":"2023-11-08T05:29:44.395350+0000","last_peered":"2023-11-08T05:29:44.395350+0000","last_clean":"2023-11-08T05:29:44.395350+0000","last_became_active":"2023-11-08T05:29:44.395012+0000","last_became_peered":"2023-11-08T05:29:44.395012+0000","last_unstale":"2023-11-08T05:29:44.395350+0000","last_undegraded":"2023-11-08T05:29:44.395350+0000","last_fullsized":"2023-11-08T05:29:44.395350+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":456,"reported_epoch":455,"state":"active+undersized","last_fresh":"2023-11-08T05:29:43.389390+0000","last_change":"2023-11-08T05:28:13.563357+0000","last_active":"2023-11-08T05:29:43.389390+0000","last_peered":"2023-11-08T05:29:43.389390+0000","last_clean":"2023-11-08T05:28:01.457691+0000","last_became_active":"2023-11-08T05:28:13.563357+0000","last_became_peered":"2023-11-08T05:28:13.563357+0000","last_unstale":"2023-11-08T05:29:43.389390+0000","last_undegraded":"2023-11-08T05:29:43.389390+0000","last_fullsized":"2023-11-08T05:28:13.541614+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":456,"reported_epoch":455,"state":"active+undersized","last_fresh":"2023-11-08T05:29:43.389311+0000","last_change":"2023-11-08T05:28:13.550672+0000","last_active":"2023-11-08T05:29:43.389311+0000","last_peered":"2023-11-08T05:29:43.389311+0000","last_clean":"2023-11-08T05:28:01.457904+0000","last_became_active":"2023-11-08T05:28:13.550672+0000","last_became_peered":"2023-11-08T05:28:13.550672+0000","last_unstale":"2023-11-08T05:29:43.389311+0000","last_undegraded":"2023-11-08T05:29:43.389311+0000","last_fullsized":"2023-11-08T05:28:13.539595+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":456,"reported_epoch":455,"state":"active+undersized","last_fresh":"2023-11-08T05:29:43.389474+0000","last_change":"2023-11-08T05:28:13.557968+0000","last_active":"2023-11-08T05:29:43.389474+0000","last_peered":"2023-11-08T05:29:43.389474+0000","last_clean":"2023-11-08T05:28:01.458433+0000","last_became_active":"2023-11-08T05:28:13.557968+0000","last_became_peered":"2023-11-08T05:28:13.557968+0000","last_unstale":"2023-11-08T05:29:43.389474+0000","last_undegraded":"2023-11-08T05:29:43.389474+0000","last_fullsized":"2023-11-08T05:28:13.542832+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":1,"acting":1,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":10,"acting":11,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738820,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":645004,"kb_used_data":5316,"kb_used_omap":275,"kb_used_meta":639404,"kb_avail":93726836,"statfs":{"total":96636764160,"available":95976280064,"internally_reserved":0,"allocated":5443584,"data_stored":2678316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":281650,"internal_metadata":654750670},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:59 2023","interfaces":[{"interface":"back","average":{"1min":0.30499999999999999,"5min":0.439,"15min":0.36099999999999999},"min":{"1min":0.17399999999999999,"5min":0.14499999999999999,"15min":0.14499999999999999},"max":{"1min":0.60099999999999998,"5min":4.9379999999999997,"15min":4.9379999999999997},"last":0.379},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.49399999999999999,"15min":0.42399999999999999},"min":{"1min":0.221,"5min":0.19600000000000001,"15min":0.17000000000000001},"max":{"1min":0.63200000000000001,"5min":4.9859999999999998,"15min":4.9859999999999998},"last":0.40899999999999997}]},{"osd":1,"last update":"Wed Nov 8 05:30:03 2023","interfaces":[{"interface":"back","average":{"1min":0.28499999999999998,"5min":0.65100000000000002,"15min":0.47099999999999997},"min":{"1min":0.20100000000000001,"5min":0.153,"15min":0.153},"max":{"1min":0.39800000000000002,"5min":23.242999999999999,"15min":23.242999999999999},"last":0.41099999999999998},{"interface":"front","average":{"1min":0.29499999999999998,"5min":0.63100000000000001,"15min":0.45900000000000002},"min":{"1min":0.224,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":0.436,"5min":23.251000000000001,"15min":23.251000000000001},"last":0.45300000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738798,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1339228,"kb_used_data":398304,"kb_used_omap":291,"kb_used_meta":936476,"kb_avail":93032612,"statfs":{"total":96636764160,"available":95265394688,"internally_reserved":0,"allocated":407863296,"data_stored":405103900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":298462,"internal_metadata":958951970},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.436,"15min":0.36299999999999999},"min":{"1min":0.19700000000000001,"5min":0.14699999999999999,"15min":0.13700000000000001},"max":{"1min":3.222,"5min":6.5499999999999998,"15min":6.9989999999999997},"last":0.245},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.46300000000000002,"15min":0.42299999999999999},"min":{"1min":0.17799999999999999,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":5.0030000000000001,"5min":6.0629999999999997,"15min":9.6869999999999994},"last":0.28599999999999998}]},{"osd":2,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.51600000000000001,"15min":0.47099999999999997},"min":{"1min":0.21099999999999999,"5min":0.153,"15min":0.153},"max":{"1min":5.7439999999999998,"5min":5.8280000000000003,"15min":9.7639999999999993},"last":0.252},{"interface":"front","average":{"1min":0.497,"5min":0.56200000000000006,"15min":0.42299999999999999},"min":{"1min":0.20399999999999999,"5min":0.187,"15min":0.16800000000000001},"max":{"1min":3.9319999999999999,"5min":8.8040000000000003,"15min":8.8040000000000003},"last":0.29899999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738821,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":650064,"kb_used_data":5320,"kb_used_omap":214,"kb_used_meta":644521,"kb_avail":93721776,"statfs":{"total":96636764160,"available":95971098624,"internally_reserved":0,"allocated":5447680,"data_stored":2679285,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":219335,"internal_metadata":659990329},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:56 2023","interfaces":[{"interface":"back","average":{"1min":0.30399999999999999,"5min":0.34699999999999998,"15min":0.36499999999999999},"min":{"1min":0.23999999999999999,"5min":0.14199999999999999,"15min":0.14199999999999999},"max":{"1min":0.42399999999999999,"5min":1.595,"15min":7.4249999999999998},"last":0.17100000000000001},{"interface":"front","average":{"1min":0.35499999999999998,"5min":0.40899999999999997,"15min":0.376},"min":{"1min":0.23999999999999999,"5min":0.19600000000000001,"15min":0.183},"max":{"1min":0.46999999999999997,"5min":2.6859999999999999,"15min":2.6859999999999999},"last":0.23000000000000001}]},{"osd":2,"last update":"Wed Nov 8 05:29:57 2023","interfaces":[{"interface":"back","average":{"1min":0.28199999999999997,"5min":0.36099999999999999,"15min":0.373},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":0.34399999999999997,"5min":2.2650000000000001,"15min":5.8209999999999997},"last":0.248},{"interface":"front","average":{"1min":0.28599999999999998,"5min":0.31,"15min":0.35799999999999998},"min":{"1min":0.17199999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.35699999999999998,"5min":1.137,"15min":4.2350000000000003},"last":0.28399999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-08T05:30:15.050 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-11-08T05:30:15.593 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T05:30:15.595 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2023-11-08T05:30:15.606 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":1578,"stamp":"2023-11-08T05:30:14.053864+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2634296,"kb_used_data":408940,"kb_used_omap":780,"kb_used_meta":2220403,"kb_avail":280481224,"statfs":{"total":289910292480,"available":287212773376,"internally_reserved":0,"allocated":418754560,"data_stored":410461501,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":799447,"internal_metadata":2273692969},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-536782902,"num_objects":-164,"num_object_clones":0,"num_object_copies":-328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-115,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-164,"num_whiteouts":0,"num_read":-122056,"num_read_kb":-39011942,"num_write":-413642,"num_write_kb":-151705339,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-8,"num_bytes_recovered":-16922624,"num_keys_recovered":-2,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.424817"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":456,"reported_epoch":455,"state":"active+undersized","last_fresh":"2023-11-08T05:29:43.389354+0000","last_change":"2023-11-08T05:28:13.557071+0000","last_active":"2023-11-08T05:29:43.389354+0000","last_peered":"2023-11-08T05:29:43.389354+0000","last_clean":"2023-11-08T05:28:01.458439+0000","last_became_active":"2023-11-08T05:28:13.557071+0000","last_became_peered":"2023-11-08T05:28:13.557071+0000","last_unstale":"2023-11-08T05:29:43.389354+0000","last_undegraded":"2023-11-08T05:29:43.389354+0000","last_fullsized":"2023-11-08T05:28:13.547025+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":18,"reported_epoch":455,"state":"active+undersized","last_fresh":"2023-11-08T05:29:43.389262+0000","last_change":"2023-11-08T05:28:13.559513+0000","last_active":"2023-11-08T05:29:43.389262+0000","last_peered":"2023-11-08T05:29:43.389262+0000","last_clean":"2023-11-08T04:52:46.531964+0000","last_became_active":"2023-11-08T05:28:13.559513+0000","last_became_peered":"2023-11-08T05:28:13.559513+0000","last_unstale":"2023-11-08T05:29:43.389262+0000","last_undegraded":"2023-11-08T05:29:43.389262+0000","last_fullsized":"2023-11-08T05:28:13.540852+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":453,"reported_epoch":455,"state":"active+clean","last_fresh":"2023-11-08T05:29:43.390991+0000","last_change":"2023-11-08T04:52:47.546100+0000","last_active":"2023-11-08T05:29:43.390991+0000","last_peered":"2023-11-08T05:29:43.390991+0000","last_clean":"2023-11-08T05:29:43.390991+0000","last_became_active":"2023-11-08T04:52:47.545840+0000","last_became_peered":"2023-11-08T04:52:47.545840+0000","last_unstale":"2023-11-08T05:29:43.390991+0000","last_undegraded":"2023-11-08T05:29:43.390991+0000","last_fullsized":"2023-11-08T05:29:43.390991+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":451,"reported_epoch":455,"state":"active+clean","last_fresh":"2023-11-08T05:29:43.389261+0000","last_change":"2023-11-08T04:52:47.548597+0000","last_active":"2023-11-08T05:29:43.389261+0000","last_peered":"2023-11-08T05:29:43.389261+0000","last_clean":"2023-11-08T05:29:43.389261+0000","last_became_active":"2023-11-08T04:52:47.548294+0000","last_became_peered":"2023-11-08T04:52:47.548294+0000","last_unstale":"2023-11-08T05:29:43.389261+0000","last_undegraded":"2023-11-08T05:29:43.389261+0000","last_fullsized":"2023-11-08T05:29:43.389261+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":457,"reported_epoch":455,"state":"active+undersized","last_fresh":"2023-11-08T05:29:43.389822+0000","last_change":"2023-11-08T05:28:13.545468+0000","last_active":"2023-11-08T05:29:43.389822+0000","last_peered":"2023-11-08T05:29:43.389822+0000","last_clean":"2023-11-08T05:28:01.456723+0000","last_became_active":"2023-11-08T05:28:13.545468+0000","last_became_peered":"2023-11-08T05:28:13.545468+0000","last_unstale":"2023-11-08T05:29:43.389822+0000","last_undegraded":"2023-11-08T05:29:43.389822+0000","last_fullsized":"2023-11-08T05:28:13.537207+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_clean_scrub_stamp":"2023-11-08T04:52:45.528796+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":462,"reported_epoch":456,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:29:44.395350+0000","last_change":"2023-11-08T05:29:44.395350+0000","last_active":"2023-11-08T05:29:44.395350+0000","last_peered":"2023-11-08T05:29:44.395350+0000","last_clean":"2023-11-08T05:29:44.395350+0000","last_became_active":"2023-11-08T05:29:44.395012+0000","last_became_peered":"2023-11-08T05:29:44.395012+0000","last_unstale":"2023-11-08T05:29:44.395350+0000","last_undegraded":"2023-11-08T05:29:44.395350+0000","last_fullsized":"2023-11-08T05:29:44.395350+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":456,"reported_epoch":455,"state":"active+undersized","last_fresh":"2023-11-08T05:29:43.389390+0000","last_change":"2023-11-08T05:28:13.563357+0000","last_active":"2023-11-08T05:29:43.389390+0000","last_peered":"2023-11-08T05:29:43.389390+0000","last_clean":"2023-11-08T05:28:01.457691+0000","last_became_active":"2023-11-08T05:28:13.563357+0000","last_became_peered":"2023-11-08T05:28:13.563357+0000","last_unstale":"2023-11-08T05:29:43.389390+0000","last_undegraded":"2023-11-08T05:29:43.389390+0000","last_fullsized":"2023-11-08T05:28:13.541614+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":456,"reported_epoch":455,"state":"active+undersized","last_fresh":"2023-11-08T05:29:43.389311+0000","last_change":"2023-11-08T05:28:13.550672+0000","last_active":"2023-11-08T05:29:43.389311+0000","last_peered":"2023-11-08T05:29:43.389311+0000","last_clean":"2023-11-08T05:28:01.457904+0000","last_became_active":"2023-11-08T05:28:13.550672+0000","last_became_peered":"2023-11-08T05:28:13.550672+0000","last_unstale":"2023-11-08T05:29:43.389311+0000","last_undegraded":"2023-11-08T05:29:43.389311+0000","last_fullsized":"2023-11-08T05:28:13.539595+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":456,"reported_epoch":455,"state":"active+undersized","last_fresh":"2023-11-08T05:29:43.389474+0000","last_change":"2023-11-08T05:28:13.557968+0000","last_active":"2023-11-08T05:29:43.389474+0000","last_peered":"2023-11-08T05:29:43.389474+0000","last_clean":"2023-11-08T05:28:01.458433+0000","last_became_active":"2023-11-08T05:28:13.557968+0000","last_became_peered":"2023-11-08T05:28:13.557968+0000","last_unstale":"2023-11-08T05:29:43.389474+0000","last_undegraded":"2023-11-08T05:29:43.389474+0000","last_fullsized":"2023-11-08T05:28:13.542832+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":1,"acting":1,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":10,"acting":11,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738820,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":645004,"kb_used_data":5316,"kb_used_omap":275,"kb_used_meta":639404,"kb_avail":93726836,"statfs":{"total":96636764160,"available":95976280064,"internally_reserved":0,"allocated":5443584,"data_stored":2678316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":281650,"internal_metadata":654750670},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:59 2023","interfaces":[{"interface":"back","average":{"1min":0.30499999999999999,"5min":0.439,"15min":0.36099999999999999},"min":{"1min":0.17399999999999999,"5min":0.14499999999999999,"15min":0.14499999999999999},"max":{"1min":0.60099999999999998,"5min":4.9379999999999997,"15min":4.9379999999999997},"last":0.379},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.49399999999999999,"15min":0.42399999999999999},"min":{"1min":0.221,"5min":0.19600000000000001,"15min":0.17000000000000001},"max":{"1min":0.63200000000000001,"5min":4.9859999999999998,"15min":4.9859999999999998},"last":0.40899999999999997}]},{"osd":1,"last update":"Wed Nov 8 05:30:03 2023","interfaces":[{"interface":"back","average":{"1min":0.28499999999999998,"5min":0.65100000000000002,"15min":0.47099999999999997},"min":{"1min":0.20100000000000001,"5min":0.153,"15min":0.153},"max":{"1min":0.39800000000000002,"5min":23.242999999999999,"15min":23.242999999999999},"last":0.41099999999999998},{"interface":"front","average":{"1min":0.29499999999999998,"5min":0.63100000000000001,"15min":0.45900000000000002},"min":{"1min":0.224,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":0.436,"5min":23.251000000000001,"15min":23.251000000000001},"last":0.45300000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738798,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1339228,"kb_used_data":398304,"kb_used_omap":291,"kb_used_meta":936476,"kb_avail":93032612,"statfs":{"total":96636764160,"available":95265394688,"internally_reserved":0,"allocated":407863296,"data_stored":405103900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":298462,"internal_metadata":958951970},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.436,"15min":0.36299999999999999},"min":{"1min":0.19700000000000001,"5min":0.14699999999999999,"15min":0.13700000000000001},"max":{"1min":3.222,"5min":6.5499999999999998,"15min":6.9989999999999997},"last":0.245},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.46300000000000002,"15min":0.42299999999999999},"min":{"1min":0.17799999999999999,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":5.0030000000000001,"5min":6.0629999999999997,"15min":9.6869999999999994},"last":0.28599999999999998}]},{"osd":2,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.51600000000000001,"15min":0.47099999999999997},"min":{"1min":0.21099999999999999,"5min":0.153,"15min":0.153},"max":{"1min":5.7439999999999998,"5min":5.8280000000000003,"15min":9.7639999999999993},"last":0.252},{"interface":"front","average":{"1min":0.497,"5min":0.56200000000000006,"15min":0.42299999999999999},"min":{"1min":0.20399999999999999,"5min":0.187,"15min":0.16800000000000001},"max":{"1min":3.9319999999999999,"5min":8.8040000000000003,"15min":8.8040000000000003},"last":0.29899999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738822,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":650064,"kb_used_data":5320,"kb_used_omap":214,"kb_used_meta":644521,"kb_avail":93721776,"statfs":{"total":96636764160,"available":95971098624,"internally_reserved":0,"allocated":5447680,"data_stored":2679285,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":219335,"internal_metadata":659990329},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:56 2023","interfaces":[{"interface":"back","average":{"1min":0.30399999999999999,"5min":0.34699999999999998,"15min":0.36499999999999999},"min":{"1min":0.23999999999999999,"5min":0.14199999999999999,"15min":0.14199999999999999},"max":{"1min":0.42399999999999999,"5min":1.595,"15min":7.4249999999999998},"last":0.17100000000000001},{"interface":"front","average":{"1min":0.35499999999999998,"5min":0.40899999999999997,"15min":0.376},"min":{"1min":0.23999999999999999,"5min":0.19600000000000001,"15min":0.183},"max":{"1min":0.46999999999999997,"5min":2.6859999999999999,"15min":2.6859999999999999},"last":0.23000000000000001}]},{"osd":2,"last update":"Wed Nov 8 05:29:57 2023","interfaces":[{"interface":"back","average":{"1min":0.28199999999999997,"5min":0.36099999999999999,"15min":0.373},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":0.34399999999999997,"5min":2.2650000000000001,"15min":5.8209999999999997},"last":0.248},{"interface":"front","average":{"1min":0.28599999999999998,"5min":0.31,"15min":0.35799999999999998},"min":{"1min":0.17199999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.35699999999999998,"5min":1.137,"15min":4.2350000000000003},"last":0.28399999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-08T05:30:15.607 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-11-08T05:30:15.890 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T05:30:15.892 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2023-11-08T05:30:15.903 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":1578,"stamp":"2023-11-08T05:30:14.053864+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2634296,"kb_used_data":408940,"kb_used_omap":780,"kb_used_meta":2220403,"kb_avail":280481224,"statfs":{"total":289910292480,"available":287212773376,"internally_reserved":0,"allocated":418754560,"data_stored":410461501,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":799447,"internal_metadata":2273692969},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-536782902,"num_objects":-164,"num_object_clones":0,"num_object_copies":-328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-115,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-164,"num_whiteouts":0,"num_read":-122056,"num_read_kb":-39011942,"num_write":-413642,"num_write_kb":-151705339,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-8,"num_bytes_recovered":-16922624,"num_keys_recovered":-2,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.424817"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":456,"reported_epoch":455,"state":"active+undersized","last_fresh":"2023-11-08T05:29:43.389354+0000","last_change":"2023-11-08T05:28:13.557071+0000","last_active":"2023-11-08T05:29:43.389354+0000","last_peered":"2023-11-08T05:29:43.389354+0000","last_clean":"2023-11-08T05:28:01.458439+0000","last_became_active":"2023-11-08T05:28:13.557071+0000","last_became_peered":"2023-11-08T05:28:13.557071+0000","last_unstale":"2023-11-08T05:29:43.389354+0000","last_undegraded":"2023-11-08T05:29:43.389354+0000","last_fullsized":"2023-11-08T05:28:13.547025+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":18,"reported_epoch":455,"state":"active+undersized","last_fresh":"2023-11-08T05:29:43.389262+0000","last_change":"2023-11-08T05:28:13.559513+0000","last_active":"2023-11-08T05:29:43.389262+0000","last_peered":"2023-11-08T05:29:43.389262+0000","last_clean":"2023-11-08T04:52:46.531964+0000","last_became_active":"2023-11-08T05:28:13.559513+0000","last_became_peered":"2023-11-08T05:28:13.559513+0000","last_unstale":"2023-11-08T05:29:43.389262+0000","last_undegraded":"2023-11-08T05:29:43.389262+0000","last_fullsized":"2023-11-08T05:28:13.540852+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":453,"reported_epoch":455,"state":"active+clean","last_fresh":"2023-11-08T05:29:43.390991+0000","last_change":"2023-11-08T04:52:47.546100+0000","last_active":"2023-11-08T05:29:43.390991+0000","last_peered":"2023-11-08T05:29:43.390991+0000","last_clean":"2023-11-08T05:29:43.390991+0000","last_became_active":"2023-11-08T04:52:47.545840+0000","last_became_peered":"2023-11-08T04:52:47.545840+0000","last_unstale":"2023-11-08T05:29:43.390991+0000","last_undegraded":"2023-11-08T05:29:43.390991+0000","last_fullsized":"2023-11-08T05:29:43.390991+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":451,"reported_epoch":455,"state":"active+clean","last_fresh":"2023-11-08T05:29:43.389261+0000","last_change":"2023-11-08T04:52:47.548597+0000","last_active":"2023-11-08T05:29:43.389261+0000","last_peered":"2023-11-08T05:29:43.389261+0000","last_clean":"2023-11-08T05:29:43.389261+0000","last_became_active":"2023-11-08T04:52:47.548294+0000","last_became_peered":"2023-11-08T04:52:47.548294+0000","last_unstale":"2023-11-08T05:29:43.389261+0000","last_undegraded":"2023-11-08T05:29:43.389261+0000","last_fullsized":"2023-11-08T05:29:43.389261+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":457,"reported_epoch":455,"state":"active+undersized","last_fresh":"2023-11-08T05:29:43.389822+0000","last_change":"2023-11-08T05:28:13.545468+0000","last_active":"2023-11-08T05:29:43.389822+0000","last_peered":"2023-11-08T05:29:43.389822+0000","last_clean":"2023-11-08T05:28:01.456723+0000","last_became_active":"2023-11-08T05:28:13.545468+0000","last_became_peered":"2023-11-08T05:28:13.545468+0000","last_unstale":"2023-11-08T05:29:43.389822+0000","last_undegraded":"2023-11-08T05:29:43.389822+0000","last_fullsized":"2023-11-08T05:28:13.537207+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_clean_scrub_stamp":"2023-11-08T04:52:45.528796+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":462,"reported_epoch":456,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:29:44.395350+0000","last_change":"2023-11-08T05:29:44.395350+0000","last_active":"2023-11-08T05:29:44.395350+0000","last_peered":"2023-11-08T05:29:44.395350+0000","last_clean":"2023-11-08T05:29:44.395350+0000","last_became_active":"2023-11-08T05:29:44.395012+0000","last_became_peered":"2023-11-08T05:29:44.395012+0000","last_unstale":"2023-11-08T05:29:44.395350+0000","last_undegraded":"2023-11-08T05:29:44.395350+0000","last_fullsized":"2023-11-08T05:29:44.395350+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":456,"reported_epoch":455,"state":"active+undersized","last_fresh":"2023-11-08T05:29:43.389390+0000","last_change":"2023-11-08T05:28:13.563357+0000","last_active":"2023-11-08T05:29:43.389390+0000","last_peered":"2023-11-08T05:29:43.389390+0000","last_clean":"2023-11-08T05:28:01.457691+0000","last_became_active":"2023-11-08T05:28:13.563357+0000","last_became_peered":"2023-11-08T05:28:13.563357+0000","last_unstale":"2023-11-08T05:29:43.389390+0000","last_undegraded":"2023-11-08T05:29:43.389390+0000","last_fullsized":"2023-11-08T05:28:13.541614+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":456,"reported_epoch":455,"state":"active+undersized","last_fresh":"2023-11-08T05:29:43.389311+0000","last_change":"2023-11-08T05:28:13.550672+0000","last_active":"2023-11-08T05:29:43.389311+0000","last_peered":"2023-11-08T05:29:43.389311+0000","last_clean":"2023-11-08T05:28:01.457904+0000","last_became_active":"2023-11-08T05:28:13.550672+0000","last_became_peered":"2023-11-08T05:28:13.550672+0000","last_unstale":"2023-11-08T05:29:43.389311+0000","last_undegraded":"2023-11-08T05:29:43.389311+0000","last_fullsized":"2023-11-08T05:28:13.539595+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":456,"reported_epoch":455,"state":"active+undersized","last_fresh":"2023-11-08T05:29:43.389474+0000","last_change":"2023-11-08T05:28:13.557968+0000","last_active":"2023-11-08T05:29:43.389474+0000","last_peered":"2023-11-08T05:29:43.389474+0000","last_clean":"2023-11-08T05:28:01.458433+0000","last_became_active":"2023-11-08T05:28:13.557968+0000","last_became_peered":"2023-11-08T05:28:13.557968+0000","last_unstale":"2023-11-08T05:29:43.389474+0000","last_undegraded":"2023-11-08T05:29:43.389474+0000","last_fullsized":"2023-11-08T05:28:13.542832+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":1,"acting":1,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":10,"acting":11,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738820,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":645004,"kb_used_data":5316,"kb_used_omap":275,"kb_used_meta":639404,"kb_avail":93726836,"statfs":{"total":96636764160,"available":95976280064,"internally_reserved":0,"allocated":5443584,"data_stored":2678316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":281650,"internal_metadata":654750670},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:59 2023","interfaces":[{"interface":"back","average":{"1min":0.30499999999999999,"5min":0.439,"15min":0.36099999999999999},"min":{"1min":0.17399999999999999,"5min":0.14499999999999999,"15min":0.14499999999999999},"max":{"1min":0.60099999999999998,"5min":4.9379999999999997,"15min":4.9379999999999997},"last":0.379},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.49399999999999999,"15min":0.42399999999999999},"min":{"1min":0.221,"5min":0.19600000000000001,"15min":0.17000000000000001},"max":{"1min":0.63200000000000001,"5min":4.9859999999999998,"15min":4.9859999999999998},"last":0.40899999999999997}]},{"osd":1,"last update":"Wed Nov 8 05:30:03 2023","interfaces":[{"interface":"back","average":{"1min":0.28499999999999998,"5min":0.65100000000000002,"15min":0.47099999999999997},"min":{"1min":0.20100000000000001,"5min":0.153,"15min":0.153},"max":{"1min":0.39800000000000002,"5min":23.242999999999999,"15min":23.242999999999999},"last":0.41099999999999998},{"interface":"front","average":{"1min":0.29499999999999998,"5min":0.63100000000000001,"15min":0.45900000000000002},"min":{"1min":0.224,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":0.436,"5min":23.251000000000001,"15min":23.251000000000001},"last":0.45300000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738798,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1339228,"kb_used_data":398304,"kb_used_omap":291,"kb_used_meta":936476,"kb_avail":93032612,"statfs":{"total":96636764160,"available":95265394688,"internally_reserved":0,"allocated":407863296,"data_stored":405103900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":298462,"internal_metadata":958951970},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.436,"15min":0.36299999999999999},"min":{"1min":0.19700000000000001,"5min":0.14699999999999999,"15min":0.13700000000000001},"max":{"1min":3.222,"5min":6.5499999999999998,"15min":6.9989999999999997},"last":0.245},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.46300000000000002,"15min":0.42299999999999999},"min":{"1min":0.17799999999999999,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":5.0030000000000001,"5min":6.0629999999999997,"15min":9.6869999999999994},"last":0.28599999999999998}]},{"osd":2,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.51600000000000001,"15min":0.47099999999999997},"min":{"1min":0.21099999999999999,"5min":0.153,"15min":0.153},"max":{"1min":5.7439999999999998,"5min":5.8280000000000003,"15min":9.7639999999999993},"last":0.252},{"interface":"front","average":{"1min":0.497,"5min":0.56200000000000006,"15min":0.42299999999999999},"min":{"1min":0.20399999999999999,"5min":0.187,"15min":0.16800000000000001},"max":{"1min":3.9319999999999999,"5min":8.8040000000000003,"15min":8.8040000000000003},"last":0.29899999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738822,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":650064,"kb_used_data":5320,"kb_used_omap":214,"kb_used_meta":644521,"kb_avail":93721776,"statfs":{"total":96636764160,"available":95971098624,"internally_reserved":0,"allocated":5447680,"data_stored":2679285,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":219335,"internal_metadata":659990329},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:56 2023","interfaces":[{"interface":"back","average":{"1min":0.30399999999999999,"5min":0.34699999999999998,"15min":0.36499999999999999},"min":{"1min":0.23999999999999999,"5min":0.14199999999999999,"15min":0.14199999999999999},"max":{"1min":0.42399999999999999,"5min":1.595,"15min":7.4249999999999998},"last":0.17100000000000001},{"interface":"front","average":{"1min":0.35499999999999998,"5min":0.40899999999999997,"15min":0.376},"min":{"1min":0.23999999999999999,"5min":0.19600000000000001,"15min":0.183},"max":{"1min":0.46999999999999997,"5min":2.6859999999999999,"15min":2.6859999999999999},"last":0.23000000000000001}]},{"osd":2,"last update":"Wed Nov 8 05:29:57 2023","interfaces":[{"interface":"back","average":{"1min":0.28199999999999997,"5min":0.36099999999999999,"15min":0.373},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":0.34399999999999997,"5min":2.2650000000000001,"15min":5.8209999999999997},"last":0.248},{"interface":"front","average":{"1min":0.28599999999999998,"5min":0.31,"15min":0.35799999999999998},"min":{"1min":0.17199999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.35699999999999998,"5min":1.137,"15min":4.2350000000000003},"last":0.28399999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-08T05:30:15.904 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-11-08T05:30:15.904 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 456, 'reported_epoch': 455, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:29:43.389354+0000', 'last_change': '2023-11-08T05:28:13.557071+0000', 'last_active': '2023-11-08T05:29:43.389354+0000', 'last_peered': '2023-11-08T05:29:43.389354+0000', 'last_clean': '2023-11-08T05:28:01.458439+0000', 'last_became_active': '2023-11-08T05:28:13.557071+0000', 'last_became_peered': '2023-11-08T05:28:13.557071+0000', 'last_unstale': '2023-11-08T05:29:43.389354+0000', 'last_undegraded': '2023-11-08T05:29:43.389354+0000', 'last_fullsized': '2023-11-08T05:28:13.547025+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:30:15.905 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-11-08T05:30:15.905 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 18, 'reported_epoch': 455, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:29:43.389262+0000', 'last_change': '2023-11-08T05:28:13.559513+0000', 'last_active': '2023-11-08T05:29:43.389262+0000', 'last_peered': '2023-11-08T05:29:43.389262+0000', 'last_clean': '2023-11-08T04:52:46.531964+0000', 'last_became_active': '2023-11-08T05:28:13.559513+0000', 'last_became_peered': '2023-11-08T05:28:13.559513+0000', 'last_unstale': '2023-11-08T05:29:43.389262+0000', 'last_undegraded': '2023-11-08T05:29:43.389262+0000', 'last_fullsized': '2023-11-08T05:28:13.540852+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:30:15.906 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-11-08T05:30:15.906 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "0'0", 'reported_seq': 457, 'reported_epoch': 455, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:29:43.389822+0000', 'last_change': '2023-11-08T05:28:13.545468+0000', 'last_active': '2023-11-08T05:29:43.389822+0000', 'last_peered': '2023-11-08T05:29:43.389822+0000', 'last_clean': '2023-11-08T05:28:01.456723+0000', 'last_became_active': '2023-11-08T05:28:13.545468+0000', 'last_became_peered': '2023-11-08T05:28:13.545468+0000', 'last_unstale': '2023-11-08T05:29:43.389822+0000', 'last_undegraded': '2023-11-08T05:29:43.389822+0000', 'last_fullsized': '2023-11-08T05:28:13.537207+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 9, 'last_epoch_clean': 10, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:45.528796+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:45.528796+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:45.528796+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:30:15.906 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.0 is not active+clean 2023-11-08T05:30:15.906 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.0', 'version': "0'0", 'reported_seq': 462, 'reported_epoch': 456, 'state': 'active+clean+remapped', 'last_fresh': '2023-11-08T05:29:44.395350+0000', 'last_change': '2023-11-08T05:29:44.395350+0000', 'last_active': '2023-11-08T05:29:44.395350+0000', 'last_peered': '2023-11-08T05:29:44.395350+0000', 'last_clean': '2023-11-08T05:29:44.395350+0000', 'last_became_active': '2023-11-08T05:29:44.395012+0000', 'last_became_peered': '2023-11-08T05:29:44.395012+0000', 'last_unstale': '2023-11-08T05:29:44.395350+0000', 'last_undegraded': '2023-11-08T05:29:44.395350+0000', 'last_fullsized': '2023-11-08T05:29:44.395350+0000', 'mapping_epoch': 455, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 456, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2], 'acting': [2, 1], 'avail_no_missing': ['2', '1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2023-11-08T05:30:15.907 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-11-08T05:30:15.907 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 456, 'reported_epoch': 455, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:29:43.389390+0000', 'last_change': '2023-11-08T05:28:13.563357+0000', 'last_active': '2023-11-08T05:29:43.389390+0000', 'last_peered': '2023-11-08T05:29:43.389390+0000', 'last_clean': '2023-11-08T05:28:01.457691+0000', 'last_became_active': '2023-11-08T05:28:13.563357+0000', 'last_became_peered': '2023-11-08T05:28:13.563357+0000', 'last_unstale': '2023-11-08T05:29:43.389390+0000', 'last_undegraded': '2023-11-08T05:29:43.389390+0000', 'last_fullsized': '2023-11-08T05:28:13.541614+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:30:15.907 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-11-08T05:30:15.908 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 456, 'reported_epoch': 455, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:29:43.389311+0000', 'last_change': '2023-11-08T05:28:13.550672+0000', 'last_active': '2023-11-08T05:29:43.389311+0000', 'last_peered': '2023-11-08T05:29:43.389311+0000', 'last_clean': '2023-11-08T05:28:01.457904+0000', 'last_became_active': '2023-11-08T05:28:13.550672+0000', 'last_became_peered': '2023-11-08T05:28:13.550672+0000', 'last_unstale': '2023-11-08T05:29:43.389311+0000', 'last_undegraded': '2023-11-08T05:29:43.389311+0000', 'last_fullsized': '2023-11-08T05:28:13.539595+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:30:15.908 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-11-08T05:30:15.908 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 456, 'reported_epoch': 455, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:29:43.389474+0000', 'last_change': '2023-11-08T05:28:13.557968+0000', 'last_active': '2023-11-08T05:29:43.389474+0000', 'last_peered': '2023-11-08T05:29:43.389474+0000', 'last_clean': '2023-11-08T05:28:01.458433+0000', 'last_became_active': '2023-11-08T05:28:13.557968+0000', 'last_became_peered': '2023-11-08T05:28:13.557968+0000', 'last_unstale': '2023-11-08T05:29:43.389474+0000', 'last_undegraded': '2023-11-08T05:29:43.389474+0000', 'last_fullsized': '2023-11-08T05:28:13.542832+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:30:15.909 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-11-08T05:30:16.260 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T05:30:16.261 INFO:teuthology.orchestra.run.smithi087.stdout:{"fsid":"ab4cbc06-449d-497d-b513-220d9db2b590","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 6 pgs undersized","count":6},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2258,"monmap":{"epoch":1,"min_mon_release_name":"pacific","num_mons":1},"osdmap":{"epoch":456,"num_osds":3,"num_up_osds":2,"osd_up_since":1699421292,"num_in_osds":3,"osd_in_since":1699419158,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":6},{"state_name":"active+clean","count":2},{"state_name":"active+clean+remapped","count":1}],"num_pgs":9,"num_pools":2,"num_objects":0,"data_bytes":0,"bytes_used":2697519104,"bytes_avail":287212773376,"bytes_total":289910292480},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":41,"modified":"2023-11-08T05:29:32.044544+0000","services":{}},"progress_events":{}} 2023-11-08T05:30:16.274 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-11-08T05:30:16.274 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-11-08T05:30:16.558 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T05:30:16.560 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2023-11-08T05:30:16.571 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":1579,"stamp":"2023-11-08T05:30:16.054252+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2634296,"kb_used_data":408940,"kb_used_omap":780,"kb_used_meta":2220403,"kb_avail":280481224,"statfs":{"total":289910292480,"available":287212773376,"internally_reserved":0,"allocated":418754560,"data_stored":410461501,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":799447,"internal_metadata":2273692969},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-536782902,"num_objects":-164,"num_object_clones":0,"num_object_copies":-328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-115,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-164,"num_whiteouts":0,"num_read":-122056,"num_read_kb":-39011942,"num_write":-413642,"num_write_kb":-151705339,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-8,"num_bytes_recovered":-16922624,"num_keys_recovered":-2,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.424817"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":456,"reported_epoch":455,"state":"active+undersized","last_fresh":"2023-11-08T05:29:43.389354+0000","last_change":"2023-11-08T05:28:13.557071+0000","last_active":"2023-11-08T05:29:43.389354+0000","last_peered":"2023-11-08T05:29:43.389354+0000","last_clean":"2023-11-08T05:28:01.458439+0000","last_became_active":"2023-11-08T05:28:13.557071+0000","last_became_peered":"2023-11-08T05:28:13.557071+0000","last_unstale":"2023-11-08T05:29:43.389354+0000","last_undegraded":"2023-11-08T05:29:43.389354+0000","last_fullsized":"2023-11-08T05:28:13.547025+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":18,"reported_epoch":455,"state":"active+undersized","last_fresh":"2023-11-08T05:29:43.389262+0000","last_change":"2023-11-08T05:28:13.559513+0000","last_active":"2023-11-08T05:29:43.389262+0000","last_peered":"2023-11-08T05:29:43.389262+0000","last_clean":"2023-11-08T04:52:46.531964+0000","last_became_active":"2023-11-08T05:28:13.559513+0000","last_became_peered":"2023-11-08T05:28:13.559513+0000","last_unstale":"2023-11-08T05:29:43.389262+0000","last_undegraded":"2023-11-08T05:29:43.389262+0000","last_fullsized":"2023-11-08T05:28:13.540852+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":453,"reported_epoch":455,"state":"active+clean","last_fresh":"2023-11-08T05:29:43.390991+0000","last_change":"2023-11-08T04:52:47.546100+0000","last_active":"2023-11-08T05:29:43.390991+0000","last_peered":"2023-11-08T05:29:43.390991+0000","last_clean":"2023-11-08T05:29:43.390991+0000","last_became_active":"2023-11-08T04:52:47.545840+0000","last_became_peered":"2023-11-08T04:52:47.545840+0000","last_unstale":"2023-11-08T05:29:43.390991+0000","last_undegraded":"2023-11-08T05:29:43.390991+0000","last_fullsized":"2023-11-08T05:29:43.390991+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":451,"reported_epoch":455,"state":"active+clean","last_fresh":"2023-11-08T05:29:43.389261+0000","last_change":"2023-11-08T04:52:47.548597+0000","last_active":"2023-11-08T05:29:43.389261+0000","last_peered":"2023-11-08T05:29:43.389261+0000","last_clean":"2023-11-08T05:29:43.389261+0000","last_became_active":"2023-11-08T04:52:47.548294+0000","last_became_peered":"2023-11-08T04:52:47.548294+0000","last_unstale":"2023-11-08T05:29:43.389261+0000","last_undegraded":"2023-11-08T05:29:43.389261+0000","last_fullsized":"2023-11-08T05:29:43.389261+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":457,"reported_epoch":455,"state":"active+undersized","last_fresh":"2023-11-08T05:29:43.389822+0000","last_change":"2023-11-08T05:28:13.545468+0000","last_active":"2023-11-08T05:29:43.389822+0000","last_peered":"2023-11-08T05:29:43.389822+0000","last_clean":"2023-11-08T05:28:01.456723+0000","last_became_active":"2023-11-08T05:28:13.545468+0000","last_became_peered":"2023-11-08T05:28:13.545468+0000","last_unstale":"2023-11-08T05:29:43.389822+0000","last_undegraded":"2023-11-08T05:29:43.389822+0000","last_fullsized":"2023-11-08T05:28:13.537207+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_clean_scrub_stamp":"2023-11-08T04:52:45.528796+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":462,"reported_epoch":456,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:29:44.395350+0000","last_change":"2023-11-08T05:29:44.395350+0000","last_active":"2023-11-08T05:29:44.395350+0000","last_peered":"2023-11-08T05:29:44.395350+0000","last_clean":"2023-11-08T05:29:44.395350+0000","last_became_active":"2023-11-08T05:29:44.395012+0000","last_became_peered":"2023-11-08T05:29:44.395012+0000","last_unstale":"2023-11-08T05:29:44.395350+0000","last_undegraded":"2023-11-08T05:29:44.395350+0000","last_fullsized":"2023-11-08T05:29:44.395350+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":456,"reported_epoch":455,"state":"active+undersized","last_fresh":"2023-11-08T05:29:43.389390+0000","last_change":"2023-11-08T05:28:13.563357+0000","last_active":"2023-11-08T05:29:43.389390+0000","last_peered":"2023-11-08T05:29:43.389390+0000","last_clean":"2023-11-08T05:28:01.457691+0000","last_became_active":"2023-11-08T05:28:13.563357+0000","last_became_peered":"2023-11-08T05:28:13.563357+0000","last_unstale":"2023-11-08T05:29:43.389390+0000","last_undegraded":"2023-11-08T05:29:43.389390+0000","last_fullsized":"2023-11-08T05:28:13.541614+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":456,"reported_epoch":455,"state":"active+undersized","last_fresh":"2023-11-08T05:29:43.389311+0000","last_change":"2023-11-08T05:28:13.550672+0000","last_active":"2023-11-08T05:29:43.389311+0000","last_peered":"2023-11-08T05:29:43.389311+0000","last_clean":"2023-11-08T05:28:01.457904+0000","last_became_active":"2023-11-08T05:28:13.550672+0000","last_became_peered":"2023-11-08T05:28:13.550672+0000","last_unstale":"2023-11-08T05:29:43.389311+0000","last_undegraded":"2023-11-08T05:29:43.389311+0000","last_fullsized":"2023-11-08T05:28:13.539595+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":456,"reported_epoch":455,"state":"active+undersized","last_fresh":"2023-11-08T05:29:43.389474+0000","last_change":"2023-11-08T05:28:13.557968+0000","last_active":"2023-11-08T05:29:43.389474+0000","last_peered":"2023-11-08T05:29:43.389474+0000","last_clean":"2023-11-08T05:28:01.458433+0000","last_became_active":"2023-11-08T05:28:13.557968+0000","last_became_peered":"2023-11-08T05:28:13.557968+0000","last_unstale":"2023-11-08T05:29:43.389474+0000","last_undegraded":"2023-11-08T05:29:43.389474+0000","last_fullsized":"2023-11-08T05:28:13.542832+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":1,"acting":1,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":10,"acting":11,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738821,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":645004,"kb_used_data":5316,"kb_used_omap":275,"kb_used_meta":639404,"kb_avail":93726836,"statfs":{"total":96636764160,"available":95976280064,"internally_reserved":0,"allocated":5443584,"data_stored":2678316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":281650,"internal_metadata":654750670},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:59 2023","interfaces":[{"interface":"back","average":{"1min":0.30499999999999999,"5min":0.439,"15min":0.36099999999999999},"min":{"1min":0.17399999999999999,"5min":0.14499999999999999,"15min":0.14499999999999999},"max":{"1min":0.60099999999999998,"5min":4.9379999999999997,"15min":4.9379999999999997},"last":0.379},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.49399999999999999,"15min":0.42399999999999999},"min":{"1min":0.221,"5min":0.19600000000000001,"15min":0.17000000000000001},"max":{"1min":0.63200000000000001,"5min":4.9859999999999998,"15min":4.9859999999999998},"last":0.40899999999999997}]},{"osd":1,"last update":"Wed Nov 8 05:30:03 2023","interfaces":[{"interface":"back","average":{"1min":0.28499999999999998,"5min":0.65100000000000002,"15min":0.47099999999999997},"min":{"1min":0.20100000000000001,"5min":0.153,"15min":0.153},"max":{"1min":0.39800000000000002,"5min":23.242999999999999,"15min":23.242999999999999},"last":0.27300000000000002},{"interface":"front","average":{"1min":0.29499999999999998,"5min":0.63100000000000001,"15min":0.45900000000000002},"min":{"1min":0.224,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":0.436,"5min":23.251000000000001,"15min":23.251000000000001},"last":0.28599999999999998}]}]},{"osd":0,"up_from":8,"seq":34359738798,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1339228,"kb_used_data":398304,"kb_used_omap":291,"kb_used_meta":936476,"kb_avail":93032612,"statfs":{"total":96636764160,"available":95265394688,"internally_reserved":0,"allocated":407863296,"data_stored":405103900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":298462,"internal_metadata":958951970},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.436,"15min":0.36299999999999999},"min":{"1min":0.19700000000000001,"5min":0.14699999999999999,"15min":0.13700000000000001},"max":{"1min":3.222,"5min":6.5499999999999998,"15min":6.9989999999999997},"last":0.245},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.46300000000000002,"15min":0.42299999999999999},"min":{"1min":0.17799999999999999,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":5.0030000000000001,"5min":6.0629999999999997,"15min":9.6869999999999994},"last":0.28599999999999998}]},{"osd":2,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.51600000000000001,"15min":0.47099999999999997},"min":{"1min":0.21099999999999999,"5min":0.153,"15min":0.153},"max":{"1min":5.7439999999999998,"5min":5.8280000000000003,"15min":9.7639999999999993},"last":0.252},{"interface":"front","average":{"1min":0.497,"5min":0.56200000000000006,"15min":0.42299999999999999},"min":{"1min":0.20399999999999999,"5min":0.187,"15min":0.16800000000000001},"max":{"1min":3.9319999999999999,"5min":8.8040000000000003,"15min":8.8040000000000003},"last":0.29899999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738823,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":650064,"kb_used_data":5320,"kb_used_omap":214,"kb_used_meta":644521,"kb_avail":93721776,"statfs":{"total":96636764160,"available":95971098624,"internally_reserved":0,"allocated":5447680,"data_stored":2679285,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":219335,"internal_metadata":659990329},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:56 2023","interfaces":[{"interface":"back","average":{"1min":0.30399999999999999,"5min":0.34699999999999998,"15min":0.36499999999999999},"min":{"1min":0.23999999999999999,"5min":0.14199999999999999,"15min":0.14199999999999999},"max":{"1min":0.42399999999999999,"5min":1.595,"15min":7.4249999999999998},"last":0.17100000000000001},{"interface":"front","average":{"1min":0.35499999999999998,"5min":0.40899999999999997,"15min":0.376},"min":{"1min":0.23999999999999999,"5min":0.19600000000000001,"15min":0.183},"max":{"1min":0.46999999999999997,"5min":2.6859999999999999,"15min":2.6859999999999999},"last":0.23000000000000001}]},{"osd":2,"last update":"Wed Nov 8 05:29:57 2023","interfaces":[{"interface":"back","average":{"1min":0.28199999999999997,"5min":0.36099999999999999,"15min":0.373},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":0.34399999999999997,"5min":2.2650000000000001,"15min":5.8209999999999997},"last":0.39300000000000002},{"interface":"front","average":{"1min":0.28599999999999998,"5min":0.31,"15min":0.35799999999999998},"min":{"1min":0.17199999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.35699999999999998,"5min":1.137,"15min":4.2350000000000003},"last":0.41999999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-08T05:30:16.933 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~119s 2023-11-08T05:30:19.573 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-11-08T05:30:19.871 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T05:30:19.872 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2023-11-08T05:30:19.884 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":1580,"stamp":"2023-11-08T05:30:18.054738+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2634296,"kb_used_data":408940,"kb_used_omap":780,"kb_used_meta":2220403,"kb_avail":280481224,"statfs":{"total":289910292480,"available":287212773376,"internally_reserved":0,"allocated":418754560,"data_stored":410461501,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":799447,"internal_metadata":2273692969},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-536782902,"num_objects":-164,"num_object_clones":0,"num_object_copies":-328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-115,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-164,"num_whiteouts":0,"num_read":-122056,"num_read_kb":-39011942,"num_write":-413642,"num_write_kb":-151705339,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-8,"num_bytes_recovered":-16922624,"num_keys_recovered":-2,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.424817"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":456,"reported_epoch":455,"state":"active+undersized","last_fresh":"2023-11-08T05:29:43.389354+0000","last_change":"2023-11-08T05:28:13.557071+0000","last_active":"2023-11-08T05:29:43.389354+0000","last_peered":"2023-11-08T05:29:43.389354+0000","last_clean":"2023-11-08T05:28:01.458439+0000","last_became_active":"2023-11-08T05:28:13.557071+0000","last_became_peered":"2023-11-08T05:28:13.557071+0000","last_unstale":"2023-11-08T05:29:43.389354+0000","last_undegraded":"2023-11-08T05:29:43.389354+0000","last_fullsized":"2023-11-08T05:28:13.547025+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":18,"reported_epoch":455,"state":"active+undersized","last_fresh":"2023-11-08T05:29:43.389262+0000","last_change":"2023-11-08T05:28:13.559513+0000","last_active":"2023-11-08T05:29:43.389262+0000","last_peered":"2023-11-08T05:29:43.389262+0000","last_clean":"2023-11-08T04:52:46.531964+0000","last_became_active":"2023-11-08T05:28:13.559513+0000","last_became_peered":"2023-11-08T05:28:13.559513+0000","last_unstale":"2023-11-08T05:29:43.389262+0000","last_undegraded":"2023-11-08T05:29:43.389262+0000","last_fullsized":"2023-11-08T05:28:13.540852+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":453,"reported_epoch":455,"state":"active+clean","last_fresh":"2023-11-08T05:29:43.390991+0000","last_change":"2023-11-08T04:52:47.546100+0000","last_active":"2023-11-08T05:29:43.390991+0000","last_peered":"2023-11-08T05:29:43.390991+0000","last_clean":"2023-11-08T05:29:43.390991+0000","last_became_active":"2023-11-08T04:52:47.545840+0000","last_became_peered":"2023-11-08T04:52:47.545840+0000","last_unstale":"2023-11-08T05:29:43.390991+0000","last_undegraded":"2023-11-08T05:29:43.390991+0000","last_fullsized":"2023-11-08T05:29:43.390991+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":451,"reported_epoch":455,"state":"active+clean","last_fresh":"2023-11-08T05:29:43.389261+0000","last_change":"2023-11-08T04:52:47.548597+0000","last_active":"2023-11-08T05:29:43.389261+0000","last_peered":"2023-11-08T05:29:43.389261+0000","last_clean":"2023-11-08T05:29:43.389261+0000","last_became_active":"2023-11-08T04:52:47.548294+0000","last_became_peered":"2023-11-08T04:52:47.548294+0000","last_unstale":"2023-11-08T05:29:43.389261+0000","last_undegraded":"2023-11-08T05:29:43.389261+0000","last_fullsized":"2023-11-08T05:29:43.389261+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":457,"reported_epoch":455,"state":"active+undersized","last_fresh":"2023-11-08T05:29:43.389822+0000","last_change":"2023-11-08T05:28:13.545468+0000","last_active":"2023-11-08T05:29:43.389822+0000","last_peered":"2023-11-08T05:29:43.389822+0000","last_clean":"2023-11-08T05:28:01.456723+0000","last_became_active":"2023-11-08T05:28:13.545468+0000","last_became_peered":"2023-11-08T05:28:13.545468+0000","last_unstale":"2023-11-08T05:29:43.389822+0000","last_undegraded":"2023-11-08T05:29:43.389822+0000","last_fullsized":"2023-11-08T05:28:13.537207+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_clean_scrub_stamp":"2023-11-08T04:52:45.528796+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":462,"reported_epoch":456,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:29:44.395350+0000","last_change":"2023-11-08T05:29:44.395350+0000","last_active":"2023-11-08T05:29:44.395350+0000","last_peered":"2023-11-08T05:29:44.395350+0000","last_clean":"2023-11-08T05:29:44.395350+0000","last_became_active":"2023-11-08T05:29:44.395012+0000","last_became_peered":"2023-11-08T05:29:44.395012+0000","last_unstale":"2023-11-08T05:29:44.395350+0000","last_undegraded":"2023-11-08T05:29:44.395350+0000","last_fullsized":"2023-11-08T05:29:44.395350+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":456,"reported_epoch":455,"state":"active+undersized","last_fresh":"2023-11-08T05:29:43.389390+0000","last_change":"2023-11-08T05:28:13.563357+0000","last_active":"2023-11-08T05:29:43.389390+0000","last_peered":"2023-11-08T05:29:43.389390+0000","last_clean":"2023-11-08T05:28:01.457691+0000","last_became_active":"2023-11-08T05:28:13.563357+0000","last_became_peered":"2023-11-08T05:28:13.563357+0000","last_unstale":"2023-11-08T05:29:43.389390+0000","last_undegraded":"2023-11-08T05:29:43.389390+0000","last_fullsized":"2023-11-08T05:28:13.541614+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":456,"reported_epoch":455,"state":"active+undersized","last_fresh":"2023-11-08T05:29:43.389311+0000","last_change":"2023-11-08T05:28:13.550672+0000","last_active":"2023-11-08T05:29:43.389311+0000","last_peered":"2023-11-08T05:29:43.389311+0000","last_clean":"2023-11-08T05:28:01.457904+0000","last_became_active":"2023-11-08T05:28:13.550672+0000","last_became_peered":"2023-11-08T05:28:13.550672+0000","last_unstale":"2023-11-08T05:29:43.389311+0000","last_undegraded":"2023-11-08T05:29:43.389311+0000","last_fullsized":"2023-11-08T05:28:13.539595+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":456,"reported_epoch":455,"state":"active+undersized","last_fresh":"2023-11-08T05:29:43.389474+0000","last_change":"2023-11-08T05:28:13.557968+0000","last_active":"2023-11-08T05:29:43.389474+0000","last_peered":"2023-11-08T05:29:43.389474+0000","last_clean":"2023-11-08T05:28:01.458433+0000","last_became_active":"2023-11-08T05:28:13.557968+0000","last_became_peered":"2023-11-08T05:28:13.557968+0000","last_unstale":"2023-11-08T05:29:43.389474+0000","last_undegraded":"2023-11-08T05:29:43.389474+0000","last_fullsized":"2023-11-08T05:28:13.542832+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":1,"acting":1,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":10,"acting":11,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738821,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":645004,"kb_used_data":5316,"kb_used_omap":275,"kb_used_meta":639404,"kb_avail":93726836,"statfs":{"total":96636764160,"available":95976280064,"internally_reserved":0,"allocated":5443584,"data_stored":2678316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":281650,"internal_metadata":654750670},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:59 2023","interfaces":[{"interface":"back","average":{"1min":0.30499999999999999,"5min":0.439,"15min":0.36099999999999999},"min":{"1min":0.17399999999999999,"5min":0.14499999999999999,"15min":0.14499999999999999},"max":{"1min":0.60099999999999998,"5min":4.9379999999999997,"15min":4.9379999999999997},"last":0.379},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.49399999999999999,"15min":0.42399999999999999},"min":{"1min":0.221,"5min":0.19600000000000001,"15min":0.17000000000000001},"max":{"1min":0.63200000000000001,"5min":4.9859999999999998,"15min":4.9859999999999998},"last":0.40899999999999997}]},{"osd":1,"last update":"Wed Nov 8 05:30:03 2023","interfaces":[{"interface":"back","average":{"1min":0.28499999999999998,"5min":0.65100000000000002,"15min":0.47099999999999997},"min":{"1min":0.20100000000000001,"5min":0.153,"15min":0.153},"max":{"1min":0.39800000000000002,"5min":23.242999999999999,"15min":23.242999999999999},"last":0.27300000000000002},{"interface":"front","average":{"1min":0.29499999999999998,"5min":0.63100000000000001,"15min":0.45900000000000002},"min":{"1min":0.224,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":0.436,"5min":23.251000000000001,"15min":23.251000000000001},"last":0.28599999999999998}]}]},{"osd":0,"up_from":8,"seq":34359738798,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1339228,"kb_used_data":398304,"kb_used_omap":291,"kb_used_meta":936476,"kb_avail":93032612,"statfs":{"total":96636764160,"available":95265394688,"internally_reserved":0,"allocated":407863296,"data_stored":405103900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":298462,"internal_metadata":958951970},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.436,"15min":0.36299999999999999},"min":{"1min":0.19700000000000001,"5min":0.14699999999999999,"15min":0.13700000000000001},"max":{"1min":3.222,"5min":6.5499999999999998,"15min":6.9989999999999997},"last":0.245},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.46300000000000002,"15min":0.42299999999999999},"min":{"1min":0.17799999999999999,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":5.0030000000000001,"5min":6.0629999999999997,"15min":9.6869999999999994},"last":0.28599999999999998}]},{"osd":2,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.51600000000000001,"15min":0.47099999999999997},"min":{"1min":0.21099999999999999,"5min":0.153,"15min":0.153},"max":{"1min":5.7439999999999998,"5min":5.8280000000000003,"15min":9.7639999999999993},"last":0.252},{"interface":"front","average":{"1min":0.497,"5min":0.56200000000000006,"15min":0.42299999999999999},"min":{"1min":0.20399999999999999,"5min":0.187,"15min":0.16800000000000001},"max":{"1min":3.9319999999999999,"5min":8.8040000000000003,"15min":8.8040000000000003},"last":0.29899999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738823,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":650064,"kb_used_data":5320,"kb_used_omap":214,"kb_used_meta":644521,"kb_avail":93721776,"statfs":{"total":96636764160,"available":95971098624,"internally_reserved":0,"allocated":5447680,"data_stored":2679285,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":219335,"internal_metadata":659990329},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:56 2023","interfaces":[{"interface":"back","average":{"1min":0.30399999999999999,"5min":0.34699999999999998,"15min":0.36499999999999999},"min":{"1min":0.23999999999999999,"5min":0.14199999999999999,"15min":0.14199999999999999},"max":{"1min":0.42399999999999999,"5min":1.595,"15min":7.4249999999999998},"last":0.17100000000000001},{"interface":"front","average":{"1min":0.35499999999999998,"5min":0.40899999999999997,"15min":0.376},"min":{"1min":0.23999999999999999,"5min":0.19600000000000001,"15min":0.183},"max":{"1min":0.46999999999999997,"5min":2.6859999999999999,"15min":2.6859999999999999},"last":0.23000000000000001}]},{"osd":2,"last update":"Wed Nov 8 05:29:57 2023","interfaces":[{"interface":"back","average":{"1min":0.28199999999999997,"5min":0.36099999999999999,"15min":0.373},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":0.34399999999999997,"5min":2.2650000000000001,"15min":5.8209999999999997},"last":0.39300000000000002},{"interface":"front","average":{"1min":0.28599999999999998,"5min":0.31,"15min":0.35799999999999998},"min":{"1min":0.17199999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.35699999999999998,"5min":1.137,"15min":4.2350000000000003},"last":0.41999999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-08T05:30:19.885 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-11-08T05:30:20.168 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T05:30:20.170 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2023-11-08T05:30:20.181 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":1581,"stamp":"2023-11-08T05:30:20.055133+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2634296,"kb_used_data":408940,"kb_used_omap":780,"kb_used_meta":2220403,"kb_avail":280481224,"statfs":{"total":289910292480,"available":287212773376,"internally_reserved":0,"allocated":418754560,"data_stored":410461501,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":799447,"internal_metadata":2273692969},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-536782902,"num_objects":-164,"num_object_clones":0,"num_object_copies":-328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-115,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-164,"num_whiteouts":0,"num_read":-122056,"num_read_kb":-39011942,"num_write":-413642,"num_write_kb":-151705339,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-8,"num_bytes_recovered":-16922624,"num_keys_recovered":-2,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.424817"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":456,"reported_epoch":455,"state":"active+undersized","last_fresh":"2023-11-08T05:29:43.389354+0000","last_change":"2023-11-08T05:28:13.557071+0000","last_active":"2023-11-08T05:29:43.389354+0000","last_peered":"2023-11-08T05:29:43.389354+0000","last_clean":"2023-11-08T05:28:01.458439+0000","last_became_active":"2023-11-08T05:28:13.557071+0000","last_became_peered":"2023-11-08T05:28:13.557071+0000","last_unstale":"2023-11-08T05:29:43.389354+0000","last_undegraded":"2023-11-08T05:29:43.389354+0000","last_fullsized":"2023-11-08T05:28:13.547025+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":18,"reported_epoch":455,"state":"active+undersized","last_fresh":"2023-11-08T05:29:43.389262+0000","last_change":"2023-11-08T05:28:13.559513+0000","last_active":"2023-11-08T05:29:43.389262+0000","last_peered":"2023-11-08T05:29:43.389262+0000","last_clean":"2023-11-08T04:52:46.531964+0000","last_became_active":"2023-11-08T05:28:13.559513+0000","last_became_peered":"2023-11-08T05:28:13.559513+0000","last_unstale":"2023-11-08T05:29:43.389262+0000","last_undegraded":"2023-11-08T05:29:43.389262+0000","last_fullsized":"2023-11-08T05:28:13.540852+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":453,"reported_epoch":455,"state":"active+clean","last_fresh":"2023-11-08T05:29:43.390991+0000","last_change":"2023-11-08T04:52:47.546100+0000","last_active":"2023-11-08T05:29:43.390991+0000","last_peered":"2023-11-08T05:29:43.390991+0000","last_clean":"2023-11-08T05:29:43.390991+0000","last_became_active":"2023-11-08T04:52:47.545840+0000","last_became_peered":"2023-11-08T04:52:47.545840+0000","last_unstale":"2023-11-08T05:29:43.390991+0000","last_undegraded":"2023-11-08T05:29:43.390991+0000","last_fullsized":"2023-11-08T05:29:43.390991+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":451,"reported_epoch":455,"state":"active+clean","last_fresh":"2023-11-08T05:29:43.389261+0000","last_change":"2023-11-08T04:52:47.548597+0000","last_active":"2023-11-08T05:29:43.389261+0000","last_peered":"2023-11-08T05:29:43.389261+0000","last_clean":"2023-11-08T05:29:43.389261+0000","last_became_active":"2023-11-08T04:52:47.548294+0000","last_became_peered":"2023-11-08T04:52:47.548294+0000","last_unstale":"2023-11-08T05:29:43.389261+0000","last_undegraded":"2023-11-08T05:29:43.389261+0000","last_fullsized":"2023-11-08T05:29:43.389261+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":457,"reported_epoch":455,"state":"active+undersized","last_fresh":"2023-11-08T05:29:43.389822+0000","last_change":"2023-11-08T05:28:13.545468+0000","last_active":"2023-11-08T05:29:43.389822+0000","last_peered":"2023-11-08T05:29:43.389822+0000","last_clean":"2023-11-08T05:28:01.456723+0000","last_became_active":"2023-11-08T05:28:13.545468+0000","last_became_peered":"2023-11-08T05:28:13.545468+0000","last_unstale":"2023-11-08T05:29:43.389822+0000","last_undegraded":"2023-11-08T05:29:43.389822+0000","last_fullsized":"2023-11-08T05:28:13.537207+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_clean_scrub_stamp":"2023-11-08T04:52:45.528796+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":462,"reported_epoch":456,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:29:44.395350+0000","last_change":"2023-11-08T05:29:44.395350+0000","last_active":"2023-11-08T05:29:44.395350+0000","last_peered":"2023-11-08T05:29:44.395350+0000","last_clean":"2023-11-08T05:29:44.395350+0000","last_became_active":"2023-11-08T05:29:44.395012+0000","last_became_peered":"2023-11-08T05:29:44.395012+0000","last_unstale":"2023-11-08T05:29:44.395350+0000","last_undegraded":"2023-11-08T05:29:44.395350+0000","last_fullsized":"2023-11-08T05:29:44.395350+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":456,"reported_epoch":455,"state":"active+undersized","last_fresh":"2023-11-08T05:29:43.389390+0000","last_change":"2023-11-08T05:28:13.563357+0000","last_active":"2023-11-08T05:29:43.389390+0000","last_peered":"2023-11-08T05:29:43.389390+0000","last_clean":"2023-11-08T05:28:01.457691+0000","last_became_active":"2023-11-08T05:28:13.563357+0000","last_became_peered":"2023-11-08T05:28:13.563357+0000","last_unstale":"2023-11-08T05:29:43.389390+0000","last_undegraded":"2023-11-08T05:29:43.389390+0000","last_fullsized":"2023-11-08T05:28:13.541614+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":456,"reported_epoch":455,"state":"active+undersized","last_fresh":"2023-11-08T05:29:43.389311+0000","last_change":"2023-11-08T05:28:13.550672+0000","last_active":"2023-11-08T05:29:43.389311+0000","last_peered":"2023-11-08T05:29:43.389311+0000","last_clean":"2023-11-08T05:28:01.457904+0000","last_became_active":"2023-11-08T05:28:13.550672+0000","last_became_peered":"2023-11-08T05:28:13.550672+0000","last_unstale":"2023-11-08T05:29:43.389311+0000","last_undegraded":"2023-11-08T05:29:43.389311+0000","last_fullsized":"2023-11-08T05:28:13.539595+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":456,"reported_epoch":455,"state":"active+undersized","last_fresh":"2023-11-08T05:29:43.389474+0000","last_change":"2023-11-08T05:28:13.557968+0000","last_active":"2023-11-08T05:29:43.389474+0000","last_peered":"2023-11-08T05:29:43.389474+0000","last_clean":"2023-11-08T05:28:01.458433+0000","last_became_active":"2023-11-08T05:28:13.557968+0000","last_became_peered":"2023-11-08T05:28:13.557968+0000","last_unstale":"2023-11-08T05:29:43.389474+0000","last_undegraded":"2023-11-08T05:29:43.389474+0000","last_fullsized":"2023-11-08T05:28:13.542832+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":1,"acting":1,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":10,"acting":11,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738822,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":645004,"kb_used_data":5316,"kb_used_omap":275,"kb_used_meta":639404,"kb_avail":93726836,"statfs":{"total":96636764160,"available":95976280064,"internally_reserved":0,"allocated":5443584,"data_stored":2678316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":281650,"internal_metadata":654750670},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:59 2023","interfaces":[{"interface":"back","average":{"1min":0.30499999999999999,"5min":0.439,"15min":0.36099999999999999},"min":{"1min":0.17399999999999999,"5min":0.14499999999999999,"15min":0.14499999999999999},"max":{"1min":0.60099999999999998,"5min":4.9379999999999997,"15min":4.9379999999999997},"last":0.379},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.49399999999999999,"15min":0.42399999999999999},"min":{"1min":0.221,"5min":0.19600000000000001,"15min":0.17000000000000001},"max":{"1min":0.63200000000000001,"5min":4.9859999999999998,"15min":4.9859999999999998},"last":0.40899999999999997}]},{"osd":1,"last update":"Wed Nov 8 05:30:03 2023","interfaces":[{"interface":"back","average":{"1min":0.28499999999999998,"5min":0.65100000000000002,"15min":0.47099999999999997},"min":{"1min":0.20100000000000001,"5min":0.153,"15min":0.153},"max":{"1min":0.39800000000000002,"5min":23.242999999999999,"15min":23.242999999999999},"last":0.29399999999999998},{"interface":"front","average":{"1min":0.29499999999999998,"5min":0.63100000000000001,"15min":0.45900000000000002},"min":{"1min":0.224,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":0.436,"5min":23.251000000000001,"15min":23.251000000000001},"last":0.32900000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738798,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1339228,"kb_used_data":398304,"kb_used_omap":291,"kb_used_meta":936476,"kb_avail":93032612,"statfs":{"total":96636764160,"available":95265394688,"internally_reserved":0,"allocated":407863296,"data_stored":405103900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":298462,"internal_metadata":958951970},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.436,"15min":0.36299999999999999},"min":{"1min":0.19700000000000001,"5min":0.14699999999999999,"15min":0.13700000000000001},"max":{"1min":3.222,"5min":6.5499999999999998,"15min":6.9989999999999997},"last":0.245},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.46300000000000002,"15min":0.42299999999999999},"min":{"1min":0.17799999999999999,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":5.0030000000000001,"5min":6.0629999999999997,"15min":9.6869999999999994},"last":0.28599999999999998}]},{"osd":2,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.51600000000000001,"15min":0.47099999999999997},"min":{"1min":0.21099999999999999,"5min":0.153,"15min":0.153},"max":{"1min":5.7439999999999998,"5min":5.8280000000000003,"15min":9.7639999999999993},"last":0.252},{"interface":"front","average":{"1min":0.497,"5min":0.56200000000000006,"15min":0.42299999999999999},"min":{"1min":0.20399999999999999,"5min":0.187,"15min":0.16800000000000001},"max":{"1min":3.9319999999999999,"5min":8.8040000000000003,"15min":8.8040000000000003},"last":0.29899999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738823,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":650064,"kb_used_data":5320,"kb_used_omap":214,"kb_used_meta":644521,"kb_avail":93721776,"statfs":{"total":96636764160,"available":95971098624,"internally_reserved":0,"allocated":5447680,"data_stored":2679285,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":219335,"internal_metadata":659990329},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:56 2023","interfaces":[{"interface":"back","average":{"1min":0.30399999999999999,"5min":0.34699999999999998,"15min":0.36499999999999999},"min":{"1min":0.23999999999999999,"5min":0.14199999999999999,"15min":0.14199999999999999},"max":{"1min":0.42399999999999999,"5min":1.595,"15min":7.4249999999999998},"last":0.17100000000000001},{"interface":"front","average":{"1min":0.35499999999999998,"5min":0.40899999999999997,"15min":0.376},"min":{"1min":0.23999999999999999,"5min":0.19600000000000001,"15min":0.183},"max":{"1min":0.46999999999999997,"5min":2.6859999999999999,"15min":2.6859999999999999},"last":0.23000000000000001}]},{"osd":2,"last update":"Wed Nov 8 05:29:57 2023","interfaces":[{"interface":"back","average":{"1min":0.28199999999999997,"5min":0.36099999999999999,"15min":0.373},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":0.34399999999999997,"5min":2.2650000000000001,"15min":5.8209999999999997},"last":0.39300000000000002},{"interface":"front","average":{"1min":0.28599999999999998,"5min":0.31,"15min":0.35799999999999998},"min":{"1min":0.17199999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.35699999999999998,"5min":1.137,"15min":4.2350000000000003},"last":0.41999999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-08T05:30:20.182 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-11-08T05:30:20.183 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 456, 'reported_epoch': 455, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:29:43.389354+0000', 'last_change': '2023-11-08T05:28:13.557071+0000', 'last_active': '2023-11-08T05:29:43.389354+0000', 'last_peered': '2023-11-08T05:29:43.389354+0000', 'last_clean': '2023-11-08T05:28:01.458439+0000', 'last_became_active': '2023-11-08T05:28:13.557071+0000', 'last_became_peered': '2023-11-08T05:28:13.557071+0000', 'last_unstale': '2023-11-08T05:29:43.389354+0000', 'last_undegraded': '2023-11-08T05:29:43.389354+0000', 'last_fullsized': '2023-11-08T05:28:13.547025+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:30:20.183 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-11-08T05:30:20.184 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 18, 'reported_epoch': 455, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:29:43.389262+0000', 'last_change': '2023-11-08T05:28:13.559513+0000', 'last_active': '2023-11-08T05:29:43.389262+0000', 'last_peered': '2023-11-08T05:29:43.389262+0000', 'last_clean': '2023-11-08T04:52:46.531964+0000', 'last_became_active': '2023-11-08T05:28:13.559513+0000', 'last_became_peered': '2023-11-08T05:28:13.559513+0000', 'last_unstale': '2023-11-08T05:29:43.389262+0000', 'last_undegraded': '2023-11-08T05:29:43.389262+0000', 'last_fullsized': '2023-11-08T05:28:13.540852+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:30:20.184 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-11-08T05:30:20.184 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "0'0", 'reported_seq': 457, 'reported_epoch': 455, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:29:43.389822+0000', 'last_change': '2023-11-08T05:28:13.545468+0000', 'last_active': '2023-11-08T05:29:43.389822+0000', 'last_peered': '2023-11-08T05:29:43.389822+0000', 'last_clean': '2023-11-08T05:28:01.456723+0000', 'last_became_active': '2023-11-08T05:28:13.545468+0000', 'last_became_peered': '2023-11-08T05:28:13.545468+0000', 'last_unstale': '2023-11-08T05:29:43.389822+0000', 'last_undegraded': '2023-11-08T05:29:43.389822+0000', 'last_fullsized': '2023-11-08T05:28:13.537207+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 9, 'last_epoch_clean': 10, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:45.528796+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:45.528796+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:45.528796+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:30:20.185 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.0 is not active+clean 2023-11-08T05:30:20.185 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.0', 'version': "0'0", 'reported_seq': 462, 'reported_epoch': 456, 'state': 'active+clean+remapped', 'last_fresh': '2023-11-08T05:29:44.395350+0000', 'last_change': '2023-11-08T05:29:44.395350+0000', 'last_active': '2023-11-08T05:29:44.395350+0000', 'last_peered': '2023-11-08T05:29:44.395350+0000', 'last_clean': '2023-11-08T05:29:44.395350+0000', 'last_became_active': '2023-11-08T05:29:44.395012+0000', 'last_became_peered': '2023-11-08T05:29:44.395012+0000', 'last_unstale': '2023-11-08T05:29:44.395350+0000', 'last_undegraded': '2023-11-08T05:29:44.395350+0000', 'last_fullsized': '2023-11-08T05:29:44.395350+0000', 'mapping_epoch': 455, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 456, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2], 'acting': [2, 1], 'avail_no_missing': ['2', '1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2023-11-08T05:30:20.185 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-11-08T05:30:20.186 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 456, 'reported_epoch': 455, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:29:43.389390+0000', 'last_change': '2023-11-08T05:28:13.563357+0000', 'last_active': '2023-11-08T05:29:43.389390+0000', 'last_peered': '2023-11-08T05:29:43.389390+0000', 'last_clean': '2023-11-08T05:28:01.457691+0000', 'last_became_active': '2023-11-08T05:28:13.563357+0000', 'last_became_peered': '2023-11-08T05:28:13.563357+0000', 'last_unstale': '2023-11-08T05:29:43.389390+0000', 'last_undegraded': '2023-11-08T05:29:43.389390+0000', 'last_fullsized': '2023-11-08T05:28:13.541614+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:30:20.186 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-11-08T05:30:20.186 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 456, 'reported_epoch': 455, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:29:43.389311+0000', 'last_change': '2023-11-08T05:28:13.550672+0000', 'last_active': '2023-11-08T05:29:43.389311+0000', 'last_peered': '2023-11-08T05:29:43.389311+0000', 'last_clean': '2023-11-08T05:28:01.457904+0000', 'last_became_active': '2023-11-08T05:28:13.550672+0000', 'last_became_peered': '2023-11-08T05:28:13.550672+0000', 'last_unstale': '2023-11-08T05:29:43.389311+0000', 'last_undegraded': '2023-11-08T05:29:43.389311+0000', 'last_fullsized': '2023-11-08T05:28:13.539595+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:30:20.187 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-11-08T05:30:20.187 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 456, 'reported_epoch': 455, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:29:43.389474+0000', 'last_change': '2023-11-08T05:28:13.557968+0000', 'last_active': '2023-11-08T05:29:43.389474+0000', 'last_peered': '2023-11-08T05:29:43.389474+0000', 'last_clean': '2023-11-08T05:28:01.458433+0000', 'last_became_active': '2023-11-08T05:28:13.557968+0000', 'last_became_peered': '2023-11-08T05:28:13.557968+0000', 'last_unstale': '2023-11-08T05:29:43.389474+0000', 'last_undegraded': '2023-11-08T05:29:43.389474+0000', 'last_fullsized': '2023-11-08T05:28:13.542832+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:30:20.187 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-11-08T05:30:20.536 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T05:30:20.536 INFO:teuthology.orchestra.run.smithi087.stdout:{"fsid":"ab4cbc06-449d-497d-b513-220d9db2b590","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 6 pgs undersized","count":6},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2263,"monmap":{"epoch":1,"min_mon_release_name":"pacific","num_mons":1},"osdmap":{"epoch":456,"num_osds":3,"num_up_osds":2,"osd_up_since":1699421292,"num_in_osds":3,"osd_in_since":1699419158,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":6},{"state_name":"active+clean","count":2},{"state_name":"active+clean+remapped","count":1}],"num_pgs":9,"num_pools":2,"num_objects":0,"data_bytes":0,"bytes_used":2697519104,"bytes_avail":287212773376,"bytes_total":289910292480},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":41,"modified":"2023-11-08T05:29:32.044544+0000","services":{}},"progress_events":{}} 2023-11-08T05:30:20.548 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-11-08T05:30:20.549 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-11-08T05:30:20.831 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T05:30:20.832 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2023-11-08T05:30:20.843 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":1581,"stamp":"2023-11-08T05:30:20.055133+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2634296,"kb_used_data":408940,"kb_used_omap":780,"kb_used_meta":2220403,"kb_avail":280481224,"statfs":{"total":289910292480,"available":287212773376,"internally_reserved":0,"allocated":418754560,"data_stored":410461501,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":799447,"internal_metadata":2273692969},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-536782902,"num_objects":-164,"num_object_clones":0,"num_object_copies":-328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-115,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-164,"num_whiteouts":0,"num_read":-122056,"num_read_kb":-39011942,"num_write":-413642,"num_write_kb":-151705339,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-8,"num_bytes_recovered":-16922624,"num_keys_recovered":-2,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.424817"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":456,"reported_epoch":455,"state":"active+undersized","last_fresh":"2023-11-08T05:29:43.389354+0000","last_change":"2023-11-08T05:28:13.557071+0000","last_active":"2023-11-08T05:29:43.389354+0000","last_peered":"2023-11-08T05:29:43.389354+0000","last_clean":"2023-11-08T05:28:01.458439+0000","last_became_active":"2023-11-08T05:28:13.557071+0000","last_became_peered":"2023-11-08T05:28:13.557071+0000","last_unstale":"2023-11-08T05:29:43.389354+0000","last_undegraded":"2023-11-08T05:29:43.389354+0000","last_fullsized":"2023-11-08T05:28:13.547025+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":18,"reported_epoch":455,"state":"active+undersized","last_fresh":"2023-11-08T05:29:43.389262+0000","last_change":"2023-11-08T05:28:13.559513+0000","last_active":"2023-11-08T05:29:43.389262+0000","last_peered":"2023-11-08T05:29:43.389262+0000","last_clean":"2023-11-08T04:52:46.531964+0000","last_became_active":"2023-11-08T05:28:13.559513+0000","last_became_peered":"2023-11-08T05:28:13.559513+0000","last_unstale":"2023-11-08T05:29:43.389262+0000","last_undegraded":"2023-11-08T05:29:43.389262+0000","last_fullsized":"2023-11-08T05:28:13.540852+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":453,"reported_epoch":455,"state":"active+clean","last_fresh":"2023-11-08T05:29:43.390991+0000","last_change":"2023-11-08T04:52:47.546100+0000","last_active":"2023-11-08T05:29:43.390991+0000","last_peered":"2023-11-08T05:29:43.390991+0000","last_clean":"2023-11-08T05:29:43.390991+0000","last_became_active":"2023-11-08T04:52:47.545840+0000","last_became_peered":"2023-11-08T04:52:47.545840+0000","last_unstale":"2023-11-08T05:29:43.390991+0000","last_undegraded":"2023-11-08T05:29:43.390991+0000","last_fullsized":"2023-11-08T05:29:43.390991+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":451,"reported_epoch":455,"state":"active+clean","last_fresh":"2023-11-08T05:29:43.389261+0000","last_change":"2023-11-08T04:52:47.548597+0000","last_active":"2023-11-08T05:29:43.389261+0000","last_peered":"2023-11-08T05:29:43.389261+0000","last_clean":"2023-11-08T05:29:43.389261+0000","last_became_active":"2023-11-08T04:52:47.548294+0000","last_became_peered":"2023-11-08T04:52:47.548294+0000","last_unstale":"2023-11-08T05:29:43.389261+0000","last_undegraded":"2023-11-08T05:29:43.389261+0000","last_fullsized":"2023-11-08T05:29:43.389261+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":457,"reported_epoch":455,"state":"active+undersized","last_fresh":"2023-11-08T05:29:43.389822+0000","last_change":"2023-11-08T05:28:13.545468+0000","last_active":"2023-11-08T05:29:43.389822+0000","last_peered":"2023-11-08T05:29:43.389822+0000","last_clean":"2023-11-08T05:28:01.456723+0000","last_became_active":"2023-11-08T05:28:13.545468+0000","last_became_peered":"2023-11-08T05:28:13.545468+0000","last_unstale":"2023-11-08T05:29:43.389822+0000","last_undegraded":"2023-11-08T05:29:43.389822+0000","last_fullsized":"2023-11-08T05:28:13.537207+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_clean_scrub_stamp":"2023-11-08T04:52:45.528796+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":462,"reported_epoch":456,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:29:44.395350+0000","last_change":"2023-11-08T05:29:44.395350+0000","last_active":"2023-11-08T05:29:44.395350+0000","last_peered":"2023-11-08T05:29:44.395350+0000","last_clean":"2023-11-08T05:29:44.395350+0000","last_became_active":"2023-11-08T05:29:44.395012+0000","last_became_peered":"2023-11-08T05:29:44.395012+0000","last_unstale":"2023-11-08T05:29:44.395350+0000","last_undegraded":"2023-11-08T05:29:44.395350+0000","last_fullsized":"2023-11-08T05:29:44.395350+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":456,"reported_epoch":455,"state":"active+undersized","last_fresh":"2023-11-08T05:29:43.389390+0000","last_change":"2023-11-08T05:28:13.563357+0000","last_active":"2023-11-08T05:29:43.389390+0000","last_peered":"2023-11-08T05:29:43.389390+0000","last_clean":"2023-11-08T05:28:01.457691+0000","last_became_active":"2023-11-08T05:28:13.563357+0000","last_became_peered":"2023-11-08T05:28:13.563357+0000","last_unstale":"2023-11-08T05:29:43.389390+0000","last_undegraded":"2023-11-08T05:29:43.389390+0000","last_fullsized":"2023-11-08T05:28:13.541614+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":456,"reported_epoch":455,"state":"active+undersized","last_fresh":"2023-11-08T05:29:43.389311+0000","last_change":"2023-11-08T05:28:13.550672+0000","last_active":"2023-11-08T05:29:43.389311+0000","last_peered":"2023-11-08T05:29:43.389311+0000","last_clean":"2023-11-08T05:28:01.457904+0000","last_became_active":"2023-11-08T05:28:13.550672+0000","last_became_peered":"2023-11-08T05:28:13.550672+0000","last_unstale":"2023-11-08T05:29:43.389311+0000","last_undegraded":"2023-11-08T05:29:43.389311+0000","last_fullsized":"2023-11-08T05:28:13.539595+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":456,"reported_epoch":455,"state":"active+undersized","last_fresh":"2023-11-08T05:29:43.389474+0000","last_change":"2023-11-08T05:28:13.557968+0000","last_active":"2023-11-08T05:29:43.389474+0000","last_peered":"2023-11-08T05:29:43.389474+0000","last_clean":"2023-11-08T05:28:01.458433+0000","last_became_active":"2023-11-08T05:28:13.557968+0000","last_became_peered":"2023-11-08T05:28:13.557968+0000","last_unstale":"2023-11-08T05:29:43.389474+0000","last_undegraded":"2023-11-08T05:29:43.389474+0000","last_fullsized":"2023-11-08T05:28:13.542832+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":1,"acting":1,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":10,"acting":11,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738822,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":645004,"kb_used_data":5316,"kb_used_omap":275,"kb_used_meta":639404,"kb_avail":93726836,"statfs":{"total":96636764160,"available":95976280064,"internally_reserved":0,"allocated":5443584,"data_stored":2678316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":281650,"internal_metadata":654750670},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:59 2023","interfaces":[{"interface":"back","average":{"1min":0.30499999999999999,"5min":0.439,"15min":0.36099999999999999},"min":{"1min":0.17399999999999999,"5min":0.14499999999999999,"15min":0.14499999999999999},"max":{"1min":0.60099999999999998,"5min":4.9379999999999997,"15min":4.9379999999999997},"last":0.379},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.49399999999999999,"15min":0.42399999999999999},"min":{"1min":0.221,"5min":0.19600000000000001,"15min":0.17000000000000001},"max":{"1min":0.63200000000000001,"5min":4.9859999999999998,"15min":4.9859999999999998},"last":0.40899999999999997}]},{"osd":1,"last update":"Wed Nov 8 05:30:03 2023","interfaces":[{"interface":"back","average":{"1min":0.28499999999999998,"5min":0.65100000000000002,"15min":0.47099999999999997},"min":{"1min":0.20100000000000001,"5min":0.153,"15min":0.153},"max":{"1min":0.39800000000000002,"5min":23.242999999999999,"15min":23.242999999999999},"last":0.29399999999999998},{"interface":"front","average":{"1min":0.29499999999999998,"5min":0.63100000000000001,"15min":0.45900000000000002},"min":{"1min":0.224,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":0.436,"5min":23.251000000000001,"15min":23.251000000000001},"last":0.32900000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738798,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1339228,"kb_used_data":398304,"kb_used_omap":291,"kb_used_meta":936476,"kb_avail":93032612,"statfs":{"total":96636764160,"available":95265394688,"internally_reserved":0,"allocated":407863296,"data_stored":405103900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":298462,"internal_metadata":958951970},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.436,"15min":0.36299999999999999},"min":{"1min":0.19700000000000001,"5min":0.14699999999999999,"15min":0.13700000000000001},"max":{"1min":3.222,"5min":6.5499999999999998,"15min":6.9989999999999997},"last":0.245},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.46300000000000002,"15min":0.42299999999999999},"min":{"1min":0.17799999999999999,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":5.0030000000000001,"5min":6.0629999999999997,"15min":9.6869999999999994},"last":0.28599999999999998}]},{"osd":2,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.51600000000000001,"15min":0.47099999999999997},"min":{"1min":0.21099999999999999,"5min":0.153,"15min":0.153},"max":{"1min":5.7439999999999998,"5min":5.8280000000000003,"15min":9.7639999999999993},"last":0.252},{"interface":"front","average":{"1min":0.497,"5min":0.56200000000000006,"15min":0.42299999999999999},"min":{"1min":0.20399999999999999,"5min":0.187,"15min":0.16800000000000001},"max":{"1min":3.9319999999999999,"5min":8.8040000000000003,"15min":8.8040000000000003},"last":0.29899999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738823,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":650064,"kb_used_data":5320,"kb_used_omap":214,"kb_used_meta":644521,"kb_avail":93721776,"statfs":{"total":96636764160,"available":95971098624,"internally_reserved":0,"allocated":5447680,"data_stored":2679285,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":219335,"internal_metadata":659990329},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:56 2023","interfaces":[{"interface":"back","average":{"1min":0.30399999999999999,"5min":0.34699999999999998,"15min":0.36499999999999999},"min":{"1min":0.23999999999999999,"5min":0.14199999999999999,"15min":0.14199999999999999},"max":{"1min":0.42399999999999999,"5min":1.595,"15min":7.4249999999999998},"last":0.17100000000000001},{"interface":"front","average":{"1min":0.35499999999999998,"5min":0.40899999999999997,"15min":0.376},"min":{"1min":0.23999999999999999,"5min":0.19600000000000001,"15min":0.183},"max":{"1min":0.46999999999999997,"5min":2.6859999999999999,"15min":2.6859999999999999},"last":0.23000000000000001}]},{"osd":2,"last update":"Wed Nov 8 05:29:57 2023","interfaces":[{"interface":"back","average":{"1min":0.28199999999999997,"5min":0.36099999999999999,"15min":0.373},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":0.34399999999999997,"5min":2.2650000000000001,"15min":5.8209999999999997},"last":0.39300000000000002},{"interface":"front","average":{"1min":0.28599999999999998,"5min":0.31,"15min":0.35799999999999998},"min":{"1min":0.17199999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.35699999999999998,"5min":1.137,"15min":4.2350000000000003},"last":0.41999999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-08T05:30:22.336 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~124s 2023-11-08T05:30:23.846 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-11-08T05:30:24.141 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T05:30:24.143 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2023-11-08T05:30:24.154 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":1583,"stamp":"2023-11-08T05:30:24.056119+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2634296,"kb_used_data":408940,"kb_used_omap":780,"kb_used_meta":2220403,"kb_avail":280481224,"statfs":{"total":289910292480,"available":287212773376,"internally_reserved":0,"allocated":418754560,"data_stored":410461501,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":799447,"internal_metadata":2273692969},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-536782902,"num_objects":-164,"num_object_clones":0,"num_object_copies":-328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-115,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-164,"num_whiteouts":0,"num_read":-122056,"num_read_kb":-39011942,"num_write":-413642,"num_write_kb":-151705339,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-8,"num_bytes_recovered":-16922624,"num_keys_recovered":-2,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.424817"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":456,"reported_epoch":455,"state":"active+undersized","last_fresh":"2023-11-08T05:29:43.389354+0000","last_change":"2023-11-08T05:28:13.557071+0000","last_active":"2023-11-08T05:29:43.389354+0000","last_peered":"2023-11-08T05:29:43.389354+0000","last_clean":"2023-11-08T05:28:01.458439+0000","last_became_active":"2023-11-08T05:28:13.557071+0000","last_became_peered":"2023-11-08T05:28:13.557071+0000","last_unstale":"2023-11-08T05:29:43.389354+0000","last_undegraded":"2023-11-08T05:29:43.389354+0000","last_fullsized":"2023-11-08T05:28:13.547025+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":18,"reported_epoch":455,"state":"active+undersized","last_fresh":"2023-11-08T05:29:43.389262+0000","last_change":"2023-11-08T05:28:13.559513+0000","last_active":"2023-11-08T05:29:43.389262+0000","last_peered":"2023-11-08T05:29:43.389262+0000","last_clean":"2023-11-08T04:52:46.531964+0000","last_became_active":"2023-11-08T05:28:13.559513+0000","last_became_peered":"2023-11-08T05:28:13.559513+0000","last_unstale":"2023-11-08T05:29:43.389262+0000","last_undegraded":"2023-11-08T05:29:43.389262+0000","last_fullsized":"2023-11-08T05:28:13.540852+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":453,"reported_epoch":455,"state":"active+clean","last_fresh":"2023-11-08T05:29:43.390991+0000","last_change":"2023-11-08T04:52:47.546100+0000","last_active":"2023-11-08T05:29:43.390991+0000","last_peered":"2023-11-08T05:29:43.390991+0000","last_clean":"2023-11-08T05:29:43.390991+0000","last_became_active":"2023-11-08T04:52:47.545840+0000","last_became_peered":"2023-11-08T04:52:47.545840+0000","last_unstale":"2023-11-08T05:29:43.390991+0000","last_undegraded":"2023-11-08T05:29:43.390991+0000","last_fullsized":"2023-11-08T05:29:43.390991+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":451,"reported_epoch":455,"state":"active+clean","last_fresh":"2023-11-08T05:29:43.389261+0000","last_change":"2023-11-08T04:52:47.548597+0000","last_active":"2023-11-08T05:29:43.389261+0000","last_peered":"2023-11-08T05:29:43.389261+0000","last_clean":"2023-11-08T05:29:43.389261+0000","last_became_active":"2023-11-08T04:52:47.548294+0000","last_became_peered":"2023-11-08T04:52:47.548294+0000","last_unstale":"2023-11-08T05:29:43.389261+0000","last_undegraded":"2023-11-08T05:29:43.389261+0000","last_fullsized":"2023-11-08T05:29:43.389261+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":457,"reported_epoch":455,"state":"active+undersized","last_fresh":"2023-11-08T05:29:43.389822+0000","last_change":"2023-11-08T05:28:13.545468+0000","last_active":"2023-11-08T05:29:43.389822+0000","last_peered":"2023-11-08T05:29:43.389822+0000","last_clean":"2023-11-08T05:28:01.456723+0000","last_became_active":"2023-11-08T05:28:13.545468+0000","last_became_peered":"2023-11-08T05:28:13.545468+0000","last_unstale":"2023-11-08T05:29:43.389822+0000","last_undegraded":"2023-11-08T05:29:43.389822+0000","last_fullsized":"2023-11-08T05:28:13.537207+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_clean_scrub_stamp":"2023-11-08T04:52:45.528796+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":462,"reported_epoch":456,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:29:44.395350+0000","last_change":"2023-11-08T05:29:44.395350+0000","last_active":"2023-11-08T05:29:44.395350+0000","last_peered":"2023-11-08T05:29:44.395350+0000","last_clean":"2023-11-08T05:29:44.395350+0000","last_became_active":"2023-11-08T05:29:44.395012+0000","last_became_peered":"2023-11-08T05:29:44.395012+0000","last_unstale":"2023-11-08T05:29:44.395350+0000","last_undegraded":"2023-11-08T05:29:44.395350+0000","last_fullsized":"2023-11-08T05:29:44.395350+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":456,"reported_epoch":455,"state":"active+undersized","last_fresh":"2023-11-08T05:29:43.389390+0000","last_change":"2023-11-08T05:28:13.563357+0000","last_active":"2023-11-08T05:29:43.389390+0000","last_peered":"2023-11-08T05:29:43.389390+0000","last_clean":"2023-11-08T05:28:01.457691+0000","last_became_active":"2023-11-08T05:28:13.563357+0000","last_became_peered":"2023-11-08T05:28:13.563357+0000","last_unstale":"2023-11-08T05:29:43.389390+0000","last_undegraded":"2023-11-08T05:29:43.389390+0000","last_fullsized":"2023-11-08T05:28:13.541614+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":456,"reported_epoch":455,"state":"active+undersized","last_fresh":"2023-11-08T05:29:43.389311+0000","last_change":"2023-11-08T05:28:13.550672+0000","last_active":"2023-11-08T05:29:43.389311+0000","last_peered":"2023-11-08T05:29:43.389311+0000","last_clean":"2023-11-08T05:28:01.457904+0000","last_became_active":"2023-11-08T05:28:13.550672+0000","last_became_peered":"2023-11-08T05:28:13.550672+0000","last_unstale":"2023-11-08T05:29:43.389311+0000","last_undegraded":"2023-11-08T05:29:43.389311+0000","last_fullsized":"2023-11-08T05:28:13.539595+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":456,"reported_epoch":455,"state":"active+undersized","last_fresh":"2023-11-08T05:29:43.389474+0000","last_change":"2023-11-08T05:28:13.557968+0000","last_active":"2023-11-08T05:29:43.389474+0000","last_peered":"2023-11-08T05:29:43.389474+0000","last_clean":"2023-11-08T05:28:01.458433+0000","last_became_active":"2023-11-08T05:28:13.557968+0000","last_became_peered":"2023-11-08T05:28:13.557968+0000","last_unstale":"2023-11-08T05:29:43.389474+0000","last_undegraded":"2023-11-08T05:29:43.389474+0000","last_fullsized":"2023-11-08T05:28:13.542832+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":1,"acting":1,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":10,"acting":11,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738822,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":645004,"kb_used_data":5316,"kb_used_omap":275,"kb_used_meta":639404,"kb_avail":93726836,"statfs":{"total":96636764160,"available":95976280064,"internally_reserved":0,"allocated":5443584,"data_stored":2678316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":281650,"internal_metadata":654750670},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:59 2023","interfaces":[{"interface":"back","average":{"1min":0.30499999999999999,"5min":0.439,"15min":0.36099999999999999},"min":{"1min":0.17399999999999999,"5min":0.14499999999999999,"15min":0.14499999999999999},"max":{"1min":0.60099999999999998,"5min":4.9379999999999997,"15min":4.9379999999999997},"last":0.379},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.49399999999999999,"15min":0.42399999999999999},"min":{"1min":0.221,"5min":0.19600000000000001,"15min":0.17000000000000001},"max":{"1min":0.63200000000000001,"5min":4.9859999999999998,"15min":4.9859999999999998},"last":0.40899999999999997}]},{"osd":1,"last update":"Wed Nov 8 05:30:03 2023","interfaces":[{"interface":"back","average":{"1min":0.28499999999999998,"5min":0.65100000000000002,"15min":0.47099999999999997},"min":{"1min":0.20100000000000001,"5min":0.153,"15min":0.153},"max":{"1min":0.39800000000000002,"5min":23.242999999999999,"15min":23.242999999999999},"last":0.29399999999999998},{"interface":"front","average":{"1min":0.29499999999999998,"5min":0.63100000000000001,"15min":0.45900000000000002},"min":{"1min":0.224,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":0.436,"5min":23.251000000000001,"15min":23.251000000000001},"last":0.32900000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738798,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1339228,"kb_used_data":398304,"kb_used_omap":291,"kb_used_meta":936476,"kb_avail":93032612,"statfs":{"total":96636764160,"available":95265394688,"internally_reserved":0,"allocated":407863296,"data_stored":405103900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":298462,"internal_metadata":958951970},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.436,"15min":0.36299999999999999},"min":{"1min":0.19700000000000001,"5min":0.14699999999999999,"15min":0.13700000000000001},"max":{"1min":3.222,"5min":6.5499999999999998,"15min":6.9989999999999997},"last":0.245},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.46300000000000002,"15min":0.42299999999999999},"min":{"1min":0.17799999999999999,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":5.0030000000000001,"5min":6.0629999999999997,"15min":9.6869999999999994},"last":0.28599999999999998}]},{"osd":2,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.51600000000000001,"15min":0.47099999999999997},"min":{"1min":0.21099999999999999,"5min":0.153,"15min":0.153},"max":{"1min":5.7439999999999998,"5min":5.8280000000000003,"15min":9.7639999999999993},"last":0.252},{"interface":"front","average":{"1min":0.497,"5min":0.56200000000000006,"15min":0.42299999999999999},"min":{"1min":0.20399999999999999,"5min":0.187,"15min":0.16800000000000001},"max":{"1min":3.9319999999999999,"5min":8.8040000000000003,"15min":8.8040000000000003},"last":0.29899999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738824,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":650064,"kb_used_data":5320,"kb_used_omap":214,"kb_used_meta":644521,"kb_avail":93721776,"statfs":{"total":96636764160,"available":95971098624,"internally_reserved":0,"allocated":5447680,"data_stored":2679285,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":219335,"internal_metadata":659990329},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:56 2023","interfaces":[{"interface":"back","average":{"1min":0.30399999999999999,"5min":0.34699999999999998,"15min":0.36499999999999999},"min":{"1min":0.23999999999999999,"5min":0.14199999999999999,"15min":0.14199999999999999},"max":{"1min":0.42399999999999999,"5min":1.595,"15min":7.4249999999999998},"last":0.17100000000000001},{"interface":"front","average":{"1min":0.35499999999999998,"5min":0.40899999999999997,"15min":0.376},"min":{"1min":0.23999999999999999,"5min":0.19600000000000001,"15min":0.183},"max":{"1min":0.46999999999999997,"5min":2.6859999999999999,"15min":2.6859999999999999},"last":0.23000000000000001}]},{"osd":2,"last update":"Wed Nov 8 05:29:57 2023","interfaces":[{"interface":"back","average":{"1min":0.28199999999999997,"5min":0.36099999999999999,"15min":0.373},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":0.34399999999999997,"5min":2.2650000000000001,"15min":5.8209999999999997},"last":0.20200000000000001},{"interface":"front","average":{"1min":0.28599999999999998,"5min":0.31,"15min":0.35799999999999998},"min":{"1min":0.17199999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.35699999999999998,"5min":1.137,"15min":4.2350000000000003},"last":0.26400000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-08T05:30:24.155 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-11-08T05:30:24.437 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T05:30:24.438 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2023-11-08T05:30:24.450 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":1583,"stamp":"2023-11-08T05:30:24.056119+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2634296,"kb_used_data":408940,"kb_used_omap":780,"kb_used_meta":2220403,"kb_avail":280481224,"statfs":{"total":289910292480,"available":287212773376,"internally_reserved":0,"allocated":418754560,"data_stored":410461501,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":799447,"internal_metadata":2273692969},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-536782902,"num_objects":-164,"num_object_clones":0,"num_object_copies":-328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-115,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-164,"num_whiteouts":0,"num_read":-122056,"num_read_kb":-39011942,"num_write":-413642,"num_write_kb":-151705339,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-8,"num_bytes_recovered":-16922624,"num_keys_recovered":-2,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.424817"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":456,"reported_epoch":455,"state":"active+undersized","last_fresh":"2023-11-08T05:29:43.389354+0000","last_change":"2023-11-08T05:28:13.557071+0000","last_active":"2023-11-08T05:29:43.389354+0000","last_peered":"2023-11-08T05:29:43.389354+0000","last_clean":"2023-11-08T05:28:01.458439+0000","last_became_active":"2023-11-08T05:28:13.557071+0000","last_became_peered":"2023-11-08T05:28:13.557071+0000","last_unstale":"2023-11-08T05:29:43.389354+0000","last_undegraded":"2023-11-08T05:29:43.389354+0000","last_fullsized":"2023-11-08T05:28:13.547025+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":18,"reported_epoch":455,"state":"active+undersized","last_fresh":"2023-11-08T05:29:43.389262+0000","last_change":"2023-11-08T05:28:13.559513+0000","last_active":"2023-11-08T05:29:43.389262+0000","last_peered":"2023-11-08T05:29:43.389262+0000","last_clean":"2023-11-08T04:52:46.531964+0000","last_became_active":"2023-11-08T05:28:13.559513+0000","last_became_peered":"2023-11-08T05:28:13.559513+0000","last_unstale":"2023-11-08T05:29:43.389262+0000","last_undegraded":"2023-11-08T05:29:43.389262+0000","last_fullsized":"2023-11-08T05:28:13.540852+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":453,"reported_epoch":455,"state":"active+clean","last_fresh":"2023-11-08T05:29:43.390991+0000","last_change":"2023-11-08T04:52:47.546100+0000","last_active":"2023-11-08T05:29:43.390991+0000","last_peered":"2023-11-08T05:29:43.390991+0000","last_clean":"2023-11-08T05:29:43.390991+0000","last_became_active":"2023-11-08T04:52:47.545840+0000","last_became_peered":"2023-11-08T04:52:47.545840+0000","last_unstale":"2023-11-08T05:29:43.390991+0000","last_undegraded":"2023-11-08T05:29:43.390991+0000","last_fullsized":"2023-11-08T05:29:43.390991+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":451,"reported_epoch":455,"state":"active+clean","last_fresh":"2023-11-08T05:29:43.389261+0000","last_change":"2023-11-08T04:52:47.548597+0000","last_active":"2023-11-08T05:29:43.389261+0000","last_peered":"2023-11-08T05:29:43.389261+0000","last_clean":"2023-11-08T05:29:43.389261+0000","last_became_active":"2023-11-08T04:52:47.548294+0000","last_became_peered":"2023-11-08T04:52:47.548294+0000","last_unstale":"2023-11-08T05:29:43.389261+0000","last_undegraded":"2023-11-08T05:29:43.389261+0000","last_fullsized":"2023-11-08T05:29:43.389261+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":457,"reported_epoch":455,"state":"active+undersized","last_fresh":"2023-11-08T05:29:43.389822+0000","last_change":"2023-11-08T05:28:13.545468+0000","last_active":"2023-11-08T05:29:43.389822+0000","last_peered":"2023-11-08T05:29:43.389822+0000","last_clean":"2023-11-08T05:28:01.456723+0000","last_became_active":"2023-11-08T05:28:13.545468+0000","last_became_peered":"2023-11-08T05:28:13.545468+0000","last_unstale":"2023-11-08T05:29:43.389822+0000","last_undegraded":"2023-11-08T05:29:43.389822+0000","last_fullsized":"2023-11-08T05:28:13.537207+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_clean_scrub_stamp":"2023-11-08T04:52:45.528796+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":462,"reported_epoch":456,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:29:44.395350+0000","last_change":"2023-11-08T05:29:44.395350+0000","last_active":"2023-11-08T05:29:44.395350+0000","last_peered":"2023-11-08T05:29:44.395350+0000","last_clean":"2023-11-08T05:29:44.395350+0000","last_became_active":"2023-11-08T05:29:44.395012+0000","last_became_peered":"2023-11-08T05:29:44.395012+0000","last_unstale":"2023-11-08T05:29:44.395350+0000","last_undegraded":"2023-11-08T05:29:44.395350+0000","last_fullsized":"2023-11-08T05:29:44.395350+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":456,"reported_epoch":455,"state":"active+undersized","last_fresh":"2023-11-08T05:29:43.389390+0000","last_change":"2023-11-08T05:28:13.563357+0000","last_active":"2023-11-08T05:29:43.389390+0000","last_peered":"2023-11-08T05:29:43.389390+0000","last_clean":"2023-11-08T05:28:01.457691+0000","last_became_active":"2023-11-08T05:28:13.563357+0000","last_became_peered":"2023-11-08T05:28:13.563357+0000","last_unstale":"2023-11-08T05:29:43.389390+0000","last_undegraded":"2023-11-08T05:29:43.389390+0000","last_fullsized":"2023-11-08T05:28:13.541614+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":456,"reported_epoch":455,"state":"active+undersized","last_fresh":"2023-11-08T05:29:43.389311+0000","last_change":"2023-11-08T05:28:13.550672+0000","last_active":"2023-11-08T05:29:43.389311+0000","last_peered":"2023-11-08T05:29:43.389311+0000","last_clean":"2023-11-08T05:28:01.457904+0000","last_became_active":"2023-11-08T05:28:13.550672+0000","last_became_peered":"2023-11-08T05:28:13.550672+0000","last_unstale":"2023-11-08T05:29:43.389311+0000","last_undegraded":"2023-11-08T05:29:43.389311+0000","last_fullsized":"2023-11-08T05:28:13.539595+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":456,"reported_epoch":455,"state":"active+undersized","last_fresh":"2023-11-08T05:29:43.389474+0000","last_change":"2023-11-08T05:28:13.557968+0000","last_active":"2023-11-08T05:29:43.389474+0000","last_peered":"2023-11-08T05:29:43.389474+0000","last_clean":"2023-11-08T05:28:01.458433+0000","last_became_active":"2023-11-08T05:28:13.557968+0000","last_became_peered":"2023-11-08T05:28:13.557968+0000","last_unstale":"2023-11-08T05:29:43.389474+0000","last_undegraded":"2023-11-08T05:29:43.389474+0000","last_fullsized":"2023-11-08T05:28:13.542832+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":1,"acting":1,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":10,"acting":11,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738822,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":645004,"kb_used_data":5316,"kb_used_omap":275,"kb_used_meta":639404,"kb_avail":93726836,"statfs":{"total":96636764160,"available":95976280064,"internally_reserved":0,"allocated":5443584,"data_stored":2678316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":281650,"internal_metadata":654750670},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:59 2023","interfaces":[{"interface":"back","average":{"1min":0.30499999999999999,"5min":0.439,"15min":0.36099999999999999},"min":{"1min":0.17399999999999999,"5min":0.14499999999999999,"15min":0.14499999999999999},"max":{"1min":0.60099999999999998,"5min":4.9379999999999997,"15min":4.9379999999999997},"last":0.379},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.49399999999999999,"15min":0.42399999999999999},"min":{"1min":0.221,"5min":0.19600000000000001,"15min":0.17000000000000001},"max":{"1min":0.63200000000000001,"5min":4.9859999999999998,"15min":4.9859999999999998},"last":0.40899999999999997}]},{"osd":1,"last update":"Wed Nov 8 05:30:03 2023","interfaces":[{"interface":"back","average":{"1min":0.28499999999999998,"5min":0.65100000000000002,"15min":0.47099999999999997},"min":{"1min":0.20100000000000001,"5min":0.153,"15min":0.153},"max":{"1min":0.39800000000000002,"5min":23.242999999999999,"15min":23.242999999999999},"last":0.29399999999999998},{"interface":"front","average":{"1min":0.29499999999999998,"5min":0.63100000000000001,"15min":0.45900000000000002},"min":{"1min":0.224,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":0.436,"5min":23.251000000000001,"15min":23.251000000000001},"last":0.32900000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738798,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1339228,"kb_used_data":398304,"kb_used_omap":291,"kb_used_meta":936476,"kb_avail":93032612,"statfs":{"total":96636764160,"available":95265394688,"internally_reserved":0,"allocated":407863296,"data_stored":405103900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":298462,"internal_metadata":958951970},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.436,"15min":0.36299999999999999},"min":{"1min":0.19700000000000001,"5min":0.14699999999999999,"15min":0.13700000000000001},"max":{"1min":3.222,"5min":6.5499999999999998,"15min":6.9989999999999997},"last":0.245},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.46300000000000002,"15min":0.42299999999999999},"min":{"1min":0.17799999999999999,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":5.0030000000000001,"5min":6.0629999999999997,"15min":9.6869999999999994},"last":0.28599999999999998}]},{"osd":2,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.51600000000000001,"15min":0.47099999999999997},"min":{"1min":0.21099999999999999,"5min":0.153,"15min":0.153},"max":{"1min":5.7439999999999998,"5min":5.8280000000000003,"15min":9.7639999999999993},"last":0.252},{"interface":"front","average":{"1min":0.497,"5min":0.56200000000000006,"15min":0.42299999999999999},"min":{"1min":0.20399999999999999,"5min":0.187,"15min":0.16800000000000001},"max":{"1min":3.9319999999999999,"5min":8.8040000000000003,"15min":8.8040000000000003},"last":0.29899999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738824,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":650064,"kb_used_data":5320,"kb_used_omap":214,"kb_used_meta":644521,"kb_avail":93721776,"statfs":{"total":96636764160,"available":95971098624,"internally_reserved":0,"allocated":5447680,"data_stored":2679285,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":219335,"internal_metadata":659990329},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:56 2023","interfaces":[{"interface":"back","average":{"1min":0.30399999999999999,"5min":0.34699999999999998,"15min":0.36499999999999999},"min":{"1min":0.23999999999999999,"5min":0.14199999999999999,"15min":0.14199999999999999},"max":{"1min":0.42399999999999999,"5min":1.595,"15min":7.4249999999999998},"last":0.17100000000000001},{"interface":"front","average":{"1min":0.35499999999999998,"5min":0.40899999999999997,"15min":0.376},"min":{"1min":0.23999999999999999,"5min":0.19600000000000001,"15min":0.183},"max":{"1min":0.46999999999999997,"5min":2.6859999999999999,"15min":2.6859999999999999},"last":0.23000000000000001}]},{"osd":2,"last update":"Wed Nov 8 05:29:57 2023","interfaces":[{"interface":"back","average":{"1min":0.28199999999999997,"5min":0.36099999999999999,"15min":0.373},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":0.34399999999999997,"5min":2.2650000000000001,"15min":5.8209999999999997},"last":0.20200000000000001},{"interface":"front","average":{"1min":0.28599999999999998,"5min":0.31,"15min":0.35799999999999998},"min":{"1min":0.17199999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.35699999999999998,"5min":1.137,"15min":4.2350000000000003},"last":0.26400000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-08T05:30:24.451 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-11-08T05:30:24.451 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 456, 'reported_epoch': 455, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:29:43.389354+0000', 'last_change': '2023-11-08T05:28:13.557071+0000', 'last_active': '2023-11-08T05:29:43.389354+0000', 'last_peered': '2023-11-08T05:29:43.389354+0000', 'last_clean': '2023-11-08T05:28:01.458439+0000', 'last_became_active': '2023-11-08T05:28:13.557071+0000', 'last_became_peered': '2023-11-08T05:28:13.557071+0000', 'last_unstale': '2023-11-08T05:29:43.389354+0000', 'last_undegraded': '2023-11-08T05:29:43.389354+0000', 'last_fullsized': '2023-11-08T05:28:13.547025+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:30:24.452 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-11-08T05:30:24.452 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 18, 'reported_epoch': 455, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:29:43.389262+0000', 'last_change': '2023-11-08T05:28:13.559513+0000', 'last_active': '2023-11-08T05:29:43.389262+0000', 'last_peered': '2023-11-08T05:29:43.389262+0000', 'last_clean': '2023-11-08T04:52:46.531964+0000', 'last_became_active': '2023-11-08T05:28:13.559513+0000', 'last_became_peered': '2023-11-08T05:28:13.559513+0000', 'last_unstale': '2023-11-08T05:29:43.389262+0000', 'last_undegraded': '2023-11-08T05:29:43.389262+0000', 'last_fullsized': '2023-11-08T05:28:13.540852+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:30:24.452 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-11-08T05:30:24.452 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "0'0", 'reported_seq': 457, 'reported_epoch': 455, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:29:43.389822+0000', 'last_change': '2023-11-08T05:28:13.545468+0000', 'last_active': '2023-11-08T05:29:43.389822+0000', 'last_peered': '2023-11-08T05:29:43.389822+0000', 'last_clean': '2023-11-08T05:28:01.456723+0000', 'last_became_active': '2023-11-08T05:28:13.545468+0000', 'last_became_peered': '2023-11-08T05:28:13.545468+0000', 'last_unstale': '2023-11-08T05:29:43.389822+0000', 'last_undegraded': '2023-11-08T05:29:43.389822+0000', 'last_fullsized': '2023-11-08T05:28:13.537207+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 9, 'last_epoch_clean': 10, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:45.528796+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:45.528796+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:45.528796+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:30:24.453 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.0 is not active+clean 2023-11-08T05:30:24.453 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.0', 'version': "0'0", 'reported_seq': 462, 'reported_epoch': 456, 'state': 'active+clean+remapped', 'last_fresh': '2023-11-08T05:29:44.395350+0000', 'last_change': '2023-11-08T05:29:44.395350+0000', 'last_active': '2023-11-08T05:29:44.395350+0000', 'last_peered': '2023-11-08T05:29:44.395350+0000', 'last_clean': '2023-11-08T05:29:44.395350+0000', 'last_became_active': '2023-11-08T05:29:44.395012+0000', 'last_became_peered': '2023-11-08T05:29:44.395012+0000', 'last_unstale': '2023-11-08T05:29:44.395350+0000', 'last_undegraded': '2023-11-08T05:29:44.395350+0000', 'last_fullsized': '2023-11-08T05:29:44.395350+0000', 'mapping_epoch': 455, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 456, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2], 'acting': [2, 1], 'avail_no_missing': ['2', '1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2023-11-08T05:30:24.453 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-11-08T05:30:24.454 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 456, 'reported_epoch': 455, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:29:43.389390+0000', 'last_change': '2023-11-08T05:28:13.563357+0000', 'last_active': '2023-11-08T05:29:43.389390+0000', 'last_peered': '2023-11-08T05:29:43.389390+0000', 'last_clean': '2023-11-08T05:28:01.457691+0000', 'last_became_active': '2023-11-08T05:28:13.563357+0000', 'last_became_peered': '2023-11-08T05:28:13.563357+0000', 'last_unstale': '2023-11-08T05:29:43.389390+0000', 'last_undegraded': '2023-11-08T05:29:43.389390+0000', 'last_fullsized': '2023-11-08T05:28:13.541614+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:30:24.454 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-11-08T05:30:24.454 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 456, 'reported_epoch': 455, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:29:43.389311+0000', 'last_change': '2023-11-08T05:28:13.550672+0000', 'last_active': '2023-11-08T05:29:43.389311+0000', 'last_peered': '2023-11-08T05:29:43.389311+0000', 'last_clean': '2023-11-08T05:28:01.457904+0000', 'last_became_active': '2023-11-08T05:28:13.550672+0000', 'last_became_peered': '2023-11-08T05:28:13.550672+0000', 'last_unstale': '2023-11-08T05:29:43.389311+0000', 'last_undegraded': '2023-11-08T05:29:43.389311+0000', 'last_fullsized': '2023-11-08T05:28:13.539595+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:30:24.455 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-11-08T05:30:24.455 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 456, 'reported_epoch': 455, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:29:43.389474+0000', 'last_change': '2023-11-08T05:28:13.557968+0000', 'last_active': '2023-11-08T05:29:43.389474+0000', 'last_peered': '2023-11-08T05:29:43.389474+0000', 'last_clean': '2023-11-08T05:28:01.458433+0000', 'last_became_active': '2023-11-08T05:28:13.557968+0000', 'last_became_peered': '2023-11-08T05:28:13.557968+0000', 'last_unstale': '2023-11-08T05:29:43.389474+0000', 'last_undegraded': '2023-11-08T05:29:43.389474+0000', 'last_fullsized': '2023-11-08T05:28:13.542832+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:30:24.455 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-11-08T05:30:24.802 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T05:30:24.802 INFO:teuthology.orchestra.run.smithi087.stdout:{"fsid":"ab4cbc06-449d-497d-b513-220d9db2b590","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 6 pgs undersized","count":6},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2267,"monmap":{"epoch":1,"min_mon_release_name":"pacific","num_mons":1},"osdmap":{"epoch":456,"num_osds":3,"num_up_osds":2,"osd_up_since":1699421292,"num_in_osds":3,"osd_in_since":1699419158,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":6},{"state_name":"active+clean","count":2},{"state_name":"active+clean+remapped","count":1}],"num_pgs":9,"num_pools":2,"num_objects":0,"data_bytes":0,"bytes_used":2697519104,"bytes_avail":287212773376,"bytes_total":289910292480},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":41,"modified":"2023-11-08T05:29:32.044544+0000","services":{}},"progress_events":{}} 2023-11-08T05:30:24.815 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-11-08T05:30:24.815 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-11-08T05:30:25.096 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T05:30:25.098 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2023-11-08T05:30:25.109 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":1583,"stamp":"2023-11-08T05:30:24.056119+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2634296,"kb_used_data":408940,"kb_used_omap":780,"kb_used_meta":2220403,"kb_avail":280481224,"statfs":{"total":289910292480,"available":287212773376,"internally_reserved":0,"allocated":418754560,"data_stored":410461501,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":799447,"internal_metadata":2273692969},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-536782902,"num_objects":-164,"num_object_clones":0,"num_object_copies":-328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-115,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-164,"num_whiteouts":0,"num_read":-122056,"num_read_kb":-39011942,"num_write":-413642,"num_write_kb":-151705339,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-8,"num_bytes_recovered":-16922624,"num_keys_recovered":-2,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.424817"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":456,"reported_epoch":455,"state":"active+undersized","last_fresh":"2023-11-08T05:29:43.389354+0000","last_change":"2023-11-08T05:28:13.557071+0000","last_active":"2023-11-08T05:29:43.389354+0000","last_peered":"2023-11-08T05:29:43.389354+0000","last_clean":"2023-11-08T05:28:01.458439+0000","last_became_active":"2023-11-08T05:28:13.557071+0000","last_became_peered":"2023-11-08T05:28:13.557071+0000","last_unstale":"2023-11-08T05:29:43.389354+0000","last_undegraded":"2023-11-08T05:29:43.389354+0000","last_fullsized":"2023-11-08T05:28:13.547025+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":18,"reported_epoch":455,"state":"active+undersized","last_fresh":"2023-11-08T05:29:43.389262+0000","last_change":"2023-11-08T05:28:13.559513+0000","last_active":"2023-11-08T05:29:43.389262+0000","last_peered":"2023-11-08T05:29:43.389262+0000","last_clean":"2023-11-08T04:52:46.531964+0000","last_became_active":"2023-11-08T05:28:13.559513+0000","last_became_peered":"2023-11-08T05:28:13.559513+0000","last_unstale":"2023-11-08T05:29:43.389262+0000","last_undegraded":"2023-11-08T05:29:43.389262+0000","last_fullsized":"2023-11-08T05:28:13.540852+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":453,"reported_epoch":455,"state":"active+clean","last_fresh":"2023-11-08T05:29:43.390991+0000","last_change":"2023-11-08T04:52:47.546100+0000","last_active":"2023-11-08T05:29:43.390991+0000","last_peered":"2023-11-08T05:29:43.390991+0000","last_clean":"2023-11-08T05:29:43.390991+0000","last_became_active":"2023-11-08T04:52:47.545840+0000","last_became_peered":"2023-11-08T04:52:47.545840+0000","last_unstale":"2023-11-08T05:29:43.390991+0000","last_undegraded":"2023-11-08T05:29:43.390991+0000","last_fullsized":"2023-11-08T05:29:43.390991+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":451,"reported_epoch":455,"state":"active+clean","last_fresh":"2023-11-08T05:29:43.389261+0000","last_change":"2023-11-08T04:52:47.548597+0000","last_active":"2023-11-08T05:29:43.389261+0000","last_peered":"2023-11-08T05:29:43.389261+0000","last_clean":"2023-11-08T05:29:43.389261+0000","last_became_active":"2023-11-08T04:52:47.548294+0000","last_became_peered":"2023-11-08T04:52:47.548294+0000","last_unstale":"2023-11-08T05:29:43.389261+0000","last_undegraded":"2023-11-08T05:29:43.389261+0000","last_fullsized":"2023-11-08T05:29:43.389261+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":457,"reported_epoch":455,"state":"active+undersized","last_fresh":"2023-11-08T05:29:43.389822+0000","last_change":"2023-11-08T05:28:13.545468+0000","last_active":"2023-11-08T05:29:43.389822+0000","last_peered":"2023-11-08T05:29:43.389822+0000","last_clean":"2023-11-08T05:28:01.456723+0000","last_became_active":"2023-11-08T05:28:13.545468+0000","last_became_peered":"2023-11-08T05:28:13.545468+0000","last_unstale":"2023-11-08T05:29:43.389822+0000","last_undegraded":"2023-11-08T05:29:43.389822+0000","last_fullsized":"2023-11-08T05:28:13.537207+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_clean_scrub_stamp":"2023-11-08T04:52:45.528796+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":462,"reported_epoch":456,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:29:44.395350+0000","last_change":"2023-11-08T05:29:44.395350+0000","last_active":"2023-11-08T05:29:44.395350+0000","last_peered":"2023-11-08T05:29:44.395350+0000","last_clean":"2023-11-08T05:29:44.395350+0000","last_became_active":"2023-11-08T05:29:44.395012+0000","last_became_peered":"2023-11-08T05:29:44.395012+0000","last_unstale":"2023-11-08T05:29:44.395350+0000","last_undegraded":"2023-11-08T05:29:44.395350+0000","last_fullsized":"2023-11-08T05:29:44.395350+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":456,"reported_epoch":455,"state":"active+undersized","last_fresh":"2023-11-08T05:29:43.389390+0000","last_change":"2023-11-08T05:28:13.563357+0000","last_active":"2023-11-08T05:29:43.389390+0000","last_peered":"2023-11-08T05:29:43.389390+0000","last_clean":"2023-11-08T05:28:01.457691+0000","last_became_active":"2023-11-08T05:28:13.563357+0000","last_became_peered":"2023-11-08T05:28:13.563357+0000","last_unstale":"2023-11-08T05:29:43.389390+0000","last_undegraded":"2023-11-08T05:29:43.389390+0000","last_fullsized":"2023-11-08T05:28:13.541614+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":456,"reported_epoch":455,"state":"active+undersized","last_fresh":"2023-11-08T05:29:43.389311+0000","last_change":"2023-11-08T05:28:13.550672+0000","last_active":"2023-11-08T05:29:43.389311+0000","last_peered":"2023-11-08T05:29:43.389311+0000","last_clean":"2023-11-08T05:28:01.457904+0000","last_became_active":"2023-11-08T05:28:13.550672+0000","last_became_peered":"2023-11-08T05:28:13.550672+0000","last_unstale":"2023-11-08T05:29:43.389311+0000","last_undegraded":"2023-11-08T05:29:43.389311+0000","last_fullsized":"2023-11-08T05:28:13.539595+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":456,"reported_epoch":455,"state":"active+undersized","last_fresh":"2023-11-08T05:29:43.389474+0000","last_change":"2023-11-08T05:28:13.557968+0000","last_active":"2023-11-08T05:29:43.389474+0000","last_peered":"2023-11-08T05:29:43.389474+0000","last_clean":"2023-11-08T05:28:01.458433+0000","last_became_active":"2023-11-08T05:28:13.557968+0000","last_became_peered":"2023-11-08T05:28:13.557968+0000","last_unstale":"2023-11-08T05:29:43.389474+0000","last_undegraded":"2023-11-08T05:29:43.389474+0000","last_fullsized":"2023-11-08T05:28:13.542832+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":1,"acting":1,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":10,"acting":11,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738822,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":645004,"kb_used_data":5316,"kb_used_omap":275,"kb_used_meta":639404,"kb_avail":93726836,"statfs":{"total":96636764160,"available":95976280064,"internally_reserved":0,"allocated":5443584,"data_stored":2678316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":281650,"internal_metadata":654750670},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:59 2023","interfaces":[{"interface":"back","average":{"1min":0.30499999999999999,"5min":0.439,"15min":0.36099999999999999},"min":{"1min":0.17399999999999999,"5min":0.14499999999999999,"15min":0.14499999999999999},"max":{"1min":0.60099999999999998,"5min":4.9379999999999997,"15min":4.9379999999999997},"last":0.379},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.49399999999999999,"15min":0.42399999999999999},"min":{"1min":0.221,"5min":0.19600000000000001,"15min":0.17000000000000001},"max":{"1min":0.63200000000000001,"5min":4.9859999999999998,"15min":4.9859999999999998},"last":0.40899999999999997}]},{"osd":1,"last update":"Wed Nov 8 05:30:03 2023","interfaces":[{"interface":"back","average":{"1min":0.28499999999999998,"5min":0.65100000000000002,"15min":0.47099999999999997},"min":{"1min":0.20100000000000001,"5min":0.153,"15min":0.153},"max":{"1min":0.39800000000000002,"5min":23.242999999999999,"15min":23.242999999999999},"last":0.29399999999999998},{"interface":"front","average":{"1min":0.29499999999999998,"5min":0.63100000000000001,"15min":0.45900000000000002},"min":{"1min":0.224,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":0.436,"5min":23.251000000000001,"15min":23.251000000000001},"last":0.32900000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738798,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1339228,"kb_used_data":398304,"kb_used_omap":291,"kb_used_meta":936476,"kb_avail":93032612,"statfs":{"total":96636764160,"available":95265394688,"internally_reserved":0,"allocated":407863296,"data_stored":405103900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":298462,"internal_metadata":958951970},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.436,"15min":0.36299999999999999},"min":{"1min":0.19700000000000001,"5min":0.14699999999999999,"15min":0.13700000000000001},"max":{"1min":3.222,"5min":6.5499999999999998,"15min":6.9989999999999997},"last":0.245},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.46300000000000002,"15min":0.42299999999999999},"min":{"1min":0.17799999999999999,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":5.0030000000000001,"5min":6.0629999999999997,"15min":9.6869999999999994},"last":0.28599999999999998}]},{"osd":2,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.51600000000000001,"15min":0.47099999999999997},"min":{"1min":0.21099999999999999,"5min":0.153,"15min":0.153},"max":{"1min":5.7439999999999998,"5min":5.8280000000000003,"15min":9.7639999999999993},"last":0.252},{"interface":"front","average":{"1min":0.497,"5min":0.56200000000000006,"15min":0.42299999999999999},"min":{"1min":0.20399999999999999,"5min":0.187,"15min":0.16800000000000001},"max":{"1min":3.9319999999999999,"5min":8.8040000000000003,"15min":8.8040000000000003},"last":0.29899999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738824,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":650064,"kb_used_data":5320,"kb_used_omap":214,"kb_used_meta":644521,"kb_avail":93721776,"statfs":{"total":96636764160,"available":95971098624,"internally_reserved":0,"allocated":5447680,"data_stored":2679285,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":219335,"internal_metadata":659990329},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:56 2023","interfaces":[{"interface":"back","average":{"1min":0.30399999999999999,"5min":0.34699999999999998,"15min":0.36499999999999999},"min":{"1min":0.23999999999999999,"5min":0.14199999999999999,"15min":0.14199999999999999},"max":{"1min":0.42399999999999999,"5min":1.595,"15min":7.4249999999999998},"last":0.17100000000000001},{"interface":"front","average":{"1min":0.35499999999999998,"5min":0.40899999999999997,"15min":0.376},"min":{"1min":0.23999999999999999,"5min":0.19600000000000001,"15min":0.183},"max":{"1min":0.46999999999999997,"5min":2.6859999999999999,"15min":2.6859999999999999},"last":0.23000000000000001}]},{"osd":2,"last update":"Wed Nov 8 05:29:57 2023","interfaces":[{"interface":"back","average":{"1min":0.28199999999999997,"5min":0.36099999999999999,"15min":0.373},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":0.34399999999999997,"5min":2.2650000000000001,"15min":5.8209999999999997},"last":0.20200000000000001},{"interface":"front","average":{"1min":0.28599999999999998,"5min":0.31,"15min":0.35799999999999998},"min":{"1min":0.17199999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.35699999999999998,"5min":1.137,"15min":4.2350000000000003},"last":0.26400000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-08T05:30:27.741 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~130s 2023-11-08T05:30:28.112 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-11-08T05:30:28.408 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T05:30:28.410 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2023-11-08T05:30:28.421 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":1585,"stamp":"2023-11-08T05:30:28.056918+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2634296,"kb_used_data":408940,"kb_used_omap":780,"kb_used_meta":2220403,"kb_avail":280481224,"statfs":{"total":289910292480,"available":287212773376,"internally_reserved":0,"allocated":418754560,"data_stored":410461501,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":799447,"internal_metadata":2273692969},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-536782902,"num_objects":-164,"num_object_clones":0,"num_object_copies":-328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-115,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-164,"num_whiteouts":0,"num_read":-122056,"num_read_kb":-39011942,"num_write":-413642,"num_write_kb":-151705339,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-8,"num_bytes_recovered":-16922624,"num_keys_recovered":-2,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.424817"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":456,"reported_epoch":455,"state":"active+undersized","last_fresh":"2023-11-08T05:29:43.389354+0000","last_change":"2023-11-08T05:28:13.557071+0000","last_active":"2023-11-08T05:29:43.389354+0000","last_peered":"2023-11-08T05:29:43.389354+0000","last_clean":"2023-11-08T05:28:01.458439+0000","last_became_active":"2023-11-08T05:28:13.557071+0000","last_became_peered":"2023-11-08T05:28:13.557071+0000","last_unstale":"2023-11-08T05:29:43.389354+0000","last_undegraded":"2023-11-08T05:29:43.389354+0000","last_fullsized":"2023-11-08T05:28:13.547025+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":18,"reported_epoch":455,"state":"active+undersized","last_fresh":"2023-11-08T05:29:43.389262+0000","last_change":"2023-11-08T05:28:13.559513+0000","last_active":"2023-11-08T05:29:43.389262+0000","last_peered":"2023-11-08T05:29:43.389262+0000","last_clean":"2023-11-08T04:52:46.531964+0000","last_became_active":"2023-11-08T05:28:13.559513+0000","last_became_peered":"2023-11-08T05:28:13.559513+0000","last_unstale":"2023-11-08T05:29:43.389262+0000","last_undegraded":"2023-11-08T05:29:43.389262+0000","last_fullsized":"2023-11-08T05:28:13.540852+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":453,"reported_epoch":455,"state":"active+clean","last_fresh":"2023-11-08T05:29:43.390991+0000","last_change":"2023-11-08T04:52:47.546100+0000","last_active":"2023-11-08T05:29:43.390991+0000","last_peered":"2023-11-08T05:29:43.390991+0000","last_clean":"2023-11-08T05:29:43.390991+0000","last_became_active":"2023-11-08T04:52:47.545840+0000","last_became_peered":"2023-11-08T04:52:47.545840+0000","last_unstale":"2023-11-08T05:29:43.390991+0000","last_undegraded":"2023-11-08T05:29:43.390991+0000","last_fullsized":"2023-11-08T05:29:43.390991+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":451,"reported_epoch":455,"state":"active+clean","last_fresh":"2023-11-08T05:29:43.389261+0000","last_change":"2023-11-08T04:52:47.548597+0000","last_active":"2023-11-08T05:29:43.389261+0000","last_peered":"2023-11-08T05:29:43.389261+0000","last_clean":"2023-11-08T05:29:43.389261+0000","last_became_active":"2023-11-08T04:52:47.548294+0000","last_became_peered":"2023-11-08T04:52:47.548294+0000","last_unstale":"2023-11-08T05:29:43.389261+0000","last_undegraded":"2023-11-08T05:29:43.389261+0000","last_fullsized":"2023-11-08T05:29:43.389261+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":457,"reported_epoch":455,"state":"active+undersized","last_fresh":"2023-11-08T05:29:43.389822+0000","last_change":"2023-11-08T05:28:13.545468+0000","last_active":"2023-11-08T05:29:43.389822+0000","last_peered":"2023-11-08T05:29:43.389822+0000","last_clean":"2023-11-08T05:28:01.456723+0000","last_became_active":"2023-11-08T05:28:13.545468+0000","last_became_peered":"2023-11-08T05:28:13.545468+0000","last_unstale":"2023-11-08T05:29:43.389822+0000","last_undegraded":"2023-11-08T05:29:43.389822+0000","last_fullsized":"2023-11-08T05:28:13.537207+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_clean_scrub_stamp":"2023-11-08T04:52:45.528796+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":462,"reported_epoch":456,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:29:44.395350+0000","last_change":"2023-11-08T05:29:44.395350+0000","last_active":"2023-11-08T05:29:44.395350+0000","last_peered":"2023-11-08T05:29:44.395350+0000","last_clean":"2023-11-08T05:29:44.395350+0000","last_became_active":"2023-11-08T05:29:44.395012+0000","last_became_peered":"2023-11-08T05:29:44.395012+0000","last_unstale":"2023-11-08T05:29:44.395350+0000","last_undegraded":"2023-11-08T05:29:44.395350+0000","last_fullsized":"2023-11-08T05:29:44.395350+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":456,"reported_epoch":455,"state":"active+undersized","last_fresh":"2023-11-08T05:29:43.389390+0000","last_change":"2023-11-08T05:28:13.563357+0000","last_active":"2023-11-08T05:29:43.389390+0000","last_peered":"2023-11-08T05:29:43.389390+0000","last_clean":"2023-11-08T05:28:01.457691+0000","last_became_active":"2023-11-08T05:28:13.563357+0000","last_became_peered":"2023-11-08T05:28:13.563357+0000","last_unstale":"2023-11-08T05:29:43.389390+0000","last_undegraded":"2023-11-08T05:29:43.389390+0000","last_fullsized":"2023-11-08T05:28:13.541614+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":456,"reported_epoch":455,"state":"active+undersized","last_fresh":"2023-11-08T05:29:43.389311+0000","last_change":"2023-11-08T05:28:13.550672+0000","last_active":"2023-11-08T05:29:43.389311+0000","last_peered":"2023-11-08T05:29:43.389311+0000","last_clean":"2023-11-08T05:28:01.457904+0000","last_became_active":"2023-11-08T05:28:13.550672+0000","last_became_peered":"2023-11-08T05:28:13.550672+0000","last_unstale":"2023-11-08T05:29:43.389311+0000","last_undegraded":"2023-11-08T05:29:43.389311+0000","last_fullsized":"2023-11-08T05:28:13.539595+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":456,"reported_epoch":455,"state":"active+undersized","last_fresh":"2023-11-08T05:29:43.389474+0000","last_change":"2023-11-08T05:28:13.557968+0000","last_active":"2023-11-08T05:29:43.389474+0000","last_peered":"2023-11-08T05:29:43.389474+0000","last_clean":"2023-11-08T05:28:01.458433+0000","last_became_active":"2023-11-08T05:28:13.557968+0000","last_became_peered":"2023-11-08T05:28:13.557968+0000","last_unstale":"2023-11-08T05:29:43.389474+0000","last_undegraded":"2023-11-08T05:29:43.389474+0000","last_fullsized":"2023-11-08T05:28:13.542832+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":1,"acting":1,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":10,"acting":11,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738823,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":645004,"kb_used_data":5316,"kb_used_omap":275,"kb_used_meta":639404,"kb_avail":93726836,"statfs":{"total":96636764160,"available":95976280064,"internally_reserved":0,"allocated":5443584,"data_stored":2678316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":281650,"internal_metadata":654750670},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:59 2023","interfaces":[{"interface":"back","average":{"1min":0.30499999999999999,"5min":0.439,"15min":0.36099999999999999},"min":{"1min":0.17399999999999999,"5min":0.14499999999999999,"15min":0.14499999999999999},"max":{"1min":0.60099999999999998,"5min":4.9379999999999997,"15min":4.9379999999999997},"last":0.379},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.49399999999999999,"15min":0.42399999999999999},"min":{"1min":0.221,"5min":0.19600000000000001,"15min":0.17000000000000001},"max":{"1min":0.63200000000000001,"5min":4.9859999999999998,"15min":4.9859999999999998},"last":0.40899999999999997}]},{"osd":1,"last update":"Wed Nov 8 05:30:03 2023","interfaces":[{"interface":"back","average":{"1min":0.28499999999999998,"5min":0.65100000000000002,"15min":0.47099999999999997},"min":{"1min":0.20100000000000001,"5min":0.153,"15min":0.153},"max":{"1min":0.39800000000000002,"5min":23.242999999999999,"15min":23.242999999999999},"last":0.248},{"interface":"front","average":{"1min":0.29499999999999998,"5min":0.63100000000000001,"15min":0.45900000000000002},"min":{"1min":0.224,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":0.436,"5min":23.251000000000001,"15min":23.251000000000001},"last":0.23799999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738798,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1339228,"kb_used_data":398304,"kb_used_omap":291,"kb_used_meta":936476,"kb_avail":93032612,"statfs":{"total":96636764160,"available":95265394688,"internally_reserved":0,"allocated":407863296,"data_stored":405103900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":298462,"internal_metadata":958951970},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.436,"15min":0.36299999999999999},"min":{"1min":0.19700000000000001,"5min":0.14699999999999999,"15min":0.13700000000000001},"max":{"1min":3.222,"5min":6.5499999999999998,"15min":6.9989999999999997},"last":0.245},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.46300000000000002,"15min":0.42299999999999999},"min":{"1min":0.17799999999999999,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":5.0030000000000001,"5min":6.0629999999999997,"15min":9.6869999999999994},"last":0.28599999999999998}]},{"osd":2,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.51600000000000001,"15min":0.47099999999999997},"min":{"1min":0.21099999999999999,"5min":0.153,"15min":0.153},"max":{"1min":5.7439999999999998,"5min":5.8280000000000003,"15min":9.7639999999999993},"last":0.252},{"interface":"front","average":{"1min":0.497,"5min":0.56200000000000006,"15min":0.42299999999999999},"min":{"1min":0.20399999999999999,"5min":0.187,"15min":0.16800000000000001},"max":{"1min":3.9319999999999999,"5min":8.8040000000000003,"15min":8.8040000000000003},"last":0.29899999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738825,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":650064,"kb_used_data":5320,"kb_used_omap":214,"kb_used_meta":644521,"kb_avail":93721776,"statfs":{"total":96636764160,"available":95971098624,"internally_reserved":0,"allocated":5447680,"data_stored":2679285,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":219335,"internal_metadata":659990329},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:56 2023","interfaces":[{"interface":"back","average":{"1min":0.30399999999999999,"5min":0.34699999999999998,"15min":0.36499999999999999},"min":{"1min":0.23999999999999999,"5min":0.14199999999999999,"15min":0.14199999999999999},"max":{"1min":0.42399999999999999,"5min":1.595,"15min":7.4249999999999998},"last":0.17100000000000001},{"interface":"front","average":{"1min":0.35499999999999998,"5min":0.40899999999999997,"15min":0.376},"min":{"1min":0.23999999999999999,"5min":0.19600000000000001,"15min":0.183},"max":{"1min":0.46999999999999997,"5min":2.6859999999999999,"15min":2.6859999999999999},"last":0.23000000000000001}]},{"osd":2,"last update":"Wed Nov 8 05:29:57 2023","interfaces":[{"interface":"back","average":{"1min":0.28199999999999997,"5min":0.36099999999999999,"15min":0.373},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":0.34399999999999997,"5min":2.2650000000000001,"15min":5.8209999999999997},"last":0.21099999999999999},{"interface":"front","average":{"1min":0.28599999999999998,"5min":0.31,"15min":0.35799999999999998},"min":{"1min":0.17199999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.35699999999999998,"5min":1.137,"15min":4.2350000000000003},"last":0.27100000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-08T05:30:28.422 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-11-08T05:30:28.704 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T05:30:28.706 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2023-11-08T05:30:28.718 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":1585,"stamp":"2023-11-08T05:30:28.056918+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2634296,"kb_used_data":408940,"kb_used_omap":780,"kb_used_meta":2220403,"kb_avail":280481224,"statfs":{"total":289910292480,"available":287212773376,"internally_reserved":0,"allocated":418754560,"data_stored":410461501,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":799447,"internal_metadata":2273692969},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-536782902,"num_objects":-164,"num_object_clones":0,"num_object_copies":-328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-115,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-164,"num_whiteouts":0,"num_read":-122056,"num_read_kb":-39011942,"num_write":-413642,"num_write_kb":-151705339,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-8,"num_bytes_recovered":-16922624,"num_keys_recovered":-2,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.424817"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":456,"reported_epoch":455,"state":"active+undersized","last_fresh":"2023-11-08T05:29:43.389354+0000","last_change":"2023-11-08T05:28:13.557071+0000","last_active":"2023-11-08T05:29:43.389354+0000","last_peered":"2023-11-08T05:29:43.389354+0000","last_clean":"2023-11-08T05:28:01.458439+0000","last_became_active":"2023-11-08T05:28:13.557071+0000","last_became_peered":"2023-11-08T05:28:13.557071+0000","last_unstale":"2023-11-08T05:29:43.389354+0000","last_undegraded":"2023-11-08T05:29:43.389354+0000","last_fullsized":"2023-11-08T05:28:13.547025+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":18,"reported_epoch":455,"state":"active+undersized","last_fresh":"2023-11-08T05:29:43.389262+0000","last_change":"2023-11-08T05:28:13.559513+0000","last_active":"2023-11-08T05:29:43.389262+0000","last_peered":"2023-11-08T05:29:43.389262+0000","last_clean":"2023-11-08T04:52:46.531964+0000","last_became_active":"2023-11-08T05:28:13.559513+0000","last_became_peered":"2023-11-08T05:28:13.559513+0000","last_unstale":"2023-11-08T05:29:43.389262+0000","last_undegraded":"2023-11-08T05:29:43.389262+0000","last_fullsized":"2023-11-08T05:28:13.540852+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":453,"reported_epoch":455,"state":"active+clean","last_fresh":"2023-11-08T05:29:43.390991+0000","last_change":"2023-11-08T04:52:47.546100+0000","last_active":"2023-11-08T05:29:43.390991+0000","last_peered":"2023-11-08T05:29:43.390991+0000","last_clean":"2023-11-08T05:29:43.390991+0000","last_became_active":"2023-11-08T04:52:47.545840+0000","last_became_peered":"2023-11-08T04:52:47.545840+0000","last_unstale":"2023-11-08T05:29:43.390991+0000","last_undegraded":"2023-11-08T05:29:43.390991+0000","last_fullsized":"2023-11-08T05:29:43.390991+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":451,"reported_epoch":455,"state":"active+clean","last_fresh":"2023-11-08T05:29:43.389261+0000","last_change":"2023-11-08T04:52:47.548597+0000","last_active":"2023-11-08T05:29:43.389261+0000","last_peered":"2023-11-08T05:29:43.389261+0000","last_clean":"2023-11-08T05:29:43.389261+0000","last_became_active":"2023-11-08T04:52:47.548294+0000","last_became_peered":"2023-11-08T04:52:47.548294+0000","last_unstale":"2023-11-08T05:29:43.389261+0000","last_undegraded":"2023-11-08T05:29:43.389261+0000","last_fullsized":"2023-11-08T05:29:43.389261+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":457,"reported_epoch":455,"state":"active+undersized","last_fresh":"2023-11-08T05:29:43.389822+0000","last_change":"2023-11-08T05:28:13.545468+0000","last_active":"2023-11-08T05:29:43.389822+0000","last_peered":"2023-11-08T05:29:43.389822+0000","last_clean":"2023-11-08T05:28:01.456723+0000","last_became_active":"2023-11-08T05:28:13.545468+0000","last_became_peered":"2023-11-08T05:28:13.545468+0000","last_unstale":"2023-11-08T05:29:43.389822+0000","last_undegraded":"2023-11-08T05:29:43.389822+0000","last_fullsized":"2023-11-08T05:28:13.537207+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_clean_scrub_stamp":"2023-11-08T04:52:45.528796+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":462,"reported_epoch":456,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:29:44.395350+0000","last_change":"2023-11-08T05:29:44.395350+0000","last_active":"2023-11-08T05:29:44.395350+0000","last_peered":"2023-11-08T05:29:44.395350+0000","last_clean":"2023-11-08T05:29:44.395350+0000","last_became_active":"2023-11-08T05:29:44.395012+0000","last_became_peered":"2023-11-08T05:29:44.395012+0000","last_unstale":"2023-11-08T05:29:44.395350+0000","last_undegraded":"2023-11-08T05:29:44.395350+0000","last_fullsized":"2023-11-08T05:29:44.395350+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":456,"reported_epoch":455,"state":"active+undersized","last_fresh":"2023-11-08T05:29:43.389390+0000","last_change":"2023-11-08T05:28:13.563357+0000","last_active":"2023-11-08T05:29:43.389390+0000","last_peered":"2023-11-08T05:29:43.389390+0000","last_clean":"2023-11-08T05:28:01.457691+0000","last_became_active":"2023-11-08T05:28:13.563357+0000","last_became_peered":"2023-11-08T05:28:13.563357+0000","last_unstale":"2023-11-08T05:29:43.389390+0000","last_undegraded":"2023-11-08T05:29:43.389390+0000","last_fullsized":"2023-11-08T05:28:13.541614+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":456,"reported_epoch":455,"state":"active+undersized","last_fresh":"2023-11-08T05:29:43.389311+0000","last_change":"2023-11-08T05:28:13.550672+0000","last_active":"2023-11-08T05:29:43.389311+0000","last_peered":"2023-11-08T05:29:43.389311+0000","last_clean":"2023-11-08T05:28:01.457904+0000","last_became_active":"2023-11-08T05:28:13.550672+0000","last_became_peered":"2023-11-08T05:28:13.550672+0000","last_unstale":"2023-11-08T05:29:43.389311+0000","last_undegraded":"2023-11-08T05:29:43.389311+0000","last_fullsized":"2023-11-08T05:28:13.539595+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":456,"reported_epoch":455,"state":"active+undersized","last_fresh":"2023-11-08T05:29:43.389474+0000","last_change":"2023-11-08T05:28:13.557968+0000","last_active":"2023-11-08T05:29:43.389474+0000","last_peered":"2023-11-08T05:29:43.389474+0000","last_clean":"2023-11-08T05:28:01.458433+0000","last_became_active":"2023-11-08T05:28:13.557968+0000","last_became_peered":"2023-11-08T05:28:13.557968+0000","last_unstale":"2023-11-08T05:29:43.389474+0000","last_undegraded":"2023-11-08T05:29:43.389474+0000","last_fullsized":"2023-11-08T05:28:13.542832+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":1,"acting":1,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":10,"acting":11,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738823,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":645004,"kb_used_data":5316,"kb_used_omap":275,"kb_used_meta":639404,"kb_avail":93726836,"statfs":{"total":96636764160,"available":95976280064,"internally_reserved":0,"allocated":5443584,"data_stored":2678316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":281650,"internal_metadata":654750670},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:59 2023","interfaces":[{"interface":"back","average":{"1min":0.30499999999999999,"5min":0.439,"15min":0.36099999999999999},"min":{"1min":0.17399999999999999,"5min":0.14499999999999999,"15min":0.14499999999999999},"max":{"1min":0.60099999999999998,"5min":4.9379999999999997,"15min":4.9379999999999997},"last":0.379},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.49399999999999999,"15min":0.42399999999999999},"min":{"1min":0.221,"5min":0.19600000000000001,"15min":0.17000000000000001},"max":{"1min":0.63200000000000001,"5min":4.9859999999999998,"15min":4.9859999999999998},"last":0.40899999999999997}]},{"osd":1,"last update":"Wed Nov 8 05:30:03 2023","interfaces":[{"interface":"back","average":{"1min":0.28499999999999998,"5min":0.65100000000000002,"15min":0.47099999999999997},"min":{"1min":0.20100000000000001,"5min":0.153,"15min":0.153},"max":{"1min":0.39800000000000002,"5min":23.242999999999999,"15min":23.242999999999999},"last":0.248},{"interface":"front","average":{"1min":0.29499999999999998,"5min":0.63100000000000001,"15min":0.45900000000000002},"min":{"1min":0.224,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":0.436,"5min":23.251000000000001,"15min":23.251000000000001},"last":0.23799999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738798,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1339228,"kb_used_data":398304,"kb_used_omap":291,"kb_used_meta":936476,"kb_avail":93032612,"statfs":{"total":96636764160,"available":95265394688,"internally_reserved":0,"allocated":407863296,"data_stored":405103900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":298462,"internal_metadata":958951970},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.436,"15min":0.36299999999999999},"min":{"1min":0.19700000000000001,"5min":0.14699999999999999,"15min":0.13700000000000001},"max":{"1min":3.222,"5min":6.5499999999999998,"15min":6.9989999999999997},"last":0.245},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.46300000000000002,"15min":0.42299999999999999},"min":{"1min":0.17799999999999999,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":5.0030000000000001,"5min":6.0629999999999997,"15min":9.6869999999999994},"last":0.28599999999999998}]},{"osd":2,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.51600000000000001,"15min":0.47099999999999997},"min":{"1min":0.21099999999999999,"5min":0.153,"15min":0.153},"max":{"1min":5.7439999999999998,"5min":5.8280000000000003,"15min":9.7639999999999993},"last":0.252},{"interface":"front","average":{"1min":0.497,"5min":0.56200000000000006,"15min":0.42299999999999999},"min":{"1min":0.20399999999999999,"5min":0.187,"15min":0.16800000000000001},"max":{"1min":3.9319999999999999,"5min":8.8040000000000003,"15min":8.8040000000000003},"last":0.29899999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738825,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":650064,"kb_used_data":5320,"kb_used_omap":214,"kb_used_meta":644521,"kb_avail":93721776,"statfs":{"total":96636764160,"available":95971098624,"internally_reserved":0,"allocated":5447680,"data_stored":2679285,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":219335,"internal_metadata":659990329},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:56 2023","interfaces":[{"interface":"back","average":{"1min":0.30399999999999999,"5min":0.34699999999999998,"15min":0.36499999999999999},"min":{"1min":0.23999999999999999,"5min":0.14199999999999999,"15min":0.14199999999999999},"max":{"1min":0.42399999999999999,"5min":1.595,"15min":7.4249999999999998},"last":0.17100000000000001},{"interface":"front","average":{"1min":0.35499999999999998,"5min":0.40899999999999997,"15min":0.376},"min":{"1min":0.23999999999999999,"5min":0.19600000000000001,"15min":0.183},"max":{"1min":0.46999999999999997,"5min":2.6859999999999999,"15min":2.6859999999999999},"last":0.23000000000000001}]},{"osd":2,"last update":"Wed Nov 8 05:29:57 2023","interfaces":[{"interface":"back","average":{"1min":0.28199999999999997,"5min":0.36099999999999999,"15min":0.373},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":0.34399999999999997,"5min":2.2650000000000001,"15min":5.8209999999999997},"last":0.21099999999999999},{"interface":"front","average":{"1min":0.28599999999999998,"5min":0.31,"15min":0.35799999999999998},"min":{"1min":0.17199999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.35699999999999998,"5min":1.137,"15min":4.2350000000000003},"last":0.27100000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-08T05:30:28.719 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-11-08T05:30:28.720 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 456, 'reported_epoch': 455, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:29:43.389354+0000', 'last_change': '2023-11-08T05:28:13.557071+0000', 'last_active': '2023-11-08T05:29:43.389354+0000', 'last_peered': '2023-11-08T05:29:43.389354+0000', 'last_clean': '2023-11-08T05:28:01.458439+0000', 'last_became_active': '2023-11-08T05:28:13.557071+0000', 'last_became_peered': '2023-11-08T05:28:13.557071+0000', 'last_unstale': '2023-11-08T05:29:43.389354+0000', 'last_undegraded': '2023-11-08T05:29:43.389354+0000', 'last_fullsized': '2023-11-08T05:28:13.547025+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:30:28.720 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-11-08T05:30:28.720 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 18, 'reported_epoch': 455, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:29:43.389262+0000', 'last_change': '2023-11-08T05:28:13.559513+0000', 'last_active': '2023-11-08T05:29:43.389262+0000', 'last_peered': '2023-11-08T05:29:43.389262+0000', 'last_clean': '2023-11-08T04:52:46.531964+0000', 'last_became_active': '2023-11-08T05:28:13.559513+0000', 'last_became_peered': '2023-11-08T05:28:13.559513+0000', 'last_unstale': '2023-11-08T05:29:43.389262+0000', 'last_undegraded': '2023-11-08T05:29:43.389262+0000', 'last_fullsized': '2023-11-08T05:28:13.540852+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:30:28.721 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-11-08T05:30:28.721 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "0'0", 'reported_seq': 457, 'reported_epoch': 455, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:29:43.389822+0000', 'last_change': '2023-11-08T05:28:13.545468+0000', 'last_active': '2023-11-08T05:29:43.389822+0000', 'last_peered': '2023-11-08T05:29:43.389822+0000', 'last_clean': '2023-11-08T05:28:01.456723+0000', 'last_became_active': '2023-11-08T05:28:13.545468+0000', 'last_became_peered': '2023-11-08T05:28:13.545468+0000', 'last_unstale': '2023-11-08T05:29:43.389822+0000', 'last_undegraded': '2023-11-08T05:29:43.389822+0000', 'last_fullsized': '2023-11-08T05:28:13.537207+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 9, 'last_epoch_clean': 10, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:45.528796+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:45.528796+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:45.528796+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:30:28.721 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.0 is not active+clean 2023-11-08T05:30:28.721 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.0', 'version': "0'0", 'reported_seq': 462, 'reported_epoch': 456, 'state': 'active+clean+remapped', 'last_fresh': '2023-11-08T05:29:44.395350+0000', 'last_change': '2023-11-08T05:29:44.395350+0000', 'last_active': '2023-11-08T05:29:44.395350+0000', 'last_peered': '2023-11-08T05:29:44.395350+0000', 'last_clean': '2023-11-08T05:29:44.395350+0000', 'last_became_active': '2023-11-08T05:29:44.395012+0000', 'last_became_peered': '2023-11-08T05:29:44.395012+0000', 'last_unstale': '2023-11-08T05:29:44.395350+0000', 'last_undegraded': '2023-11-08T05:29:44.395350+0000', 'last_fullsized': '2023-11-08T05:29:44.395350+0000', 'mapping_epoch': 455, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 456, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2], 'acting': [2, 1], 'avail_no_missing': ['2', '1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2023-11-08T05:30:28.722 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-11-08T05:30:28.722 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 456, 'reported_epoch': 455, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:29:43.389390+0000', 'last_change': '2023-11-08T05:28:13.563357+0000', 'last_active': '2023-11-08T05:29:43.389390+0000', 'last_peered': '2023-11-08T05:29:43.389390+0000', 'last_clean': '2023-11-08T05:28:01.457691+0000', 'last_became_active': '2023-11-08T05:28:13.563357+0000', 'last_became_peered': '2023-11-08T05:28:13.563357+0000', 'last_unstale': '2023-11-08T05:29:43.389390+0000', 'last_undegraded': '2023-11-08T05:29:43.389390+0000', 'last_fullsized': '2023-11-08T05:28:13.541614+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:30:28.722 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-11-08T05:30:28.723 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 456, 'reported_epoch': 455, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:29:43.389311+0000', 'last_change': '2023-11-08T05:28:13.550672+0000', 'last_active': '2023-11-08T05:29:43.389311+0000', 'last_peered': '2023-11-08T05:29:43.389311+0000', 'last_clean': '2023-11-08T05:28:01.457904+0000', 'last_became_active': '2023-11-08T05:28:13.550672+0000', 'last_became_peered': '2023-11-08T05:28:13.550672+0000', 'last_unstale': '2023-11-08T05:29:43.389311+0000', 'last_undegraded': '2023-11-08T05:29:43.389311+0000', 'last_fullsized': '2023-11-08T05:28:13.539595+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:30:28.723 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-11-08T05:30:28.723 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 456, 'reported_epoch': 455, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:29:43.389474+0000', 'last_change': '2023-11-08T05:28:13.557968+0000', 'last_active': '2023-11-08T05:29:43.389474+0000', 'last_peered': '2023-11-08T05:29:43.389474+0000', 'last_clean': '2023-11-08T05:28:01.458433+0000', 'last_became_active': '2023-11-08T05:28:13.557968+0000', 'last_became_peered': '2023-11-08T05:28:13.557968+0000', 'last_unstale': '2023-11-08T05:29:43.389474+0000', 'last_undegraded': '2023-11-08T05:29:43.389474+0000', 'last_fullsized': '2023-11-08T05:28:13.542832+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:30:28.724 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-11-08T05:30:29.070 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T05:30:29.071 INFO:teuthology.orchestra.run.smithi087.stdout:{"fsid":"ab4cbc06-449d-497d-b513-220d9db2b590","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 6 pgs undersized","count":6},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2271,"monmap":{"epoch":1,"min_mon_release_name":"pacific","num_mons":1},"osdmap":{"epoch":456,"num_osds":3,"num_up_osds":2,"osd_up_since":1699421292,"num_in_osds":3,"osd_in_since":1699419158,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":6},{"state_name":"active+clean","count":2},{"state_name":"active+clean+remapped","count":1}],"num_pgs":9,"num_pools":2,"num_objects":0,"data_bytes":0,"bytes_used":2697519104,"bytes_avail":287212773376,"bytes_total":289910292480},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":41,"modified":"2023-11-08T05:29:32.044544+0000","services":{}},"progress_events":{}} 2023-11-08T05:30:29.084 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-11-08T05:30:29.084 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-11-08T05:30:29.365 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T05:30:29.367 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2023-11-08T05:30:29.378 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":1585,"stamp":"2023-11-08T05:30:28.056918+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2634296,"kb_used_data":408940,"kb_used_omap":780,"kb_used_meta":2220403,"kb_avail":280481224,"statfs":{"total":289910292480,"available":287212773376,"internally_reserved":0,"allocated":418754560,"data_stored":410461501,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":799447,"internal_metadata":2273692969},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-536782902,"num_objects":-164,"num_object_clones":0,"num_object_copies":-328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-115,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-164,"num_whiteouts":0,"num_read":-122056,"num_read_kb":-39011942,"num_write":-413642,"num_write_kb":-151705339,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-8,"num_bytes_recovered":-16922624,"num_keys_recovered":-2,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.424817"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":456,"reported_epoch":455,"state":"active+undersized","last_fresh":"2023-11-08T05:29:43.389354+0000","last_change":"2023-11-08T05:28:13.557071+0000","last_active":"2023-11-08T05:29:43.389354+0000","last_peered":"2023-11-08T05:29:43.389354+0000","last_clean":"2023-11-08T05:28:01.458439+0000","last_became_active":"2023-11-08T05:28:13.557071+0000","last_became_peered":"2023-11-08T05:28:13.557071+0000","last_unstale":"2023-11-08T05:29:43.389354+0000","last_undegraded":"2023-11-08T05:29:43.389354+0000","last_fullsized":"2023-11-08T05:28:13.547025+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":18,"reported_epoch":455,"state":"active+undersized","last_fresh":"2023-11-08T05:29:43.389262+0000","last_change":"2023-11-08T05:28:13.559513+0000","last_active":"2023-11-08T05:29:43.389262+0000","last_peered":"2023-11-08T05:29:43.389262+0000","last_clean":"2023-11-08T04:52:46.531964+0000","last_became_active":"2023-11-08T05:28:13.559513+0000","last_became_peered":"2023-11-08T05:28:13.559513+0000","last_unstale":"2023-11-08T05:29:43.389262+0000","last_undegraded":"2023-11-08T05:29:43.389262+0000","last_fullsized":"2023-11-08T05:28:13.540852+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":453,"reported_epoch":455,"state":"active+clean","last_fresh":"2023-11-08T05:29:43.390991+0000","last_change":"2023-11-08T04:52:47.546100+0000","last_active":"2023-11-08T05:29:43.390991+0000","last_peered":"2023-11-08T05:29:43.390991+0000","last_clean":"2023-11-08T05:29:43.390991+0000","last_became_active":"2023-11-08T04:52:47.545840+0000","last_became_peered":"2023-11-08T04:52:47.545840+0000","last_unstale":"2023-11-08T05:29:43.390991+0000","last_undegraded":"2023-11-08T05:29:43.390991+0000","last_fullsized":"2023-11-08T05:29:43.390991+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":451,"reported_epoch":455,"state":"active+clean","last_fresh":"2023-11-08T05:29:43.389261+0000","last_change":"2023-11-08T04:52:47.548597+0000","last_active":"2023-11-08T05:29:43.389261+0000","last_peered":"2023-11-08T05:29:43.389261+0000","last_clean":"2023-11-08T05:29:43.389261+0000","last_became_active":"2023-11-08T04:52:47.548294+0000","last_became_peered":"2023-11-08T04:52:47.548294+0000","last_unstale":"2023-11-08T05:29:43.389261+0000","last_undegraded":"2023-11-08T05:29:43.389261+0000","last_fullsized":"2023-11-08T05:29:43.389261+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":457,"reported_epoch":455,"state":"active+undersized","last_fresh":"2023-11-08T05:29:43.389822+0000","last_change":"2023-11-08T05:28:13.545468+0000","last_active":"2023-11-08T05:29:43.389822+0000","last_peered":"2023-11-08T05:29:43.389822+0000","last_clean":"2023-11-08T05:28:01.456723+0000","last_became_active":"2023-11-08T05:28:13.545468+0000","last_became_peered":"2023-11-08T05:28:13.545468+0000","last_unstale":"2023-11-08T05:29:43.389822+0000","last_undegraded":"2023-11-08T05:29:43.389822+0000","last_fullsized":"2023-11-08T05:28:13.537207+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_clean_scrub_stamp":"2023-11-08T04:52:45.528796+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":462,"reported_epoch":456,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:29:44.395350+0000","last_change":"2023-11-08T05:29:44.395350+0000","last_active":"2023-11-08T05:29:44.395350+0000","last_peered":"2023-11-08T05:29:44.395350+0000","last_clean":"2023-11-08T05:29:44.395350+0000","last_became_active":"2023-11-08T05:29:44.395012+0000","last_became_peered":"2023-11-08T05:29:44.395012+0000","last_unstale":"2023-11-08T05:29:44.395350+0000","last_undegraded":"2023-11-08T05:29:44.395350+0000","last_fullsized":"2023-11-08T05:29:44.395350+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":456,"reported_epoch":455,"state":"active+undersized","last_fresh":"2023-11-08T05:29:43.389390+0000","last_change":"2023-11-08T05:28:13.563357+0000","last_active":"2023-11-08T05:29:43.389390+0000","last_peered":"2023-11-08T05:29:43.389390+0000","last_clean":"2023-11-08T05:28:01.457691+0000","last_became_active":"2023-11-08T05:28:13.563357+0000","last_became_peered":"2023-11-08T05:28:13.563357+0000","last_unstale":"2023-11-08T05:29:43.389390+0000","last_undegraded":"2023-11-08T05:29:43.389390+0000","last_fullsized":"2023-11-08T05:28:13.541614+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":456,"reported_epoch":455,"state":"active+undersized","last_fresh":"2023-11-08T05:29:43.389311+0000","last_change":"2023-11-08T05:28:13.550672+0000","last_active":"2023-11-08T05:29:43.389311+0000","last_peered":"2023-11-08T05:29:43.389311+0000","last_clean":"2023-11-08T05:28:01.457904+0000","last_became_active":"2023-11-08T05:28:13.550672+0000","last_became_peered":"2023-11-08T05:28:13.550672+0000","last_unstale":"2023-11-08T05:29:43.389311+0000","last_undegraded":"2023-11-08T05:29:43.389311+0000","last_fullsized":"2023-11-08T05:28:13.539595+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":456,"reported_epoch":455,"state":"active+undersized","last_fresh":"2023-11-08T05:29:43.389474+0000","last_change":"2023-11-08T05:28:13.557968+0000","last_active":"2023-11-08T05:29:43.389474+0000","last_peered":"2023-11-08T05:29:43.389474+0000","last_clean":"2023-11-08T05:28:01.458433+0000","last_became_active":"2023-11-08T05:28:13.557968+0000","last_became_peered":"2023-11-08T05:28:13.557968+0000","last_unstale":"2023-11-08T05:29:43.389474+0000","last_undegraded":"2023-11-08T05:29:43.389474+0000","last_fullsized":"2023-11-08T05:28:13.542832+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":1,"acting":1,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":10,"acting":11,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738823,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":645004,"kb_used_data":5316,"kb_used_omap":275,"kb_used_meta":639404,"kb_avail":93726836,"statfs":{"total":96636764160,"available":95976280064,"internally_reserved":0,"allocated":5443584,"data_stored":2678316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":281650,"internal_metadata":654750670},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:59 2023","interfaces":[{"interface":"back","average":{"1min":0.30499999999999999,"5min":0.439,"15min":0.36099999999999999},"min":{"1min":0.17399999999999999,"5min":0.14499999999999999,"15min":0.14499999999999999},"max":{"1min":0.60099999999999998,"5min":4.9379999999999997,"15min":4.9379999999999997},"last":0.379},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.49399999999999999,"15min":0.42399999999999999},"min":{"1min":0.221,"5min":0.19600000000000001,"15min":0.17000000000000001},"max":{"1min":0.63200000000000001,"5min":4.9859999999999998,"15min":4.9859999999999998},"last":0.40899999999999997}]},{"osd":1,"last update":"Wed Nov 8 05:30:03 2023","interfaces":[{"interface":"back","average":{"1min":0.28499999999999998,"5min":0.65100000000000002,"15min":0.47099999999999997},"min":{"1min":0.20100000000000001,"5min":0.153,"15min":0.153},"max":{"1min":0.39800000000000002,"5min":23.242999999999999,"15min":23.242999999999999},"last":0.248},{"interface":"front","average":{"1min":0.29499999999999998,"5min":0.63100000000000001,"15min":0.45900000000000002},"min":{"1min":0.224,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":0.436,"5min":23.251000000000001,"15min":23.251000000000001},"last":0.23799999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738798,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1339228,"kb_used_data":398304,"kb_used_omap":291,"kb_used_meta":936476,"kb_avail":93032612,"statfs":{"total":96636764160,"available":95265394688,"internally_reserved":0,"allocated":407863296,"data_stored":405103900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":298462,"internal_metadata":958951970},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.436,"15min":0.36299999999999999},"min":{"1min":0.19700000000000001,"5min":0.14699999999999999,"15min":0.13700000000000001},"max":{"1min":3.222,"5min":6.5499999999999998,"15min":6.9989999999999997},"last":0.245},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.46300000000000002,"15min":0.42299999999999999},"min":{"1min":0.17799999999999999,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":5.0030000000000001,"5min":6.0629999999999997,"15min":9.6869999999999994},"last":0.28599999999999998}]},{"osd":2,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.51600000000000001,"15min":0.47099999999999997},"min":{"1min":0.21099999999999999,"5min":0.153,"15min":0.153},"max":{"1min":5.7439999999999998,"5min":5.8280000000000003,"15min":9.7639999999999993},"last":0.252},{"interface":"front","average":{"1min":0.497,"5min":0.56200000000000006,"15min":0.42299999999999999},"min":{"1min":0.20399999999999999,"5min":0.187,"15min":0.16800000000000001},"max":{"1min":3.9319999999999999,"5min":8.8040000000000003,"15min":8.8040000000000003},"last":0.29899999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738825,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":650064,"kb_used_data":5320,"kb_used_omap":214,"kb_used_meta":644521,"kb_avail":93721776,"statfs":{"total":96636764160,"available":95971098624,"internally_reserved":0,"allocated":5447680,"data_stored":2679285,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":219335,"internal_metadata":659990329},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:56 2023","interfaces":[{"interface":"back","average":{"1min":0.30399999999999999,"5min":0.34699999999999998,"15min":0.36499999999999999},"min":{"1min":0.23999999999999999,"5min":0.14199999999999999,"15min":0.14199999999999999},"max":{"1min":0.42399999999999999,"5min":1.595,"15min":7.4249999999999998},"last":0.17100000000000001},{"interface":"front","average":{"1min":0.35499999999999998,"5min":0.40899999999999997,"15min":0.376},"min":{"1min":0.23999999999999999,"5min":0.19600000000000001,"15min":0.183},"max":{"1min":0.46999999999999997,"5min":2.6859999999999999,"15min":2.6859999999999999},"last":0.23000000000000001}]},{"osd":2,"last update":"Wed Nov 8 05:29:57 2023","interfaces":[{"interface":"back","average":{"1min":0.28199999999999997,"5min":0.36099999999999999,"15min":0.373},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":0.34399999999999997,"5min":2.2650000000000001,"15min":5.8209999999999997},"last":0.21099999999999999},{"interface":"front","average":{"1min":0.28599999999999998,"5min":0.31,"15min":0.35799999999999998},"min":{"1min":0.17199999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.35699999999999998,"5min":1.137,"15min":4.2350000000000003},"last":0.27100000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-08T05:30:32.380 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-11-08T05:30:32.674 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T05:30:32.676 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2023-11-08T05:30:32.687 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":1587,"stamp":"2023-11-08T05:30:32.057701+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2634296,"kb_used_data":408940,"kb_used_omap":780,"kb_used_meta":2220403,"kb_avail":280481224,"statfs":{"total":289910292480,"available":287212773376,"internally_reserved":0,"allocated":418754560,"data_stored":410461501,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":799447,"internal_metadata":2273692969},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-536782902,"num_objects":-164,"num_object_clones":0,"num_object_copies":-328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-115,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-164,"num_whiteouts":0,"num_read":-122056,"num_read_kb":-39011942,"num_write":-413642,"num_write_kb":-151705339,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-8,"num_bytes_recovered":-16922624,"num_keys_recovered":-2,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.424817"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":456,"reported_epoch":455,"state":"active+undersized","last_fresh":"2023-11-08T05:29:43.389354+0000","last_change":"2023-11-08T05:28:13.557071+0000","last_active":"2023-11-08T05:29:43.389354+0000","last_peered":"2023-11-08T05:29:43.389354+0000","last_clean":"2023-11-08T05:28:01.458439+0000","last_became_active":"2023-11-08T05:28:13.557071+0000","last_became_peered":"2023-11-08T05:28:13.557071+0000","last_unstale":"2023-11-08T05:29:43.389354+0000","last_undegraded":"2023-11-08T05:29:43.389354+0000","last_fullsized":"2023-11-08T05:28:13.547025+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":18,"reported_epoch":455,"state":"active+undersized","last_fresh":"2023-11-08T05:29:43.389262+0000","last_change":"2023-11-08T05:28:13.559513+0000","last_active":"2023-11-08T05:29:43.389262+0000","last_peered":"2023-11-08T05:29:43.389262+0000","last_clean":"2023-11-08T04:52:46.531964+0000","last_became_active":"2023-11-08T05:28:13.559513+0000","last_became_peered":"2023-11-08T05:28:13.559513+0000","last_unstale":"2023-11-08T05:29:43.389262+0000","last_undegraded":"2023-11-08T05:29:43.389262+0000","last_fullsized":"2023-11-08T05:28:13.540852+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":453,"reported_epoch":455,"state":"active+clean","last_fresh":"2023-11-08T05:29:43.390991+0000","last_change":"2023-11-08T04:52:47.546100+0000","last_active":"2023-11-08T05:29:43.390991+0000","last_peered":"2023-11-08T05:29:43.390991+0000","last_clean":"2023-11-08T05:29:43.390991+0000","last_became_active":"2023-11-08T04:52:47.545840+0000","last_became_peered":"2023-11-08T04:52:47.545840+0000","last_unstale":"2023-11-08T05:29:43.390991+0000","last_undegraded":"2023-11-08T05:29:43.390991+0000","last_fullsized":"2023-11-08T05:29:43.390991+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":451,"reported_epoch":455,"state":"active+clean","last_fresh":"2023-11-08T05:29:43.389261+0000","last_change":"2023-11-08T04:52:47.548597+0000","last_active":"2023-11-08T05:29:43.389261+0000","last_peered":"2023-11-08T05:29:43.389261+0000","last_clean":"2023-11-08T05:29:43.389261+0000","last_became_active":"2023-11-08T04:52:47.548294+0000","last_became_peered":"2023-11-08T04:52:47.548294+0000","last_unstale":"2023-11-08T05:29:43.389261+0000","last_undegraded":"2023-11-08T05:29:43.389261+0000","last_fullsized":"2023-11-08T05:29:43.389261+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":457,"reported_epoch":455,"state":"active+undersized","last_fresh":"2023-11-08T05:29:43.389822+0000","last_change":"2023-11-08T05:28:13.545468+0000","last_active":"2023-11-08T05:29:43.389822+0000","last_peered":"2023-11-08T05:29:43.389822+0000","last_clean":"2023-11-08T05:28:01.456723+0000","last_became_active":"2023-11-08T05:28:13.545468+0000","last_became_peered":"2023-11-08T05:28:13.545468+0000","last_unstale":"2023-11-08T05:29:43.389822+0000","last_undegraded":"2023-11-08T05:29:43.389822+0000","last_fullsized":"2023-11-08T05:28:13.537207+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_clean_scrub_stamp":"2023-11-08T04:52:45.528796+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":462,"reported_epoch":456,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:29:44.395350+0000","last_change":"2023-11-08T05:29:44.395350+0000","last_active":"2023-11-08T05:29:44.395350+0000","last_peered":"2023-11-08T05:29:44.395350+0000","last_clean":"2023-11-08T05:29:44.395350+0000","last_became_active":"2023-11-08T05:29:44.395012+0000","last_became_peered":"2023-11-08T05:29:44.395012+0000","last_unstale":"2023-11-08T05:29:44.395350+0000","last_undegraded":"2023-11-08T05:29:44.395350+0000","last_fullsized":"2023-11-08T05:29:44.395350+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":456,"reported_epoch":455,"state":"active+undersized","last_fresh":"2023-11-08T05:29:43.389390+0000","last_change":"2023-11-08T05:28:13.563357+0000","last_active":"2023-11-08T05:29:43.389390+0000","last_peered":"2023-11-08T05:29:43.389390+0000","last_clean":"2023-11-08T05:28:01.457691+0000","last_became_active":"2023-11-08T05:28:13.563357+0000","last_became_peered":"2023-11-08T05:28:13.563357+0000","last_unstale":"2023-11-08T05:29:43.389390+0000","last_undegraded":"2023-11-08T05:29:43.389390+0000","last_fullsized":"2023-11-08T05:28:13.541614+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":456,"reported_epoch":455,"state":"active+undersized","last_fresh":"2023-11-08T05:29:43.389311+0000","last_change":"2023-11-08T05:28:13.550672+0000","last_active":"2023-11-08T05:29:43.389311+0000","last_peered":"2023-11-08T05:29:43.389311+0000","last_clean":"2023-11-08T05:28:01.457904+0000","last_became_active":"2023-11-08T05:28:13.550672+0000","last_became_peered":"2023-11-08T05:28:13.550672+0000","last_unstale":"2023-11-08T05:29:43.389311+0000","last_undegraded":"2023-11-08T05:29:43.389311+0000","last_fullsized":"2023-11-08T05:28:13.539595+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":456,"reported_epoch":455,"state":"active+undersized","last_fresh":"2023-11-08T05:29:43.389474+0000","last_change":"2023-11-08T05:28:13.557968+0000","last_active":"2023-11-08T05:29:43.389474+0000","last_peered":"2023-11-08T05:29:43.389474+0000","last_clean":"2023-11-08T05:28:01.458433+0000","last_became_active":"2023-11-08T05:28:13.557968+0000","last_became_peered":"2023-11-08T05:28:13.557968+0000","last_unstale":"2023-11-08T05:29:43.389474+0000","last_undegraded":"2023-11-08T05:29:43.389474+0000","last_fullsized":"2023-11-08T05:28:13.542832+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":1,"acting":1,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":10,"acting":11,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738824,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":645004,"kb_used_data":5316,"kb_used_omap":275,"kb_used_meta":639404,"kb_avail":93726836,"statfs":{"total":96636764160,"available":95976280064,"internally_reserved":0,"allocated":5443584,"data_stored":2678316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":281650,"internal_metadata":654750670},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:59 2023","interfaces":[{"interface":"back","average":{"1min":0.30499999999999999,"5min":0.439,"15min":0.36099999999999999},"min":{"1min":0.17399999999999999,"5min":0.14499999999999999,"15min":0.14499999999999999},"max":{"1min":0.60099999999999998,"5min":4.9379999999999997,"15min":4.9379999999999997},"last":0.379},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.49399999999999999,"15min":0.42399999999999999},"min":{"1min":0.221,"5min":0.19600000000000001,"15min":0.17000000000000001},"max":{"1min":0.63200000000000001,"5min":4.9859999999999998,"15min":4.9859999999999998},"last":0.40899999999999997}]},{"osd":1,"last update":"Wed Nov 8 05:30:03 2023","interfaces":[{"interface":"back","average":{"1min":0.28499999999999998,"5min":0.65100000000000002,"15min":0.47099999999999997},"min":{"1min":0.20100000000000001,"5min":0.153,"15min":0.153},"max":{"1min":0.39800000000000002,"5min":23.242999999999999,"15min":23.242999999999999},"last":0.20899999999999999},{"interface":"front","average":{"1min":0.29499999999999998,"5min":0.63100000000000001,"15min":0.45900000000000002},"min":{"1min":0.224,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":0.436,"5min":23.251000000000001,"15min":23.251000000000001},"last":0.24099999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738798,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1339228,"kb_used_data":398304,"kb_used_omap":291,"kb_used_meta":936476,"kb_avail":93032612,"statfs":{"total":96636764160,"available":95265394688,"internally_reserved":0,"allocated":407863296,"data_stored":405103900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":298462,"internal_metadata":958951970},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.436,"15min":0.36299999999999999},"min":{"1min":0.19700000000000001,"5min":0.14699999999999999,"15min":0.13700000000000001},"max":{"1min":3.222,"5min":6.5499999999999998,"15min":6.9989999999999997},"last":0.245},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.46300000000000002,"15min":0.42299999999999999},"min":{"1min":0.17799999999999999,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":5.0030000000000001,"5min":6.0629999999999997,"15min":9.6869999999999994},"last":0.28599999999999998}]},{"osd":2,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.51600000000000001,"15min":0.47099999999999997},"min":{"1min":0.21099999999999999,"5min":0.153,"15min":0.153},"max":{"1min":5.7439999999999998,"5min":5.8280000000000003,"15min":9.7639999999999993},"last":0.252},{"interface":"front","average":{"1min":0.497,"5min":0.56200000000000006,"15min":0.42299999999999999},"min":{"1min":0.20399999999999999,"5min":0.187,"15min":0.16800000000000001},"max":{"1min":3.9319999999999999,"5min":8.8040000000000003,"15min":8.8040000000000003},"last":0.29899999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738826,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":650064,"kb_used_data":5320,"kb_used_omap":214,"kb_used_meta":644521,"kb_avail":93721776,"statfs":{"total":96636764160,"available":95971098624,"internally_reserved":0,"allocated":5447680,"data_stored":2679285,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":219335,"internal_metadata":659990329},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:56 2023","interfaces":[{"interface":"back","average":{"1min":0.30399999999999999,"5min":0.34699999999999998,"15min":0.36499999999999999},"min":{"1min":0.23999999999999999,"5min":0.14199999999999999,"15min":0.14199999999999999},"max":{"1min":0.42399999999999999,"5min":1.595,"15min":7.4249999999999998},"last":0.17100000000000001},{"interface":"front","average":{"1min":0.35499999999999998,"5min":0.40899999999999997,"15min":0.376},"min":{"1min":0.23999999999999999,"5min":0.19600000000000001,"15min":0.183},"max":{"1min":0.46999999999999997,"5min":2.6859999999999999,"15min":2.6859999999999999},"last":0.23000000000000001}]},{"osd":2,"last update":"Wed Nov 8 05:29:57 2023","interfaces":[{"interface":"back","average":{"1min":0.28199999999999997,"5min":0.36099999999999999,"15min":0.373},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":0.34399999999999997,"5min":2.2650000000000001,"15min":5.8209999999999997},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.28599999999999998,"5min":0.31,"15min":0.35799999999999998},"min":{"1min":0.17199999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.35699999999999998,"5min":1.137,"15min":4.2350000000000003},"last":0.42499999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-08T05:30:32.688 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-11-08T05:30:32.972 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T05:30:32.974 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2023-11-08T05:30:32.985 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":1587,"stamp":"2023-11-08T05:30:32.057701+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2634296,"kb_used_data":408940,"kb_used_omap":780,"kb_used_meta":2220403,"kb_avail":280481224,"statfs":{"total":289910292480,"available":287212773376,"internally_reserved":0,"allocated":418754560,"data_stored":410461501,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":799447,"internal_metadata":2273692969},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-536782902,"num_objects":-164,"num_object_clones":0,"num_object_copies":-328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-115,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-164,"num_whiteouts":0,"num_read":-122056,"num_read_kb":-39011942,"num_write":-413642,"num_write_kb":-151705339,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-8,"num_bytes_recovered":-16922624,"num_keys_recovered":-2,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.424817"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":456,"reported_epoch":455,"state":"active+undersized","last_fresh":"2023-11-08T05:29:43.389354+0000","last_change":"2023-11-08T05:28:13.557071+0000","last_active":"2023-11-08T05:29:43.389354+0000","last_peered":"2023-11-08T05:29:43.389354+0000","last_clean":"2023-11-08T05:28:01.458439+0000","last_became_active":"2023-11-08T05:28:13.557071+0000","last_became_peered":"2023-11-08T05:28:13.557071+0000","last_unstale":"2023-11-08T05:29:43.389354+0000","last_undegraded":"2023-11-08T05:29:43.389354+0000","last_fullsized":"2023-11-08T05:28:13.547025+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":18,"reported_epoch":455,"state":"active+undersized","last_fresh":"2023-11-08T05:29:43.389262+0000","last_change":"2023-11-08T05:28:13.559513+0000","last_active":"2023-11-08T05:29:43.389262+0000","last_peered":"2023-11-08T05:29:43.389262+0000","last_clean":"2023-11-08T04:52:46.531964+0000","last_became_active":"2023-11-08T05:28:13.559513+0000","last_became_peered":"2023-11-08T05:28:13.559513+0000","last_unstale":"2023-11-08T05:29:43.389262+0000","last_undegraded":"2023-11-08T05:29:43.389262+0000","last_fullsized":"2023-11-08T05:28:13.540852+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":453,"reported_epoch":455,"state":"active+clean","last_fresh":"2023-11-08T05:29:43.390991+0000","last_change":"2023-11-08T04:52:47.546100+0000","last_active":"2023-11-08T05:29:43.390991+0000","last_peered":"2023-11-08T05:29:43.390991+0000","last_clean":"2023-11-08T05:29:43.390991+0000","last_became_active":"2023-11-08T04:52:47.545840+0000","last_became_peered":"2023-11-08T04:52:47.545840+0000","last_unstale":"2023-11-08T05:29:43.390991+0000","last_undegraded":"2023-11-08T05:29:43.390991+0000","last_fullsized":"2023-11-08T05:29:43.390991+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":451,"reported_epoch":455,"state":"active+clean","last_fresh":"2023-11-08T05:29:43.389261+0000","last_change":"2023-11-08T04:52:47.548597+0000","last_active":"2023-11-08T05:29:43.389261+0000","last_peered":"2023-11-08T05:29:43.389261+0000","last_clean":"2023-11-08T05:29:43.389261+0000","last_became_active":"2023-11-08T04:52:47.548294+0000","last_became_peered":"2023-11-08T04:52:47.548294+0000","last_unstale":"2023-11-08T05:29:43.389261+0000","last_undegraded":"2023-11-08T05:29:43.389261+0000","last_fullsized":"2023-11-08T05:29:43.389261+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":457,"reported_epoch":455,"state":"active+undersized","last_fresh":"2023-11-08T05:29:43.389822+0000","last_change":"2023-11-08T05:28:13.545468+0000","last_active":"2023-11-08T05:29:43.389822+0000","last_peered":"2023-11-08T05:29:43.389822+0000","last_clean":"2023-11-08T05:28:01.456723+0000","last_became_active":"2023-11-08T05:28:13.545468+0000","last_became_peered":"2023-11-08T05:28:13.545468+0000","last_unstale":"2023-11-08T05:29:43.389822+0000","last_undegraded":"2023-11-08T05:29:43.389822+0000","last_fullsized":"2023-11-08T05:28:13.537207+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_clean_scrub_stamp":"2023-11-08T04:52:45.528796+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":462,"reported_epoch":456,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:29:44.395350+0000","last_change":"2023-11-08T05:29:44.395350+0000","last_active":"2023-11-08T05:29:44.395350+0000","last_peered":"2023-11-08T05:29:44.395350+0000","last_clean":"2023-11-08T05:29:44.395350+0000","last_became_active":"2023-11-08T05:29:44.395012+0000","last_became_peered":"2023-11-08T05:29:44.395012+0000","last_unstale":"2023-11-08T05:29:44.395350+0000","last_undegraded":"2023-11-08T05:29:44.395350+0000","last_fullsized":"2023-11-08T05:29:44.395350+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":456,"reported_epoch":455,"state":"active+undersized","last_fresh":"2023-11-08T05:29:43.389390+0000","last_change":"2023-11-08T05:28:13.563357+0000","last_active":"2023-11-08T05:29:43.389390+0000","last_peered":"2023-11-08T05:29:43.389390+0000","last_clean":"2023-11-08T05:28:01.457691+0000","last_became_active":"2023-11-08T05:28:13.563357+0000","last_became_peered":"2023-11-08T05:28:13.563357+0000","last_unstale":"2023-11-08T05:29:43.389390+0000","last_undegraded":"2023-11-08T05:29:43.389390+0000","last_fullsized":"2023-11-08T05:28:13.541614+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":456,"reported_epoch":455,"state":"active+undersized","last_fresh":"2023-11-08T05:29:43.389311+0000","last_change":"2023-11-08T05:28:13.550672+0000","last_active":"2023-11-08T05:29:43.389311+0000","last_peered":"2023-11-08T05:29:43.389311+0000","last_clean":"2023-11-08T05:28:01.457904+0000","last_became_active":"2023-11-08T05:28:13.550672+0000","last_became_peered":"2023-11-08T05:28:13.550672+0000","last_unstale":"2023-11-08T05:29:43.389311+0000","last_undegraded":"2023-11-08T05:29:43.389311+0000","last_fullsized":"2023-11-08T05:28:13.539595+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":456,"reported_epoch":455,"state":"active+undersized","last_fresh":"2023-11-08T05:29:43.389474+0000","last_change":"2023-11-08T05:28:13.557968+0000","last_active":"2023-11-08T05:29:43.389474+0000","last_peered":"2023-11-08T05:29:43.389474+0000","last_clean":"2023-11-08T05:28:01.458433+0000","last_became_active":"2023-11-08T05:28:13.557968+0000","last_became_peered":"2023-11-08T05:28:13.557968+0000","last_unstale":"2023-11-08T05:29:43.389474+0000","last_undegraded":"2023-11-08T05:29:43.389474+0000","last_fullsized":"2023-11-08T05:28:13.542832+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":1,"acting":1,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":10,"acting":11,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738824,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":645004,"kb_used_data":5316,"kb_used_omap":275,"kb_used_meta":639404,"kb_avail":93726836,"statfs":{"total":96636764160,"available":95976280064,"internally_reserved":0,"allocated":5443584,"data_stored":2678316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":281650,"internal_metadata":654750670},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:59 2023","interfaces":[{"interface":"back","average":{"1min":0.30499999999999999,"5min":0.439,"15min":0.36099999999999999},"min":{"1min":0.17399999999999999,"5min":0.14499999999999999,"15min":0.14499999999999999},"max":{"1min":0.60099999999999998,"5min":4.9379999999999997,"15min":4.9379999999999997},"last":0.379},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.49399999999999999,"15min":0.42399999999999999},"min":{"1min":0.221,"5min":0.19600000000000001,"15min":0.17000000000000001},"max":{"1min":0.63200000000000001,"5min":4.9859999999999998,"15min":4.9859999999999998},"last":0.40899999999999997}]},{"osd":1,"last update":"Wed Nov 8 05:30:03 2023","interfaces":[{"interface":"back","average":{"1min":0.28499999999999998,"5min":0.65100000000000002,"15min":0.47099999999999997},"min":{"1min":0.20100000000000001,"5min":0.153,"15min":0.153},"max":{"1min":0.39800000000000002,"5min":23.242999999999999,"15min":23.242999999999999},"last":0.20899999999999999},{"interface":"front","average":{"1min":0.29499999999999998,"5min":0.63100000000000001,"15min":0.45900000000000002},"min":{"1min":0.224,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":0.436,"5min":23.251000000000001,"15min":23.251000000000001},"last":0.24099999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738798,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1339228,"kb_used_data":398304,"kb_used_omap":291,"kb_used_meta":936476,"kb_avail":93032612,"statfs":{"total":96636764160,"available":95265394688,"internally_reserved":0,"allocated":407863296,"data_stored":405103900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":298462,"internal_metadata":958951970},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.436,"15min":0.36299999999999999},"min":{"1min":0.19700000000000001,"5min":0.14699999999999999,"15min":0.13700000000000001},"max":{"1min":3.222,"5min":6.5499999999999998,"15min":6.9989999999999997},"last":0.245},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.46300000000000002,"15min":0.42299999999999999},"min":{"1min":0.17799999999999999,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":5.0030000000000001,"5min":6.0629999999999997,"15min":9.6869999999999994},"last":0.28599999999999998}]},{"osd":2,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.51600000000000001,"15min":0.47099999999999997},"min":{"1min":0.21099999999999999,"5min":0.153,"15min":0.153},"max":{"1min":5.7439999999999998,"5min":5.8280000000000003,"15min":9.7639999999999993},"last":0.252},{"interface":"front","average":{"1min":0.497,"5min":0.56200000000000006,"15min":0.42299999999999999},"min":{"1min":0.20399999999999999,"5min":0.187,"15min":0.16800000000000001},"max":{"1min":3.9319999999999999,"5min":8.8040000000000003,"15min":8.8040000000000003},"last":0.29899999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738826,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":650064,"kb_used_data":5320,"kb_used_omap":214,"kb_used_meta":644521,"kb_avail":93721776,"statfs":{"total":96636764160,"available":95971098624,"internally_reserved":0,"allocated":5447680,"data_stored":2679285,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":219335,"internal_metadata":659990329},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:56 2023","interfaces":[{"interface":"back","average":{"1min":0.30399999999999999,"5min":0.34699999999999998,"15min":0.36499999999999999},"min":{"1min":0.23999999999999999,"5min":0.14199999999999999,"15min":0.14199999999999999},"max":{"1min":0.42399999999999999,"5min":1.595,"15min":7.4249999999999998},"last":0.17100000000000001},{"interface":"front","average":{"1min":0.35499999999999998,"5min":0.40899999999999997,"15min":0.376},"min":{"1min":0.23999999999999999,"5min":0.19600000000000001,"15min":0.183},"max":{"1min":0.46999999999999997,"5min":2.6859999999999999,"15min":2.6859999999999999},"last":0.23000000000000001}]},{"osd":2,"last update":"Wed Nov 8 05:29:57 2023","interfaces":[{"interface":"back","average":{"1min":0.28199999999999997,"5min":0.36099999999999999,"15min":0.373},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":0.34399999999999997,"5min":2.2650000000000001,"15min":5.8209999999999997},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.28599999999999998,"5min":0.31,"15min":0.35799999999999998},"min":{"1min":0.17199999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.35699999999999998,"5min":1.137,"15min":4.2350000000000003},"last":0.42499999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-08T05:30:32.986 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-11-08T05:30:32.986 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 456, 'reported_epoch': 455, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:29:43.389354+0000', 'last_change': '2023-11-08T05:28:13.557071+0000', 'last_active': '2023-11-08T05:29:43.389354+0000', 'last_peered': '2023-11-08T05:29:43.389354+0000', 'last_clean': '2023-11-08T05:28:01.458439+0000', 'last_became_active': '2023-11-08T05:28:13.557071+0000', 'last_became_peered': '2023-11-08T05:28:13.557071+0000', 'last_unstale': '2023-11-08T05:29:43.389354+0000', 'last_undegraded': '2023-11-08T05:29:43.389354+0000', 'last_fullsized': '2023-11-08T05:28:13.547025+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:30:32.987 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-11-08T05:30:32.987 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 18, 'reported_epoch': 455, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:29:43.389262+0000', 'last_change': '2023-11-08T05:28:13.559513+0000', 'last_active': '2023-11-08T05:29:43.389262+0000', 'last_peered': '2023-11-08T05:29:43.389262+0000', 'last_clean': '2023-11-08T04:52:46.531964+0000', 'last_became_active': '2023-11-08T05:28:13.559513+0000', 'last_became_peered': '2023-11-08T05:28:13.559513+0000', 'last_unstale': '2023-11-08T05:29:43.389262+0000', 'last_undegraded': '2023-11-08T05:29:43.389262+0000', 'last_fullsized': '2023-11-08T05:28:13.540852+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:30:32.987 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-11-08T05:30:32.987 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "0'0", 'reported_seq': 457, 'reported_epoch': 455, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:29:43.389822+0000', 'last_change': '2023-11-08T05:28:13.545468+0000', 'last_active': '2023-11-08T05:29:43.389822+0000', 'last_peered': '2023-11-08T05:29:43.389822+0000', 'last_clean': '2023-11-08T05:28:01.456723+0000', 'last_became_active': '2023-11-08T05:28:13.545468+0000', 'last_became_peered': '2023-11-08T05:28:13.545468+0000', 'last_unstale': '2023-11-08T05:29:43.389822+0000', 'last_undegraded': '2023-11-08T05:29:43.389822+0000', 'last_fullsized': '2023-11-08T05:28:13.537207+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 9, 'last_epoch_clean': 10, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:45.528796+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:45.528796+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:45.528796+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:30:32.988 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.0 is not active+clean 2023-11-08T05:30:32.988 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.0', 'version': "0'0", 'reported_seq': 462, 'reported_epoch': 456, 'state': 'active+clean+remapped', 'last_fresh': '2023-11-08T05:29:44.395350+0000', 'last_change': '2023-11-08T05:29:44.395350+0000', 'last_active': '2023-11-08T05:29:44.395350+0000', 'last_peered': '2023-11-08T05:29:44.395350+0000', 'last_clean': '2023-11-08T05:29:44.395350+0000', 'last_became_active': '2023-11-08T05:29:44.395012+0000', 'last_became_peered': '2023-11-08T05:29:44.395012+0000', 'last_unstale': '2023-11-08T05:29:44.395350+0000', 'last_undegraded': '2023-11-08T05:29:44.395350+0000', 'last_fullsized': '2023-11-08T05:29:44.395350+0000', 'mapping_epoch': 455, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 456, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2], 'acting': [2, 1], 'avail_no_missing': ['2', '1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2023-11-08T05:30:32.988 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-11-08T05:30:32.989 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 456, 'reported_epoch': 455, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:29:43.389390+0000', 'last_change': '2023-11-08T05:28:13.563357+0000', 'last_active': '2023-11-08T05:29:43.389390+0000', 'last_peered': '2023-11-08T05:29:43.389390+0000', 'last_clean': '2023-11-08T05:28:01.457691+0000', 'last_became_active': '2023-11-08T05:28:13.563357+0000', 'last_became_peered': '2023-11-08T05:28:13.563357+0000', 'last_unstale': '2023-11-08T05:29:43.389390+0000', 'last_undegraded': '2023-11-08T05:29:43.389390+0000', 'last_fullsized': '2023-11-08T05:28:13.541614+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:30:32.989 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-11-08T05:30:32.989 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 456, 'reported_epoch': 455, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:29:43.389311+0000', 'last_change': '2023-11-08T05:28:13.550672+0000', 'last_active': '2023-11-08T05:29:43.389311+0000', 'last_peered': '2023-11-08T05:29:43.389311+0000', 'last_clean': '2023-11-08T05:28:01.457904+0000', 'last_became_active': '2023-11-08T05:28:13.550672+0000', 'last_became_peered': '2023-11-08T05:28:13.550672+0000', 'last_unstale': '2023-11-08T05:29:43.389311+0000', 'last_undegraded': '2023-11-08T05:29:43.389311+0000', 'last_fullsized': '2023-11-08T05:28:13.539595+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:30:32.990 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-11-08T05:30:32.990 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 456, 'reported_epoch': 455, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:29:43.389474+0000', 'last_change': '2023-11-08T05:28:13.557968+0000', 'last_active': '2023-11-08T05:29:43.389474+0000', 'last_peered': '2023-11-08T05:29:43.389474+0000', 'last_clean': '2023-11-08T05:28:01.458433+0000', 'last_became_active': '2023-11-08T05:28:13.557968+0000', 'last_became_peered': '2023-11-08T05:28:13.557968+0000', 'last_unstale': '2023-11-08T05:29:43.389474+0000', 'last_undegraded': '2023-11-08T05:29:43.389474+0000', 'last_fullsized': '2023-11-08T05:28:13.542832+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:30:32.990 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-11-08T05:30:33.145 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~135s 2023-11-08T05:30:33.333 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T05:30:33.334 INFO:teuthology.orchestra.run.smithi087.stdout:{"fsid":"ab4cbc06-449d-497d-b513-220d9db2b590","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 6 pgs undersized","count":6},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2275,"monmap":{"epoch":1,"min_mon_release_name":"pacific","num_mons":1},"osdmap":{"epoch":456,"num_osds":3,"num_up_osds":2,"osd_up_since":1699421292,"num_in_osds":3,"osd_in_since":1699419158,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":6},{"state_name":"active+clean","count":2},{"state_name":"active+clean+remapped","count":1}],"num_pgs":9,"num_pools":2,"num_objects":0,"data_bytes":0,"bytes_used":2697519104,"bytes_avail":287212773376,"bytes_total":289910292480},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":41,"modified":"2023-11-08T05:29:32.044544+0000","services":{}},"progress_events":{}} 2023-11-08T05:30:33.347 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-11-08T05:30:33.347 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-11-08T05:30:33.631 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T05:30:33.633 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2023-11-08T05:30:33.644 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":1587,"stamp":"2023-11-08T05:30:32.057701+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2634296,"kb_used_data":408940,"kb_used_omap":780,"kb_used_meta":2220403,"kb_avail":280481224,"statfs":{"total":289910292480,"available":287212773376,"internally_reserved":0,"allocated":418754560,"data_stored":410461501,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":799447,"internal_metadata":2273692969},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-536782902,"num_objects":-164,"num_object_clones":0,"num_object_copies":-328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-115,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-164,"num_whiteouts":0,"num_read":-122056,"num_read_kb":-39011942,"num_write":-413642,"num_write_kb":-151705339,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-8,"num_bytes_recovered":-16922624,"num_keys_recovered":-2,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.424817"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":456,"reported_epoch":455,"state":"active+undersized","last_fresh":"2023-11-08T05:29:43.389354+0000","last_change":"2023-11-08T05:28:13.557071+0000","last_active":"2023-11-08T05:29:43.389354+0000","last_peered":"2023-11-08T05:29:43.389354+0000","last_clean":"2023-11-08T05:28:01.458439+0000","last_became_active":"2023-11-08T05:28:13.557071+0000","last_became_peered":"2023-11-08T05:28:13.557071+0000","last_unstale":"2023-11-08T05:29:43.389354+0000","last_undegraded":"2023-11-08T05:29:43.389354+0000","last_fullsized":"2023-11-08T05:28:13.547025+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":18,"reported_epoch":455,"state":"active+undersized","last_fresh":"2023-11-08T05:29:43.389262+0000","last_change":"2023-11-08T05:28:13.559513+0000","last_active":"2023-11-08T05:29:43.389262+0000","last_peered":"2023-11-08T05:29:43.389262+0000","last_clean":"2023-11-08T04:52:46.531964+0000","last_became_active":"2023-11-08T05:28:13.559513+0000","last_became_peered":"2023-11-08T05:28:13.559513+0000","last_unstale":"2023-11-08T05:29:43.389262+0000","last_undegraded":"2023-11-08T05:29:43.389262+0000","last_fullsized":"2023-11-08T05:28:13.540852+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":453,"reported_epoch":455,"state":"active+clean","last_fresh":"2023-11-08T05:29:43.390991+0000","last_change":"2023-11-08T04:52:47.546100+0000","last_active":"2023-11-08T05:29:43.390991+0000","last_peered":"2023-11-08T05:29:43.390991+0000","last_clean":"2023-11-08T05:29:43.390991+0000","last_became_active":"2023-11-08T04:52:47.545840+0000","last_became_peered":"2023-11-08T04:52:47.545840+0000","last_unstale":"2023-11-08T05:29:43.390991+0000","last_undegraded":"2023-11-08T05:29:43.390991+0000","last_fullsized":"2023-11-08T05:29:43.390991+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":451,"reported_epoch":455,"state":"active+clean","last_fresh":"2023-11-08T05:29:43.389261+0000","last_change":"2023-11-08T04:52:47.548597+0000","last_active":"2023-11-08T05:29:43.389261+0000","last_peered":"2023-11-08T05:29:43.389261+0000","last_clean":"2023-11-08T05:29:43.389261+0000","last_became_active":"2023-11-08T04:52:47.548294+0000","last_became_peered":"2023-11-08T04:52:47.548294+0000","last_unstale":"2023-11-08T05:29:43.389261+0000","last_undegraded":"2023-11-08T05:29:43.389261+0000","last_fullsized":"2023-11-08T05:29:43.389261+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":457,"reported_epoch":455,"state":"active+undersized","last_fresh":"2023-11-08T05:29:43.389822+0000","last_change":"2023-11-08T05:28:13.545468+0000","last_active":"2023-11-08T05:29:43.389822+0000","last_peered":"2023-11-08T05:29:43.389822+0000","last_clean":"2023-11-08T05:28:01.456723+0000","last_became_active":"2023-11-08T05:28:13.545468+0000","last_became_peered":"2023-11-08T05:28:13.545468+0000","last_unstale":"2023-11-08T05:29:43.389822+0000","last_undegraded":"2023-11-08T05:29:43.389822+0000","last_fullsized":"2023-11-08T05:28:13.537207+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_clean_scrub_stamp":"2023-11-08T04:52:45.528796+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":462,"reported_epoch":456,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:29:44.395350+0000","last_change":"2023-11-08T05:29:44.395350+0000","last_active":"2023-11-08T05:29:44.395350+0000","last_peered":"2023-11-08T05:29:44.395350+0000","last_clean":"2023-11-08T05:29:44.395350+0000","last_became_active":"2023-11-08T05:29:44.395012+0000","last_became_peered":"2023-11-08T05:29:44.395012+0000","last_unstale":"2023-11-08T05:29:44.395350+0000","last_undegraded":"2023-11-08T05:29:44.395350+0000","last_fullsized":"2023-11-08T05:29:44.395350+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":456,"reported_epoch":455,"state":"active+undersized","last_fresh":"2023-11-08T05:29:43.389390+0000","last_change":"2023-11-08T05:28:13.563357+0000","last_active":"2023-11-08T05:29:43.389390+0000","last_peered":"2023-11-08T05:29:43.389390+0000","last_clean":"2023-11-08T05:28:01.457691+0000","last_became_active":"2023-11-08T05:28:13.563357+0000","last_became_peered":"2023-11-08T05:28:13.563357+0000","last_unstale":"2023-11-08T05:29:43.389390+0000","last_undegraded":"2023-11-08T05:29:43.389390+0000","last_fullsized":"2023-11-08T05:28:13.541614+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":456,"reported_epoch":455,"state":"active+undersized","last_fresh":"2023-11-08T05:29:43.389311+0000","last_change":"2023-11-08T05:28:13.550672+0000","last_active":"2023-11-08T05:29:43.389311+0000","last_peered":"2023-11-08T05:29:43.389311+0000","last_clean":"2023-11-08T05:28:01.457904+0000","last_became_active":"2023-11-08T05:28:13.550672+0000","last_became_peered":"2023-11-08T05:28:13.550672+0000","last_unstale":"2023-11-08T05:29:43.389311+0000","last_undegraded":"2023-11-08T05:29:43.389311+0000","last_fullsized":"2023-11-08T05:28:13.539595+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":456,"reported_epoch":455,"state":"active+undersized","last_fresh":"2023-11-08T05:29:43.389474+0000","last_change":"2023-11-08T05:28:13.557968+0000","last_active":"2023-11-08T05:29:43.389474+0000","last_peered":"2023-11-08T05:29:43.389474+0000","last_clean":"2023-11-08T05:28:01.458433+0000","last_became_active":"2023-11-08T05:28:13.557968+0000","last_became_peered":"2023-11-08T05:28:13.557968+0000","last_unstale":"2023-11-08T05:29:43.389474+0000","last_undegraded":"2023-11-08T05:29:43.389474+0000","last_fullsized":"2023-11-08T05:28:13.542832+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":1,"acting":1,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":10,"acting":11,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738824,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":645004,"kb_used_data":5316,"kb_used_omap":275,"kb_used_meta":639404,"kb_avail":93726836,"statfs":{"total":96636764160,"available":95976280064,"internally_reserved":0,"allocated":5443584,"data_stored":2678316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":281650,"internal_metadata":654750670},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:59 2023","interfaces":[{"interface":"back","average":{"1min":0.30499999999999999,"5min":0.439,"15min":0.36099999999999999},"min":{"1min":0.17399999999999999,"5min":0.14499999999999999,"15min":0.14499999999999999},"max":{"1min":0.60099999999999998,"5min":4.9379999999999997,"15min":4.9379999999999997},"last":0.379},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.49399999999999999,"15min":0.42399999999999999},"min":{"1min":0.221,"5min":0.19600000000000001,"15min":0.17000000000000001},"max":{"1min":0.63200000000000001,"5min":4.9859999999999998,"15min":4.9859999999999998},"last":0.40899999999999997}]},{"osd":1,"last update":"Wed Nov 8 05:30:03 2023","interfaces":[{"interface":"back","average":{"1min":0.28499999999999998,"5min":0.65100000000000002,"15min":0.47099999999999997},"min":{"1min":0.20100000000000001,"5min":0.153,"15min":0.153},"max":{"1min":0.39800000000000002,"5min":23.242999999999999,"15min":23.242999999999999},"last":0.20899999999999999},{"interface":"front","average":{"1min":0.29499999999999998,"5min":0.63100000000000001,"15min":0.45900000000000002},"min":{"1min":0.224,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":0.436,"5min":23.251000000000001,"15min":23.251000000000001},"last":0.24099999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738798,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1339228,"kb_used_data":398304,"kb_used_omap":291,"kb_used_meta":936476,"kb_avail":93032612,"statfs":{"total":96636764160,"available":95265394688,"internally_reserved":0,"allocated":407863296,"data_stored":405103900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":298462,"internal_metadata":958951970},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.436,"15min":0.36299999999999999},"min":{"1min":0.19700000000000001,"5min":0.14699999999999999,"15min":0.13700000000000001},"max":{"1min":3.222,"5min":6.5499999999999998,"15min":6.9989999999999997},"last":0.245},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.46300000000000002,"15min":0.42299999999999999},"min":{"1min":0.17799999999999999,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":5.0030000000000001,"5min":6.0629999999999997,"15min":9.6869999999999994},"last":0.28599999999999998}]},{"osd":2,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.51600000000000001,"15min":0.47099999999999997},"min":{"1min":0.21099999999999999,"5min":0.153,"15min":0.153},"max":{"1min":5.7439999999999998,"5min":5.8280000000000003,"15min":9.7639999999999993},"last":0.252},{"interface":"front","average":{"1min":0.497,"5min":0.56200000000000006,"15min":0.42299999999999999},"min":{"1min":0.20399999999999999,"5min":0.187,"15min":0.16800000000000001},"max":{"1min":3.9319999999999999,"5min":8.8040000000000003,"15min":8.8040000000000003},"last":0.29899999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738826,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":650064,"kb_used_data":5320,"kb_used_omap":214,"kb_used_meta":644521,"kb_avail":93721776,"statfs":{"total":96636764160,"available":95971098624,"internally_reserved":0,"allocated":5447680,"data_stored":2679285,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":219335,"internal_metadata":659990329},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:56 2023","interfaces":[{"interface":"back","average":{"1min":0.30399999999999999,"5min":0.34699999999999998,"15min":0.36499999999999999},"min":{"1min":0.23999999999999999,"5min":0.14199999999999999,"15min":0.14199999999999999},"max":{"1min":0.42399999999999999,"5min":1.595,"15min":7.4249999999999998},"last":0.17100000000000001},{"interface":"front","average":{"1min":0.35499999999999998,"5min":0.40899999999999997,"15min":0.376},"min":{"1min":0.23999999999999999,"5min":0.19600000000000001,"15min":0.183},"max":{"1min":0.46999999999999997,"5min":2.6859999999999999,"15min":2.6859999999999999},"last":0.23000000000000001}]},{"osd":2,"last update":"Wed Nov 8 05:29:57 2023","interfaces":[{"interface":"back","average":{"1min":0.28199999999999997,"5min":0.36099999999999999,"15min":0.373},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":0.34399999999999997,"5min":2.2650000000000001,"15min":5.8209999999999997},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.28599999999999998,"5min":0.31,"15min":0.35799999999999998},"min":{"1min":0.17199999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.35699999999999998,"5min":1.137,"15min":4.2350000000000003},"last":0.42499999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-08T05:30:36.646 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-11-08T05:30:36.941 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T05:30:36.943 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2023-11-08T05:30:36.954 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":1589,"stamp":"2023-11-08T05:30:36.058469+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2634296,"kb_used_data":408940,"kb_used_omap":780,"kb_used_meta":2220403,"kb_avail":280481224,"statfs":{"total":289910292480,"available":287212773376,"internally_reserved":0,"allocated":418754560,"data_stored":410461501,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":799447,"internal_metadata":2273692969},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-536782902,"num_objects":-164,"num_object_clones":0,"num_object_copies":-328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-115,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-164,"num_whiteouts":0,"num_read":-122056,"num_read_kb":-39011942,"num_write":-413642,"num_write_kb":-151705339,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-8,"num_bytes_recovered":-16922624,"num_keys_recovered":-2,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.424817"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":456,"reported_epoch":455,"state":"active+undersized","last_fresh":"2023-11-08T05:29:43.389354+0000","last_change":"2023-11-08T05:28:13.557071+0000","last_active":"2023-11-08T05:29:43.389354+0000","last_peered":"2023-11-08T05:29:43.389354+0000","last_clean":"2023-11-08T05:28:01.458439+0000","last_became_active":"2023-11-08T05:28:13.557071+0000","last_became_peered":"2023-11-08T05:28:13.557071+0000","last_unstale":"2023-11-08T05:29:43.389354+0000","last_undegraded":"2023-11-08T05:29:43.389354+0000","last_fullsized":"2023-11-08T05:28:13.547025+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":18,"reported_epoch":455,"state":"active+undersized","last_fresh":"2023-11-08T05:29:43.389262+0000","last_change":"2023-11-08T05:28:13.559513+0000","last_active":"2023-11-08T05:29:43.389262+0000","last_peered":"2023-11-08T05:29:43.389262+0000","last_clean":"2023-11-08T04:52:46.531964+0000","last_became_active":"2023-11-08T05:28:13.559513+0000","last_became_peered":"2023-11-08T05:28:13.559513+0000","last_unstale":"2023-11-08T05:29:43.389262+0000","last_undegraded":"2023-11-08T05:29:43.389262+0000","last_fullsized":"2023-11-08T05:28:13.540852+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":453,"reported_epoch":455,"state":"active+clean","last_fresh":"2023-11-08T05:29:43.390991+0000","last_change":"2023-11-08T04:52:47.546100+0000","last_active":"2023-11-08T05:29:43.390991+0000","last_peered":"2023-11-08T05:29:43.390991+0000","last_clean":"2023-11-08T05:29:43.390991+0000","last_became_active":"2023-11-08T04:52:47.545840+0000","last_became_peered":"2023-11-08T04:52:47.545840+0000","last_unstale":"2023-11-08T05:29:43.390991+0000","last_undegraded":"2023-11-08T05:29:43.390991+0000","last_fullsized":"2023-11-08T05:29:43.390991+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":451,"reported_epoch":455,"state":"active+clean","last_fresh":"2023-11-08T05:29:43.389261+0000","last_change":"2023-11-08T04:52:47.548597+0000","last_active":"2023-11-08T05:29:43.389261+0000","last_peered":"2023-11-08T05:29:43.389261+0000","last_clean":"2023-11-08T05:29:43.389261+0000","last_became_active":"2023-11-08T04:52:47.548294+0000","last_became_peered":"2023-11-08T04:52:47.548294+0000","last_unstale":"2023-11-08T05:29:43.389261+0000","last_undegraded":"2023-11-08T05:29:43.389261+0000","last_fullsized":"2023-11-08T05:29:43.389261+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":457,"reported_epoch":455,"state":"active+undersized","last_fresh":"2023-11-08T05:29:43.389822+0000","last_change":"2023-11-08T05:28:13.545468+0000","last_active":"2023-11-08T05:29:43.389822+0000","last_peered":"2023-11-08T05:29:43.389822+0000","last_clean":"2023-11-08T05:28:01.456723+0000","last_became_active":"2023-11-08T05:28:13.545468+0000","last_became_peered":"2023-11-08T05:28:13.545468+0000","last_unstale":"2023-11-08T05:29:43.389822+0000","last_undegraded":"2023-11-08T05:29:43.389822+0000","last_fullsized":"2023-11-08T05:28:13.537207+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_clean_scrub_stamp":"2023-11-08T04:52:45.528796+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":462,"reported_epoch":456,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:29:44.395350+0000","last_change":"2023-11-08T05:29:44.395350+0000","last_active":"2023-11-08T05:29:44.395350+0000","last_peered":"2023-11-08T05:29:44.395350+0000","last_clean":"2023-11-08T05:29:44.395350+0000","last_became_active":"2023-11-08T05:29:44.395012+0000","last_became_peered":"2023-11-08T05:29:44.395012+0000","last_unstale":"2023-11-08T05:29:44.395350+0000","last_undegraded":"2023-11-08T05:29:44.395350+0000","last_fullsized":"2023-11-08T05:29:44.395350+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":456,"reported_epoch":455,"state":"active+undersized","last_fresh":"2023-11-08T05:29:43.389390+0000","last_change":"2023-11-08T05:28:13.563357+0000","last_active":"2023-11-08T05:29:43.389390+0000","last_peered":"2023-11-08T05:29:43.389390+0000","last_clean":"2023-11-08T05:28:01.457691+0000","last_became_active":"2023-11-08T05:28:13.563357+0000","last_became_peered":"2023-11-08T05:28:13.563357+0000","last_unstale":"2023-11-08T05:29:43.389390+0000","last_undegraded":"2023-11-08T05:29:43.389390+0000","last_fullsized":"2023-11-08T05:28:13.541614+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":456,"reported_epoch":455,"state":"active+undersized","last_fresh":"2023-11-08T05:29:43.389311+0000","last_change":"2023-11-08T05:28:13.550672+0000","last_active":"2023-11-08T05:29:43.389311+0000","last_peered":"2023-11-08T05:29:43.389311+0000","last_clean":"2023-11-08T05:28:01.457904+0000","last_became_active":"2023-11-08T05:28:13.550672+0000","last_became_peered":"2023-11-08T05:28:13.550672+0000","last_unstale":"2023-11-08T05:29:43.389311+0000","last_undegraded":"2023-11-08T05:29:43.389311+0000","last_fullsized":"2023-11-08T05:28:13.539595+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":456,"reported_epoch":455,"state":"active+undersized","last_fresh":"2023-11-08T05:29:43.389474+0000","last_change":"2023-11-08T05:28:13.557968+0000","last_active":"2023-11-08T05:29:43.389474+0000","last_peered":"2023-11-08T05:29:43.389474+0000","last_clean":"2023-11-08T05:28:01.458433+0000","last_became_active":"2023-11-08T05:28:13.557968+0000","last_became_peered":"2023-11-08T05:28:13.557968+0000","last_unstale":"2023-11-08T05:29:43.389474+0000","last_undegraded":"2023-11-08T05:29:43.389474+0000","last_fullsized":"2023-11-08T05:28:13.542832+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":1,"acting":1,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":10,"acting":11,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738825,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":645004,"kb_used_data":5316,"kb_used_omap":275,"kb_used_meta":639404,"kb_avail":93726836,"statfs":{"total":96636764160,"available":95976280064,"internally_reserved":0,"allocated":5443584,"data_stored":2678316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":281650,"internal_metadata":654750670},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:59 2023","interfaces":[{"interface":"back","average":{"1min":0.30499999999999999,"5min":0.439,"15min":0.36099999999999999},"min":{"1min":0.17399999999999999,"5min":0.14499999999999999,"15min":0.14499999999999999},"max":{"1min":0.60099999999999998,"5min":4.9379999999999997,"15min":4.9379999999999997},"last":0.379},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.49399999999999999,"15min":0.42399999999999999},"min":{"1min":0.221,"5min":0.19600000000000001,"15min":0.17000000000000001},"max":{"1min":0.63200000000000001,"5min":4.9859999999999998,"15min":4.9859999999999998},"last":0.40899999999999997}]},{"osd":1,"last update":"Wed Nov 8 05:30:03 2023","interfaces":[{"interface":"back","average":{"1min":0.28499999999999998,"5min":0.65100000000000002,"15min":0.47099999999999997},"min":{"1min":0.20100000000000001,"5min":0.153,"15min":0.153},"max":{"1min":0.39800000000000002,"5min":23.242999999999999,"15min":23.242999999999999},"last":0.33400000000000002},{"interface":"front","average":{"1min":0.29499999999999998,"5min":0.63100000000000001,"15min":0.45900000000000002},"min":{"1min":0.224,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":0.436,"5min":23.251000000000001,"15min":23.251000000000001},"last":0.34999999999999998}]}]},{"osd":0,"up_from":8,"seq":34359738798,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1339228,"kb_used_data":398304,"kb_used_omap":291,"kb_used_meta":936476,"kb_avail":93032612,"statfs":{"total":96636764160,"available":95265394688,"internally_reserved":0,"allocated":407863296,"data_stored":405103900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":298462,"internal_metadata":958951970},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.436,"15min":0.36299999999999999},"min":{"1min":0.19700000000000001,"5min":0.14699999999999999,"15min":0.13700000000000001},"max":{"1min":3.222,"5min":6.5499999999999998,"15min":6.9989999999999997},"last":0.245},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.46300000000000002,"15min":0.42299999999999999},"min":{"1min":0.17799999999999999,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":5.0030000000000001,"5min":6.0629999999999997,"15min":9.6869999999999994},"last":0.28599999999999998}]},{"osd":2,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.51600000000000001,"15min":0.47099999999999997},"min":{"1min":0.21099999999999999,"5min":0.153,"15min":0.153},"max":{"1min":5.7439999999999998,"5min":5.8280000000000003,"15min":9.7639999999999993},"last":0.252},{"interface":"front","average":{"1min":0.497,"5min":0.56200000000000006,"15min":0.42299999999999999},"min":{"1min":0.20399999999999999,"5min":0.187,"15min":0.16800000000000001},"max":{"1min":3.9319999999999999,"5min":8.8040000000000003,"15min":8.8040000000000003},"last":0.29899999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738827,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":650064,"kb_used_data":5320,"kb_used_omap":214,"kb_used_meta":644521,"kb_avail":93721776,"statfs":{"total":96636764160,"available":95971098624,"internally_reserved":0,"allocated":5447680,"data_stored":2679285,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":219335,"internal_metadata":659990329},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:56 2023","interfaces":[{"interface":"back","average":{"1min":0.30399999999999999,"5min":0.34699999999999998,"15min":0.36499999999999999},"min":{"1min":0.23999999999999999,"5min":0.14199999999999999,"15min":0.14199999999999999},"max":{"1min":0.42399999999999999,"5min":1.595,"15min":7.4249999999999998},"last":0.17100000000000001},{"interface":"front","average":{"1min":0.35499999999999998,"5min":0.40899999999999997,"15min":0.376},"min":{"1min":0.23999999999999999,"5min":0.19600000000000001,"15min":0.183},"max":{"1min":0.46999999999999997,"5min":2.6859999999999999,"15min":2.6859999999999999},"last":0.23000000000000001}]},{"osd":2,"last update":"Wed Nov 8 05:29:57 2023","interfaces":[{"interface":"back","average":{"1min":0.28199999999999997,"5min":0.36099999999999999,"15min":0.373},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":0.34399999999999997,"5min":2.2650000000000001,"15min":5.8209999999999997},"last":0.27000000000000002},{"interface":"front","average":{"1min":0.28599999999999998,"5min":0.31,"15min":0.35799999999999998},"min":{"1min":0.17199999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.35699999999999998,"5min":1.137,"15min":4.2350000000000003},"last":0.223}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-08T05:30:36.955 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-11-08T05:30:37.238 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T05:30:37.240 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2023-11-08T05:30:37.251 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":1589,"stamp":"2023-11-08T05:30:36.058469+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2634296,"kb_used_data":408940,"kb_used_omap":780,"kb_used_meta":2220403,"kb_avail":280481224,"statfs":{"total":289910292480,"available":287212773376,"internally_reserved":0,"allocated":418754560,"data_stored":410461501,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":799447,"internal_metadata":2273692969},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-536782902,"num_objects":-164,"num_object_clones":0,"num_object_copies":-328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-115,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-164,"num_whiteouts":0,"num_read":-122056,"num_read_kb":-39011942,"num_write":-413642,"num_write_kb":-151705339,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-8,"num_bytes_recovered":-16922624,"num_keys_recovered":-2,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.424817"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":456,"reported_epoch":455,"state":"active+undersized","last_fresh":"2023-11-08T05:29:43.389354+0000","last_change":"2023-11-08T05:28:13.557071+0000","last_active":"2023-11-08T05:29:43.389354+0000","last_peered":"2023-11-08T05:29:43.389354+0000","last_clean":"2023-11-08T05:28:01.458439+0000","last_became_active":"2023-11-08T05:28:13.557071+0000","last_became_peered":"2023-11-08T05:28:13.557071+0000","last_unstale":"2023-11-08T05:29:43.389354+0000","last_undegraded":"2023-11-08T05:29:43.389354+0000","last_fullsized":"2023-11-08T05:28:13.547025+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":18,"reported_epoch":455,"state":"active+undersized","last_fresh":"2023-11-08T05:29:43.389262+0000","last_change":"2023-11-08T05:28:13.559513+0000","last_active":"2023-11-08T05:29:43.389262+0000","last_peered":"2023-11-08T05:29:43.389262+0000","last_clean":"2023-11-08T04:52:46.531964+0000","last_became_active":"2023-11-08T05:28:13.559513+0000","last_became_peered":"2023-11-08T05:28:13.559513+0000","last_unstale":"2023-11-08T05:29:43.389262+0000","last_undegraded":"2023-11-08T05:29:43.389262+0000","last_fullsized":"2023-11-08T05:28:13.540852+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":453,"reported_epoch":455,"state":"active+clean","last_fresh":"2023-11-08T05:29:43.390991+0000","last_change":"2023-11-08T04:52:47.546100+0000","last_active":"2023-11-08T05:29:43.390991+0000","last_peered":"2023-11-08T05:29:43.390991+0000","last_clean":"2023-11-08T05:29:43.390991+0000","last_became_active":"2023-11-08T04:52:47.545840+0000","last_became_peered":"2023-11-08T04:52:47.545840+0000","last_unstale":"2023-11-08T05:29:43.390991+0000","last_undegraded":"2023-11-08T05:29:43.390991+0000","last_fullsized":"2023-11-08T05:29:43.390991+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":451,"reported_epoch":455,"state":"active+clean","last_fresh":"2023-11-08T05:29:43.389261+0000","last_change":"2023-11-08T04:52:47.548597+0000","last_active":"2023-11-08T05:29:43.389261+0000","last_peered":"2023-11-08T05:29:43.389261+0000","last_clean":"2023-11-08T05:29:43.389261+0000","last_became_active":"2023-11-08T04:52:47.548294+0000","last_became_peered":"2023-11-08T04:52:47.548294+0000","last_unstale":"2023-11-08T05:29:43.389261+0000","last_undegraded":"2023-11-08T05:29:43.389261+0000","last_fullsized":"2023-11-08T05:29:43.389261+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":457,"reported_epoch":455,"state":"active+undersized","last_fresh":"2023-11-08T05:29:43.389822+0000","last_change":"2023-11-08T05:28:13.545468+0000","last_active":"2023-11-08T05:29:43.389822+0000","last_peered":"2023-11-08T05:29:43.389822+0000","last_clean":"2023-11-08T05:28:01.456723+0000","last_became_active":"2023-11-08T05:28:13.545468+0000","last_became_peered":"2023-11-08T05:28:13.545468+0000","last_unstale":"2023-11-08T05:29:43.389822+0000","last_undegraded":"2023-11-08T05:29:43.389822+0000","last_fullsized":"2023-11-08T05:28:13.537207+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_clean_scrub_stamp":"2023-11-08T04:52:45.528796+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":462,"reported_epoch":456,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:29:44.395350+0000","last_change":"2023-11-08T05:29:44.395350+0000","last_active":"2023-11-08T05:29:44.395350+0000","last_peered":"2023-11-08T05:29:44.395350+0000","last_clean":"2023-11-08T05:29:44.395350+0000","last_became_active":"2023-11-08T05:29:44.395012+0000","last_became_peered":"2023-11-08T05:29:44.395012+0000","last_unstale":"2023-11-08T05:29:44.395350+0000","last_undegraded":"2023-11-08T05:29:44.395350+0000","last_fullsized":"2023-11-08T05:29:44.395350+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":456,"reported_epoch":455,"state":"active+undersized","last_fresh":"2023-11-08T05:29:43.389390+0000","last_change":"2023-11-08T05:28:13.563357+0000","last_active":"2023-11-08T05:29:43.389390+0000","last_peered":"2023-11-08T05:29:43.389390+0000","last_clean":"2023-11-08T05:28:01.457691+0000","last_became_active":"2023-11-08T05:28:13.563357+0000","last_became_peered":"2023-11-08T05:28:13.563357+0000","last_unstale":"2023-11-08T05:29:43.389390+0000","last_undegraded":"2023-11-08T05:29:43.389390+0000","last_fullsized":"2023-11-08T05:28:13.541614+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":456,"reported_epoch":455,"state":"active+undersized","last_fresh":"2023-11-08T05:29:43.389311+0000","last_change":"2023-11-08T05:28:13.550672+0000","last_active":"2023-11-08T05:29:43.389311+0000","last_peered":"2023-11-08T05:29:43.389311+0000","last_clean":"2023-11-08T05:28:01.457904+0000","last_became_active":"2023-11-08T05:28:13.550672+0000","last_became_peered":"2023-11-08T05:28:13.550672+0000","last_unstale":"2023-11-08T05:29:43.389311+0000","last_undegraded":"2023-11-08T05:29:43.389311+0000","last_fullsized":"2023-11-08T05:28:13.539595+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":456,"reported_epoch":455,"state":"active+undersized","last_fresh":"2023-11-08T05:29:43.389474+0000","last_change":"2023-11-08T05:28:13.557968+0000","last_active":"2023-11-08T05:29:43.389474+0000","last_peered":"2023-11-08T05:29:43.389474+0000","last_clean":"2023-11-08T05:28:01.458433+0000","last_became_active":"2023-11-08T05:28:13.557968+0000","last_became_peered":"2023-11-08T05:28:13.557968+0000","last_unstale":"2023-11-08T05:29:43.389474+0000","last_undegraded":"2023-11-08T05:29:43.389474+0000","last_fullsized":"2023-11-08T05:28:13.542832+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":1,"acting":1,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":10,"acting":11,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738825,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":645004,"kb_used_data":5316,"kb_used_omap":275,"kb_used_meta":639404,"kb_avail":93726836,"statfs":{"total":96636764160,"available":95976280064,"internally_reserved":0,"allocated":5443584,"data_stored":2678316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":281650,"internal_metadata":654750670},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:59 2023","interfaces":[{"interface":"back","average":{"1min":0.30499999999999999,"5min":0.439,"15min":0.36099999999999999},"min":{"1min":0.17399999999999999,"5min":0.14499999999999999,"15min":0.14499999999999999},"max":{"1min":0.60099999999999998,"5min":4.9379999999999997,"15min":4.9379999999999997},"last":0.379},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.49399999999999999,"15min":0.42399999999999999},"min":{"1min":0.221,"5min":0.19600000000000001,"15min":0.17000000000000001},"max":{"1min":0.63200000000000001,"5min":4.9859999999999998,"15min":4.9859999999999998},"last":0.40899999999999997}]},{"osd":1,"last update":"Wed Nov 8 05:30:03 2023","interfaces":[{"interface":"back","average":{"1min":0.28499999999999998,"5min":0.65100000000000002,"15min":0.47099999999999997},"min":{"1min":0.20100000000000001,"5min":0.153,"15min":0.153},"max":{"1min":0.39800000000000002,"5min":23.242999999999999,"15min":23.242999999999999},"last":0.33400000000000002},{"interface":"front","average":{"1min":0.29499999999999998,"5min":0.63100000000000001,"15min":0.45900000000000002},"min":{"1min":0.224,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":0.436,"5min":23.251000000000001,"15min":23.251000000000001},"last":0.34999999999999998}]}]},{"osd":0,"up_from":8,"seq":34359738798,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1339228,"kb_used_data":398304,"kb_used_omap":291,"kb_used_meta":936476,"kb_avail":93032612,"statfs":{"total":96636764160,"available":95265394688,"internally_reserved":0,"allocated":407863296,"data_stored":405103900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":298462,"internal_metadata":958951970},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.436,"15min":0.36299999999999999},"min":{"1min":0.19700000000000001,"5min":0.14699999999999999,"15min":0.13700000000000001},"max":{"1min":3.222,"5min":6.5499999999999998,"15min":6.9989999999999997},"last":0.245},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.46300000000000002,"15min":0.42299999999999999},"min":{"1min":0.17799999999999999,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":5.0030000000000001,"5min":6.0629999999999997,"15min":9.6869999999999994},"last":0.28599999999999998}]},{"osd":2,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.51600000000000001,"15min":0.47099999999999997},"min":{"1min":0.21099999999999999,"5min":0.153,"15min":0.153},"max":{"1min":5.7439999999999998,"5min":5.8280000000000003,"15min":9.7639999999999993},"last":0.252},{"interface":"front","average":{"1min":0.497,"5min":0.56200000000000006,"15min":0.42299999999999999},"min":{"1min":0.20399999999999999,"5min":0.187,"15min":0.16800000000000001},"max":{"1min":3.9319999999999999,"5min":8.8040000000000003,"15min":8.8040000000000003},"last":0.29899999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738827,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":650064,"kb_used_data":5320,"kb_used_omap":214,"kb_used_meta":644521,"kb_avail":93721776,"statfs":{"total":96636764160,"available":95971098624,"internally_reserved":0,"allocated":5447680,"data_stored":2679285,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":219335,"internal_metadata":659990329},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:56 2023","interfaces":[{"interface":"back","average":{"1min":0.30399999999999999,"5min":0.34699999999999998,"15min":0.36499999999999999},"min":{"1min":0.23999999999999999,"5min":0.14199999999999999,"15min":0.14199999999999999},"max":{"1min":0.42399999999999999,"5min":1.595,"15min":7.4249999999999998},"last":0.17100000000000001},{"interface":"front","average":{"1min":0.35499999999999998,"5min":0.40899999999999997,"15min":0.376},"min":{"1min":0.23999999999999999,"5min":0.19600000000000001,"15min":0.183},"max":{"1min":0.46999999999999997,"5min":2.6859999999999999,"15min":2.6859999999999999},"last":0.23000000000000001}]},{"osd":2,"last update":"Wed Nov 8 05:29:57 2023","interfaces":[{"interface":"back","average":{"1min":0.28199999999999997,"5min":0.36099999999999999,"15min":0.373},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":0.34399999999999997,"5min":2.2650000000000001,"15min":5.8209999999999997},"last":0.27000000000000002},{"interface":"front","average":{"1min":0.28599999999999998,"5min":0.31,"15min":0.35799999999999998},"min":{"1min":0.17199999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.35699999999999998,"5min":1.137,"15min":4.2350000000000003},"last":0.223}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-08T05:30:37.252 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-11-08T05:30:37.252 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 456, 'reported_epoch': 455, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:29:43.389354+0000', 'last_change': '2023-11-08T05:28:13.557071+0000', 'last_active': '2023-11-08T05:29:43.389354+0000', 'last_peered': '2023-11-08T05:29:43.389354+0000', 'last_clean': '2023-11-08T05:28:01.458439+0000', 'last_became_active': '2023-11-08T05:28:13.557071+0000', 'last_became_peered': '2023-11-08T05:28:13.557071+0000', 'last_unstale': '2023-11-08T05:29:43.389354+0000', 'last_undegraded': '2023-11-08T05:29:43.389354+0000', 'last_fullsized': '2023-11-08T05:28:13.547025+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:30:37.253 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-11-08T05:30:37.253 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 18, 'reported_epoch': 455, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:29:43.389262+0000', 'last_change': '2023-11-08T05:28:13.559513+0000', 'last_active': '2023-11-08T05:29:43.389262+0000', 'last_peered': '2023-11-08T05:29:43.389262+0000', 'last_clean': '2023-11-08T04:52:46.531964+0000', 'last_became_active': '2023-11-08T05:28:13.559513+0000', 'last_became_peered': '2023-11-08T05:28:13.559513+0000', 'last_unstale': '2023-11-08T05:29:43.389262+0000', 'last_undegraded': '2023-11-08T05:29:43.389262+0000', 'last_fullsized': '2023-11-08T05:28:13.540852+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:30:37.253 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-11-08T05:30:37.254 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "0'0", 'reported_seq': 457, 'reported_epoch': 455, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:29:43.389822+0000', 'last_change': '2023-11-08T05:28:13.545468+0000', 'last_active': '2023-11-08T05:29:43.389822+0000', 'last_peered': '2023-11-08T05:29:43.389822+0000', 'last_clean': '2023-11-08T05:28:01.456723+0000', 'last_became_active': '2023-11-08T05:28:13.545468+0000', 'last_became_peered': '2023-11-08T05:28:13.545468+0000', 'last_unstale': '2023-11-08T05:29:43.389822+0000', 'last_undegraded': '2023-11-08T05:29:43.389822+0000', 'last_fullsized': '2023-11-08T05:28:13.537207+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 9, 'last_epoch_clean': 10, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:45.528796+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:45.528796+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:45.528796+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:30:37.254 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.0 is not active+clean 2023-11-08T05:30:37.254 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.0', 'version': "0'0", 'reported_seq': 462, 'reported_epoch': 456, 'state': 'active+clean+remapped', 'last_fresh': '2023-11-08T05:29:44.395350+0000', 'last_change': '2023-11-08T05:29:44.395350+0000', 'last_active': '2023-11-08T05:29:44.395350+0000', 'last_peered': '2023-11-08T05:29:44.395350+0000', 'last_clean': '2023-11-08T05:29:44.395350+0000', 'last_became_active': '2023-11-08T05:29:44.395012+0000', 'last_became_peered': '2023-11-08T05:29:44.395012+0000', 'last_unstale': '2023-11-08T05:29:44.395350+0000', 'last_undegraded': '2023-11-08T05:29:44.395350+0000', 'last_fullsized': '2023-11-08T05:29:44.395350+0000', 'mapping_epoch': 455, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 456, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2], 'acting': [2, 1], 'avail_no_missing': ['2', '1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2023-11-08T05:30:37.255 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-11-08T05:30:37.255 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 456, 'reported_epoch': 455, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:29:43.389390+0000', 'last_change': '2023-11-08T05:28:13.563357+0000', 'last_active': '2023-11-08T05:29:43.389390+0000', 'last_peered': '2023-11-08T05:29:43.389390+0000', 'last_clean': '2023-11-08T05:28:01.457691+0000', 'last_became_active': '2023-11-08T05:28:13.563357+0000', 'last_became_peered': '2023-11-08T05:28:13.563357+0000', 'last_unstale': '2023-11-08T05:29:43.389390+0000', 'last_undegraded': '2023-11-08T05:29:43.389390+0000', 'last_fullsized': '2023-11-08T05:28:13.541614+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:30:37.255 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-11-08T05:30:37.256 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 456, 'reported_epoch': 455, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:29:43.389311+0000', 'last_change': '2023-11-08T05:28:13.550672+0000', 'last_active': '2023-11-08T05:29:43.389311+0000', 'last_peered': '2023-11-08T05:29:43.389311+0000', 'last_clean': '2023-11-08T05:28:01.457904+0000', 'last_became_active': '2023-11-08T05:28:13.550672+0000', 'last_became_peered': '2023-11-08T05:28:13.550672+0000', 'last_unstale': '2023-11-08T05:29:43.389311+0000', 'last_undegraded': '2023-11-08T05:29:43.389311+0000', 'last_fullsized': '2023-11-08T05:28:13.539595+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:30:37.256 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-11-08T05:30:37.256 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 456, 'reported_epoch': 455, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:29:43.389474+0000', 'last_change': '2023-11-08T05:28:13.557968+0000', 'last_active': '2023-11-08T05:29:43.389474+0000', 'last_peered': '2023-11-08T05:29:43.389474+0000', 'last_clean': '2023-11-08T05:28:01.458433+0000', 'last_became_active': '2023-11-08T05:28:13.557968+0000', 'last_became_peered': '2023-11-08T05:28:13.557968+0000', 'last_unstale': '2023-11-08T05:29:43.389474+0000', 'last_undegraded': '2023-11-08T05:29:43.389474+0000', 'last_fullsized': '2023-11-08T05:28:13.542832+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:30:37.257 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-11-08T05:30:37.605 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T05:30:37.605 INFO:teuthology.orchestra.run.smithi087.stdout:{"fsid":"ab4cbc06-449d-497d-b513-220d9db2b590","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 6 pgs undersized","count":6},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2280,"monmap":{"epoch":1,"min_mon_release_name":"pacific","num_mons":1},"osdmap":{"epoch":456,"num_osds":3,"num_up_osds":2,"osd_up_since":1699421292,"num_in_osds":3,"osd_in_since":1699419158,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":6},{"state_name":"active+clean","count":2},{"state_name":"active+clean+remapped","count":1}],"num_pgs":9,"num_pools":2,"num_objects":0,"data_bytes":0,"bytes_used":2697519104,"bytes_avail":287212773376,"bytes_total":289910292480},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":41,"modified":"2023-11-08T05:29:32.044544+0000","services":{}},"progress_events":{}} 2023-11-08T05:30:37.619 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-11-08T05:30:37.619 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-11-08T05:30:37.903 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T05:30:37.904 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2023-11-08T05:30:37.915 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":1589,"stamp":"2023-11-08T05:30:36.058469+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2634296,"kb_used_data":408940,"kb_used_omap":780,"kb_used_meta":2220403,"kb_avail":280481224,"statfs":{"total":289910292480,"available":287212773376,"internally_reserved":0,"allocated":418754560,"data_stored":410461501,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":799447,"internal_metadata":2273692969},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-536782902,"num_objects":-164,"num_object_clones":0,"num_object_copies":-328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-115,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-164,"num_whiteouts":0,"num_read":-122056,"num_read_kb":-39011942,"num_write":-413642,"num_write_kb":-151705339,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-8,"num_bytes_recovered":-16922624,"num_keys_recovered":-2,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.424817"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":456,"reported_epoch":455,"state":"active+undersized","last_fresh":"2023-11-08T05:29:43.389354+0000","last_change":"2023-11-08T05:28:13.557071+0000","last_active":"2023-11-08T05:29:43.389354+0000","last_peered":"2023-11-08T05:29:43.389354+0000","last_clean":"2023-11-08T05:28:01.458439+0000","last_became_active":"2023-11-08T05:28:13.557071+0000","last_became_peered":"2023-11-08T05:28:13.557071+0000","last_unstale":"2023-11-08T05:29:43.389354+0000","last_undegraded":"2023-11-08T05:29:43.389354+0000","last_fullsized":"2023-11-08T05:28:13.547025+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":18,"reported_epoch":455,"state":"active+undersized","last_fresh":"2023-11-08T05:29:43.389262+0000","last_change":"2023-11-08T05:28:13.559513+0000","last_active":"2023-11-08T05:29:43.389262+0000","last_peered":"2023-11-08T05:29:43.389262+0000","last_clean":"2023-11-08T04:52:46.531964+0000","last_became_active":"2023-11-08T05:28:13.559513+0000","last_became_peered":"2023-11-08T05:28:13.559513+0000","last_unstale":"2023-11-08T05:29:43.389262+0000","last_undegraded":"2023-11-08T05:29:43.389262+0000","last_fullsized":"2023-11-08T05:28:13.540852+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":453,"reported_epoch":455,"state":"active+clean","last_fresh":"2023-11-08T05:29:43.390991+0000","last_change":"2023-11-08T04:52:47.546100+0000","last_active":"2023-11-08T05:29:43.390991+0000","last_peered":"2023-11-08T05:29:43.390991+0000","last_clean":"2023-11-08T05:29:43.390991+0000","last_became_active":"2023-11-08T04:52:47.545840+0000","last_became_peered":"2023-11-08T04:52:47.545840+0000","last_unstale":"2023-11-08T05:29:43.390991+0000","last_undegraded":"2023-11-08T05:29:43.390991+0000","last_fullsized":"2023-11-08T05:29:43.390991+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":451,"reported_epoch":455,"state":"active+clean","last_fresh":"2023-11-08T05:29:43.389261+0000","last_change":"2023-11-08T04:52:47.548597+0000","last_active":"2023-11-08T05:29:43.389261+0000","last_peered":"2023-11-08T05:29:43.389261+0000","last_clean":"2023-11-08T05:29:43.389261+0000","last_became_active":"2023-11-08T04:52:47.548294+0000","last_became_peered":"2023-11-08T04:52:47.548294+0000","last_unstale":"2023-11-08T05:29:43.389261+0000","last_undegraded":"2023-11-08T05:29:43.389261+0000","last_fullsized":"2023-11-08T05:29:43.389261+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":457,"reported_epoch":455,"state":"active+undersized","last_fresh":"2023-11-08T05:29:43.389822+0000","last_change":"2023-11-08T05:28:13.545468+0000","last_active":"2023-11-08T05:29:43.389822+0000","last_peered":"2023-11-08T05:29:43.389822+0000","last_clean":"2023-11-08T05:28:01.456723+0000","last_became_active":"2023-11-08T05:28:13.545468+0000","last_became_peered":"2023-11-08T05:28:13.545468+0000","last_unstale":"2023-11-08T05:29:43.389822+0000","last_undegraded":"2023-11-08T05:29:43.389822+0000","last_fullsized":"2023-11-08T05:28:13.537207+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_clean_scrub_stamp":"2023-11-08T04:52:45.528796+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":462,"reported_epoch":456,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:29:44.395350+0000","last_change":"2023-11-08T05:29:44.395350+0000","last_active":"2023-11-08T05:29:44.395350+0000","last_peered":"2023-11-08T05:29:44.395350+0000","last_clean":"2023-11-08T05:29:44.395350+0000","last_became_active":"2023-11-08T05:29:44.395012+0000","last_became_peered":"2023-11-08T05:29:44.395012+0000","last_unstale":"2023-11-08T05:29:44.395350+0000","last_undegraded":"2023-11-08T05:29:44.395350+0000","last_fullsized":"2023-11-08T05:29:44.395350+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":456,"reported_epoch":455,"state":"active+undersized","last_fresh":"2023-11-08T05:29:43.389390+0000","last_change":"2023-11-08T05:28:13.563357+0000","last_active":"2023-11-08T05:29:43.389390+0000","last_peered":"2023-11-08T05:29:43.389390+0000","last_clean":"2023-11-08T05:28:01.457691+0000","last_became_active":"2023-11-08T05:28:13.563357+0000","last_became_peered":"2023-11-08T05:28:13.563357+0000","last_unstale":"2023-11-08T05:29:43.389390+0000","last_undegraded":"2023-11-08T05:29:43.389390+0000","last_fullsized":"2023-11-08T05:28:13.541614+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":456,"reported_epoch":455,"state":"active+undersized","last_fresh":"2023-11-08T05:29:43.389311+0000","last_change":"2023-11-08T05:28:13.550672+0000","last_active":"2023-11-08T05:29:43.389311+0000","last_peered":"2023-11-08T05:29:43.389311+0000","last_clean":"2023-11-08T05:28:01.457904+0000","last_became_active":"2023-11-08T05:28:13.550672+0000","last_became_peered":"2023-11-08T05:28:13.550672+0000","last_unstale":"2023-11-08T05:29:43.389311+0000","last_undegraded":"2023-11-08T05:29:43.389311+0000","last_fullsized":"2023-11-08T05:28:13.539595+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":456,"reported_epoch":455,"state":"active+undersized","last_fresh":"2023-11-08T05:29:43.389474+0000","last_change":"2023-11-08T05:28:13.557968+0000","last_active":"2023-11-08T05:29:43.389474+0000","last_peered":"2023-11-08T05:29:43.389474+0000","last_clean":"2023-11-08T05:28:01.458433+0000","last_became_active":"2023-11-08T05:28:13.557968+0000","last_became_peered":"2023-11-08T05:28:13.557968+0000","last_unstale":"2023-11-08T05:29:43.389474+0000","last_undegraded":"2023-11-08T05:29:43.389474+0000","last_fullsized":"2023-11-08T05:28:13.542832+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":1,"acting":1,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":10,"acting":11,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738825,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":645004,"kb_used_data":5316,"kb_used_omap":275,"kb_used_meta":639404,"kb_avail":93726836,"statfs":{"total":96636764160,"available":95976280064,"internally_reserved":0,"allocated":5443584,"data_stored":2678316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":281650,"internal_metadata":654750670},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:59 2023","interfaces":[{"interface":"back","average":{"1min":0.30499999999999999,"5min":0.439,"15min":0.36099999999999999},"min":{"1min":0.17399999999999999,"5min":0.14499999999999999,"15min":0.14499999999999999},"max":{"1min":0.60099999999999998,"5min":4.9379999999999997,"15min":4.9379999999999997},"last":0.379},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.49399999999999999,"15min":0.42399999999999999},"min":{"1min":0.221,"5min":0.19600000000000001,"15min":0.17000000000000001},"max":{"1min":0.63200000000000001,"5min":4.9859999999999998,"15min":4.9859999999999998},"last":0.40899999999999997}]},{"osd":1,"last update":"Wed Nov 8 05:30:03 2023","interfaces":[{"interface":"back","average":{"1min":0.28499999999999998,"5min":0.65100000000000002,"15min":0.47099999999999997},"min":{"1min":0.20100000000000001,"5min":0.153,"15min":0.153},"max":{"1min":0.39800000000000002,"5min":23.242999999999999,"15min":23.242999999999999},"last":0.33400000000000002},{"interface":"front","average":{"1min":0.29499999999999998,"5min":0.63100000000000001,"15min":0.45900000000000002},"min":{"1min":0.224,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":0.436,"5min":23.251000000000001,"15min":23.251000000000001},"last":0.34999999999999998}]}]},{"osd":0,"up_from":8,"seq":34359738798,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1339228,"kb_used_data":398304,"kb_used_omap":291,"kb_used_meta":936476,"kb_avail":93032612,"statfs":{"total":96636764160,"available":95265394688,"internally_reserved":0,"allocated":407863296,"data_stored":405103900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":298462,"internal_metadata":958951970},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.436,"15min":0.36299999999999999},"min":{"1min":0.19700000000000001,"5min":0.14699999999999999,"15min":0.13700000000000001},"max":{"1min":3.222,"5min":6.5499999999999998,"15min":6.9989999999999997},"last":0.245},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.46300000000000002,"15min":0.42299999999999999},"min":{"1min":0.17799999999999999,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":5.0030000000000001,"5min":6.0629999999999997,"15min":9.6869999999999994},"last":0.28599999999999998}]},{"osd":2,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.51600000000000001,"15min":0.47099999999999997},"min":{"1min":0.21099999999999999,"5min":0.153,"15min":0.153},"max":{"1min":5.7439999999999998,"5min":5.8280000000000003,"15min":9.7639999999999993},"last":0.252},{"interface":"front","average":{"1min":0.497,"5min":0.56200000000000006,"15min":0.42299999999999999},"min":{"1min":0.20399999999999999,"5min":0.187,"15min":0.16800000000000001},"max":{"1min":3.9319999999999999,"5min":8.8040000000000003,"15min":8.8040000000000003},"last":0.29899999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738827,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":650064,"kb_used_data":5320,"kb_used_omap":214,"kb_used_meta":644521,"kb_avail":93721776,"statfs":{"total":96636764160,"available":95971098624,"internally_reserved":0,"allocated":5447680,"data_stored":2679285,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":219335,"internal_metadata":659990329},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:56 2023","interfaces":[{"interface":"back","average":{"1min":0.30399999999999999,"5min":0.34699999999999998,"15min":0.36499999999999999},"min":{"1min":0.23999999999999999,"5min":0.14199999999999999,"15min":0.14199999999999999},"max":{"1min":0.42399999999999999,"5min":1.595,"15min":7.4249999999999998},"last":0.17100000000000001},{"interface":"front","average":{"1min":0.35499999999999998,"5min":0.40899999999999997,"15min":0.376},"min":{"1min":0.23999999999999999,"5min":0.19600000000000001,"15min":0.183},"max":{"1min":0.46999999999999997,"5min":2.6859999999999999,"15min":2.6859999999999999},"last":0.23000000000000001}]},{"osd":2,"last update":"Wed Nov 8 05:29:57 2023","interfaces":[{"interface":"back","average":{"1min":0.28199999999999997,"5min":0.36099999999999999,"15min":0.373},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":0.34399999999999997,"5min":2.2650000000000001,"15min":5.8209999999999997},"last":0.27000000000000002},{"interface":"front","average":{"1min":0.28599999999999998,"5min":0.31,"15min":0.35799999999999998},"min":{"1min":0.17199999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.35699999999999998,"5min":1.137,"15min":4.2350000000000003},"last":0.223}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-08T05:30:38.549 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~141s 2023-11-08T05:30:40.917 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-11-08T05:30:41.215 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T05:30:41.217 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2023-11-08T05:30:41.228 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":1591,"stamp":"2023-11-08T05:30:40.059101+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2634296,"kb_used_data":408940,"kb_used_omap":780,"kb_used_meta":2220403,"kb_avail":280481224,"statfs":{"total":289910292480,"available":287212773376,"internally_reserved":0,"allocated":418754560,"data_stored":410461501,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":799447,"internal_metadata":2273692969},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-536782902,"num_objects":-164,"num_object_clones":0,"num_object_copies":-328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-115,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-164,"num_whiteouts":0,"num_read":-122056,"num_read_kb":-39011942,"num_write":-413642,"num_write_kb":-151705339,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-8,"num_bytes_recovered":-16922624,"num_keys_recovered":-2,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.424817"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":456,"reported_epoch":455,"state":"active+undersized","last_fresh":"2023-11-08T05:29:43.389354+0000","last_change":"2023-11-08T05:28:13.557071+0000","last_active":"2023-11-08T05:29:43.389354+0000","last_peered":"2023-11-08T05:29:43.389354+0000","last_clean":"2023-11-08T05:28:01.458439+0000","last_became_active":"2023-11-08T05:28:13.557071+0000","last_became_peered":"2023-11-08T05:28:13.557071+0000","last_unstale":"2023-11-08T05:29:43.389354+0000","last_undegraded":"2023-11-08T05:29:43.389354+0000","last_fullsized":"2023-11-08T05:28:13.547025+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":18,"reported_epoch":455,"state":"active+undersized","last_fresh":"2023-11-08T05:29:43.389262+0000","last_change":"2023-11-08T05:28:13.559513+0000","last_active":"2023-11-08T05:29:43.389262+0000","last_peered":"2023-11-08T05:29:43.389262+0000","last_clean":"2023-11-08T04:52:46.531964+0000","last_became_active":"2023-11-08T05:28:13.559513+0000","last_became_peered":"2023-11-08T05:28:13.559513+0000","last_unstale":"2023-11-08T05:29:43.389262+0000","last_undegraded":"2023-11-08T05:29:43.389262+0000","last_fullsized":"2023-11-08T05:28:13.540852+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":453,"reported_epoch":455,"state":"active+clean","last_fresh":"2023-11-08T05:29:43.390991+0000","last_change":"2023-11-08T04:52:47.546100+0000","last_active":"2023-11-08T05:29:43.390991+0000","last_peered":"2023-11-08T05:29:43.390991+0000","last_clean":"2023-11-08T05:29:43.390991+0000","last_became_active":"2023-11-08T04:52:47.545840+0000","last_became_peered":"2023-11-08T04:52:47.545840+0000","last_unstale":"2023-11-08T05:29:43.390991+0000","last_undegraded":"2023-11-08T05:29:43.390991+0000","last_fullsized":"2023-11-08T05:29:43.390991+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":451,"reported_epoch":455,"state":"active+clean","last_fresh":"2023-11-08T05:29:43.389261+0000","last_change":"2023-11-08T04:52:47.548597+0000","last_active":"2023-11-08T05:29:43.389261+0000","last_peered":"2023-11-08T05:29:43.389261+0000","last_clean":"2023-11-08T05:29:43.389261+0000","last_became_active":"2023-11-08T04:52:47.548294+0000","last_became_peered":"2023-11-08T04:52:47.548294+0000","last_unstale":"2023-11-08T05:29:43.389261+0000","last_undegraded":"2023-11-08T05:29:43.389261+0000","last_fullsized":"2023-11-08T05:29:43.389261+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":457,"reported_epoch":455,"state":"active+undersized","last_fresh":"2023-11-08T05:29:43.389822+0000","last_change":"2023-11-08T05:28:13.545468+0000","last_active":"2023-11-08T05:29:43.389822+0000","last_peered":"2023-11-08T05:29:43.389822+0000","last_clean":"2023-11-08T05:28:01.456723+0000","last_became_active":"2023-11-08T05:28:13.545468+0000","last_became_peered":"2023-11-08T05:28:13.545468+0000","last_unstale":"2023-11-08T05:29:43.389822+0000","last_undegraded":"2023-11-08T05:29:43.389822+0000","last_fullsized":"2023-11-08T05:28:13.537207+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_clean_scrub_stamp":"2023-11-08T04:52:45.528796+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":462,"reported_epoch":456,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:29:44.395350+0000","last_change":"2023-11-08T05:29:44.395350+0000","last_active":"2023-11-08T05:29:44.395350+0000","last_peered":"2023-11-08T05:29:44.395350+0000","last_clean":"2023-11-08T05:29:44.395350+0000","last_became_active":"2023-11-08T05:29:44.395012+0000","last_became_peered":"2023-11-08T05:29:44.395012+0000","last_unstale":"2023-11-08T05:29:44.395350+0000","last_undegraded":"2023-11-08T05:29:44.395350+0000","last_fullsized":"2023-11-08T05:29:44.395350+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":456,"reported_epoch":455,"state":"active+undersized","last_fresh":"2023-11-08T05:29:43.389390+0000","last_change":"2023-11-08T05:28:13.563357+0000","last_active":"2023-11-08T05:29:43.389390+0000","last_peered":"2023-11-08T05:29:43.389390+0000","last_clean":"2023-11-08T05:28:01.457691+0000","last_became_active":"2023-11-08T05:28:13.563357+0000","last_became_peered":"2023-11-08T05:28:13.563357+0000","last_unstale":"2023-11-08T05:29:43.389390+0000","last_undegraded":"2023-11-08T05:29:43.389390+0000","last_fullsized":"2023-11-08T05:28:13.541614+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":456,"reported_epoch":455,"state":"active+undersized","last_fresh":"2023-11-08T05:29:43.389311+0000","last_change":"2023-11-08T05:28:13.550672+0000","last_active":"2023-11-08T05:29:43.389311+0000","last_peered":"2023-11-08T05:29:43.389311+0000","last_clean":"2023-11-08T05:28:01.457904+0000","last_became_active":"2023-11-08T05:28:13.550672+0000","last_became_peered":"2023-11-08T05:28:13.550672+0000","last_unstale":"2023-11-08T05:29:43.389311+0000","last_undegraded":"2023-11-08T05:29:43.389311+0000","last_fullsized":"2023-11-08T05:28:13.539595+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":456,"reported_epoch":455,"state":"active+undersized","last_fresh":"2023-11-08T05:29:43.389474+0000","last_change":"2023-11-08T05:28:13.557968+0000","last_active":"2023-11-08T05:29:43.389474+0000","last_peered":"2023-11-08T05:29:43.389474+0000","last_clean":"2023-11-08T05:28:01.458433+0000","last_became_active":"2023-11-08T05:28:13.557968+0000","last_became_peered":"2023-11-08T05:28:13.557968+0000","last_unstale":"2023-11-08T05:29:43.389474+0000","last_undegraded":"2023-11-08T05:29:43.389474+0000","last_fullsized":"2023-11-08T05:28:13.542832+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":1,"acting":1,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":10,"acting":11,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738826,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":645004,"kb_used_data":5316,"kb_used_omap":275,"kb_used_meta":639404,"kb_avail":93726836,"statfs":{"total":96636764160,"available":95976280064,"internally_reserved":0,"allocated":5443584,"data_stored":2678316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":281650,"internal_metadata":654750670},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:59 2023","interfaces":[{"interface":"back","average":{"1min":0.30499999999999999,"5min":0.439,"15min":0.36099999999999999},"min":{"1min":0.17399999999999999,"5min":0.14499999999999999,"15min":0.14499999999999999},"max":{"1min":0.60099999999999998,"5min":4.9379999999999997,"15min":4.9379999999999997},"last":0.379},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.49399999999999999,"15min":0.42399999999999999},"min":{"1min":0.221,"5min":0.19600000000000001,"15min":0.17000000000000001},"max":{"1min":0.63200000000000001,"5min":4.9859999999999998,"15min":4.9859999999999998},"last":0.40899999999999997}]},{"osd":1,"last update":"Wed Nov 8 05:30:03 2023","interfaces":[{"interface":"back","average":{"1min":0.28499999999999998,"5min":0.65100000000000002,"15min":0.47099999999999997},"min":{"1min":0.20100000000000001,"5min":0.153,"15min":0.153},"max":{"1min":0.39800000000000002,"5min":23.242999999999999,"15min":23.242999999999999},"last":0.29999999999999999},{"interface":"front","average":{"1min":0.29499999999999998,"5min":0.63100000000000001,"15min":0.45900000000000002},"min":{"1min":0.224,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":0.436,"5min":23.251000000000001,"15min":23.251000000000001},"last":0.309}]}]},{"osd":0,"up_from":8,"seq":34359738798,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1339228,"kb_used_data":398304,"kb_used_omap":291,"kb_used_meta":936476,"kb_avail":93032612,"statfs":{"total":96636764160,"available":95265394688,"internally_reserved":0,"allocated":407863296,"data_stored":405103900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":298462,"internal_metadata":958951970},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.436,"15min":0.36299999999999999},"min":{"1min":0.19700000000000001,"5min":0.14699999999999999,"15min":0.13700000000000001},"max":{"1min":3.222,"5min":6.5499999999999998,"15min":6.9989999999999997},"last":0.245},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.46300000000000002,"15min":0.42299999999999999},"min":{"1min":0.17799999999999999,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":5.0030000000000001,"5min":6.0629999999999997,"15min":9.6869999999999994},"last":0.28599999999999998}]},{"osd":2,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.51600000000000001,"15min":0.47099999999999997},"min":{"1min":0.21099999999999999,"5min":0.153,"15min":0.153},"max":{"1min":5.7439999999999998,"5min":5.8280000000000003,"15min":9.7639999999999993},"last":0.252},{"interface":"front","average":{"1min":0.497,"5min":0.56200000000000006,"15min":0.42299999999999999},"min":{"1min":0.20399999999999999,"5min":0.187,"15min":0.16800000000000001},"max":{"1min":3.9319999999999999,"5min":8.8040000000000003,"15min":8.8040000000000003},"last":0.29899999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738827,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":650064,"kb_used_data":5320,"kb_used_omap":214,"kb_used_meta":644521,"kb_avail":93721776,"statfs":{"total":96636764160,"available":95971098624,"internally_reserved":0,"allocated":5447680,"data_stored":2679285,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":219335,"internal_metadata":659990329},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:56 2023","interfaces":[{"interface":"back","average":{"1min":0.30399999999999999,"5min":0.34699999999999998,"15min":0.36499999999999999},"min":{"1min":0.23999999999999999,"5min":0.14199999999999999,"15min":0.14199999999999999},"max":{"1min":0.42399999999999999,"5min":1.595,"15min":7.4249999999999998},"last":0.17100000000000001},{"interface":"front","average":{"1min":0.35499999999999998,"5min":0.40899999999999997,"15min":0.376},"min":{"1min":0.23999999999999999,"5min":0.19600000000000001,"15min":0.183},"max":{"1min":0.46999999999999997,"5min":2.6859999999999999,"15min":2.6859999999999999},"last":0.23000000000000001}]},{"osd":2,"last update":"Wed Nov 8 05:29:57 2023","interfaces":[{"interface":"back","average":{"1min":0.28199999999999997,"5min":0.36099999999999999,"15min":0.373},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":0.34399999999999997,"5min":2.2650000000000001,"15min":5.8209999999999997},"last":0.27000000000000002},{"interface":"front","average":{"1min":0.28599999999999998,"5min":0.31,"15min":0.35799999999999998},"min":{"1min":0.17199999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.35699999999999998,"5min":1.137,"15min":4.2350000000000003},"last":0.223}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-08T05:30:41.229 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-11-08T05:30:41.511 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T05:30:41.513 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2023-11-08T05:30:41.524 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":1591,"stamp":"2023-11-08T05:30:40.059101+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2634296,"kb_used_data":408940,"kb_used_omap":780,"kb_used_meta":2220403,"kb_avail":280481224,"statfs":{"total":289910292480,"available":287212773376,"internally_reserved":0,"allocated":418754560,"data_stored":410461501,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":799447,"internal_metadata":2273692969},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-536782902,"num_objects":-164,"num_object_clones":0,"num_object_copies":-328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-115,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-164,"num_whiteouts":0,"num_read":-122056,"num_read_kb":-39011942,"num_write":-413642,"num_write_kb":-151705339,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-8,"num_bytes_recovered":-16922624,"num_keys_recovered":-2,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.424817"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":456,"reported_epoch":455,"state":"active+undersized","last_fresh":"2023-11-08T05:29:43.389354+0000","last_change":"2023-11-08T05:28:13.557071+0000","last_active":"2023-11-08T05:29:43.389354+0000","last_peered":"2023-11-08T05:29:43.389354+0000","last_clean":"2023-11-08T05:28:01.458439+0000","last_became_active":"2023-11-08T05:28:13.557071+0000","last_became_peered":"2023-11-08T05:28:13.557071+0000","last_unstale":"2023-11-08T05:29:43.389354+0000","last_undegraded":"2023-11-08T05:29:43.389354+0000","last_fullsized":"2023-11-08T05:28:13.547025+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":18,"reported_epoch":455,"state":"active+undersized","last_fresh":"2023-11-08T05:29:43.389262+0000","last_change":"2023-11-08T05:28:13.559513+0000","last_active":"2023-11-08T05:29:43.389262+0000","last_peered":"2023-11-08T05:29:43.389262+0000","last_clean":"2023-11-08T04:52:46.531964+0000","last_became_active":"2023-11-08T05:28:13.559513+0000","last_became_peered":"2023-11-08T05:28:13.559513+0000","last_unstale":"2023-11-08T05:29:43.389262+0000","last_undegraded":"2023-11-08T05:29:43.389262+0000","last_fullsized":"2023-11-08T05:28:13.540852+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":453,"reported_epoch":455,"state":"active+clean","last_fresh":"2023-11-08T05:29:43.390991+0000","last_change":"2023-11-08T04:52:47.546100+0000","last_active":"2023-11-08T05:29:43.390991+0000","last_peered":"2023-11-08T05:29:43.390991+0000","last_clean":"2023-11-08T05:29:43.390991+0000","last_became_active":"2023-11-08T04:52:47.545840+0000","last_became_peered":"2023-11-08T04:52:47.545840+0000","last_unstale":"2023-11-08T05:29:43.390991+0000","last_undegraded":"2023-11-08T05:29:43.390991+0000","last_fullsized":"2023-11-08T05:29:43.390991+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":451,"reported_epoch":455,"state":"active+clean","last_fresh":"2023-11-08T05:29:43.389261+0000","last_change":"2023-11-08T04:52:47.548597+0000","last_active":"2023-11-08T05:29:43.389261+0000","last_peered":"2023-11-08T05:29:43.389261+0000","last_clean":"2023-11-08T05:29:43.389261+0000","last_became_active":"2023-11-08T04:52:47.548294+0000","last_became_peered":"2023-11-08T04:52:47.548294+0000","last_unstale":"2023-11-08T05:29:43.389261+0000","last_undegraded":"2023-11-08T05:29:43.389261+0000","last_fullsized":"2023-11-08T05:29:43.389261+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":457,"reported_epoch":455,"state":"active+undersized","last_fresh":"2023-11-08T05:29:43.389822+0000","last_change":"2023-11-08T05:28:13.545468+0000","last_active":"2023-11-08T05:29:43.389822+0000","last_peered":"2023-11-08T05:29:43.389822+0000","last_clean":"2023-11-08T05:28:01.456723+0000","last_became_active":"2023-11-08T05:28:13.545468+0000","last_became_peered":"2023-11-08T05:28:13.545468+0000","last_unstale":"2023-11-08T05:29:43.389822+0000","last_undegraded":"2023-11-08T05:29:43.389822+0000","last_fullsized":"2023-11-08T05:28:13.537207+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_clean_scrub_stamp":"2023-11-08T04:52:45.528796+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":462,"reported_epoch":456,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:29:44.395350+0000","last_change":"2023-11-08T05:29:44.395350+0000","last_active":"2023-11-08T05:29:44.395350+0000","last_peered":"2023-11-08T05:29:44.395350+0000","last_clean":"2023-11-08T05:29:44.395350+0000","last_became_active":"2023-11-08T05:29:44.395012+0000","last_became_peered":"2023-11-08T05:29:44.395012+0000","last_unstale":"2023-11-08T05:29:44.395350+0000","last_undegraded":"2023-11-08T05:29:44.395350+0000","last_fullsized":"2023-11-08T05:29:44.395350+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":456,"reported_epoch":455,"state":"active+undersized","last_fresh":"2023-11-08T05:29:43.389390+0000","last_change":"2023-11-08T05:28:13.563357+0000","last_active":"2023-11-08T05:29:43.389390+0000","last_peered":"2023-11-08T05:29:43.389390+0000","last_clean":"2023-11-08T05:28:01.457691+0000","last_became_active":"2023-11-08T05:28:13.563357+0000","last_became_peered":"2023-11-08T05:28:13.563357+0000","last_unstale":"2023-11-08T05:29:43.389390+0000","last_undegraded":"2023-11-08T05:29:43.389390+0000","last_fullsized":"2023-11-08T05:28:13.541614+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":456,"reported_epoch":455,"state":"active+undersized","last_fresh":"2023-11-08T05:29:43.389311+0000","last_change":"2023-11-08T05:28:13.550672+0000","last_active":"2023-11-08T05:29:43.389311+0000","last_peered":"2023-11-08T05:29:43.389311+0000","last_clean":"2023-11-08T05:28:01.457904+0000","last_became_active":"2023-11-08T05:28:13.550672+0000","last_became_peered":"2023-11-08T05:28:13.550672+0000","last_unstale":"2023-11-08T05:29:43.389311+0000","last_undegraded":"2023-11-08T05:29:43.389311+0000","last_fullsized":"2023-11-08T05:28:13.539595+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":456,"reported_epoch":455,"state":"active+undersized","last_fresh":"2023-11-08T05:29:43.389474+0000","last_change":"2023-11-08T05:28:13.557968+0000","last_active":"2023-11-08T05:29:43.389474+0000","last_peered":"2023-11-08T05:29:43.389474+0000","last_clean":"2023-11-08T05:28:01.458433+0000","last_became_active":"2023-11-08T05:28:13.557968+0000","last_became_peered":"2023-11-08T05:28:13.557968+0000","last_unstale":"2023-11-08T05:29:43.389474+0000","last_undegraded":"2023-11-08T05:29:43.389474+0000","last_fullsized":"2023-11-08T05:28:13.542832+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":1,"acting":1,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":10,"acting":11,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738826,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":645004,"kb_used_data":5316,"kb_used_omap":275,"kb_used_meta":639404,"kb_avail":93726836,"statfs":{"total":96636764160,"available":95976280064,"internally_reserved":0,"allocated":5443584,"data_stored":2678316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":281650,"internal_metadata":654750670},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:59 2023","interfaces":[{"interface":"back","average":{"1min":0.30499999999999999,"5min":0.439,"15min":0.36099999999999999},"min":{"1min":0.17399999999999999,"5min":0.14499999999999999,"15min":0.14499999999999999},"max":{"1min":0.60099999999999998,"5min":4.9379999999999997,"15min":4.9379999999999997},"last":0.379},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.49399999999999999,"15min":0.42399999999999999},"min":{"1min":0.221,"5min":0.19600000000000001,"15min":0.17000000000000001},"max":{"1min":0.63200000000000001,"5min":4.9859999999999998,"15min":4.9859999999999998},"last":0.40899999999999997}]},{"osd":1,"last update":"Wed Nov 8 05:30:03 2023","interfaces":[{"interface":"back","average":{"1min":0.28499999999999998,"5min":0.65100000000000002,"15min":0.47099999999999997},"min":{"1min":0.20100000000000001,"5min":0.153,"15min":0.153},"max":{"1min":0.39800000000000002,"5min":23.242999999999999,"15min":23.242999999999999},"last":0.29999999999999999},{"interface":"front","average":{"1min":0.29499999999999998,"5min":0.63100000000000001,"15min":0.45900000000000002},"min":{"1min":0.224,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":0.436,"5min":23.251000000000001,"15min":23.251000000000001},"last":0.309}]}]},{"osd":0,"up_from":8,"seq":34359738798,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1339228,"kb_used_data":398304,"kb_used_omap":291,"kb_used_meta":936476,"kb_avail":93032612,"statfs":{"total":96636764160,"available":95265394688,"internally_reserved":0,"allocated":407863296,"data_stored":405103900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":298462,"internal_metadata":958951970},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.436,"15min":0.36299999999999999},"min":{"1min":0.19700000000000001,"5min":0.14699999999999999,"15min":0.13700000000000001},"max":{"1min":3.222,"5min":6.5499999999999998,"15min":6.9989999999999997},"last":0.245},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.46300000000000002,"15min":0.42299999999999999},"min":{"1min":0.17799999999999999,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":5.0030000000000001,"5min":6.0629999999999997,"15min":9.6869999999999994},"last":0.28599999999999998}]},{"osd":2,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.51600000000000001,"15min":0.47099999999999997},"min":{"1min":0.21099999999999999,"5min":0.153,"15min":0.153},"max":{"1min":5.7439999999999998,"5min":5.8280000000000003,"15min":9.7639999999999993},"last":0.252},{"interface":"front","average":{"1min":0.497,"5min":0.56200000000000006,"15min":0.42299999999999999},"min":{"1min":0.20399999999999999,"5min":0.187,"15min":0.16800000000000001},"max":{"1min":3.9319999999999999,"5min":8.8040000000000003,"15min":8.8040000000000003},"last":0.29899999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738827,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":650064,"kb_used_data":5320,"kb_used_omap":214,"kb_used_meta":644521,"kb_avail":93721776,"statfs":{"total":96636764160,"available":95971098624,"internally_reserved":0,"allocated":5447680,"data_stored":2679285,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":219335,"internal_metadata":659990329},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:56 2023","interfaces":[{"interface":"back","average":{"1min":0.30399999999999999,"5min":0.34699999999999998,"15min":0.36499999999999999},"min":{"1min":0.23999999999999999,"5min":0.14199999999999999,"15min":0.14199999999999999},"max":{"1min":0.42399999999999999,"5min":1.595,"15min":7.4249999999999998},"last":0.17100000000000001},{"interface":"front","average":{"1min":0.35499999999999998,"5min":0.40899999999999997,"15min":0.376},"min":{"1min":0.23999999999999999,"5min":0.19600000000000001,"15min":0.183},"max":{"1min":0.46999999999999997,"5min":2.6859999999999999,"15min":2.6859999999999999},"last":0.23000000000000001}]},{"osd":2,"last update":"Wed Nov 8 05:29:57 2023","interfaces":[{"interface":"back","average":{"1min":0.28199999999999997,"5min":0.36099999999999999,"15min":0.373},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":0.34399999999999997,"5min":2.2650000000000001,"15min":5.8209999999999997},"last":0.27000000000000002},{"interface":"front","average":{"1min":0.28599999999999998,"5min":0.31,"15min":0.35799999999999998},"min":{"1min":0.17199999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.35699999999999998,"5min":1.137,"15min":4.2350000000000003},"last":0.223}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-08T05:30:41.525 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-11-08T05:30:41.525 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 456, 'reported_epoch': 455, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:29:43.389354+0000', 'last_change': '2023-11-08T05:28:13.557071+0000', 'last_active': '2023-11-08T05:29:43.389354+0000', 'last_peered': '2023-11-08T05:29:43.389354+0000', 'last_clean': '2023-11-08T05:28:01.458439+0000', 'last_became_active': '2023-11-08T05:28:13.557071+0000', 'last_became_peered': '2023-11-08T05:28:13.557071+0000', 'last_unstale': '2023-11-08T05:29:43.389354+0000', 'last_undegraded': '2023-11-08T05:29:43.389354+0000', 'last_fullsized': '2023-11-08T05:28:13.547025+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:30:41.526 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-11-08T05:30:41.526 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 18, 'reported_epoch': 455, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:29:43.389262+0000', 'last_change': '2023-11-08T05:28:13.559513+0000', 'last_active': '2023-11-08T05:29:43.389262+0000', 'last_peered': '2023-11-08T05:29:43.389262+0000', 'last_clean': '2023-11-08T04:52:46.531964+0000', 'last_became_active': '2023-11-08T05:28:13.559513+0000', 'last_became_peered': '2023-11-08T05:28:13.559513+0000', 'last_unstale': '2023-11-08T05:29:43.389262+0000', 'last_undegraded': '2023-11-08T05:29:43.389262+0000', 'last_fullsized': '2023-11-08T05:28:13.540852+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:30:41.526 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-11-08T05:30:41.526 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "0'0", 'reported_seq': 457, 'reported_epoch': 455, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:29:43.389822+0000', 'last_change': '2023-11-08T05:28:13.545468+0000', 'last_active': '2023-11-08T05:29:43.389822+0000', 'last_peered': '2023-11-08T05:29:43.389822+0000', 'last_clean': '2023-11-08T05:28:01.456723+0000', 'last_became_active': '2023-11-08T05:28:13.545468+0000', 'last_became_peered': '2023-11-08T05:28:13.545468+0000', 'last_unstale': '2023-11-08T05:29:43.389822+0000', 'last_undegraded': '2023-11-08T05:29:43.389822+0000', 'last_fullsized': '2023-11-08T05:28:13.537207+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 9, 'last_epoch_clean': 10, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:45.528796+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:45.528796+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:45.528796+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:30:41.527 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.0 is not active+clean 2023-11-08T05:30:41.527 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.0', 'version': "0'0", 'reported_seq': 462, 'reported_epoch': 456, 'state': 'active+clean+remapped', 'last_fresh': '2023-11-08T05:29:44.395350+0000', 'last_change': '2023-11-08T05:29:44.395350+0000', 'last_active': '2023-11-08T05:29:44.395350+0000', 'last_peered': '2023-11-08T05:29:44.395350+0000', 'last_clean': '2023-11-08T05:29:44.395350+0000', 'last_became_active': '2023-11-08T05:29:44.395012+0000', 'last_became_peered': '2023-11-08T05:29:44.395012+0000', 'last_unstale': '2023-11-08T05:29:44.395350+0000', 'last_undegraded': '2023-11-08T05:29:44.395350+0000', 'last_fullsized': '2023-11-08T05:29:44.395350+0000', 'mapping_epoch': 455, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 456, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2], 'acting': [2, 1], 'avail_no_missing': ['2', '1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2023-11-08T05:30:41.527 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-11-08T05:30:41.528 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 456, 'reported_epoch': 455, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:29:43.389390+0000', 'last_change': '2023-11-08T05:28:13.563357+0000', 'last_active': '2023-11-08T05:29:43.389390+0000', 'last_peered': '2023-11-08T05:29:43.389390+0000', 'last_clean': '2023-11-08T05:28:01.457691+0000', 'last_became_active': '2023-11-08T05:28:13.563357+0000', 'last_became_peered': '2023-11-08T05:28:13.563357+0000', 'last_unstale': '2023-11-08T05:29:43.389390+0000', 'last_undegraded': '2023-11-08T05:29:43.389390+0000', 'last_fullsized': '2023-11-08T05:28:13.541614+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:30:41.528 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-11-08T05:30:41.528 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 456, 'reported_epoch': 455, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:29:43.389311+0000', 'last_change': '2023-11-08T05:28:13.550672+0000', 'last_active': '2023-11-08T05:29:43.389311+0000', 'last_peered': '2023-11-08T05:29:43.389311+0000', 'last_clean': '2023-11-08T05:28:01.457904+0000', 'last_became_active': '2023-11-08T05:28:13.550672+0000', 'last_became_peered': '2023-11-08T05:28:13.550672+0000', 'last_unstale': '2023-11-08T05:29:43.389311+0000', 'last_undegraded': '2023-11-08T05:29:43.389311+0000', 'last_fullsized': '2023-11-08T05:28:13.539595+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:30:41.529 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-11-08T05:30:41.529 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 456, 'reported_epoch': 455, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:29:43.389474+0000', 'last_change': '2023-11-08T05:28:13.557968+0000', 'last_active': '2023-11-08T05:29:43.389474+0000', 'last_peered': '2023-11-08T05:29:43.389474+0000', 'last_clean': '2023-11-08T05:28:01.458433+0000', 'last_became_active': '2023-11-08T05:28:13.557968+0000', 'last_became_peered': '2023-11-08T05:28:13.557968+0000', 'last_unstale': '2023-11-08T05:29:43.389474+0000', 'last_undegraded': '2023-11-08T05:29:43.389474+0000', 'last_fullsized': '2023-11-08T05:28:13.542832+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:30:41.529 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-11-08T05:30:41.874 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T05:30:41.874 INFO:teuthology.orchestra.run.smithi087.stdout:{"fsid":"ab4cbc06-449d-497d-b513-220d9db2b590","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 6 pgs undersized","count":6},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2284,"monmap":{"epoch":1,"min_mon_release_name":"pacific","num_mons":1},"osdmap":{"epoch":456,"num_osds":3,"num_up_osds":2,"osd_up_since":1699421292,"num_in_osds":3,"osd_in_since":1699419158,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":6},{"state_name":"active+clean","count":2},{"state_name":"active+clean+remapped","count":1}],"num_pgs":9,"num_pools":2,"num_objects":0,"data_bytes":0,"bytes_used":2697519104,"bytes_avail":287212773376,"bytes_total":289910292480},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":41,"modified":"2023-11-08T05:29:32.044544+0000","services":{}},"progress_events":{}} 2023-11-08T05:30:41.887 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-11-08T05:30:41.888 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-11-08T05:30:42.171 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T05:30:42.173 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2023-11-08T05:30:42.184 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":1592,"stamp":"2023-11-08T05:30:42.059450+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2634296,"kb_used_data":408940,"kb_used_omap":780,"kb_used_meta":2220403,"kb_avail":280481224,"statfs":{"total":289910292480,"available":287212773376,"internally_reserved":0,"allocated":418754560,"data_stored":410461501,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":799447,"internal_metadata":2273692969},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-536782902,"num_objects":-164,"num_object_clones":0,"num_object_copies":-328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-115,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-164,"num_whiteouts":0,"num_read":-122056,"num_read_kb":-39011942,"num_write":-413642,"num_write_kb":-151705339,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-8,"num_bytes_recovered":-16922624,"num_keys_recovered":-2,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.424817"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":456,"reported_epoch":455,"state":"active+undersized","last_fresh":"2023-11-08T05:29:43.389354+0000","last_change":"2023-11-08T05:28:13.557071+0000","last_active":"2023-11-08T05:29:43.389354+0000","last_peered":"2023-11-08T05:29:43.389354+0000","last_clean":"2023-11-08T05:28:01.458439+0000","last_became_active":"2023-11-08T05:28:13.557071+0000","last_became_peered":"2023-11-08T05:28:13.557071+0000","last_unstale":"2023-11-08T05:29:43.389354+0000","last_undegraded":"2023-11-08T05:29:43.389354+0000","last_fullsized":"2023-11-08T05:28:13.547025+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":18,"reported_epoch":455,"state":"active+undersized","last_fresh":"2023-11-08T05:29:43.389262+0000","last_change":"2023-11-08T05:28:13.559513+0000","last_active":"2023-11-08T05:29:43.389262+0000","last_peered":"2023-11-08T05:29:43.389262+0000","last_clean":"2023-11-08T04:52:46.531964+0000","last_became_active":"2023-11-08T05:28:13.559513+0000","last_became_peered":"2023-11-08T05:28:13.559513+0000","last_unstale":"2023-11-08T05:29:43.389262+0000","last_undegraded":"2023-11-08T05:29:43.389262+0000","last_fullsized":"2023-11-08T05:28:13.540852+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":453,"reported_epoch":455,"state":"active+clean","last_fresh":"2023-11-08T05:29:43.390991+0000","last_change":"2023-11-08T04:52:47.546100+0000","last_active":"2023-11-08T05:29:43.390991+0000","last_peered":"2023-11-08T05:29:43.390991+0000","last_clean":"2023-11-08T05:29:43.390991+0000","last_became_active":"2023-11-08T04:52:47.545840+0000","last_became_peered":"2023-11-08T04:52:47.545840+0000","last_unstale":"2023-11-08T05:29:43.390991+0000","last_undegraded":"2023-11-08T05:29:43.390991+0000","last_fullsized":"2023-11-08T05:29:43.390991+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":451,"reported_epoch":455,"state":"active+clean","last_fresh":"2023-11-08T05:29:43.389261+0000","last_change":"2023-11-08T04:52:47.548597+0000","last_active":"2023-11-08T05:29:43.389261+0000","last_peered":"2023-11-08T05:29:43.389261+0000","last_clean":"2023-11-08T05:29:43.389261+0000","last_became_active":"2023-11-08T04:52:47.548294+0000","last_became_peered":"2023-11-08T04:52:47.548294+0000","last_unstale":"2023-11-08T05:29:43.389261+0000","last_undegraded":"2023-11-08T05:29:43.389261+0000","last_fullsized":"2023-11-08T05:29:43.389261+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":457,"reported_epoch":455,"state":"active+undersized","last_fresh":"2023-11-08T05:29:43.389822+0000","last_change":"2023-11-08T05:28:13.545468+0000","last_active":"2023-11-08T05:29:43.389822+0000","last_peered":"2023-11-08T05:29:43.389822+0000","last_clean":"2023-11-08T05:28:01.456723+0000","last_became_active":"2023-11-08T05:28:13.545468+0000","last_became_peered":"2023-11-08T05:28:13.545468+0000","last_unstale":"2023-11-08T05:29:43.389822+0000","last_undegraded":"2023-11-08T05:29:43.389822+0000","last_fullsized":"2023-11-08T05:28:13.537207+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_clean_scrub_stamp":"2023-11-08T04:52:45.528796+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":462,"reported_epoch":456,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:29:44.395350+0000","last_change":"2023-11-08T05:29:44.395350+0000","last_active":"2023-11-08T05:29:44.395350+0000","last_peered":"2023-11-08T05:29:44.395350+0000","last_clean":"2023-11-08T05:29:44.395350+0000","last_became_active":"2023-11-08T05:29:44.395012+0000","last_became_peered":"2023-11-08T05:29:44.395012+0000","last_unstale":"2023-11-08T05:29:44.395350+0000","last_undegraded":"2023-11-08T05:29:44.395350+0000","last_fullsized":"2023-11-08T05:29:44.395350+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":456,"reported_epoch":455,"state":"active+undersized","last_fresh":"2023-11-08T05:29:43.389390+0000","last_change":"2023-11-08T05:28:13.563357+0000","last_active":"2023-11-08T05:29:43.389390+0000","last_peered":"2023-11-08T05:29:43.389390+0000","last_clean":"2023-11-08T05:28:01.457691+0000","last_became_active":"2023-11-08T05:28:13.563357+0000","last_became_peered":"2023-11-08T05:28:13.563357+0000","last_unstale":"2023-11-08T05:29:43.389390+0000","last_undegraded":"2023-11-08T05:29:43.389390+0000","last_fullsized":"2023-11-08T05:28:13.541614+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":456,"reported_epoch":455,"state":"active+undersized","last_fresh":"2023-11-08T05:29:43.389311+0000","last_change":"2023-11-08T05:28:13.550672+0000","last_active":"2023-11-08T05:29:43.389311+0000","last_peered":"2023-11-08T05:29:43.389311+0000","last_clean":"2023-11-08T05:28:01.457904+0000","last_became_active":"2023-11-08T05:28:13.550672+0000","last_became_peered":"2023-11-08T05:28:13.550672+0000","last_unstale":"2023-11-08T05:29:43.389311+0000","last_undegraded":"2023-11-08T05:29:43.389311+0000","last_fullsized":"2023-11-08T05:28:13.539595+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":456,"reported_epoch":455,"state":"active+undersized","last_fresh":"2023-11-08T05:29:43.389474+0000","last_change":"2023-11-08T05:28:13.557968+0000","last_active":"2023-11-08T05:29:43.389474+0000","last_peered":"2023-11-08T05:29:43.389474+0000","last_clean":"2023-11-08T05:28:01.458433+0000","last_became_active":"2023-11-08T05:28:13.557968+0000","last_became_peered":"2023-11-08T05:28:13.557968+0000","last_unstale":"2023-11-08T05:29:43.389474+0000","last_undegraded":"2023-11-08T05:29:43.389474+0000","last_fullsized":"2023-11-08T05:28:13.542832+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":1,"acting":1,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":10,"acting":11,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738826,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":645004,"kb_used_data":5316,"kb_used_omap":275,"kb_used_meta":639404,"kb_avail":93726836,"statfs":{"total":96636764160,"available":95976280064,"internally_reserved":0,"allocated":5443584,"data_stored":2678316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":281650,"internal_metadata":654750670},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:59 2023","interfaces":[{"interface":"back","average":{"1min":0.30499999999999999,"5min":0.439,"15min":0.36099999999999999},"min":{"1min":0.17399999999999999,"5min":0.14499999999999999,"15min":0.14499999999999999},"max":{"1min":0.60099999999999998,"5min":4.9379999999999997,"15min":4.9379999999999997},"last":0.379},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.49399999999999999,"15min":0.42399999999999999},"min":{"1min":0.221,"5min":0.19600000000000001,"15min":0.17000000000000001},"max":{"1min":0.63200000000000001,"5min":4.9859999999999998,"15min":4.9859999999999998},"last":0.40899999999999997}]},{"osd":1,"last update":"Wed Nov 8 05:30:03 2023","interfaces":[{"interface":"back","average":{"1min":0.28499999999999998,"5min":0.65100000000000002,"15min":0.47099999999999997},"min":{"1min":0.20100000000000001,"5min":0.153,"15min":0.153},"max":{"1min":0.39800000000000002,"5min":23.242999999999999,"15min":23.242999999999999},"last":0.29999999999999999},{"interface":"front","average":{"1min":0.29499999999999998,"5min":0.63100000000000001,"15min":0.45900000000000002},"min":{"1min":0.224,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":0.436,"5min":23.251000000000001,"15min":23.251000000000001},"last":0.309}]}]},{"osd":0,"up_from":8,"seq":34359738798,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1339228,"kb_used_data":398304,"kb_used_omap":291,"kb_used_meta":936476,"kb_avail":93032612,"statfs":{"total":96636764160,"available":95265394688,"internally_reserved":0,"allocated":407863296,"data_stored":405103900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":298462,"internal_metadata":958951970},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.436,"15min":0.36299999999999999},"min":{"1min":0.19700000000000001,"5min":0.14699999999999999,"15min":0.13700000000000001},"max":{"1min":3.222,"5min":6.5499999999999998,"15min":6.9989999999999997},"last":0.245},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.46300000000000002,"15min":0.42299999999999999},"min":{"1min":0.17799999999999999,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":5.0030000000000001,"5min":6.0629999999999997,"15min":9.6869999999999994},"last":0.28599999999999998}]},{"osd":2,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.51600000000000001,"15min":0.47099999999999997},"min":{"1min":0.21099999999999999,"5min":0.153,"15min":0.153},"max":{"1min":5.7439999999999998,"5min":5.8280000000000003,"15min":9.7639999999999993},"last":0.252},{"interface":"front","average":{"1min":0.497,"5min":0.56200000000000006,"15min":0.42299999999999999},"min":{"1min":0.20399999999999999,"5min":0.187,"15min":0.16800000000000001},"max":{"1min":3.9319999999999999,"5min":8.8040000000000003,"15min":8.8040000000000003},"last":0.29899999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738828,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":650064,"kb_used_data":5320,"kb_used_omap":214,"kb_used_meta":644521,"kb_avail":93721776,"statfs":{"total":96636764160,"available":95971098624,"internally_reserved":0,"allocated":5447680,"data_stored":2679285,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":219335,"internal_metadata":659990329},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:56 2023","interfaces":[{"interface":"back","average":{"1min":0.30399999999999999,"5min":0.34699999999999998,"15min":0.36499999999999999},"min":{"1min":0.23999999999999999,"5min":0.14199999999999999,"15min":0.14199999999999999},"max":{"1min":0.42399999999999999,"5min":1.595,"15min":7.4249999999999998},"last":0.17100000000000001},{"interface":"front","average":{"1min":0.35499999999999998,"5min":0.40899999999999997,"15min":0.376},"min":{"1min":0.23999999999999999,"5min":0.19600000000000001,"15min":0.183},"max":{"1min":0.46999999999999997,"5min":2.6859999999999999,"15min":2.6859999999999999},"last":0.23000000000000001}]},{"osd":2,"last update":"Wed Nov 8 05:29:57 2023","interfaces":[{"interface":"back","average":{"1min":0.28199999999999997,"5min":0.36099999999999999,"15min":0.373},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":0.34399999999999997,"5min":2.2650000000000001,"15min":5.8209999999999997},"last":0.216},{"interface":"front","average":{"1min":0.28599999999999998,"5min":0.31,"15min":0.35799999999999998},"min":{"1min":0.17199999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.35699999999999998,"5min":1.137,"15min":4.2350000000000003},"last":0.23499999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-08T05:30:42.485 INFO:tasks.ceph.cluster2.mgr.x.smithi077.stderr:2023-11-08T05:30:42.483+0000 7f16c17cf700 -1 calc_pg_upmaps abort due to max <= 0 2023-11-08T05:30:43.953 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~146s 2023-11-08T05:30:45.186 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-11-08T05:30:45.481 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T05:30:45.483 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2023-11-08T05:30:45.495 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":1595,"stamp":"2023-11-08T05:30:44.489232+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2634296,"kb_used_data":408940,"kb_used_omap":780,"kb_used_meta":2220403,"kb_avail":280481224,"statfs":{"total":289910292480,"available":287212773376,"internally_reserved":0,"allocated":418754560,"data_stored":410461501,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":799447,"internal_metadata":2273692969},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-536782902,"num_objects":-164,"num_object_clones":0,"num_object_copies":-328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-115,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-164,"num_whiteouts":0,"num_read":-122056,"num_read_kb":-39011942,"num_write":-413642,"num_write_kb":-151705339,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-8,"num_bytes_recovered":-16922624,"num_keys_recovered":-2,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.424817"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":456,"reported_epoch":455,"state":"active+undersized","last_fresh":"2023-11-08T05:29:43.389354+0000","last_change":"2023-11-08T05:28:13.557071+0000","last_active":"2023-11-08T05:29:43.389354+0000","last_peered":"2023-11-08T05:29:43.389354+0000","last_clean":"2023-11-08T05:28:01.458439+0000","last_became_active":"2023-11-08T05:28:13.557071+0000","last_became_peered":"2023-11-08T05:28:13.557071+0000","last_unstale":"2023-11-08T05:29:43.389354+0000","last_undegraded":"2023-11-08T05:29:43.389354+0000","last_fullsized":"2023-11-08T05:28:13.547025+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":18,"reported_epoch":455,"state":"active+undersized","last_fresh":"2023-11-08T05:29:43.389262+0000","last_change":"2023-11-08T05:28:13.559513+0000","last_active":"2023-11-08T05:29:43.389262+0000","last_peered":"2023-11-08T05:29:43.389262+0000","last_clean":"2023-11-08T04:52:46.531964+0000","last_became_active":"2023-11-08T05:28:13.559513+0000","last_became_peered":"2023-11-08T05:28:13.559513+0000","last_unstale":"2023-11-08T05:29:43.389262+0000","last_undegraded":"2023-11-08T05:29:43.389262+0000","last_fullsized":"2023-11-08T05:28:13.540852+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":453,"reported_epoch":455,"state":"active+clean","last_fresh":"2023-11-08T05:29:43.390991+0000","last_change":"2023-11-08T04:52:47.546100+0000","last_active":"2023-11-08T05:29:43.390991+0000","last_peered":"2023-11-08T05:29:43.390991+0000","last_clean":"2023-11-08T05:29:43.390991+0000","last_became_active":"2023-11-08T04:52:47.545840+0000","last_became_peered":"2023-11-08T04:52:47.545840+0000","last_unstale":"2023-11-08T05:29:43.390991+0000","last_undegraded":"2023-11-08T05:29:43.390991+0000","last_fullsized":"2023-11-08T05:29:43.390991+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":451,"reported_epoch":455,"state":"active+clean","last_fresh":"2023-11-08T05:29:43.389261+0000","last_change":"2023-11-08T04:52:47.548597+0000","last_active":"2023-11-08T05:29:43.389261+0000","last_peered":"2023-11-08T05:29:43.389261+0000","last_clean":"2023-11-08T05:29:43.389261+0000","last_became_active":"2023-11-08T04:52:47.548294+0000","last_became_peered":"2023-11-08T04:52:47.548294+0000","last_unstale":"2023-11-08T05:29:43.389261+0000","last_undegraded":"2023-11-08T05:29:43.389261+0000","last_fullsized":"2023-11-08T05:29:43.389261+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":457,"reported_epoch":455,"state":"active+undersized","last_fresh":"2023-11-08T05:29:43.389822+0000","last_change":"2023-11-08T05:28:13.545468+0000","last_active":"2023-11-08T05:29:43.389822+0000","last_peered":"2023-11-08T05:29:43.389822+0000","last_clean":"2023-11-08T05:28:01.456723+0000","last_became_active":"2023-11-08T05:28:13.545468+0000","last_became_peered":"2023-11-08T05:28:13.545468+0000","last_unstale":"2023-11-08T05:29:43.389822+0000","last_undegraded":"2023-11-08T05:29:43.389822+0000","last_fullsized":"2023-11-08T05:28:13.537207+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_clean_scrub_stamp":"2023-11-08T04:52:45.528796+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":462,"reported_epoch":456,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:29:44.395350+0000","last_change":"2023-11-08T05:29:44.395350+0000","last_active":"2023-11-08T05:29:44.395350+0000","last_peered":"2023-11-08T05:29:44.395350+0000","last_clean":"2023-11-08T05:29:44.395350+0000","last_became_active":"2023-11-08T05:29:44.395012+0000","last_became_peered":"2023-11-08T05:29:44.395012+0000","last_unstale":"2023-11-08T05:29:44.395350+0000","last_undegraded":"2023-11-08T05:29:44.395350+0000","last_fullsized":"2023-11-08T05:29:44.395350+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":456,"reported_epoch":455,"state":"active+undersized","last_fresh":"2023-11-08T05:29:43.389390+0000","last_change":"2023-11-08T05:28:13.563357+0000","last_active":"2023-11-08T05:29:43.389390+0000","last_peered":"2023-11-08T05:29:43.389390+0000","last_clean":"2023-11-08T05:28:01.457691+0000","last_became_active":"2023-11-08T05:28:13.563357+0000","last_became_peered":"2023-11-08T05:28:13.563357+0000","last_unstale":"2023-11-08T05:29:43.389390+0000","last_undegraded":"2023-11-08T05:29:43.389390+0000","last_fullsized":"2023-11-08T05:28:13.541614+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":456,"reported_epoch":455,"state":"active+undersized","last_fresh":"2023-11-08T05:29:43.389311+0000","last_change":"2023-11-08T05:28:13.550672+0000","last_active":"2023-11-08T05:29:43.389311+0000","last_peered":"2023-11-08T05:29:43.389311+0000","last_clean":"2023-11-08T05:28:01.457904+0000","last_became_active":"2023-11-08T05:28:13.550672+0000","last_became_peered":"2023-11-08T05:28:13.550672+0000","last_unstale":"2023-11-08T05:29:43.389311+0000","last_undegraded":"2023-11-08T05:29:43.389311+0000","last_fullsized":"2023-11-08T05:28:13.539595+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":456,"reported_epoch":455,"state":"active+undersized","last_fresh":"2023-11-08T05:29:43.389474+0000","last_change":"2023-11-08T05:28:13.557968+0000","last_active":"2023-11-08T05:29:43.389474+0000","last_peered":"2023-11-08T05:29:43.389474+0000","last_clean":"2023-11-08T05:28:01.458433+0000","last_became_active":"2023-11-08T05:28:13.557968+0000","last_became_peered":"2023-11-08T05:28:13.557968+0000","last_unstale":"2023-11-08T05:29:43.389474+0000","last_undegraded":"2023-11-08T05:29:43.389474+0000","last_fullsized":"2023-11-08T05:28:13.542832+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":1,"acting":1,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":10,"acting":11,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738826,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":645004,"kb_used_data":5316,"kb_used_omap":275,"kb_used_meta":639404,"kb_avail":93726836,"statfs":{"total":96636764160,"available":95976280064,"internally_reserved":0,"allocated":5443584,"data_stored":2678316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":281650,"internal_metadata":654750670},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:59 2023","interfaces":[{"interface":"back","average":{"1min":0.30499999999999999,"5min":0.439,"15min":0.36099999999999999},"min":{"1min":0.17399999999999999,"5min":0.14499999999999999,"15min":0.14499999999999999},"max":{"1min":0.60099999999999998,"5min":4.9379999999999997,"15min":4.9379999999999997},"last":0.379},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.49399999999999999,"15min":0.42399999999999999},"min":{"1min":0.221,"5min":0.19600000000000001,"15min":0.17000000000000001},"max":{"1min":0.63200000000000001,"5min":4.9859999999999998,"15min":4.9859999999999998},"last":0.40899999999999997}]},{"osd":1,"last update":"Wed Nov 8 05:30:03 2023","interfaces":[{"interface":"back","average":{"1min":0.28499999999999998,"5min":0.65100000000000002,"15min":0.47099999999999997},"min":{"1min":0.20100000000000001,"5min":0.153,"15min":0.153},"max":{"1min":0.39800000000000002,"5min":23.242999999999999,"15min":23.242999999999999},"last":0.29999999999999999},{"interface":"front","average":{"1min":0.29499999999999998,"5min":0.63100000000000001,"15min":0.45900000000000002},"min":{"1min":0.224,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":0.436,"5min":23.251000000000001,"15min":23.251000000000001},"last":0.309}]}]},{"osd":0,"up_from":8,"seq":34359738798,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1339228,"kb_used_data":398304,"kb_used_omap":291,"kb_used_meta":936476,"kb_avail":93032612,"statfs":{"total":96636764160,"available":95265394688,"internally_reserved":0,"allocated":407863296,"data_stored":405103900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":298462,"internal_metadata":958951970},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.436,"15min":0.36299999999999999},"min":{"1min":0.19700000000000001,"5min":0.14699999999999999,"15min":0.13700000000000001},"max":{"1min":3.222,"5min":6.5499999999999998,"15min":6.9989999999999997},"last":0.245},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.46300000000000002,"15min":0.42299999999999999},"min":{"1min":0.17799999999999999,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":5.0030000000000001,"5min":6.0629999999999997,"15min":9.6869999999999994},"last":0.28599999999999998}]},{"osd":2,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.51600000000000001,"15min":0.47099999999999997},"min":{"1min":0.21099999999999999,"5min":0.153,"15min":0.153},"max":{"1min":5.7439999999999998,"5min":5.8280000000000003,"15min":9.7639999999999993},"last":0.252},{"interface":"front","average":{"1min":0.497,"5min":0.56200000000000006,"15min":0.42299999999999999},"min":{"1min":0.20399999999999999,"5min":0.187,"15min":0.16800000000000001},"max":{"1min":3.9319999999999999,"5min":8.8040000000000003,"15min":8.8040000000000003},"last":0.29899999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738828,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":650064,"kb_used_data":5320,"kb_used_omap":214,"kb_used_meta":644521,"kb_avail":93721776,"statfs":{"total":96636764160,"available":95971098624,"internally_reserved":0,"allocated":5447680,"data_stored":2679285,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":219335,"internal_metadata":659990329},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:56 2023","interfaces":[{"interface":"back","average":{"1min":0.30399999999999999,"5min":0.34699999999999998,"15min":0.36499999999999999},"min":{"1min":0.23999999999999999,"5min":0.14199999999999999,"15min":0.14199999999999999},"max":{"1min":0.42399999999999999,"5min":1.595,"15min":7.4249999999999998},"last":0.17100000000000001},{"interface":"front","average":{"1min":0.35499999999999998,"5min":0.40899999999999997,"15min":0.376},"min":{"1min":0.23999999999999999,"5min":0.19600000000000001,"15min":0.183},"max":{"1min":0.46999999999999997,"5min":2.6859999999999999,"15min":2.6859999999999999},"last":0.23000000000000001}]},{"osd":2,"last update":"Wed Nov 8 05:29:57 2023","interfaces":[{"interface":"back","average":{"1min":0.28199999999999997,"5min":0.36099999999999999,"15min":0.373},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":0.34399999999999997,"5min":2.2650000000000001,"15min":5.8209999999999997},"last":0.216},{"interface":"front","average":{"1min":0.28599999999999998,"5min":0.31,"15min":0.35799999999999998},"min":{"1min":0.17199999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.35699999999999998,"5min":1.137,"15min":4.2350000000000003},"last":0.23499999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-08T05:30:45.496 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-11-08T05:30:45.778 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T05:30:45.779 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2023-11-08T05:30:45.790 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":1596,"stamp":"2023-11-08T05:30:45.489178+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":10,"acting":13,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2634320,"kb_used_data":408964,"kb_used_omap":780,"kb_used_meta":2220403,"kb_avail":280481200,"statfs":{"total":289910292480,"available":287212748800,"internally_reserved":0,"allocated":418779136,"data_stored":410474405,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":799447,"internal_metadata":2273692969},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-536782902,"num_objects":-164,"num_object_clones":0,"num_object_copies":-328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-115,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-164,"num_whiteouts":0,"num_read":-122056,"num_read_kb":-39011942,"num_write":-413642,"num_write_kb":-151705339,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-8,"num_bytes_recovered":-16922624,"num_keys_recovered":-2,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.424817"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":456,"reported_epoch":455,"state":"active+undersized","last_fresh":"2023-11-08T05:29:43.389354+0000","last_change":"2023-11-08T05:28:13.557071+0000","last_active":"2023-11-08T05:29:43.389354+0000","last_peered":"2023-11-08T05:29:43.389354+0000","last_clean":"2023-11-08T05:28:01.458439+0000","last_became_active":"2023-11-08T05:28:13.557071+0000","last_became_peered":"2023-11-08T05:28:13.557071+0000","last_unstale":"2023-11-08T05:29:43.389354+0000","last_undegraded":"2023-11-08T05:29:43.389354+0000","last_fullsized":"2023-11-08T05:28:13.547025+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":3,"reported_epoch":458,"state":"remapped+peering","last_fresh":"2023-11-08T05:30:44.492310+0000","last_change":"2023-11-08T05:30:44.492310+0000","last_active":"0.000000","last_peered":"0.000000","last_clean":"0.000000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-08T05:30:44.492310+0000","last_undegraded":"2023-11-08T05:30:44.492310+0000","last_fullsized":"2023-11-08T05:30:44.492310+0000","mapping_epoch":458,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":457,"reported_epoch":458,"state":"remapped+peering","last_fresh":"2023-11-08T05:30:44.492151+0000","last_change":"2023-11-08T05:30:44.492368+0000","last_active":"2023-11-08T05:30:43.487008+0000","last_peered":"2023-11-08T05:29:44.393948+0000","last_clean":"2023-11-08T05:29:44.393948+0000","last_became_active":"2023-11-08T04:52:47.545840+0000","last_became_peered":"2023-11-08T04:52:47.545840+0000","last_unstale":"2023-11-08T05:30:44.492151+0000","last_undegraded":"2023-11-08T05:30:44.492151+0000","last_fullsized":"2023-11-08T05:30:44.492151+0000","mapping_epoch":458,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":451,"reported_epoch":455,"state":"active+clean","last_fresh":"2023-11-08T05:29:43.389261+0000","last_change":"2023-11-08T04:52:47.548597+0000","last_active":"2023-11-08T05:29:43.389261+0000","last_peered":"2023-11-08T05:29:43.389261+0000","last_clean":"2023-11-08T05:29:43.389261+0000","last_became_active":"2023-11-08T04:52:47.548294+0000","last_became_peered":"2023-11-08T04:52:47.548294+0000","last_unstale":"2023-11-08T05:29:43.389261+0000","last_undegraded":"2023-11-08T05:29:43.389261+0000","last_fullsized":"2023-11-08T05:29:43.389261+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":457,"reported_epoch":455,"state":"active+undersized","last_fresh":"2023-11-08T05:29:43.389822+0000","last_change":"2023-11-08T05:28:13.545468+0000","last_active":"2023-11-08T05:29:43.389822+0000","last_peered":"2023-11-08T05:29:43.389822+0000","last_clean":"2023-11-08T05:28:01.456723+0000","last_became_active":"2023-11-08T05:28:13.545468+0000","last_became_peered":"2023-11-08T05:28:13.545468+0000","last_unstale":"2023-11-08T05:29:43.389822+0000","last_undegraded":"2023-11-08T05:29:43.389822+0000","last_fullsized":"2023-11-08T05:28:13.537207+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_clean_scrub_stamp":"2023-11-08T04:52:45.528796+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":464,"reported_epoch":457,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:43.487361+0000","last_change":"2023-11-08T05:29:44.395350+0000","last_active":"2023-11-08T05:30:43.487361+0000","last_peered":"2023-11-08T05:30:43.487361+0000","last_clean":"2023-11-08T05:30:43.487361+0000","last_became_active":"2023-11-08T05:29:44.395012+0000","last_became_peered":"2023-11-08T05:29:44.395012+0000","last_unstale":"2023-11-08T05:30:43.487361+0000","last_undegraded":"2023-11-08T05:30:43.487361+0000","last_fullsized":"2023-11-08T05:30:43.487361+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":456,"reported_epoch":455,"state":"active+undersized","last_fresh":"2023-11-08T05:29:43.389390+0000","last_change":"2023-11-08T05:28:13.563357+0000","last_active":"2023-11-08T05:29:43.389390+0000","last_peered":"2023-11-08T05:29:43.389390+0000","last_clean":"2023-11-08T05:28:01.457691+0000","last_became_active":"2023-11-08T05:28:13.563357+0000","last_became_peered":"2023-11-08T05:28:13.563357+0000","last_unstale":"2023-11-08T05:29:43.389390+0000","last_undegraded":"2023-11-08T05:29:43.389390+0000","last_fullsized":"2023-11-08T05:28:13.541614+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":456,"reported_epoch":455,"state":"active+undersized","last_fresh":"2023-11-08T05:29:43.389311+0000","last_change":"2023-11-08T05:28:13.550672+0000","last_active":"2023-11-08T05:29:43.389311+0000","last_peered":"2023-11-08T05:29:43.389311+0000","last_clean":"2023-11-08T05:28:01.457904+0000","last_became_active":"2023-11-08T05:28:13.550672+0000","last_became_peered":"2023-11-08T05:28:13.550672+0000","last_unstale":"2023-11-08T05:29:43.389311+0000","last_undegraded":"2023-11-08T05:29:43.389311+0000","last_fullsized":"2023-11-08T05:28:13.539595+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":456,"reported_epoch":455,"state":"active+undersized","last_fresh":"2023-11-08T05:29:43.389474+0000","last_change":"2023-11-08T05:28:13.557968+0000","last_active":"2023-11-08T05:29:43.389474+0000","last_peered":"2023-11-08T05:29:43.389474+0000","last_clean":"2023-11-08T05:28:01.458433+0000","last_became_active":"2023-11-08T05:28:13.557968+0000","last_became_peered":"2023-11-08T05:28:13.557968+0000","last_unstale":"2023-11-08T05:29:43.389474+0000","last_undegraded":"2023-11-08T05:29:43.389474+0000","last_fullsized":"2023-11-08T05:28:13.542832+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":1,"acting":1,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":12,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738827,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":645028,"kb_used_data":5340,"kb_used_omap":275,"kb_used_meta":639404,"kb_avail":93726812,"statfs":{"total":96636764160,"available":95976255488,"internally_reserved":0,"allocated":5468160,"data_stored":2691220,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":281650,"internal_metadata":654750670},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:59 2023","interfaces":[{"interface":"back","average":{"1min":0.30499999999999999,"5min":0.439,"15min":0.36099999999999999},"min":{"1min":0.17399999999999999,"5min":0.14499999999999999,"15min":0.14499999999999999},"max":{"1min":0.60099999999999998,"5min":4.9379999999999997,"15min":4.9379999999999997},"last":0.379},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.49399999999999999,"15min":0.42399999999999999},"min":{"1min":0.221,"5min":0.19600000000000001,"15min":0.17000000000000001},"max":{"1min":0.63200000000000001,"5min":4.9859999999999998,"15min":4.9859999999999998},"last":0.40899999999999997}]},{"osd":1,"last update":"Wed Nov 8 05:30:03 2023","interfaces":[{"interface":"back","average":{"1min":0.28499999999999998,"5min":0.65100000000000002,"15min":0.47099999999999997},"min":{"1min":0.20100000000000001,"5min":0.153,"15min":0.153},"max":{"1min":0.39800000000000002,"5min":23.242999999999999,"15min":23.242999999999999},"last":0.252},{"interface":"front","average":{"1min":0.29499999999999998,"5min":0.63100000000000001,"15min":0.45900000000000002},"min":{"1min":0.224,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":0.436,"5min":23.251000000000001,"15min":23.251000000000001},"last":0.23999999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738798,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1339228,"kb_used_data":398304,"kb_used_omap":291,"kb_used_meta":936476,"kb_avail":93032612,"statfs":{"total":96636764160,"available":95265394688,"internally_reserved":0,"allocated":407863296,"data_stored":405103900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":298462,"internal_metadata":958951970},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.436,"15min":0.36299999999999999},"min":{"1min":0.19700000000000001,"5min":0.14699999999999999,"15min":0.13700000000000001},"max":{"1min":3.222,"5min":6.5499999999999998,"15min":6.9989999999999997},"last":0.245},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.46300000000000002,"15min":0.42299999999999999},"min":{"1min":0.17799999999999999,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":5.0030000000000001,"5min":6.0629999999999997,"15min":9.6869999999999994},"last":0.28599999999999998}]},{"osd":2,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.51600000000000001,"15min":0.47099999999999997},"min":{"1min":0.21099999999999999,"5min":0.153,"15min":0.153},"max":{"1min":5.7439999999999998,"5min":5.8280000000000003,"15min":9.7639999999999993},"last":0.252},{"interface":"front","average":{"1min":0.497,"5min":0.56200000000000006,"15min":0.42299999999999999},"min":{"1min":0.20399999999999999,"5min":0.187,"15min":0.16800000000000001},"max":{"1min":3.9319999999999999,"5min":8.8040000000000003,"15min":8.8040000000000003},"last":0.29899999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738828,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":650064,"kb_used_data":5320,"kb_used_omap":214,"kb_used_meta":644521,"kb_avail":93721776,"statfs":{"total":96636764160,"available":95971098624,"internally_reserved":0,"allocated":5447680,"data_stored":2679285,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":219335,"internal_metadata":659990329},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:56 2023","interfaces":[{"interface":"back","average":{"1min":0.30399999999999999,"5min":0.34699999999999998,"15min":0.36499999999999999},"min":{"1min":0.23999999999999999,"5min":0.14199999999999999,"15min":0.14199999999999999},"max":{"1min":0.42399999999999999,"5min":1.595,"15min":7.4249999999999998},"last":0.17100000000000001},{"interface":"front","average":{"1min":0.35499999999999998,"5min":0.40899999999999997,"15min":0.376},"min":{"1min":0.23999999999999999,"5min":0.19600000000000001,"15min":0.183},"max":{"1min":0.46999999999999997,"5min":2.6859999999999999,"15min":2.6859999999999999},"last":0.23000000000000001}]},{"osd":2,"last update":"Wed Nov 8 05:29:57 2023","interfaces":[{"interface":"back","average":{"1min":0.28199999999999997,"5min":0.36099999999999999,"15min":0.373},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":0.34399999999999997,"5min":2.2650000000000001,"15min":5.8209999999999997},"last":0.216},{"interface":"front","average":{"1min":0.28599999999999998,"5min":0.31,"15min":0.35799999999999998},"min":{"1min":0.17199999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.35699999999999998,"5min":1.137,"15min":4.2350000000000003},"last":0.23499999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-08T05:30:45.792 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-11-08T05:30:45.792 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 456, 'reported_epoch': 455, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:29:43.389354+0000', 'last_change': '2023-11-08T05:28:13.557071+0000', 'last_active': '2023-11-08T05:29:43.389354+0000', 'last_peered': '2023-11-08T05:29:43.389354+0000', 'last_clean': '2023-11-08T05:28:01.458439+0000', 'last_became_active': '2023-11-08T05:28:13.557071+0000', 'last_became_peered': '2023-11-08T05:28:13.557071+0000', 'last_unstale': '2023-11-08T05:29:43.389354+0000', 'last_undegraded': '2023-11-08T05:29:43.389354+0000', 'last_fullsized': '2023-11-08T05:28:13.547025+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:30:45.793 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-11-08T05:30:45.793 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 3, 'reported_epoch': 458, 'state': 'remapped+peering', 'last_fresh': '2023-11-08T05:30:44.492310+0000', 'last_change': '2023-11-08T05:30:44.492310+0000', 'last_active': '0.000000', 'last_peered': '0.000000', 'last_clean': '0.000000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-08T05:30:44.492310+0000', 'last_undegraded': '2023-11-08T05:30:44.492310+0000', 'last_fullsized': '2023-11-08T05:30:44.492310+0000', 'mapping_epoch': 458, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2], 'acting': [2, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [1], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2023-11-08T05:30:45.793 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.1 is not active+clean 2023-11-08T05:30:45.793 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.1', 'version': "0'0", 'reported_seq': 457, 'reported_epoch': 458, 'state': 'remapped+peering', 'last_fresh': '2023-11-08T05:30:44.492151+0000', 'last_change': '2023-11-08T05:30:44.492368+0000', 'last_active': '2023-11-08T05:30:43.487008+0000', 'last_peered': '2023-11-08T05:29:44.393948+0000', 'last_clean': '2023-11-08T05:29:44.393948+0000', 'last_became_active': '2023-11-08T04:52:47.545840+0000', 'last_became_peered': '2023-11-08T04:52:47.545840+0000', 'last_unstale': '2023-11-08T05:30:44.492151+0000', 'last_undegraded': '2023-11-08T05:30:44.492151+0000', 'last_fullsized': '2023-11-08T05:30:44.492151+0000', 'mapping_epoch': 458, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2], 'acting': [2, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [1], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2023-11-08T05:30:45.794 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-11-08T05:30:45.794 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "0'0", 'reported_seq': 457, 'reported_epoch': 455, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:29:43.389822+0000', 'last_change': '2023-11-08T05:28:13.545468+0000', 'last_active': '2023-11-08T05:29:43.389822+0000', 'last_peered': '2023-11-08T05:29:43.389822+0000', 'last_clean': '2023-11-08T05:28:01.456723+0000', 'last_became_active': '2023-11-08T05:28:13.545468+0000', 'last_became_peered': '2023-11-08T05:28:13.545468+0000', 'last_unstale': '2023-11-08T05:29:43.389822+0000', 'last_undegraded': '2023-11-08T05:29:43.389822+0000', 'last_fullsized': '2023-11-08T05:28:13.537207+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 9, 'last_epoch_clean': 10, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:45.528796+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:45.528796+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:45.528796+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:30:45.794 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.0 is not active+clean 2023-11-08T05:30:45.795 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.0', 'version': "0'0", 'reported_seq': 464, 'reported_epoch': 457, 'state': 'active+clean+remapped', 'last_fresh': '2023-11-08T05:30:43.487361+0000', 'last_change': '2023-11-08T05:29:44.395350+0000', 'last_active': '2023-11-08T05:30:43.487361+0000', 'last_peered': '2023-11-08T05:30:43.487361+0000', 'last_clean': '2023-11-08T05:30:43.487361+0000', 'last_became_active': '2023-11-08T05:29:44.395012+0000', 'last_became_peered': '2023-11-08T05:29:44.395012+0000', 'last_unstale': '2023-11-08T05:30:43.487361+0000', 'last_undegraded': '2023-11-08T05:30:43.487361+0000', 'last_fullsized': '2023-11-08T05:30:43.487361+0000', 'mapping_epoch': 455, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 456, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2], 'acting': [2, 1], 'avail_no_missing': ['2', '1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2023-11-08T05:30:45.795 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-11-08T05:30:45.795 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 456, 'reported_epoch': 455, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:29:43.389390+0000', 'last_change': '2023-11-08T05:28:13.563357+0000', 'last_active': '2023-11-08T05:29:43.389390+0000', 'last_peered': '2023-11-08T05:29:43.389390+0000', 'last_clean': '2023-11-08T05:28:01.457691+0000', 'last_became_active': '2023-11-08T05:28:13.563357+0000', 'last_became_peered': '2023-11-08T05:28:13.563357+0000', 'last_unstale': '2023-11-08T05:29:43.389390+0000', 'last_undegraded': '2023-11-08T05:29:43.389390+0000', 'last_fullsized': '2023-11-08T05:28:13.541614+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:30:45.796 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-11-08T05:30:45.797 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 456, 'reported_epoch': 455, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:29:43.389311+0000', 'last_change': '2023-11-08T05:28:13.550672+0000', 'last_active': '2023-11-08T05:29:43.389311+0000', 'last_peered': '2023-11-08T05:29:43.389311+0000', 'last_clean': '2023-11-08T05:28:01.457904+0000', 'last_became_active': '2023-11-08T05:28:13.550672+0000', 'last_became_peered': '2023-11-08T05:28:13.550672+0000', 'last_unstale': '2023-11-08T05:29:43.389311+0000', 'last_undegraded': '2023-11-08T05:29:43.389311+0000', 'last_fullsized': '2023-11-08T05:28:13.539595+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:30:45.797 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-11-08T05:30:45.797 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 456, 'reported_epoch': 455, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:29:43.389474+0000', 'last_change': '2023-11-08T05:28:13.557968+0000', 'last_active': '2023-11-08T05:29:43.389474+0000', 'last_peered': '2023-11-08T05:29:43.389474+0000', 'last_clean': '2023-11-08T05:28:01.458433+0000', 'last_became_active': '2023-11-08T05:28:13.557968+0000', 'last_became_peered': '2023-11-08T05:28:13.557968+0000', 'last_unstale': '2023-11-08T05:29:43.389474+0000', 'last_undegraded': '2023-11-08T05:29:43.389474+0000', 'last_fullsized': '2023-11-08T05:28:13.542832+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:30:45.798 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-11-08T05:30:46.143 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T05:30:46.154 INFO:teuthology.orchestra.run.smithi087.stdout:{"fsid":"ab4cbc06-449d-497d-b513-220d9db2b590","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 6 pgs undersized","count":6},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2288,"monmap":{"epoch":1,"min_mon_release_name":"pacific","num_mons":1},"osdmap":{"epoch":459,"num_osds":3,"num_up_osds":2,"osd_up_since":1699421292,"num_in_osds":3,"osd_in_since":1699419158,"num_remapped_pgs":3},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":6},{"state_name":"active+clean","count":2},{"state_name":"active+clean+remapped","count":1}],"num_pgs":9,"num_pools":2,"num_objects":0,"data_bytes":0,"bytes_used":2697519104,"bytes_avail":287212773376,"bytes_total":289910292480},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":41,"modified":"2023-11-08T05:29:32.044544+0000","services":{}},"progress_events":{}} 2023-11-08T05:30:46.156 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-11-08T05:30:46.157 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-11-08T05:30:46.440 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T05:30:46.441 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2023-11-08T05:30:46.452 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":1597,"stamp":"2023-11-08T05:30:46.060175+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":10,"acting":13,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2634344,"kb_used_data":408988,"kb_used_omap":780,"kb_used_meta":2220403,"kb_avail":280481176,"statfs":{"total":289910292480,"available":287212724224,"internally_reserved":0,"allocated":418803712,"data_stored":410487309,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":799447,"internal_metadata":2273692969},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-536782902,"num_objects":-164,"num_object_clones":0,"num_object_copies":-328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-115,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-164,"num_whiteouts":0,"num_read":-122056,"num_read_kb":-39011942,"num_write":-413642,"num_write_kb":-151705339,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-8,"num_bytes_recovered":-16922624,"num_keys_recovered":-2,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.424817"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":458,"reported_epoch":457,"state":"active+undersized","last_fresh":"2023-11-08T05:30:43.484884+0000","last_change":"2023-11-08T05:28:13.557071+0000","last_active":"2023-11-08T05:30:43.484884+0000","last_peered":"2023-11-08T05:30:43.484884+0000","last_clean":"2023-11-08T05:28:01.458439+0000","last_became_active":"2023-11-08T05:28:13.557071+0000","last_became_peered":"2023-11-08T05:28:13.557071+0000","last_unstale":"2023-11-08T05:30:43.484884+0000","last_undegraded":"2023-11-08T05:30:43.484884+0000","last_fullsized":"2023-11-08T05:28:13.547025+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":3,"reported_epoch":458,"state":"remapped+peering","last_fresh":"2023-11-08T05:30:44.492310+0000","last_change":"2023-11-08T05:30:44.492310+0000","last_active":"0.000000","last_peered":"0.000000","last_clean":"0.000000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-08T05:30:44.492310+0000","last_undegraded":"2023-11-08T05:30:44.492310+0000","last_fullsized":"2023-11-08T05:30:44.492310+0000","mapping_epoch":458,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":457,"reported_epoch":458,"state":"remapped+peering","last_fresh":"2023-11-08T05:30:44.492151+0000","last_change":"2023-11-08T05:30:44.492368+0000","last_active":"2023-11-08T05:30:43.487008+0000","last_peered":"2023-11-08T05:29:44.393948+0000","last_clean":"2023-11-08T05:29:44.393948+0000","last_became_active":"2023-11-08T04:52:47.545840+0000","last_became_peered":"2023-11-08T04:52:47.545840+0000","last_unstale":"2023-11-08T05:30:44.492151+0000","last_undegraded":"2023-11-08T05:30:44.492151+0000","last_fullsized":"2023-11-08T05:30:44.492151+0000","mapping_epoch":458,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":453,"reported_epoch":457,"state":"active+clean","last_fresh":"2023-11-08T05:30:43.484906+0000","last_change":"2023-11-08T04:52:47.548597+0000","last_active":"2023-11-08T05:30:43.484906+0000","last_peered":"2023-11-08T05:30:43.484906+0000","last_clean":"2023-11-08T05:30:43.484906+0000","last_became_active":"2023-11-08T04:52:47.548294+0000","last_became_peered":"2023-11-08T04:52:47.548294+0000","last_unstale":"2023-11-08T05:30:43.484906+0000","last_undegraded":"2023-11-08T05:30:43.484906+0000","last_fullsized":"2023-11-08T05:30:43.484906+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":459,"reported_epoch":457,"state":"active+undersized","last_fresh":"2023-11-08T05:30:43.484799+0000","last_change":"2023-11-08T05:28:13.545468+0000","last_active":"2023-11-08T05:30:43.484799+0000","last_peered":"2023-11-08T05:30:43.484799+0000","last_clean":"2023-11-08T05:28:01.456723+0000","last_became_active":"2023-11-08T05:28:13.545468+0000","last_became_peered":"2023-11-08T05:28:13.545468+0000","last_unstale":"2023-11-08T05:30:43.484799+0000","last_undegraded":"2023-11-08T05:30:43.484799+0000","last_fullsized":"2023-11-08T05:28:13.537207+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_clean_scrub_stamp":"2023-11-08T04:52:45.528796+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":464,"reported_epoch":457,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:43.487361+0000","last_change":"2023-11-08T05:29:44.395350+0000","last_active":"2023-11-08T05:30:43.487361+0000","last_peered":"2023-11-08T05:30:43.487361+0000","last_clean":"2023-11-08T05:30:43.487361+0000","last_became_active":"2023-11-08T05:29:44.395012+0000","last_became_peered":"2023-11-08T05:29:44.395012+0000","last_unstale":"2023-11-08T05:30:43.487361+0000","last_undegraded":"2023-11-08T05:30:43.487361+0000","last_fullsized":"2023-11-08T05:30:43.487361+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":458,"reported_epoch":457,"state":"active+undersized","last_fresh":"2023-11-08T05:30:43.484912+0000","last_change":"2023-11-08T05:28:13.563357+0000","last_active":"2023-11-08T05:30:43.484912+0000","last_peered":"2023-11-08T05:30:43.484912+0000","last_clean":"2023-11-08T05:28:01.457691+0000","last_became_active":"2023-11-08T05:28:13.563357+0000","last_became_peered":"2023-11-08T05:28:13.563357+0000","last_unstale":"2023-11-08T05:30:43.484912+0000","last_undegraded":"2023-11-08T05:30:43.484912+0000","last_fullsized":"2023-11-08T05:28:13.541614+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":458,"reported_epoch":457,"state":"active+undersized","last_fresh":"2023-11-08T05:30:43.485032+0000","last_change":"2023-11-08T05:28:13.550672+0000","last_active":"2023-11-08T05:30:43.485032+0000","last_peered":"2023-11-08T05:30:43.485032+0000","last_clean":"2023-11-08T05:28:01.457904+0000","last_became_active":"2023-11-08T05:28:13.550672+0000","last_became_peered":"2023-11-08T05:28:13.550672+0000","last_unstale":"2023-11-08T05:30:43.485032+0000","last_undegraded":"2023-11-08T05:30:43.485032+0000","last_fullsized":"2023-11-08T05:28:13.539595+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":458,"reported_epoch":457,"state":"active+undersized","last_fresh":"2023-11-08T05:30:43.484992+0000","last_change":"2023-11-08T05:28:13.557968+0000","last_active":"2023-11-08T05:30:43.484992+0000","last_peered":"2023-11-08T05:30:43.484992+0000","last_clean":"2023-11-08T05:28:01.458433+0000","last_became_active":"2023-11-08T05:28:13.557968+0000","last_became_peered":"2023-11-08T05:28:13.557968+0000","last_unstale":"2023-11-08T05:30:43.484992+0000","last_undegraded":"2023-11-08T05:30:43.484992+0000","last_fullsized":"2023-11-08T05:28:13.542832+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":1,"acting":1,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":12,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738827,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":645028,"kb_used_data":5340,"kb_used_omap":275,"kb_used_meta":639404,"kb_avail":93726812,"statfs":{"total":96636764160,"available":95976255488,"internally_reserved":0,"allocated":5468160,"data_stored":2691220,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":281650,"internal_metadata":654750670},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:59 2023","interfaces":[{"interface":"back","average":{"1min":0.30499999999999999,"5min":0.439,"15min":0.36099999999999999},"min":{"1min":0.17399999999999999,"5min":0.14499999999999999,"15min":0.14499999999999999},"max":{"1min":0.60099999999999998,"5min":4.9379999999999997,"15min":4.9379999999999997},"last":0.379},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.49399999999999999,"15min":0.42399999999999999},"min":{"1min":0.221,"5min":0.19600000000000001,"15min":0.17000000000000001},"max":{"1min":0.63200000000000001,"5min":4.9859999999999998,"15min":4.9859999999999998},"last":0.40899999999999997}]},{"osd":1,"last update":"Wed Nov 8 05:30:03 2023","interfaces":[{"interface":"back","average":{"1min":0.28499999999999998,"5min":0.65100000000000002,"15min":0.47099999999999997},"min":{"1min":0.20100000000000001,"5min":0.153,"15min":0.153},"max":{"1min":0.39800000000000002,"5min":23.242999999999999,"15min":23.242999999999999},"last":0.252},{"interface":"front","average":{"1min":0.29499999999999998,"5min":0.63100000000000001,"15min":0.45900000000000002},"min":{"1min":0.224,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":0.436,"5min":23.251000000000001,"15min":23.251000000000001},"last":0.23999999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738798,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1339228,"kb_used_data":398304,"kb_used_omap":291,"kb_used_meta":936476,"kb_avail":93032612,"statfs":{"total":96636764160,"available":95265394688,"internally_reserved":0,"allocated":407863296,"data_stored":405103900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":298462,"internal_metadata":958951970},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.436,"15min":0.36299999999999999},"min":{"1min":0.19700000000000001,"5min":0.14699999999999999,"15min":0.13700000000000001},"max":{"1min":3.222,"5min":6.5499999999999998,"15min":6.9989999999999997},"last":0.245},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.46300000000000002,"15min":0.42299999999999999},"min":{"1min":0.17799999999999999,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":5.0030000000000001,"5min":6.0629999999999997,"15min":9.6869999999999994},"last":0.28599999999999998}]},{"osd":2,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.51600000000000001,"15min":0.47099999999999997},"min":{"1min":0.21099999999999999,"5min":0.153,"15min":0.153},"max":{"1min":5.7439999999999998,"5min":5.8280000000000003,"15min":9.7639999999999993},"last":0.252},{"interface":"front","average":{"1min":0.497,"5min":0.56200000000000006,"15min":0.42299999999999999},"min":{"1min":0.20399999999999999,"5min":0.187,"15min":0.16800000000000001},"max":{"1min":3.9319999999999999,"5min":8.8040000000000003,"15min":8.8040000000000003},"last":0.29899999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738829,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":650088,"kb_used_data":5344,"kb_used_omap":214,"kb_used_meta":644521,"kb_avail":93721752,"statfs":{"total":96636764160,"available":95971074048,"internally_reserved":0,"allocated":5472256,"data_stored":2692189,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":219335,"internal_metadata":659990329},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:56 2023","interfaces":[{"interface":"back","average":{"1min":0.30399999999999999,"5min":0.34699999999999998,"15min":0.36499999999999999},"min":{"1min":0.23999999999999999,"5min":0.14199999999999999,"15min":0.14199999999999999},"max":{"1min":0.42399999999999999,"5min":1.595,"15min":7.4249999999999998},"last":0.17100000000000001},{"interface":"front","average":{"1min":0.35499999999999998,"5min":0.40899999999999997,"15min":0.376},"min":{"1min":0.23999999999999999,"5min":0.19600000000000001,"15min":0.183},"max":{"1min":0.46999999999999997,"5min":2.6859999999999999,"15min":2.6859999999999999},"last":0.23000000000000001}]},{"osd":2,"last update":"Wed Nov 8 05:29:57 2023","interfaces":[{"interface":"back","average":{"1min":0.28199999999999997,"5min":0.36099999999999999,"15min":0.373},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":0.34399999999999997,"5min":2.2650000000000001,"15min":5.8209999999999997},"last":0.19700000000000001},{"interface":"front","average":{"1min":0.28599999999999998,"5min":0.31,"15min":0.35799999999999998},"min":{"1min":0.17199999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.35699999999999998,"5min":1.137,"15min":4.2350000000000003},"last":0.17799999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-08T05:30:49.358 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~151s 2023-11-08T05:30:49.454 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-11-08T05:30:49.752 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T05:30:49.754 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2023-11-08T05:30:49.765 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":1598,"stamp":"2023-11-08T05:30:48.060564+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":10,"acting":13,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2634344,"kb_used_data":408988,"kb_used_omap":780,"kb_used_meta":2220403,"kb_avail":280481176,"statfs":{"total":289910292480,"available":287212724224,"internally_reserved":0,"allocated":418803712,"data_stored":410487309,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":799447,"internal_metadata":2273692969},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-536782902,"num_objects":-164,"num_object_clones":0,"num_object_copies":-328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-115,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-164,"num_whiteouts":0,"num_read":-122056,"num_read_kb":-39011942,"num_write":-413642,"num_write_kb":-151705339,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-8,"num_bytes_recovered":-16922624,"num_keys_recovered":-2,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.424817"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":458,"reported_epoch":457,"state":"active+undersized","last_fresh":"2023-11-08T05:30:43.484884+0000","last_change":"2023-11-08T05:28:13.557071+0000","last_active":"2023-11-08T05:30:43.484884+0000","last_peered":"2023-11-08T05:30:43.484884+0000","last_clean":"2023-11-08T05:28:01.458439+0000","last_became_active":"2023-11-08T05:28:13.557071+0000","last_became_peered":"2023-11-08T05:28:13.557071+0000","last_unstale":"2023-11-08T05:30:43.484884+0000","last_undegraded":"2023-11-08T05:30:43.484884+0000","last_fullsized":"2023-11-08T05:28:13.547025+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":3,"reported_epoch":458,"state":"remapped+peering","last_fresh":"2023-11-08T05:30:44.492310+0000","last_change":"2023-11-08T05:30:44.492310+0000","last_active":"0.000000","last_peered":"0.000000","last_clean":"0.000000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-08T05:30:44.492310+0000","last_undegraded":"2023-11-08T05:30:44.492310+0000","last_fullsized":"2023-11-08T05:30:44.492310+0000","mapping_epoch":458,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":457,"reported_epoch":458,"state":"remapped+peering","last_fresh":"2023-11-08T05:30:44.492151+0000","last_change":"2023-11-08T05:30:44.492368+0000","last_active":"2023-11-08T05:30:43.487008+0000","last_peered":"2023-11-08T05:29:44.393948+0000","last_clean":"2023-11-08T05:29:44.393948+0000","last_became_active":"2023-11-08T04:52:47.545840+0000","last_became_peered":"2023-11-08T04:52:47.545840+0000","last_unstale":"2023-11-08T05:30:44.492151+0000","last_undegraded":"2023-11-08T05:30:44.492151+0000","last_fullsized":"2023-11-08T05:30:44.492151+0000","mapping_epoch":458,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":453,"reported_epoch":457,"state":"active+clean","last_fresh":"2023-11-08T05:30:43.484906+0000","last_change":"2023-11-08T04:52:47.548597+0000","last_active":"2023-11-08T05:30:43.484906+0000","last_peered":"2023-11-08T05:30:43.484906+0000","last_clean":"2023-11-08T05:30:43.484906+0000","last_became_active":"2023-11-08T04:52:47.548294+0000","last_became_peered":"2023-11-08T04:52:47.548294+0000","last_unstale":"2023-11-08T05:30:43.484906+0000","last_undegraded":"2023-11-08T05:30:43.484906+0000","last_fullsized":"2023-11-08T05:30:43.484906+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":459,"reported_epoch":457,"state":"active+undersized","last_fresh":"2023-11-08T05:30:43.484799+0000","last_change":"2023-11-08T05:28:13.545468+0000","last_active":"2023-11-08T05:30:43.484799+0000","last_peered":"2023-11-08T05:30:43.484799+0000","last_clean":"2023-11-08T05:28:01.456723+0000","last_became_active":"2023-11-08T05:28:13.545468+0000","last_became_peered":"2023-11-08T05:28:13.545468+0000","last_unstale":"2023-11-08T05:30:43.484799+0000","last_undegraded":"2023-11-08T05:30:43.484799+0000","last_fullsized":"2023-11-08T05:28:13.537207+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_clean_scrub_stamp":"2023-11-08T04:52:45.528796+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":464,"reported_epoch":457,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:43.487361+0000","last_change":"2023-11-08T05:29:44.395350+0000","last_active":"2023-11-08T05:30:43.487361+0000","last_peered":"2023-11-08T05:30:43.487361+0000","last_clean":"2023-11-08T05:30:43.487361+0000","last_became_active":"2023-11-08T05:29:44.395012+0000","last_became_peered":"2023-11-08T05:29:44.395012+0000","last_unstale":"2023-11-08T05:30:43.487361+0000","last_undegraded":"2023-11-08T05:30:43.487361+0000","last_fullsized":"2023-11-08T05:30:43.487361+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":458,"reported_epoch":457,"state":"active+undersized","last_fresh":"2023-11-08T05:30:43.484912+0000","last_change":"2023-11-08T05:28:13.563357+0000","last_active":"2023-11-08T05:30:43.484912+0000","last_peered":"2023-11-08T05:30:43.484912+0000","last_clean":"2023-11-08T05:28:01.457691+0000","last_became_active":"2023-11-08T05:28:13.563357+0000","last_became_peered":"2023-11-08T05:28:13.563357+0000","last_unstale":"2023-11-08T05:30:43.484912+0000","last_undegraded":"2023-11-08T05:30:43.484912+0000","last_fullsized":"2023-11-08T05:28:13.541614+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":458,"reported_epoch":457,"state":"active+undersized","last_fresh":"2023-11-08T05:30:43.485032+0000","last_change":"2023-11-08T05:28:13.550672+0000","last_active":"2023-11-08T05:30:43.485032+0000","last_peered":"2023-11-08T05:30:43.485032+0000","last_clean":"2023-11-08T05:28:01.457904+0000","last_became_active":"2023-11-08T05:28:13.550672+0000","last_became_peered":"2023-11-08T05:28:13.550672+0000","last_unstale":"2023-11-08T05:30:43.485032+0000","last_undegraded":"2023-11-08T05:30:43.485032+0000","last_fullsized":"2023-11-08T05:28:13.539595+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":458,"reported_epoch":457,"state":"active+undersized","last_fresh":"2023-11-08T05:30:43.484992+0000","last_change":"2023-11-08T05:28:13.557968+0000","last_active":"2023-11-08T05:30:43.484992+0000","last_peered":"2023-11-08T05:30:43.484992+0000","last_clean":"2023-11-08T05:28:01.458433+0000","last_became_active":"2023-11-08T05:28:13.557968+0000","last_became_peered":"2023-11-08T05:28:13.557968+0000","last_unstale":"2023-11-08T05:30:43.484992+0000","last_undegraded":"2023-11-08T05:30:43.484992+0000","last_fullsized":"2023-11-08T05:28:13.542832+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":1,"acting":1,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":12,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738827,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":645028,"kb_used_data":5340,"kb_used_omap":275,"kb_used_meta":639404,"kb_avail":93726812,"statfs":{"total":96636764160,"available":95976255488,"internally_reserved":0,"allocated":5468160,"data_stored":2691220,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":281650,"internal_metadata":654750670},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:59 2023","interfaces":[{"interface":"back","average":{"1min":0.30499999999999999,"5min":0.439,"15min":0.36099999999999999},"min":{"1min":0.17399999999999999,"5min":0.14499999999999999,"15min":0.14499999999999999},"max":{"1min":0.60099999999999998,"5min":4.9379999999999997,"15min":4.9379999999999997},"last":0.379},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.49399999999999999,"15min":0.42399999999999999},"min":{"1min":0.221,"5min":0.19600000000000001,"15min":0.17000000000000001},"max":{"1min":0.63200000000000001,"5min":4.9859999999999998,"15min":4.9859999999999998},"last":0.40899999999999997}]},{"osd":1,"last update":"Wed Nov 8 05:30:03 2023","interfaces":[{"interface":"back","average":{"1min":0.28499999999999998,"5min":0.65100000000000002,"15min":0.47099999999999997},"min":{"1min":0.20100000000000001,"5min":0.153,"15min":0.153},"max":{"1min":0.39800000000000002,"5min":23.242999999999999,"15min":23.242999999999999},"last":0.252},{"interface":"front","average":{"1min":0.29499999999999998,"5min":0.63100000000000001,"15min":0.45900000000000002},"min":{"1min":0.224,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":0.436,"5min":23.251000000000001,"15min":23.251000000000001},"last":0.23999999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738798,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1339228,"kb_used_data":398304,"kb_used_omap":291,"kb_used_meta":936476,"kb_avail":93032612,"statfs":{"total":96636764160,"available":95265394688,"internally_reserved":0,"allocated":407863296,"data_stored":405103900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":298462,"internal_metadata":958951970},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.436,"15min":0.36299999999999999},"min":{"1min":0.19700000000000001,"5min":0.14699999999999999,"15min":0.13700000000000001},"max":{"1min":3.222,"5min":6.5499999999999998,"15min":6.9989999999999997},"last":0.245},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.46300000000000002,"15min":0.42299999999999999},"min":{"1min":0.17799999999999999,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":5.0030000000000001,"5min":6.0629999999999997,"15min":9.6869999999999994},"last":0.28599999999999998}]},{"osd":2,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.51600000000000001,"15min":0.47099999999999997},"min":{"1min":0.21099999999999999,"5min":0.153,"15min":0.153},"max":{"1min":5.7439999999999998,"5min":5.8280000000000003,"15min":9.7639999999999993},"last":0.252},{"interface":"front","average":{"1min":0.497,"5min":0.56200000000000006,"15min":0.42299999999999999},"min":{"1min":0.20399999999999999,"5min":0.187,"15min":0.16800000000000001},"max":{"1min":3.9319999999999999,"5min":8.8040000000000003,"15min":8.8040000000000003},"last":0.29899999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738829,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":650088,"kb_used_data":5344,"kb_used_omap":214,"kb_used_meta":644521,"kb_avail":93721752,"statfs":{"total":96636764160,"available":95971074048,"internally_reserved":0,"allocated":5472256,"data_stored":2692189,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":219335,"internal_metadata":659990329},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:56 2023","interfaces":[{"interface":"back","average":{"1min":0.30399999999999999,"5min":0.34699999999999998,"15min":0.36499999999999999},"min":{"1min":0.23999999999999999,"5min":0.14199999999999999,"15min":0.14199999999999999},"max":{"1min":0.42399999999999999,"5min":1.595,"15min":7.4249999999999998},"last":0.17100000000000001},{"interface":"front","average":{"1min":0.35499999999999998,"5min":0.40899999999999997,"15min":0.376},"min":{"1min":0.23999999999999999,"5min":0.19600000000000001,"15min":0.183},"max":{"1min":0.46999999999999997,"5min":2.6859999999999999,"15min":2.6859999999999999},"last":0.23000000000000001}]},{"osd":2,"last update":"Wed Nov 8 05:29:57 2023","interfaces":[{"interface":"back","average":{"1min":0.28199999999999997,"5min":0.36099999999999999,"15min":0.373},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":0.34399999999999997,"5min":2.2650000000000001,"15min":5.8209999999999997},"last":0.19700000000000001},{"interface":"front","average":{"1min":0.28599999999999998,"5min":0.31,"15min":0.35799999999999998},"min":{"1min":0.17199999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.35699999999999998,"5min":1.137,"15min":4.2350000000000003},"last":0.17799999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-08T05:30:49.766 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-11-08T05:30:50.047 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T05:30:50.050 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2023-11-08T05:30:50.060 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":1598,"stamp":"2023-11-08T05:30:48.060564+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":10,"acting":13,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2634344,"kb_used_data":408988,"kb_used_omap":780,"kb_used_meta":2220403,"kb_avail":280481176,"statfs":{"total":289910292480,"available":287212724224,"internally_reserved":0,"allocated":418803712,"data_stored":410487309,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":799447,"internal_metadata":2273692969},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-536782902,"num_objects":-164,"num_object_clones":0,"num_object_copies":-328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-115,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-164,"num_whiteouts":0,"num_read":-122056,"num_read_kb":-39011942,"num_write":-413642,"num_write_kb":-151705339,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-8,"num_bytes_recovered":-16922624,"num_keys_recovered":-2,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.424817"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":458,"reported_epoch":457,"state":"active+undersized","last_fresh":"2023-11-08T05:30:43.484884+0000","last_change":"2023-11-08T05:28:13.557071+0000","last_active":"2023-11-08T05:30:43.484884+0000","last_peered":"2023-11-08T05:30:43.484884+0000","last_clean":"2023-11-08T05:28:01.458439+0000","last_became_active":"2023-11-08T05:28:13.557071+0000","last_became_peered":"2023-11-08T05:28:13.557071+0000","last_unstale":"2023-11-08T05:30:43.484884+0000","last_undegraded":"2023-11-08T05:30:43.484884+0000","last_fullsized":"2023-11-08T05:28:13.547025+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":3,"reported_epoch":458,"state":"remapped+peering","last_fresh":"2023-11-08T05:30:44.492310+0000","last_change":"2023-11-08T05:30:44.492310+0000","last_active":"0.000000","last_peered":"0.000000","last_clean":"0.000000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-11-08T05:30:44.492310+0000","last_undegraded":"2023-11-08T05:30:44.492310+0000","last_fullsized":"2023-11-08T05:30:44.492310+0000","mapping_epoch":458,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":457,"reported_epoch":458,"state":"remapped+peering","last_fresh":"2023-11-08T05:30:44.492151+0000","last_change":"2023-11-08T05:30:44.492368+0000","last_active":"2023-11-08T05:30:43.487008+0000","last_peered":"2023-11-08T05:29:44.393948+0000","last_clean":"2023-11-08T05:29:44.393948+0000","last_became_active":"2023-11-08T04:52:47.545840+0000","last_became_peered":"2023-11-08T04:52:47.545840+0000","last_unstale":"2023-11-08T05:30:44.492151+0000","last_undegraded":"2023-11-08T05:30:44.492151+0000","last_fullsized":"2023-11-08T05:30:44.492151+0000","mapping_epoch":458,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":453,"reported_epoch":457,"state":"active+clean","last_fresh":"2023-11-08T05:30:43.484906+0000","last_change":"2023-11-08T04:52:47.548597+0000","last_active":"2023-11-08T05:30:43.484906+0000","last_peered":"2023-11-08T05:30:43.484906+0000","last_clean":"2023-11-08T05:30:43.484906+0000","last_became_active":"2023-11-08T04:52:47.548294+0000","last_became_peered":"2023-11-08T04:52:47.548294+0000","last_unstale":"2023-11-08T05:30:43.484906+0000","last_undegraded":"2023-11-08T05:30:43.484906+0000","last_fullsized":"2023-11-08T05:30:43.484906+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":459,"reported_epoch":457,"state":"active+undersized","last_fresh":"2023-11-08T05:30:43.484799+0000","last_change":"2023-11-08T05:28:13.545468+0000","last_active":"2023-11-08T05:30:43.484799+0000","last_peered":"2023-11-08T05:30:43.484799+0000","last_clean":"2023-11-08T05:28:01.456723+0000","last_became_active":"2023-11-08T05:28:13.545468+0000","last_became_peered":"2023-11-08T05:28:13.545468+0000","last_unstale":"2023-11-08T05:30:43.484799+0000","last_undegraded":"2023-11-08T05:30:43.484799+0000","last_fullsized":"2023-11-08T05:28:13.537207+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_clean_scrub_stamp":"2023-11-08T04:52:45.528796+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":464,"reported_epoch":457,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:43.487361+0000","last_change":"2023-11-08T05:29:44.395350+0000","last_active":"2023-11-08T05:30:43.487361+0000","last_peered":"2023-11-08T05:30:43.487361+0000","last_clean":"2023-11-08T05:30:43.487361+0000","last_became_active":"2023-11-08T05:29:44.395012+0000","last_became_peered":"2023-11-08T05:29:44.395012+0000","last_unstale":"2023-11-08T05:30:43.487361+0000","last_undegraded":"2023-11-08T05:30:43.487361+0000","last_fullsized":"2023-11-08T05:30:43.487361+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":458,"reported_epoch":457,"state":"active+undersized","last_fresh":"2023-11-08T05:30:43.484912+0000","last_change":"2023-11-08T05:28:13.563357+0000","last_active":"2023-11-08T05:30:43.484912+0000","last_peered":"2023-11-08T05:30:43.484912+0000","last_clean":"2023-11-08T05:28:01.457691+0000","last_became_active":"2023-11-08T05:28:13.563357+0000","last_became_peered":"2023-11-08T05:28:13.563357+0000","last_unstale":"2023-11-08T05:30:43.484912+0000","last_undegraded":"2023-11-08T05:30:43.484912+0000","last_fullsized":"2023-11-08T05:28:13.541614+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":458,"reported_epoch":457,"state":"active+undersized","last_fresh":"2023-11-08T05:30:43.485032+0000","last_change":"2023-11-08T05:28:13.550672+0000","last_active":"2023-11-08T05:30:43.485032+0000","last_peered":"2023-11-08T05:30:43.485032+0000","last_clean":"2023-11-08T05:28:01.457904+0000","last_became_active":"2023-11-08T05:28:13.550672+0000","last_became_peered":"2023-11-08T05:28:13.550672+0000","last_unstale":"2023-11-08T05:30:43.485032+0000","last_undegraded":"2023-11-08T05:30:43.485032+0000","last_fullsized":"2023-11-08T05:28:13.539595+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":458,"reported_epoch":457,"state":"active+undersized","last_fresh":"2023-11-08T05:30:43.484992+0000","last_change":"2023-11-08T05:28:13.557968+0000","last_active":"2023-11-08T05:30:43.484992+0000","last_peered":"2023-11-08T05:30:43.484992+0000","last_clean":"2023-11-08T05:28:01.458433+0000","last_became_active":"2023-11-08T05:28:13.557968+0000","last_became_peered":"2023-11-08T05:28:13.557968+0000","last_unstale":"2023-11-08T05:30:43.484992+0000","last_undegraded":"2023-11-08T05:30:43.484992+0000","last_fullsized":"2023-11-08T05:28:13.542832+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":1,"acting":1,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":12,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738827,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":645028,"kb_used_data":5340,"kb_used_omap":275,"kb_used_meta":639404,"kb_avail":93726812,"statfs":{"total":96636764160,"available":95976255488,"internally_reserved":0,"allocated":5468160,"data_stored":2691220,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":281650,"internal_metadata":654750670},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:59 2023","interfaces":[{"interface":"back","average":{"1min":0.30499999999999999,"5min":0.439,"15min":0.36099999999999999},"min":{"1min":0.17399999999999999,"5min":0.14499999999999999,"15min":0.14499999999999999},"max":{"1min":0.60099999999999998,"5min":4.9379999999999997,"15min":4.9379999999999997},"last":0.379},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.49399999999999999,"15min":0.42399999999999999},"min":{"1min":0.221,"5min":0.19600000000000001,"15min":0.17000000000000001},"max":{"1min":0.63200000000000001,"5min":4.9859999999999998,"15min":4.9859999999999998},"last":0.40899999999999997}]},{"osd":1,"last update":"Wed Nov 8 05:30:03 2023","interfaces":[{"interface":"back","average":{"1min":0.28499999999999998,"5min":0.65100000000000002,"15min":0.47099999999999997},"min":{"1min":0.20100000000000001,"5min":0.153,"15min":0.153},"max":{"1min":0.39800000000000002,"5min":23.242999999999999,"15min":23.242999999999999},"last":0.252},{"interface":"front","average":{"1min":0.29499999999999998,"5min":0.63100000000000001,"15min":0.45900000000000002},"min":{"1min":0.224,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":0.436,"5min":23.251000000000001,"15min":23.251000000000001},"last":0.23999999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738798,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1339228,"kb_used_data":398304,"kb_used_omap":291,"kb_used_meta":936476,"kb_avail":93032612,"statfs":{"total":96636764160,"available":95265394688,"internally_reserved":0,"allocated":407863296,"data_stored":405103900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":298462,"internal_metadata":958951970},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.436,"15min":0.36299999999999999},"min":{"1min":0.19700000000000001,"5min":0.14699999999999999,"15min":0.13700000000000001},"max":{"1min":3.222,"5min":6.5499999999999998,"15min":6.9989999999999997},"last":0.245},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.46300000000000002,"15min":0.42299999999999999},"min":{"1min":0.17799999999999999,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":5.0030000000000001,"5min":6.0629999999999997,"15min":9.6869999999999994},"last":0.28599999999999998}]},{"osd":2,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.51600000000000001,"15min":0.47099999999999997},"min":{"1min":0.21099999999999999,"5min":0.153,"15min":0.153},"max":{"1min":5.7439999999999998,"5min":5.8280000000000003,"15min":9.7639999999999993},"last":0.252},{"interface":"front","average":{"1min":0.497,"5min":0.56200000000000006,"15min":0.42299999999999999},"min":{"1min":0.20399999999999999,"5min":0.187,"15min":0.16800000000000001},"max":{"1min":3.9319999999999999,"5min":8.8040000000000003,"15min":8.8040000000000003},"last":0.29899999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738829,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":650088,"kb_used_data":5344,"kb_used_omap":214,"kb_used_meta":644521,"kb_avail":93721752,"statfs":{"total":96636764160,"available":95971074048,"internally_reserved":0,"allocated":5472256,"data_stored":2692189,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":219335,"internal_metadata":659990329},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:56 2023","interfaces":[{"interface":"back","average":{"1min":0.30399999999999999,"5min":0.34699999999999998,"15min":0.36499999999999999},"min":{"1min":0.23999999999999999,"5min":0.14199999999999999,"15min":0.14199999999999999},"max":{"1min":0.42399999999999999,"5min":1.595,"15min":7.4249999999999998},"last":0.17100000000000001},{"interface":"front","average":{"1min":0.35499999999999998,"5min":0.40899999999999997,"15min":0.376},"min":{"1min":0.23999999999999999,"5min":0.19600000000000001,"15min":0.183},"max":{"1min":0.46999999999999997,"5min":2.6859999999999999,"15min":2.6859999999999999},"last":0.23000000000000001}]},{"osd":2,"last update":"Wed Nov 8 05:29:57 2023","interfaces":[{"interface":"back","average":{"1min":0.28199999999999997,"5min":0.36099999999999999,"15min":0.373},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":0.34399999999999997,"5min":2.2650000000000001,"15min":5.8209999999999997},"last":0.19700000000000001},{"interface":"front","average":{"1min":0.28599999999999998,"5min":0.31,"15min":0.35799999999999998},"min":{"1min":0.17199999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.35699999999999998,"5min":1.137,"15min":4.2350000000000003},"last":0.17799999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-08T05:30:50.061 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-11-08T05:30:50.061 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 458, 'reported_epoch': 457, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:30:43.484884+0000', 'last_change': '2023-11-08T05:28:13.557071+0000', 'last_active': '2023-11-08T05:30:43.484884+0000', 'last_peered': '2023-11-08T05:30:43.484884+0000', 'last_clean': '2023-11-08T05:28:01.458439+0000', 'last_became_active': '2023-11-08T05:28:13.557071+0000', 'last_became_peered': '2023-11-08T05:28:13.557071+0000', 'last_unstale': '2023-11-08T05:30:43.484884+0000', 'last_undegraded': '2023-11-08T05:30:43.484884+0000', 'last_fullsized': '2023-11-08T05:28:13.547025+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:30:50.062 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-11-08T05:30:50.062 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 3, 'reported_epoch': 458, 'state': 'remapped+peering', 'last_fresh': '2023-11-08T05:30:44.492310+0000', 'last_change': '2023-11-08T05:30:44.492310+0000', 'last_active': '0.000000', 'last_peered': '0.000000', 'last_clean': '0.000000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-11-08T05:30:44.492310+0000', 'last_undegraded': '2023-11-08T05:30:44.492310+0000', 'last_fullsized': '2023-11-08T05:30:44.492310+0000', 'mapping_epoch': 458, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2], 'acting': [2, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [1], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2023-11-08T05:30:50.062 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.1 is not active+clean 2023-11-08T05:30:50.063 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.1', 'version': "0'0", 'reported_seq': 457, 'reported_epoch': 458, 'state': 'remapped+peering', 'last_fresh': '2023-11-08T05:30:44.492151+0000', 'last_change': '2023-11-08T05:30:44.492368+0000', 'last_active': '2023-11-08T05:30:43.487008+0000', 'last_peered': '2023-11-08T05:29:44.393948+0000', 'last_clean': '2023-11-08T05:29:44.393948+0000', 'last_became_active': '2023-11-08T04:52:47.545840+0000', 'last_became_peered': '2023-11-08T04:52:47.545840+0000', 'last_unstale': '2023-11-08T05:30:44.492151+0000', 'last_undegraded': '2023-11-08T05:30:44.492151+0000', 'last_fullsized': '2023-11-08T05:30:44.492151+0000', 'mapping_epoch': 458, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2], 'acting': [2, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [1], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2023-11-08T05:30:50.063 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-11-08T05:30:50.063 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "0'0", 'reported_seq': 459, 'reported_epoch': 457, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:30:43.484799+0000', 'last_change': '2023-11-08T05:28:13.545468+0000', 'last_active': '2023-11-08T05:30:43.484799+0000', 'last_peered': '2023-11-08T05:30:43.484799+0000', 'last_clean': '2023-11-08T05:28:01.456723+0000', 'last_became_active': '2023-11-08T05:28:13.545468+0000', 'last_became_peered': '2023-11-08T05:28:13.545468+0000', 'last_unstale': '2023-11-08T05:30:43.484799+0000', 'last_undegraded': '2023-11-08T05:30:43.484799+0000', 'last_fullsized': '2023-11-08T05:28:13.537207+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 9, 'last_epoch_clean': 10, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:45.528796+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:45.528796+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:45.528796+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:30:50.064 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.0 is not active+clean 2023-11-08T05:30:50.064 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.0', 'version': "0'0", 'reported_seq': 464, 'reported_epoch': 457, 'state': 'active+clean+remapped', 'last_fresh': '2023-11-08T05:30:43.487361+0000', 'last_change': '2023-11-08T05:29:44.395350+0000', 'last_active': '2023-11-08T05:30:43.487361+0000', 'last_peered': '2023-11-08T05:30:43.487361+0000', 'last_clean': '2023-11-08T05:30:43.487361+0000', 'last_became_active': '2023-11-08T05:29:44.395012+0000', 'last_became_peered': '2023-11-08T05:29:44.395012+0000', 'last_unstale': '2023-11-08T05:30:43.487361+0000', 'last_undegraded': '2023-11-08T05:30:43.487361+0000', 'last_fullsized': '2023-11-08T05:30:43.487361+0000', 'mapping_epoch': 455, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 456, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2], 'acting': [2, 1], 'avail_no_missing': ['2', '1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2023-11-08T05:30:50.064 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-11-08T05:30:50.065 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 458, 'reported_epoch': 457, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:30:43.484912+0000', 'last_change': '2023-11-08T05:28:13.563357+0000', 'last_active': '2023-11-08T05:30:43.484912+0000', 'last_peered': '2023-11-08T05:30:43.484912+0000', 'last_clean': '2023-11-08T05:28:01.457691+0000', 'last_became_active': '2023-11-08T05:28:13.563357+0000', 'last_became_peered': '2023-11-08T05:28:13.563357+0000', 'last_unstale': '2023-11-08T05:30:43.484912+0000', 'last_undegraded': '2023-11-08T05:30:43.484912+0000', 'last_fullsized': '2023-11-08T05:28:13.541614+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:30:50.065 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-11-08T05:30:50.066 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 458, 'reported_epoch': 457, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:30:43.485032+0000', 'last_change': '2023-11-08T05:28:13.550672+0000', 'last_active': '2023-11-08T05:30:43.485032+0000', 'last_peered': '2023-11-08T05:30:43.485032+0000', 'last_clean': '2023-11-08T05:28:01.457904+0000', 'last_became_active': '2023-11-08T05:28:13.550672+0000', 'last_became_peered': '2023-11-08T05:28:13.550672+0000', 'last_unstale': '2023-11-08T05:30:43.485032+0000', 'last_undegraded': '2023-11-08T05:30:43.485032+0000', 'last_fullsized': '2023-11-08T05:28:13.539595+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:30:50.066 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-11-08T05:30:50.066 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 458, 'reported_epoch': 457, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:30:43.484992+0000', 'last_change': '2023-11-08T05:28:13.557968+0000', 'last_active': '2023-11-08T05:30:43.484992+0000', 'last_peered': '2023-11-08T05:30:43.484992+0000', 'last_clean': '2023-11-08T05:28:01.458433+0000', 'last_became_active': '2023-11-08T05:28:13.557968+0000', 'last_became_peered': '2023-11-08T05:28:13.557968+0000', 'last_unstale': '2023-11-08T05:30:43.484992+0000', 'last_undegraded': '2023-11-08T05:30:43.484992+0000', 'last_fullsized': '2023-11-08T05:28:13.542832+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:30:50.067 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-11-08T05:30:50.614 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T05:30:50.614 INFO:teuthology.orchestra.run.smithi087.stdout:{"fsid":"ab4cbc06-449d-497d-b513-220d9db2b590","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 5 pgs undersized","count":5},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2293,"monmap":{"epoch":1,"min_mon_release_name":"pacific","num_mons":1},"osdmap":{"epoch":459,"num_osds":3,"num_up_osds":2,"osd_up_since":1699421292,"num_in_osds":3,"osd_in_since":1699419158,"num_remapped_pgs":3},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean+remapped","count":3},{"state_name":"active+clean","count":1}],"num_pgs":9,"num_pools":2,"num_objects":0,"data_bytes":0,"bytes_used":2697580544,"bytes_avail":287212711936,"bytes_total":289910292480},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":41,"modified":"2023-11-08T05:29:32.044544+0000","services":{}},"progress_events":{}} 2023-11-08T05:30:50.627 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-11-08T05:30:50.628 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-11-08T05:30:50.911 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T05:30:50.913 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2023-11-08T05:30:50.924 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":1599,"stamp":"2023-11-08T05:30:50.060931+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":10,"acting":13,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":109,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2634356,"kb_used_data":409000,"kb_used_omap":780,"kb_used_meta":2220403,"kb_avail":280481164,"statfs":{"total":289910292480,"available":287212711936,"internally_reserved":0,"allocated":418816000,"data_stored":410493736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":799447,"internal_metadata":2273692969},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-536782902,"num_objects":-164,"num_object_clones":0,"num_object_copies":-328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-115,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-164,"num_whiteouts":0,"num_read":-122056,"num_read_kb":-39011942,"num_write":-413642,"num_write_kb":-151705339,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-8,"num_bytes_recovered":-16922624,"num_keys_recovered":-2,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.424817"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":458,"reported_epoch":457,"state":"active+undersized","last_fresh":"2023-11-08T05:30:43.484884+0000","last_change":"2023-11-08T05:28:13.557071+0000","last_active":"2023-11-08T05:30:43.484884+0000","last_peered":"2023-11-08T05:30:43.484884+0000","last_clean":"2023-11-08T05:28:01.458439+0000","last_became_active":"2023-11-08T05:28:13.557071+0000","last_became_peered":"2023-11-08T05:28:13.557071+0000","last_unstale":"2023-11-08T05:30:43.484884+0000","last_undegraded":"2023-11-08T05:30:43.484884+0000","last_fullsized":"2023-11-08T05:28:13.547025+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":459,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:45.794760+0000","last_change":"2023-11-08T05:30:45.794760+0000","last_active":"2023-11-08T05:30:45.794760+0000","last_peered":"2023-11-08T05:30:45.794760+0000","last_clean":"2023-11-08T05:30:45.794760+0000","last_became_active":"2023-11-08T05:30:45.794398+0000","last_became_peered":"2023-11-08T05:30:45.794398+0000","last_unstale":"2023-11-08T05:30:45.794760+0000","last_undegraded":"2023-11-08T05:30:45.794760+0000","last_fullsized":"2023-11-08T05:30:45.794760+0000","mapping_epoch":458,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":459,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":465,"reported_epoch":459,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:45.795085+0000","last_change":"2023-11-08T05:30:45.795085+0000","last_active":"2023-11-08T05:30:45.795085+0000","last_peered":"2023-11-08T05:30:45.795085+0000","last_clean":"2023-11-08T05:30:45.795085+0000","last_became_active":"2023-11-08T05:30:45.794743+0000","last_became_peered":"2023-11-08T05:30:45.794743+0000","last_unstale":"2023-11-08T05:30:45.795085+0000","last_undegraded":"2023-11-08T05:30:45.795085+0000","last_fullsized":"2023-11-08T05:30:45.795085+0000","mapping_epoch":458,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":459,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":453,"reported_epoch":457,"state":"active+clean","last_fresh":"2023-11-08T05:30:43.484906+0000","last_change":"2023-11-08T04:52:47.548597+0000","last_active":"2023-11-08T05:30:43.484906+0000","last_peered":"2023-11-08T05:30:43.484906+0000","last_clean":"2023-11-08T05:30:43.484906+0000","last_became_active":"2023-11-08T04:52:47.548294+0000","last_became_peered":"2023-11-08T04:52:47.548294+0000","last_unstale":"2023-11-08T05:30:43.484906+0000","last_undegraded":"2023-11-08T05:30:43.484906+0000","last_fullsized":"2023-11-08T05:30:43.484906+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":459,"reported_epoch":457,"state":"active+undersized","last_fresh":"2023-11-08T05:30:43.484799+0000","last_change":"2023-11-08T05:28:13.545468+0000","last_active":"2023-11-08T05:30:43.484799+0000","last_peered":"2023-11-08T05:30:43.484799+0000","last_clean":"2023-11-08T05:28:01.456723+0000","last_became_active":"2023-11-08T05:28:13.545468+0000","last_became_peered":"2023-11-08T05:28:13.545468+0000","last_unstale":"2023-11-08T05:30:43.484799+0000","last_undegraded":"2023-11-08T05:30:43.484799+0000","last_fullsized":"2023-11-08T05:28:13.537207+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_clean_scrub_stamp":"2023-11-08T04:52:45.528796+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":465,"reported_epoch":458,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:44.491907+0000","last_change":"2023-11-08T05:29:44.395350+0000","last_active":"2023-11-08T05:30:44.491907+0000","last_peered":"2023-11-08T05:30:44.491907+0000","last_clean":"2023-11-08T05:30:44.491907+0000","last_became_active":"2023-11-08T05:29:44.395012+0000","last_became_peered":"2023-11-08T05:29:44.395012+0000","last_unstale":"2023-11-08T05:30:44.491907+0000","last_undegraded":"2023-11-08T05:30:44.491907+0000","last_fullsized":"2023-11-08T05:30:44.491907+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":458,"reported_epoch":457,"state":"active+undersized","last_fresh":"2023-11-08T05:30:43.484912+0000","last_change":"2023-11-08T05:28:13.563357+0000","last_active":"2023-11-08T05:30:43.484912+0000","last_peered":"2023-11-08T05:30:43.484912+0000","last_clean":"2023-11-08T05:28:01.457691+0000","last_became_active":"2023-11-08T05:28:13.563357+0000","last_became_peered":"2023-11-08T05:28:13.563357+0000","last_unstale":"2023-11-08T05:30:43.484912+0000","last_undegraded":"2023-11-08T05:30:43.484912+0000","last_fullsized":"2023-11-08T05:28:13.541614+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":458,"reported_epoch":457,"state":"active+undersized","last_fresh":"2023-11-08T05:30:43.485032+0000","last_change":"2023-11-08T05:28:13.550672+0000","last_active":"2023-11-08T05:30:43.485032+0000","last_peered":"2023-11-08T05:30:43.485032+0000","last_clean":"2023-11-08T05:28:01.457904+0000","last_became_active":"2023-11-08T05:28:13.550672+0000","last_became_peered":"2023-11-08T05:28:13.550672+0000","last_unstale":"2023-11-08T05:30:43.485032+0000","last_undegraded":"2023-11-08T05:30:43.485032+0000","last_fullsized":"2023-11-08T05:28:13.539595+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":458,"reported_epoch":457,"state":"active+undersized","last_fresh":"2023-11-08T05:30:43.484992+0000","last_change":"2023-11-08T05:28:13.557968+0000","last_active":"2023-11-08T05:30:43.484992+0000","last_peered":"2023-11-08T05:30:43.484992+0000","last_clean":"2023-11-08T05:28:01.458433+0000","last_became_active":"2023-11-08T05:28:13.557968+0000","last_became_peered":"2023-11-08T05:28:13.557968+0000","last_unstale":"2023-11-08T05:30:43.484992+0000","last_undegraded":"2023-11-08T05:30:43.484992+0000","last_fullsized":"2023-11-08T05:28:13.542832+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":1,"acting":1,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":12,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738828,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":645040,"kb_used_data":5352,"kb_used_omap":275,"kb_used_meta":639404,"kb_avail":93726800,"statfs":{"total":96636764160,"available":95976243200,"internally_reserved":0,"allocated":5480448,"data_stored":2697647,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":281650,"internal_metadata":654750670},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:59 2023","interfaces":[{"interface":"back","average":{"1min":0.30499999999999999,"5min":0.439,"15min":0.36099999999999999},"min":{"1min":0.17399999999999999,"5min":0.14499999999999999,"15min":0.14499999999999999},"max":{"1min":0.60099999999999998,"5min":4.9379999999999997,"15min":4.9379999999999997},"last":0.379},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.49399999999999999,"15min":0.42399999999999999},"min":{"1min":0.221,"5min":0.19600000000000001,"15min":0.17000000000000001},"max":{"1min":0.63200000000000001,"5min":4.9859999999999998,"15min":4.9859999999999998},"last":0.40899999999999997}]},{"osd":1,"last update":"Wed Nov 8 05:30:03 2023","interfaces":[{"interface":"back","average":{"1min":0.28499999999999998,"5min":0.65100000000000002,"15min":0.47099999999999997},"min":{"1min":0.20100000000000001,"5min":0.153,"15min":0.153},"max":{"1min":0.39800000000000002,"5min":23.242999999999999,"15min":23.242999999999999},"last":0.28699999999999998},{"interface":"front","average":{"1min":0.29499999999999998,"5min":0.63100000000000001,"15min":0.45900000000000002},"min":{"1min":0.224,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":0.436,"5min":23.251000000000001,"15min":23.251000000000001},"last":0.311}]}]},{"osd":0,"up_from":8,"seq":34359738798,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1339228,"kb_used_data":398304,"kb_used_omap":291,"kb_used_meta":936476,"kb_avail":93032612,"statfs":{"total":96636764160,"available":95265394688,"internally_reserved":0,"allocated":407863296,"data_stored":405103900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":298462,"internal_metadata":958951970},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.436,"15min":0.36299999999999999},"min":{"1min":0.19700000000000001,"5min":0.14699999999999999,"15min":0.13700000000000001},"max":{"1min":3.222,"5min":6.5499999999999998,"15min":6.9989999999999997},"last":0.245},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.46300000000000002,"15min":0.42299999999999999},"min":{"1min":0.17799999999999999,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":5.0030000000000001,"5min":6.0629999999999997,"15min":9.6869999999999994},"last":0.28599999999999998}]},{"osd":2,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.51600000000000001,"15min":0.47099999999999997},"min":{"1min":0.21099999999999999,"5min":0.153,"15min":0.153},"max":{"1min":5.7439999999999998,"5min":5.8280000000000003,"15min":9.7639999999999993},"last":0.252},{"interface":"front","average":{"1min":0.497,"5min":0.56200000000000006,"15min":0.42299999999999999},"min":{"1min":0.20399999999999999,"5min":0.187,"15min":0.16800000000000001},"max":{"1min":3.9319999999999999,"5min":8.8040000000000003,"15min":8.8040000000000003},"last":0.29899999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738829,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":650088,"kb_used_data":5344,"kb_used_omap":214,"kb_used_meta":644521,"kb_avail":93721752,"statfs":{"total":96636764160,"available":95971074048,"internally_reserved":0,"allocated":5472256,"data_stored":2692189,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":219335,"internal_metadata":659990329},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:56 2023","interfaces":[{"interface":"back","average":{"1min":0.30399999999999999,"5min":0.34699999999999998,"15min":0.36499999999999999},"min":{"1min":0.23999999999999999,"5min":0.14199999999999999,"15min":0.14199999999999999},"max":{"1min":0.42399999999999999,"5min":1.595,"15min":7.4249999999999998},"last":0.17100000000000001},{"interface":"front","average":{"1min":0.35499999999999998,"5min":0.40899999999999997,"15min":0.376},"min":{"1min":0.23999999999999999,"5min":0.19600000000000001,"15min":0.183},"max":{"1min":0.46999999999999997,"5min":2.6859999999999999,"15min":2.6859999999999999},"last":0.23000000000000001}]},{"osd":2,"last update":"Wed Nov 8 05:29:57 2023","interfaces":[{"interface":"back","average":{"1min":0.28199999999999997,"5min":0.36099999999999999,"15min":0.373},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":0.34399999999999997,"5min":2.2650000000000001,"15min":5.8209999999999997},"last":0.19700000000000001},{"interface":"front","average":{"1min":0.28599999999999998,"5min":0.31,"15min":0.35799999999999998},"min":{"1min":0.17199999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.35699999999999998,"5min":1.137,"15min":4.2350000000000003},"last":0.17799999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-08T05:30:53.926 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-11-08T05:30:54.223 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T05:30:54.225 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2023-11-08T05:30:54.236 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":1601,"stamp":"2023-11-08T05:30:54.061537+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":10,"acting":13,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":109,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2634368,"kb_used_data":409012,"kb_used_omap":780,"kb_used_meta":2220403,"kb_avail":280481152,"statfs":{"total":289910292480,"available":287212699648,"internally_reserved":0,"allocated":418828288,"data_stored":410500163,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":799447,"internal_metadata":2273692969},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-536782902,"num_objects":-164,"num_object_clones":0,"num_object_copies":-328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-115,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-164,"num_whiteouts":0,"num_read":-122056,"num_read_kb":-39011942,"num_write":-413642,"num_write_kb":-151705339,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-8,"num_bytes_recovered":-16922624,"num_keys_recovered":-2,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.424817"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490644+0000","last_change":"2023-11-08T05:28:13.557071+0000","last_active":"2023-11-08T05:30:44.490644+0000","last_peered":"2023-11-08T05:30:44.490644+0000","last_clean":"2023-11-08T05:28:01.458439+0000","last_became_active":"2023-11-08T05:28:13.557071+0000","last_became_peered":"2023-11-08T05:28:13.557071+0000","last_unstale":"2023-11-08T05:30:44.490644+0000","last_undegraded":"2023-11-08T05:30:44.490644+0000","last_fullsized":"2023-11-08T05:28:13.547025+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":459,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:45.794760+0000","last_change":"2023-11-08T05:30:45.794760+0000","last_active":"2023-11-08T05:30:45.794760+0000","last_peered":"2023-11-08T05:30:45.794760+0000","last_clean":"2023-11-08T05:30:45.794760+0000","last_became_active":"2023-11-08T05:30:45.794398+0000","last_became_peered":"2023-11-08T05:30:45.794398+0000","last_unstale":"2023-11-08T05:30:45.794760+0000","last_undegraded":"2023-11-08T05:30:45.794760+0000","last_fullsized":"2023-11-08T05:30:45.794760+0000","mapping_epoch":458,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":459,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":465,"reported_epoch":459,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:45.795085+0000","last_change":"2023-11-08T05:30:45.795085+0000","last_active":"2023-11-08T05:30:45.795085+0000","last_peered":"2023-11-08T05:30:45.795085+0000","last_clean":"2023-11-08T05:30:45.795085+0000","last_became_active":"2023-11-08T05:30:45.794743+0000","last_became_peered":"2023-11-08T05:30:45.794743+0000","last_unstale":"2023-11-08T05:30:45.795085+0000","last_undegraded":"2023-11-08T05:30:45.795085+0000","last_fullsized":"2023-11-08T05:30:45.795085+0000","mapping_epoch":458,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":459,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":454,"reported_epoch":458,"state":"active+clean","last_fresh":"2023-11-08T05:30:44.490618+0000","last_change":"2023-11-08T04:52:47.548597+0000","last_active":"2023-11-08T05:30:44.490618+0000","last_peered":"2023-11-08T05:30:44.490618+0000","last_clean":"2023-11-08T05:30:44.490618+0000","last_became_active":"2023-11-08T04:52:47.548294+0000","last_became_peered":"2023-11-08T04:52:47.548294+0000","last_unstale":"2023-11-08T05:30:44.490618+0000","last_undegraded":"2023-11-08T05:30:44.490618+0000","last_fullsized":"2023-11-08T05:30:44.490618+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":460,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490523+0000","last_change":"2023-11-08T05:28:13.545468+0000","last_active":"2023-11-08T05:30:44.490523+0000","last_peered":"2023-11-08T05:30:44.490523+0000","last_clean":"2023-11-08T05:28:01.456723+0000","last_became_active":"2023-11-08T05:28:13.545468+0000","last_became_peered":"2023-11-08T05:28:13.545468+0000","last_unstale":"2023-11-08T05:30:44.490523+0000","last_undegraded":"2023-11-08T05:30:44.490523+0000","last_fullsized":"2023-11-08T05:28:13.537207+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_clean_scrub_stamp":"2023-11-08T04:52:45.528796+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":465,"reported_epoch":458,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:44.491907+0000","last_change":"2023-11-08T05:29:44.395350+0000","last_active":"2023-11-08T05:30:44.491907+0000","last_peered":"2023-11-08T05:30:44.491907+0000","last_clean":"2023-11-08T05:30:44.491907+0000","last_became_active":"2023-11-08T05:29:44.395012+0000","last_became_peered":"2023-11-08T05:29:44.395012+0000","last_unstale":"2023-11-08T05:30:44.491907+0000","last_undegraded":"2023-11-08T05:30:44.491907+0000","last_fullsized":"2023-11-08T05:30:44.491907+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490629+0000","last_change":"2023-11-08T05:28:13.563357+0000","last_active":"2023-11-08T05:30:44.490629+0000","last_peered":"2023-11-08T05:30:44.490629+0000","last_clean":"2023-11-08T05:28:01.457691+0000","last_became_active":"2023-11-08T05:28:13.563357+0000","last_became_peered":"2023-11-08T05:28:13.563357+0000","last_unstale":"2023-11-08T05:30:44.490629+0000","last_undegraded":"2023-11-08T05:30:44.490629+0000","last_fullsized":"2023-11-08T05:28:13.541614+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490691+0000","last_change":"2023-11-08T05:28:13.550672+0000","last_active":"2023-11-08T05:30:44.490691+0000","last_peered":"2023-11-08T05:30:44.490691+0000","last_clean":"2023-11-08T05:28:01.457904+0000","last_became_active":"2023-11-08T05:28:13.550672+0000","last_became_peered":"2023-11-08T05:28:13.550672+0000","last_unstale":"2023-11-08T05:30:44.490691+0000","last_undegraded":"2023-11-08T05:30:44.490691+0000","last_fullsized":"2023-11-08T05:28:13.539595+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490805+0000","last_change":"2023-11-08T05:28:13.557968+0000","last_active":"2023-11-08T05:30:44.490805+0000","last_peered":"2023-11-08T05:30:44.490805+0000","last_clean":"2023-11-08T05:28:01.458433+0000","last_became_active":"2023-11-08T05:28:13.557968+0000","last_became_peered":"2023-11-08T05:28:13.557968+0000","last_unstale":"2023-11-08T05:30:44.490805+0000","last_undegraded":"2023-11-08T05:30:44.490805+0000","last_fullsized":"2023-11-08T05:28:13.542832+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":1,"acting":1,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":12,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738828,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":645040,"kb_used_data":5352,"kb_used_omap":275,"kb_used_meta":639404,"kb_avail":93726800,"statfs":{"total":96636764160,"available":95976243200,"internally_reserved":0,"allocated":5480448,"data_stored":2697647,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":281650,"internal_metadata":654750670},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:59 2023","interfaces":[{"interface":"back","average":{"1min":0.30499999999999999,"5min":0.439,"15min":0.36099999999999999},"min":{"1min":0.17399999999999999,"5min":0.14499999999999999,"15min":0.14499999999999999},"max":{"1min":0.60099999999999998,"5min":4.9379999999999997,"15min":4.9379999999999997},"last":0.379},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.49399999999999999,"15min":0.42399999999999999},"min":{"1min":0.221,"5min":0.19600000000000001,"15min":0.17000000000000001},"max":{"1min":0.63200000000000001,"5min":4.9859999999999998,"15min":4.9859999999999998},"last":0.40899999999999997}]},{"osd":1,"last update":"Wed Nov 8 05:30:03 2023","interfaces":[{"interface":"back","average":{"1min":0.28499999999999998,"5min":0.65100000000000002,"15min":0.47099999999999997},"min":{"1min":0.20100000000000001,"5min":0.153,"15min":0.153},"max":{"1min":0.39800000000000002,"5min":23.242999999999999,"15min":23.242999999999999},"last":0.28699999999999998},{"interface":"front","average":{"1min":0.29499999999999998,"5min":0.63100000000000001,"15min":0.45900000000000002},"min":{"1min":0.224,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":0.436,"5min":23.251000000000001,"15min":23.251000000000001},"last":0.311}]}]},{"osd":0,"up_from":8,"seq":34359738798,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1339228,"kb_used_data":398304,"kb_used_omap":291,"kb_used_meta":936476,"kb_avail":93032612,"statfs":{"total":96636764160,"available":95265394688,"internally_reserved":0,"allocated":407863296,"data_stored":405103900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":298462,"internal_metadata":958951970},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.436,"15min":0.36299999999999999},"min":{"1min":0.19700000000000001,"5min":0.14699999999999999,"15min":0.13700000000000001},"max":{"1min":3.222,"5min":6.5499999999999998,"15min":6.9989999999999997},"last":0.245},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.46300000000000002,"15min":0.42299999999999999},"min":{"1min":0.17799999999999999,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":5.0030000000000001,"5min":6.0629999999999997,"15min":9.6869999999999994},"last":0.28599999999999998}]},{"osd":2,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.51600000000000001,"15min":0.47099999999999997},"min":{"1min":0.21099999999999999,"5min":0.153,"15min":0.153},"max":{"1min":5.7439999999999998,"5min":5.8280000000000003,"15min":9.7639999999999993},"last":0.252},{"interface":"front","average":{"1min":0.497,"5min":0.56200000000000006,"15min":0.42299999999999999},"min":{"1min":0.20399999999999999,"5min":0.187,"15min":0.16800000000000001},"max":{"1min":3.9319999999999999,"5min":8.8040000000000003,"15min":8.8040000000000003},"last":0.29899999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738830,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":650100,"kb_used_data":5356,"kb_used_omap":214,"kb_used_meta":644521,"kb_avail":93721740,"statfs":{"total":96636764160,"available":95971061760,"internally_reserved":0,"allocated":5484544,"data_stored":2698616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":219335,"internal_metadata":659990329},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:56 2023","interfaces":[{"interface":"back","average":{"1min":0.30399999999999999,"5min":0.34699999999999998,"15min":0.36499999999999999},"min":{"1min":0.23999999999999999,"5min":0.14199999999999999,"15min":0.14199999999999999},"max":{"1min":0.42399999999999999,"5min":1.595,"15min":7.4249999999999998},"last":0.17100000000000001},{"interface":"front","average":{"1min":0.35499999999999998,"5min":0.40899999999999997,"15min":0.376},"min":{"1min":0.23999999999999999,"5min":0.19600000000000001,"15min":0.183},"max":{"1min":0.46999999999999997,"5min":2.6859999999999999,"15min":2.6859999999999999},"last":0.23000000000000001}]},{"osd":2,"last update":"Wed Nov 8 05:29:57 2023","interfaces":[{"interface":"back","average":{"1min":0.28199999999999997,"5min":0.36099999999999999,"15min":0.373},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":0.34399999999999997,"5min":2.2650000000000001,"15min":5.8209999999999997},"last":0.31900000000000001},{"interface":"front","average":{"1min":0.28599999999999998,"5min":0.31,"15min":0.35799999999999998},"min":{"1min":0.17199999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.35699999999999998,"5min":1.137,"15min":4.2350000000000003},"last":0.35199999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-08T05:30:54.237 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-11-08T05:30:54.522 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T05:30:54.524 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2023-11-08T05:30:54.535 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":1601,"stamp":"2023-11-08T05:30:54.061537+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":10,"acting":13,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":109,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2634368,"kb_used_data":409012,"kb_used_omap":780,"kb_used_meta":2220403,"kb_avail":280481152,"statfs":{"total":289910292480,"available":287212699648,"internally_reserved":0,"allocated":418828288,"data_stored":410500163,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":799447,"internal_metadata":2273692969},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-536782902,"num_objects":-164,"num_object_clones":0,"num_object_copies":-328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-115,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-164,"num_whiteouts":0,"num_read":-122056,"num_read_kb":-39011942,"num_write":-413642,"num_write_kb":-151705339,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-8,"num_bytes_recovered":-16922624,"num_keys_recovered":-2,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.424817"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490644+0000","last_change":"2023-11-08T05:28:13.557071+0000","last_active":"2023-11-08T05:30:44.490644+0000","last_peered":"2023-11-08T05:30:44.490644+0000","last_clean":"2023-11-08T05:28:01.458439+0000","last_became_active":"2023-11-08T05:28:13.557071+0000","last_became_peered":"2023-11-08T05:28:13.557071+0000","last_unstale":"2023-11-08T05:30:44.490644+0000","last_undegraded":"2023-11-08T05:30:44.490644+0000","last_fullsized":"2023-11-08T05:28:13.547025+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":459,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:45.794760+0000","last_change":"2023-11-08T05:30:45.794760+0000","last_active":"2023-11-08T05:30:45.794760+0000","last_peered":"2023-11-08T05:30:45.794760+0000","last_clean":"2023-11-08T05:30:45.794760+0000","last_became_active":"2023-11-08T05:30:45.794398+0000","last_became_peered":"2023-11-08T05:30:45.794398+0000","last_unstale":"2023-11-08T05:30:45.794760+0000","last_undegraded":"2023-11-08T05:30:45.794760+0000","last_fullsized":"2023-11-08T05:30:45.794760+0000","mapping_epoch":458,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":459,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":465,"reported_epoch":459,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:45.795085+0000","last_change":"2023-11-08T05:30:45.795085+0000","last_active":"2023-11-08T05:30:45.795085+0000","last_peered":"2023-11-08T05:30:45.795085+0000","last_clean":"2023-11-08T05:30:45.795085+0000","last_became_active":"2023-11-08T05:30:45.794743+0000","last_became_peered":"2023-11-08T05:30:45.794743+0000","last_unstale":"2023-11-08T05:30:45.795085+0000","last_undegraded":"2023-11-08T05:30:45.795085+0000","last_fullsized":"2023-11-08T05:30:45.795085+0000","mapping_epoch":458,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":459,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":454,"reported_epoch":458,"state":"active+clean","last_fresh":"2023-11-08T05:30:44.490618+0000","last_change":"2023-11-08T04:52:47.548597+0000","last_active":"2023-11-08T05:30:44.490618+0000","last_peered":"2023-11-08T05:30:44.490618+0000","last_clean":"2023-11-08T05:30:44.490618+0000","last_became_active":"2023-11-08T04:52:47.548294+0000","last_became_peered":"2023-11-08T04:52:47.548294+0000","last_unstale":"2023-11-08T05:30:44.490618+0000","last_undegraded":"2023-11-08T05:30:44.490618+0000","last_fullsized":"2023-11-08T05:30:44.490618+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":460,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490523+0000","last_change":"2023-11-08T05:28:13.545468+0000","last_active":"2023-11-08T05:30:44.490523+0000","last_peered":"2023-11-08T05:30:44.490523+0000","last_clean":"2023-11-08T05:28:01.456723+0000","last_became_active":"2023-11-08T05:28:13.545468+0000","last_became_peered":"2023-11-08T05:28:13.545468+0000","last_unstale":"2023-11-08T05:30:44.490523+0000","last_undegraded":"2023-11-08T05:30:44.490523+0000","last_fullsized":"2023-11-08T05:28:13.537207+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_clean_scrub_stamp":"2023-11-08T04:52:45.528796+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":465,"reported_epoch":458,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:44.491907+0000","last_change":"2023-11-08T05:29:44.395350+0000","last_active":"2023-11-08T05:30:44.491907+0000","last_peered":"2023-11-08T05:30:44.491907+0000","last_clean":"2023-11-08T05:30:44.491907+0000","last_became_active":"2023-11-08T05:29:44.395012+0000","last_became_peered":"2023-11-08T05:29:44.395012+0000","last_unstale":"2023-11-08T05:30:44.491907+0000","last_undegraded":"2023-11-08T05:30:44.491907+0000","last_fullsized":"2023-11-08T05:30:44.491907+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490629+0000","last_change":"2023-11-08T05:28:13.563357+0000","last_active":"2023-11-08T05:30:44.490629+0000","last_peered":"2023-11-08T05:30:44.490629+0000","last_clean":"2023-11-08T05:28:01.457691+0000","last_became_active":"2023-11-08T05:28:13.563357+0000","last_became_peered":"2023-11-08T05:28:13.563357+0000","last_unstale":"2023-11-08T05:30:44.490629+0000","last_undegraded":"2023-11-08T05:30:44.490629+0000","last_fullsized":"2023-11-08T05:28:13.541614+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490691+0000","last_change":"2023-11-08T05:28:13.550672+0000","last_active":"2023-11-08T05:30:44.490691+0000","last_peered":"2023-11-08T05:30:44.490691+0000","last_clean":"2023-11-08T05:28:01.457904+0000","last_became_active":"2023-11-08T05:28:13.550672+0000","last_became_peered":"2023-11-08T05:28:13.550672+0000","last_unstale":"2023-11-08T05:30:44.490691+0000","last_undegraded":"2023-11-08T05:30:44.490691+0000","last_fullsized":"2023-11-08T05:28:13.539595+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490805+0000","last_change":"2023-11-08T05:28:13.557968+0000","last_active":"2023-11-08T05:30:44.490805+0000","last_peered":"2023-11-08T05:30:44.490805+0000","last_clean":"2023-11-08T05:28:01.458433+0000","last_became_active":"2023-11-08T05:28:13.557968+0000","last_became_peered":"2023-11-08T05:28:13.557968+0000","last_unstale":"2023-11-08T05:30:44.490805+0000","last_undegraded":"2023-11-08T05:30:44.490805+0000","last_fullsized":"2023-11-08T05:28:13.542832+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":1,"acting":1,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":12,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738828,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":645040,"kb_used_data":5352,"kb_used_omap":275,"kb_used_meta":639404,"kb_avail":93726800,"statfs":{"total":96636764160,"available":95976243200,"internally_reserved":0,"allocated":5480448,"data_stored":2697647,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":281650,"internal_metadata":654750670},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:59 2023","interfaces":[{"interface":"back","average":{"1min":0.30499999999999999,"5min":0.439,"15min":0.36099999999999999},"min":{"1min":0.17399999999999999,"5min":0.14499999999999999,"15min":0.14499999999999999},"max":{"1min":0.60099999999999998,"5min":4.9379999999999997,"15min":4.9379999999999997},"last":0.379},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.49399999999999999,"15min":0.42399999999999999},"min":{"1min":0.221,"5min":0.19600000000000001,"15min":0.17000000000000001},"max":{"1min":0.63200000000000001,"5min":4.9859999999999998,"15min":4.9859999999999998},"last":0.40899999999999997}]},{"osd":1,"last update":"Wed Nov 8 05:30:03 2023","interfaces":[{"interface":"back","average":{"1min":0.28499999999999998,"5min":0.65100000000000002,"15min":0.47099999999999997},"min":{"1min":0.20100000000000001,"5min":0.153,"15min":0.153},"max":{"1min":0.39800000000000002,"5min":23.242999999999999,"15min":23.242999999999999},"last":0.28699999999999998},{"interface":"front","average":{"1min":0.29499999999999998,"5min":0.63100000000000001,"15min":0.45900000000000002},"min":{"1min":0.224,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":0.436,"5min":23.251000000000001,"15min":23.251000000000001},"last":0.311}]}]},{"osd":0,"up_from":8,"seq":34359738798,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1339228,"kb_used_data":398304,"kb_used_omap":291,"kb_used_meta":936476,"kb_avail":93032612,"statfs":{"total":96636764160,"available":95265394688,"internally_reserved":0,"allocated":407863296,"data_stored":405103900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":298462,"internal_metadata":958951970},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.436,"15min":0.36299999999999999},"min":{"1min":0.19700000000000001,"5min":0.14699999999999999,"15min":0.13700000000000001},"max":{"1min":3.222,"5min":6.5499999999999998,"15min":6.9989999999999997},"last":0.245},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.46300000000000002,"15min":0.42299999999999999},"min":{"1min":0.17799999999999999,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":5.0030000000000001,"5min":6.0629999999999997,"15min":9.6869999999999994},"last":0.28599999999999998}]},{"osd":2,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.51600000000000001,"15min":0.47099999999999997},"min":{"1min":0.21099999999999999,"5min":0.153,"15min":0.153},"max":{"1min":5.7439999999999998,"5min":5.8280000000000003,"15min":9.7639999999999993},"last":0.252},{"interface":"front","average":{"1min":0.497,"5min":0.56200000000000006,"15min":0.42299999999999999},"min":{"1min":0.20399999999999999,"5min":0.187,"15min":0.16800000000000001},"max":{"1min":3.9319999999999999,"5min":8.8040000000000003,"15min":8.8040000000000003},"last":0.29899999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738830,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":650100,"kb_used_data":5356,"kb_used_omap":214,"kb_used_meta":644521,"kb_avail":93721740,"statfs":{"total":96636764160,"available":95971061760,"internally_reserved":0,"allocated":5484544,"data_stored":2698616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":219335,"internal_metadata":659990329},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:56 2023","interfaces":[{"interface":"back","average":{"1min":0.30399999999999999,"5min":0.34699999999999998,"15min":0.36499999999999999},"min":{"1min":0.23999999999999999,"5min":0.14199999999999999,"15min":0.14199999999999999},"max":{"1min":0.42399999999999999,"5min":1.595,"15min":7.4249999999999998},"last":0.17100000000000001},{"interface":"front","average":{"1min":0.35499999999999998,"5min":0.40899999999999997,"15min":0.376},"min":{"1min":0.23999999999999999,"5min":0.19600000000000001,"15min":0.183},"max":{"1min":0.46999999999999997,"5min":2.6859999999999999,"15min":2.6859999999999999},"last":0.23000000000000001}]},{"osd":2,"last update":"Wed Nov 8 05:29:57 2023","interfaces":[{"interface":"back","average":{"1min":0.28199999999999997,"5min":0.36099999999999999,"15min":0.373},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":0.34399999999999997,"5min":2.2650000000000001,"15min":5.8209999999999997},"last":0.31900000000000001},{"interface":"front","average":{"1min":0.28599999999999998,"5min":0.31,"15min":0.35799999999999998},"min":{"1min":0.17199999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.35699999999999998,"5min":1.137,"15min":4.2350000000000003},"last":0.35199999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-08T05:30:54.536 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-11-08T05:30:54.537 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 459, 'reported_epoch': 458, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:30:44.490644+0000', 'last_change': '2023-11-08T05:28:13.557071+0000', 'last_active': '2023-11-08T05:30:44.490644+0000', 'last_peered': '2023-11-08T05:30:44.490644+0000', 'last_clean': '2023-11-08T05:28:01.458439+0000', 'last_became_active': '2023-11-08T05:28:13.557071+0000', 'last_became_peered': '2023-11-08T05:28:13.557071+0000', 'last_unstale': '2023-11-08T05:30:44.490644+0000', 'last_undegraded': '2023-11-08T05:30:44.490644+0000', 'last_fullsized': '2023-11-08T05:28:13.547025+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:30:54.537 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-11-08T05:30:54.537 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 11, 'reported_epoch': 459, 'state': 'active+clean+remapped', 'last_fresh': '2023-11-08T05:30:45.794760+0000', 'last_change': '2023-11-08T05:30:45.794760+0000', 'last_active': '2023-11-08T05:30:45.794760+0000', 'last_peered': '2023-11-08T05:30:45.794760+0000', 'last_clean': '2023-11-08T05:30:45.794760+0000', 'last_became_active': '2023-11-08T05:30:45.794398+0000', 'last_became_peered': '2023-11-08T05:30:45.794398+0000', 'last_unstale': '2023-11-08T05:30:45.794760+0000', 'last_undegraded': '2023-11-08T05:30:45.794760+0000', 'last_fullsized': '2023-11-08T05:30:45.794760+0000', 'mapping_epoch': 458, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 459, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2], 'acting': [2, 1], 'avail_no_missing': ['2', '1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2023-11-08T05:30:54.538 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.1 is not active+clean 2023-11-08T05:30:54.538 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.1', 'version': "0'0", 'reported_seq': 465, 'reported_epoch': 459, 'state': 'active+clean+remapped', 'last_fresh': '2023-11-08T05:30:45.795085+0000', 'last_change': '2023-11-08T05:30:45.795085+0000', 'last_active': '2023-11-08T05:30:45.795085+0000', 'last_peered': '2023-11-08T05:30:45.795085+0000', 'last_clean': '2023-11-08T05:30:45.795085+0000', 'last_became_active': '2023-11-08T05:30:45.794743+0000', 'last_became_peered': '2023-11-08T05:30:45.794743+0000', 'last_unstale': '2023-11-08T05:30:45.795085+0000', 'last_undegraded': '2023-11-08T05:30:45.795085+0000', 'last_fullsized': '2023-11-08T05:30:45.795085+0000', 'mapping_epoch': 458, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 459, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2], 'acting': [2, 1], 'avail_no_missing': ['2', '1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2023-11-08T05:30:54.538 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-11-08T05:30:54.539 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "0'0", 'reported_seq': 460, 'reported_epoch': 458, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:30:44.490523+0000', 'last_change': '2023-11-08T05:28:13.545468+0000', 'last_active': '2023-11-08T05:30:44.490523+0000', 'last_peered': '2023-11-08T05:30:44.490523+0000', 'last_clean': '2023-11-08T05:28:01.456723+0000', 'last_became_active': '2023-11-08T05:28:13.545468+0000', 'last_became_peered': '2023-11-08T05:28:13.545468+0000', 'last_unstale': '2023-11-08T05:30:44.490523+0000', 'last_undegraded': '2023-11-08T05:30:44.490523+0000', 'last_fullsized': '2023-11-08T05:28:13.537207+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 9, 'last_epoch_clean': 10, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:45.528796+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:45.528796+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:45.528796+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:30:54.539 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.0 is not active+clean 2023-11-08T05:30:54.539 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.0', 'version': "0'0", 'reported_seq': 465, 'reported_epoch': 458, 'state': 'active+clean+remapped', 'last_fresh': '2023-11-08T05:30:44.491907+0000', 'last_change': '2023-11-08T05:29:44.395350+0000', 'last_active': '2023-11-08T05:30:44.491907+0000', 'last_peered': '2023-11-08T05:30:44.491907+0000', 'last_clean': '2023-11-08T05:30:44.491907+0000', 'last_became_active': '2023-11-08T05:29:44.395012+0000', 'last_became_peered': '2023-11-08T05:29:44.395012+0000', 'last_unstale': '2023-11-08T05:30:44.491907+0000', 'last_undegraded': '2023-11-08T05:30:44.491907+0000', 'last_fullsized': '2023-11-08T05:30:44.491907+0000', 'mapping_epoch': 455, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 456, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2], 'acting': [2, 1], 'avail_no_missing': ['2', '1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2023-11-08T05:30:54.540 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-11-08T05:30:54.540 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 459, 'reported_epoch': 458, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:30:44.490629+0000', 'last_change': '2023-11-08T05:28:13.563357+0000', 'last_active': '2023-11-08T05:30:44.490629+0000', 'last_peered': '2023-11-08T05:30:44.490629+0000', 'last_clean': '2023-11-08T05:28:01.457691+0000', 'last_became_active': '2023-11-08T05:28:13.563357+0000', 'last_became_peered': '2023-11-08T05:28:13.563357+0000', 'last_unstale': '2023-11-08T05:30:44.490629+0000', 'last_undegraded': '2023-11-08T05:30:44.490629+0000', 'last_fullsized': '2023-11-08T05:28:13.541614+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:30:54.541 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-11-08T05:30:54.541 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 459, 'reported_epoch': 458, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:30:44.490691+0000', 'last_change': '2023-11-08T05:28:13.550672+0000', 'last_active': '2023-11-08T05:30:44.490691+0000', 'last_peered': '2023-11-08T05:30:44.490691+0000', 'last_clean': '2023-11-08T05:28:01.457904+0000', 'last_became_active': '2023-11-08T05:28:13.550672+0000', 'last_became_peered': '2023-11-08T05:28:13.550672+0000', 'last_unstale': '2023-11-08T05:30:44.490691+0000', 'last_undegraded': '2023-11-08T05:30:44.490691+0000', 'last_fullsized': '2023-11-08T05:28:13.539595+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:30:54.541 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-11-08T05:30:54.541 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 459, 'reported_epoch': 458, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:30:44.490805+0000', 'last_change': '2023-11-08T05:28:13.557968+0000', 'last_active': '2023-11-08T05:30:44.490805+0000', 'last_peered': '2023-11-08T05:30:44.490805+0000', 'last_clean': '2023-11-08T05:28:01.458433+0000', 'last_became_active': '2023-11-08T05:28:13.557968+0000', 'last_became_peered': '2023-11-08T05:28:13.557968+0000', 'last_unstale': '2023-11-08T05:30:44.490805+0000', 'last_undegraded': '2023-11-08T05:30:44.490805+0000', 'last_fullsized': '2023-11-08T05:28:13.542832+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:30:54.542 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-11-08T05:30:54.763 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~157s 2023-11-08T05:30:54.887 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T05:30:54.888 INFO:teuthology.orchestra.run.smithi087.stdout:{"fsid":"ab4cbc06-449d-497d-b513-220d9db2b590","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 5 pgs undersized","count":5},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2297,"monmap":{"epoch":1,"min_mon_release_name":"pacific","num_mons":1},"osdmap":{"epoch":459,"num_osds":3,"num_up_osds":2,"osd_up_since":1699421292,"num_in_osds":3,"osd_in_since":1699419158,"num_remapped_pgs":3},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean+remapped","count":3},{"state_name":"active+clean","count":1}],"num_pgs":9,"num_pools":2,"num_objects":0,"data_bytes":0,"bytes_used":2697592832,"bytes_avail":287212699648,"bytes_total":289910292480},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":41,"modified":"2023-11-08T05:29:32.044544+0000","services":{}},"progress_events":{}} 2023-11-08T05:30:54.900 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-11-08T05:30:54.901 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-11-08T05:30:55.183 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T05:30:55.184 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2023-11-08T05:30:55.196 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":1601,"stamp":"2023-11-08T05:30:54.061537+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":10,"acting":13,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":109,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2634368,"kb_used_data":409012,"kb_used_omap":780,"kb_used_meta":2220403,"kb_avail":280481152,"statfs":{"total":289910292480,"available":287212699648,"internally_reserved":0,"allocated":418828288,"data_stored":410500163,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":799447,"internal_metadata":2273692969},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-536782902,"num_objects":-164,"num_object_clones":0,"num_object_copies":-328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-115,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-164,"num_whiteouts":0,"num_read":-122056,"num_read_kb":-39011942,"num_write":-413642,"num_write_kb":-151705339,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-8,"num_bytes_recovered":-16922624,"num_keys_recovered":-2,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.424817"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490644+0000","last_change":"2023-11-08T05:28:13.557071+0000","last_active":"2023-11-08T05:30:44.490644+0000","last_peered":"2023-11-08T05:30:44.490644+0000","last_clean":"2023-11-08T05:28:01.458439+0000","last_became_active":"2023-11-08T05:28:13.557071+0000","last_became_peered":"2023-11-08T05:28:13.557071+0000","last_unstale":"2023-11-08T05:30:44.490644+0000","last_undegraded":"2023-11-08T05:30:44.490644+0000","last_fullsized":"2023-11-08T05:28:13.547025+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":459,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:45.794760+0000","last_change":"2023-11-08T05:30:45.794760+0000","last_active":"2023-11-08T05:30:45.794760+0000","last_peered":"2023-11-08T05:30:45.794760+0000","last_clean":"2023-11-08T05:30:45.794760+0000","last_became_active":"2023-11-08T05:30:45.794398+0000","last_became_peered":"2023-11-08T05:30:45.794398+0000","last_unstale":"2023-11-08T05:30:45.794760+0000","last_undegraded":"2023-11-08T05:30:45.794760+0000","last_fullsized":"2023-11-08T05:30:45.794760+0000","mapping_epoch":458,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":459,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":465,"reported_epoch":459,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:45.795085+0000","last_change":"2023-11-08T05:30:45.795085+0000","last_active":"2023-11-08T05:30:45.795085+0000","last_peered":"2023-11-08T05:30:45.795085+0000","last_clean":"2023-11-08T05:30:45.795085+0000","last_became_active":"2023-11-08T05:30:45.794743+0000","last_became_peered":"2023-11-08T05:30:45.794743+0000","last_unstale":"2023-11-08T05:30:45.795085+0000","last_undegraded":"2023-11-08T05:30:45.795085+0000","last_fullsized":"2023-11-08T05:30:45.795085+0000","mapping_epoch":458,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":459,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":454,"reported_epoch":458,"state":"active+clean","last_fresh":"2023-11-08T05:30:44.490618+0000","last_change":"2023-11-08T04:52:47.548597+0000","last_active":"2023-11-08T05:30:44.490618+0000","last_peered":"2023-11-08T05:30:44.490618+0000","last_clean":"2023-11-08T05:30:44.490618+0000","last_became_active":"2023-11-08T04:52:47.548294+0000","last_became_peered":"2023-11-08T04:52:47.548294+0000","last_unstale":"2023-11-08T05:30:44.490618+0000","last_undegraded":"2023-11-08T05:30:44.490618+0000","last_fullsized":"2023-11-08T05:30:44.490618+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":460,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490523+0000","last_change":"2023-11-08T05:28:13.545468+0000","last_active":"2023-11-08T05:30:44.490523+0000","last_peered":"2023-11-08T05:30:44.490523+0000","last_clean":"2023-11-08T05:28:01.456723+0000","last_became_active":"2023-11-08T05:28:13.545468+0000","last_became_peered":"2023-11-08T05:28:13.545468+0000","last_unstale":"2023-11-08T05:30:44.490523+0000","last_undegraded":"2023-11-08T05:30:44.490523+0000","last_fullsized":"2023-11-08T05:28:13.537207+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_clean_scrub_stamp":"2023-11-08T04:52:45.528796+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":465,"reported_epoch":458,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:44.491907+0000","last_change":"2023-11-08T05:29:44.395350+0000","last_active":"2023-11-08T05:30:44.491907+0000","last_peered":"2023-11-08T05:30:44.491907+0000","last_clean":"2023-11-08T05:30:44.491907+0000","last_became_active":"2023-11-08T05:29:44.395012+0000","last_became_peered":"2023-11-08T05:29:44.395012+0000","last_unstale":"2023-11-08T05:30:44.491907+0000","last_undegraded":"2023-11-08T05:30:44.491907+0000","last_fullsized":"2023-11-08T05:30:44.491907+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490629+0000","last_change":"2023-11-08T05:28:13.563357+0000","last_active":"2023-11-08T05:30:44.490629+0000","last_peered":"2023-11-08T05:30:44.490629+0000","last_clean":"2023-11-08T05:28:01.457691+0000","last_became_active":"2023-11-08T05:28:13.563357+0000","last_became_peered":"2023-11-08T05:28:13.563357+0000","last_unstale":"2023-11-08T05:30:44.490629+0000","last_undegraded":"2023-11-08T05:30:44.490629+0000","last_fullsized":"2023-11-08T05:28:13.541614+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490691+0000","last_change":"2023-11-08T05:28:13.550672+0000","last_active":"2023-11-08T05:30:44.490691+0000","last_peered":"2023-11-08T05:30:44.490691+0000","last_clean":"2023-11-08T05:28:01.457904+0000","last_became_active":"2023-11-08T05:28:13.550672+0000","last_became_peered":"2023-11-08T05:28:13.550672+0000","last_unstale":"2023-11-08T05:30:44.490691+0000","last_undegraded":"2023-11-08T05:30:44.490691+0000","last_fullsized":"2023-11-08T05:28:13.539595+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490805+0000","last_change":"2023-11-08T05:28:13.557968+0000","last_active":"2023-11-08T05:30:44.490805+0000","last_peered":"2023-11-08T05:30:44.490805+0000","last_clean":"2023-11-08T05:28:01.458433+0000","last_became_active":"2023-11-08T05:28:13.557968+0000","last_became_peered":"2023-11-08T05:28:13.557968+0000","last_unstale":"2023-11-08T05:30:44.490805+0000","last_undegraded":"2023-11-08T05:30:44.490805+0000","last_fullsized":"2023-11-08T05:28:13.542832+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":1,"acting":1,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":12,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738828,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":645040,"kb_used_data":5352,"kb_used_omap":275,"kb_used_meta":639404,"kb_avail":93726800,"statfs":{"total":96636764160,"available":95976243200,"internally_reserved":0,"allocated":5480448,"data_stored":2697647,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":281650,"internal_metadata":654750670},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:59 2023","interfaces":[{"interface":"back","average":{"1min":0.30499999999999999,"5min":0.439,"15min":0.36099999999999999},"min":{"1min":0.17399999999999999,"5min":0.14499999999999999,"15min":0.14499999999999999},"max":{"1min":0.60099999999999998,"5min":4.9379999999999997,"15min":4.9379999999999997},"last":0.379},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.49399999999999999,"15min":0.42399999999999999},"min":{"1min":0.221,"5min":0.19600000000000001,"15min":0.17000000000000001},"max":{"1min":0.63200000000000001,"5min":4.9859999999999998,"15min":4.9859999999999998},"last":0.40899999999999997}]},{"osd":1,"last update":"Wed Nov 8 05:30:03 2023","interfaces":[{"interface":"back","average":{"1min":0.28499999999999998,"5min":0.65100000000000002,"15min":0.47099999999999997},"min":{"1min":0.20100000000000001,"5min":0.153,"15min":0.153},"max":{"1min":0.39800000000000002,"5min":23.242999999999999,"15min":23.242999999999999},"last":0.28699999999999998},{"interface":"front","average":{"1min":0.29499999999999998,"5min":0.63100000000000001,"15min":0.45900000000000002},"min":{"1min":0.224,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":0.436,"5min":23.251000000000001,"15min":23.251000000000001},"last":0.311}]}]},{"osd":0,"up_from":8,"seq":34359738798,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1339228,"kb_used_data":398304,"kb_used_omap":291,"kb_used_meta":936476,"kb_avail":93032612,"statfs":{"total":96636764160,"available":95265394688,"internally_reserved":0,"allocated":407863296,"data_stored":405103900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":298462,"internal_metadata":958951970},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.436,"15min":0.36299999999999999},"min":{"1min":0.19700000000000001,"5min":0.14699999999999999,"15min":0.13700000000000001},"max":{"1min":3.222,"5min":6.5499999999999998,"15min":6.9989999999999997},"last":0.245},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.46300000000000002,"15min":0.42299999999999999},"min":{"1min":0.17799999999999999,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":5.0030000000000001,"5min":6.0629999999999997,"15min":9.6869999999999994},"last":0.28599999999999998}]},{"osd":2,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.51600000000000001,"15min":0.47099999999999997},"min":{"1min":0.21099999999999999,"5min":0.153,"15min":0.153},"max":{"1min":5.7439999999999998,"5min":5.8280000000000003,"15min":9.7639999999999993},"last":0.252},{"interface":"front","average":{"1min":0.497,"5min":0.56200000000000006,"15min":0.42299999999999999},"min":{"1min":0.20399999999999999,"5min":0.187,"15min":0.16800000000000001},"max":{"1min":3.9319999999999999,"5min":8.8040000000000003,"15min":8.8040000000000003},"last":0.29899999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738830,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":650100,"kb_used_data":5356,"kb_used_omap":214,"kb_used_meta":644521,"kb_avail":93721740,"statfs":{"total":96636764160,"available":95971061760,"internally_reserved":0,"allocated":5484544,"data_stored":2698616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":219335,"internal_metadata":659990329},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:56 2023","interfaces":[{"interface":"back","average":{"1min":0.30399999999999999,"5min":0.34699999999999998,"15min":0.36499999999999999},"min":{"1min":0.23999999999999999,"5min":0.14199999999999999,"15min":0.14199999999999999},"max":{"1min":0.42399999999999999,"5min":1.595,"15min":7.4249999999999998},"last":0.17100000000000001},{"interface":"front","average":{"1min":0.35499999999999998,"5min":0.40899999999999997,"15min":0.376},"min":{"1min":0.23999999999999999,"5min":0.19600000000000001,"15min":0.183},"max":{"1min":0.46999999999999997,"5min":2.6859999999999999,"15min":2.6859999999999999},"last":0.23000000000000001}]},{"osd":2,"last update":"Wed Nov 8 05:29:57 2023","interfaces":[{"interface":"back","average":{"1min":0.28199999999999997,"5min":0.36099999999999999,"15min":0.373},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":0.34399999999999997,"5min":2.2650000000000001,"15min":5.8209999999999997},"last":0.31900000000000001},{"interface":"front","average":{"1min":0.28599999999999998,"5min":0.31,"15min":0.35799999999999998},"min":{"1min":0.17199999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.35699999999999998,"5min":1.137,"15min":4.2350000000000003},"last":0.35199999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-08T05:30:58.198 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-11-08T05:30:58.495 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T05:30:58.497 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2023-11-08T05:30:58.508 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":1603,"stamp":"2023-11-08T05:30:58.062280+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":10,"acting":13,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":109,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2634368,"kb_used_data":409012,"kb_used_omap":780,"kb_used_meta":2220403,"kb_avail":280481152,"statfs":{"total":289910292480,"available":287212699648,"internally_reserved":0,"allocated":418828288,"data_stored":410500163,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":799447,"internal_metadata":2273692969},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-536782902,"num_objects":-164,"num_object_clones":0,"num_object_copies":-328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-115,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-164,"num_whiteouts":0,"num_read":-122056,"num_read_kb":-39011942,"num_write":-413642,"num_write_kb":-151705339,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-8,"num_bytes_recovered":-16922624,"num_keys_recovered":-2,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.424817"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490644+0000","last_change":"2023-11-08T05:28:13.557071+0000","last_active":"2023-11-08T05:30:44.490644+0000","last_peered":"2023-11-08T05:30:44.490644+0000","last_clean":"2023-11-08T05:28:01.458439+0000","last_became_active":"2023-11-08T05:28:13.557071+0000","last_became_peered":"2023-11-08T05:28:13.557071+0000","last_unstale":"2023-11-08T05:30:44.490644+0000","last_undegraded":"2023-11-08T05:30:44.490644+0000","last_fullsized":"2023-11-08T05:28:13.547025+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":459,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:45.794760+0000","last_change":"2023-11-08T05:30:45.794760+0000","last_active":"2023-11-08T05:30:45.794760+0000","last_peered":"2023-11-08T05:30:45.794760+0000","last_clean":"2023-11-08T05:30:45.794760+0000","last_became_active":"2023-11-08T05:30:45.794398+0000","last_became_peered":"2023-11-08T05:30:45.794398+0000","last_unstale":"2023-11-08T05:30:45.794760+0000","last_undegraded":"2023-11-08T05:30:45.794760+0000","last_fullsized":"2023-11-08T05:30:45.794760+0000","mapping_epoch":458,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":459,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":465,"reported_epoch":459,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:45.795085+0000","last_change":"2023-11-08T05:30:45.795085+0000","last_active":"2023-11-08T05:30:45.795085+0000","last_peered":"2023-11-08T05:30:45.795085+0000","last_clean":"2023-11-08T05:30:45.795085+0000","last_became_active":"2023-11-08T05:30:45.794743+0000","last_became_peered":"2023-11-08T05:30:45.794743+0000","last_unstale":"2023-11-08T05:30:45.795085+0000","last_undegraded":"2023-11-08T05:30:45.795085+0000","last_fullsized":"2023-11-08T05:30:45.795085+0000","mapping_epoch":458,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":459,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":454,"reported_epoch":458,"state":"active+clean","last_fresh":"2023-11-08T05:30:44.490618+0000","last_change":"2023-11-08T04:52:47.548597+0000","last_active":"2023-11-08T05:30:44.490618+0000","last_peered":"2023-11-08T05:30:44.490618+0000","last_clean":"2023-11-08T05:30:44.490618+0000","last_became_active":"2023-11-08T04:52:47.548294+0000","last_became_peered":"2023-11-08T04:52:47.548294+0000","last_unstale":"2023-11-08T05:30:44.490618+0000","last_undegraded":"2023-11-08T05:30:44.490618+0000","last_fullsized":"2023-11-08T05:30:44.490618+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":460,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490523+0000","last_change":"2023-11-08T05:28:13.545468+0000","last_active":"2023-11-08T05:30:44.490523+0000","last_peered":"2023-11-08T05:30:44.490523+0000","last_clean":"2023-11-08T05:28:01.456723+0000","last_became_active":"2023-11-08T05:28:13.545468+0000","last_became_peered":"2023-11-08T05:28:13.545468+0000","last_unstale":"2023-11-08T05:30:44.490523+0000","last_undegraded":"2023-11-08T05:30:44.490523+0000","last_fullsized":"2023-11-08T05:28:13.537207+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_clean_scrub_stamp":"2023-11-08T04:52:45.528796+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":465,"reported_epoch":458,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:44.491907+0000","last_change":"2023-11-08T05:29:44.395350+0000","last_active":"2023-11-08T05:30:44.491907+0000","last_peered":"2023-11-08T05:30:44.491907+0000","last_clean":"2023-11-08T05:30:44.491907+0000","last_became_active":"2023-11-08T05:29:44.395012+0000","last_became_peered":"2023-11-08T05:29:44.395012+0000","last_unstale":"2023-11-08T05:30:44.491907+0000","last_undegraded":"2023-11-08T05:30:44.491907+0000","last_fullsized":"2023-11-08T05:30:44.491907+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490629+0000","last_change":"2023-11-08T05:28:13.563357+0000","last_active":"2023-11-08T05:30:44.490629+0000","last_peered":"2023-11-08T05:30:44.490629+0000","last_clean":"2023-11-08T05:28:01.457691+0000","last_became_active":"2023-11-08T05:28:13.563357+0000","last_became_peered":"2023-11-08T05:28:13.563357+0000","last_unstale":"2023-11-08T05:30:44.490629+0000","last_undegraded":"2023-11-08T05:30:44.490629+0000","last_fullsized":"2023-11-08T05:28:13.541614+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490691+0000","last_change":"2023-11-08T05:28:13.550672+0000","last_active":"2023-11-08T05:30:44.490691+0000","last_peered":"2023-11-08T05:30:44.490691+0000","last_clean":"2023-11-08T05:28:01.457904+0000","last_became_active":"2023-11-08T05:28:13.550672+0000","last_became_peered":"2023-11-08T05:28:13.550672+0000","last_unstale":"2023-11-08T05:30:44.490691+0000","last_undegraded":"2023-11-08T05:30:44.490691+0000","last_fullsized":"2023-11-08T05:28:13.539595+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490805+0000","last_change":"2023-11-08T05:28:13.557968+0000","last_active":"2023-11-08T05:30:44.490805+0000","last_peered":"2023-11-08T05:30:44.490805+0000","last_clean":"2023-11-08T05:28:01.458433+0000","last_became_active":"2023-11-08T05:28:13.557968+0000","last_became_peered":"2023-11-08T05:28:13.557968+0000","last_unstale":"2023-11-08T05:30:44.490805+0000","last_undegraded":"2023-11-08T05:30:44.490805+0000","last_fullsized":"2023-11-08T05:28:13.542832+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":1,"acting":1,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":12,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738829,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":645040,"kb_used_data":5352,"kb_used_omap":275,"kb_used_meta":639404,"kb_avail":93726800,"statfs":{"total":96636764160,"available":95976243200,"internally_reserved":0,"allocated":5480448,"data_stored":2697647,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":281650,"internal_metadata":654750670},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:59 2023","interfaces":[{"interface":"back","average":{"1min":0.30499999999999999,"5min":0.439,"15min":0.36099999999999999},"min":{"1min":0.17399999999999999,"5min":0.14499999999999999,"15min":0.14499999999999999},"max":{"1min":0.60099999999999998,"5min":4.9379999999999997,"15min":4.9379999999999997},"last":0.379},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.49399999999999999,"15min":0.42399999999999999},"min":{"1min":0.221,"5min":0.19600000000000001,"15min":0.17000000000000001},"max":{"1min":0.63200000000000001,"5min":4.9859999999999998,"15min":4.9859999999999998},"last":0.40899999999999997}]},{"osd":1,"last update":"Wed Nov 8 05:30:03 2023","interfaces":[{"interface":"back","average":{"1min":0.28499999999999998,"5min":0.65100000000000002,"15min":0.47099999999999997},"min":{"1min":0.20100000000000001,"5min":0.153,"15min":0.153},"max":{"1min":0.39800000000000002,"5min":23.242999999999999,"15min":23.242999999999999},"last":0.35999999999999999},{"interface":"front","average":{"1min":0.29499999999999998,"5min":0.63100000000000001,"15min":0.45900000000000002},"min":{"1min":0.224,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":0.436,"5min":23.251000000000001,"15min":23.251000000000001},"last":0.38500000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738798,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1339228,"kb_used_data":398304,"kb_used_omap":291,"kb_used_meta":936476,"kb_avail":93032612,"statfs":{"total":96636764160,"available":95265394688,"internally_reserved":0,"allocated":407863296,"data_stored":405103900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":298462,"internal_metadata":958951970},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.436,"15min":0.36299999999999999},"min":{"1min":0.19700000000000001,"5min":0.14699999999999999,"15min":0.13700000000000001},"max":{"1min":3.222,"5min":6.5499999999999998,"15min":6.9989999999999997},"last":0.245},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.46300000000000002,"15min":0.42299999999999999},"min":{"1min":0.17799999999999999,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":5.0030000000000001,"5min":6.0629999999999997,"15min":9.6869999999999994},"last":0.28599999999999998}]},{"osd":2,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.51600000000000001,"15min":0.47099999999999997},"min":{"1min":0.21099999999999999,"5min":0.153,"15min":0.153},"max":{"1min":5.7439999999999998,"5min":5.8280000000000003,"15min":9.7639999999999993},"last":0.252},{"interface":"front","average":{"1min":0.497,"5min":0.56200000000000006,"15min":0.42299999999999999},"min":{"1min":0.20399999999999999,"5min":0.187,"15min":0.16800000000000001},"max":{"1min":3.9319999999999999,"5min":8.8040000000000003,"15min":8.8040000000000003},"last":0.29899999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738831,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":650100,"kb_used_data":5356,"kb_used_omap":214,"kb_used_meta":644521,"kb_avail":93721740,"statfs":{"total":96636764160,"available":95971061760,"internally_reserved":0,"allocated":5484544,"data_stored":2698616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":219335,"internal_metadata":659990329},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:56 2023","interfaces":[{"interface":"back","average":{"1min":0.30399999999999999,"5min":0.34699999999999998,"15min":0.36499999999999999},"min":{"1min":0.23999999999999999,"5min":0.14199999999999999,"15min":0.14199999999999999},"max":{"1min":0.42399999999999999,"5min":1.595,"15min":7.4249999999999998},"last":0.17100000000000001},{"interface":"front","average":{"1min":0.35499999999999998,"5min":0.40899999999999997,"15min":0.376},"min":{"1min":0.23999999999999999,"5min":0.19600000000000001,"15min":0.183},"max":{"1min":0.46999999999999997,"5min":2.6859999999999999,"15min":2.6859999999999999},"last":0.23000000000000001}]},{"osd":2,"last update":"Wed Nov 8 05:29:57 2023","interfaces":[{"interface":"back","average":{"1min":0.28199999999999997,"5min":0.36099999999999999,"15min":0.373},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":0.34399999999999997,"5min":2.2650000000000001,"15min":5.8209999999999997},"last":0.24099999999999999},{"interface":"front","average":{"1min":0.28599999999999998,"5min":0.31,"15min":0.35799999999999998},"min":{"1min":0.17199999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.35699999999999998,"5min":1.137,"15min":4.2350000000000003},"last":0.223}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-08T05:30:58.509 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-11-08T05:30:58.792 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T05:30:58.794 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2023-11-08T05:30:58.805 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":1603,"stamp":"2023-11-08T05:30:58.062280+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":10,"acting":13,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":109,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2634368,"kb_used_data":409012,"kb_used_omap":780,"kb_used_meta":2220403,"kb_avail":280481152,"statfs":{"total":289910292480,"available":287212699648,"internally_reserved":0,"allocated":418828288,"data_stored":410500163,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":799447,"internal_metadata":2273692969},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-536782902,"num_objects":-164,"num_object_clones":0,"num_object_copies":-328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-115,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-164,"num_whiteouts":0,"num_read":-122056,"num_read_kb":-39011942,"num_write":-413642,"num_write_kb":-151705339,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-8,"num_bytes_recovered":-16922624,"num_keys_recovered":-2,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.424817"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490644+0000","last_change":"2023-11-08T05:28:13.557071+0000","last_active":"2023-11-08T05:30:44.490644+0000","last_peered":"2023-11-08T05:30:44.490644+0000","last_clean":"2023-11-08T05:28:01.458439+0000","last_became_active":"2023-11-08T05:28:13.557071+0000","last_became_peered":"2023-11-08T05:28:13.557071+0000","last_unstale":"2023-11-08T05:30:44.490644+0000","last_undegraded":"2023-11-08T05:30:44.490644+0000","last_fullsized":"2023-11-08T05:28:13.547025+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":459,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:45.794760+0000","last_change":"2023-11-08T05:30:45.794760+0000","last_active":"2023-11-08T05:30:45.794760+0000","last_peered":"2023-11-08T05:30:45.794760+0000","last_clean":"2023-11-08T05:30:45.794760+0000","last_became_active":"2023-11-08T05:30:45.794398+0000","last_became_peered":"2023-11-08T05:30:45.794398+0000","last_unstale":"2023-11-08T05:30:45.794760+0000","last_undegraded":"2023-11-08T05:30:45.794760+0000","last_fullsized":"2023-11-08T05:30:45.794760+0000","mapping_epoch":458,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":459,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":465,"reported_epoch":459,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:45.795085+0000","last_change":"2023-11-08T05:30:45.795085+0000","last_active":"2023-11-08T05:30:45.795085+0000","last_peered":"2023-11-08T05:30:45.795085+0000","last_clean":"2023-11-08T05:30:45.795085+0000","last_became_active":"2023-11-08T05:30:45.794743+0000","last_became_peered":"2023-11-08T05:30:45.794743+0000","last_unstale":"2023-11-08T05:30:45.795085+0000","last_undegraded":"2023-11-08T05:30:45.795085+0000","last_fullsized":"2023-11-08T05:30:45.795085+0000","mapping_epoch":458,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":459,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":454,"reported_epoch":458,"state":"active+clean","last_fresh":"2023-11-08T05:30:44.490618+0000","last_change":"2023-11-08T04:52:47.548597+0000","last_active":"2023-11-08T05:30:44.490618+0000","last_peered":"2023-11-08T05:30:44.490618+0000","last_clean":"2023-11-08T05:30:44.490618+0000","last_became_active":"2023-11-08T04:52:47.548294+0000","last_became_peered":"2023-11-08T04:52:47.548294+0000","last_unstale":"2023-11-08T05:30:44.490618+0000","last_undegraded":"2023-11-08T05:30:44.490618+0000","last_fullsized":"2023-11-08T05:30:44.490618+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":460,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490523+0000","last_change":"2023-11-08T05:28:13.545468+0000","last_active":"2023-11-08T05:30:44.490523+0000","last_peered":"2023-11-08T05:30:44.490523+0000","last_clean":"2023-11-08T05:28:01.456723+0000","last_became_active":"2023-11-08T05:28:13.545468+0000","last_became_peered":"2023-11-08T05:28:13.545468+0000","last_unstale":"2023-11-08T05:30:44.490523+0000","last_undegraded":"2023-11-08T05:30:44.490523+0000","last_fullsized":"2023-11-08T05:28:13.537207+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_clean_scrub_stamp":"2023-11-08T04:52:45.528796+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":465,"reported_epoch":458,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:44.491907+0000","last_change":"2023-11-08T05:29:44.395350+0000","last_active":"2023-11-08T05:30:44.491907+0000","last_peered":"2023-11-08T05:30:44.491907+0000","last_clean":"2023-11-08T05:30:44.491907+0000","last_became_active":"2023-11-08T05:29:44.395012+0000","last_became_peered":"2023-11-08T05:29:44.395012+0000","last_unstale":"2023-11-08T05:30:44.491907+0000","last_undegraded":"2023-11-08T05:30:44.491907+0000","last_fullsized":"2023-11-08T05:30:44.491907+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490629+0000","last_change":"2023-11-08T05:28:13.563357+0000","last_active":"2023-11-08T05:30:44.490629+0000","last_peered":"2023-11-08T05:30:44.490629+0000","last_clean":"2023-11-08T05:28:01.457691+0000","last_became_active":"2023-11-08T05:28:13.563357+0000","last_became_peered":"2023-11-08T05:28:13.563357+0000","last_unstale":"2023-11-08T05:30:44.490629+0000","last_undegraded":"2023-11-08T05:30:44.490629+0000","last_fullsized":"2023-11-08T05:28:13.541614+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490691+0000","last_change":"2023-11-08T05:28:13.550672+0000","last_active":"2023-11-08T05:30:44.490691+0000","last_peered":"2023-11-08T05:30:44.490691+0000","last_clean":"2023-11-08T05:28:01.457904+0000","last_became_active":"2023-11-08T05:28:13.550672+0000","last_became_peered":"2023-11-08T05:28:13.550672+0000","last_unstale":"2023-11-08T05:30:44.490691+0000","last_undegraded":"2023-11-08T05:30:44.490691+0000","last_fullsized":"2023-11-08T05:28:13.539595+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490805+0000","last_change":"2023-11-08T05:28:13.557968+0000","last_active":"2023-11-08T05:30:44.490805+0000","last_peered":"2023-11-08T05:30:44.490805+0000","last_clean":"2023-11-08T05:28:01.458433+0000","last_became_active":"2023-11-08T05:28:13.557968+0000","last_became_peered":"2023-11-08T05:28:13.557968+0000","last_unstale":"2023-11-08T05:30:44.490805+0000","last_undegraded":"2023-11-08T05:30:44.490805+0000","last_fullsized":"2023-11-08T05:28:13.542832+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":1,"acting":1,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":12,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738829,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":645040,"kb_used_data":5352,"kb_used_omap":275,"kb_used_meta":639404,"kb_avail":93726800,"statfs":{"total":96636764160,"available":95976243200,"internally_reserved":0,"allocated":5480448,"data_stored":2697647,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":281650,"internal_metadata":654750670},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:59 2023","interfaces":[{"interface":"back","average":{"1min":0.30499999999999999,"5min":0.439,"15min":0.36099999999999999},"min":{"1min":0.17399999999999999,"5min":0.14499999999999999,"15min":0.14499999999999999},"max":{"1min":0.60099999999999998,"5min":4.9379999999999997,"15min":4.9379999999999997},"last":0.379},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.49399999999999999,"15min":0.42399999999999999},"min":{"1min":0.221,"5min":0.19600000000000001,"15min":0.17000000000000001},"max":{"1min":0.63200000000000001,"5min":4.9859999999999998,"15min":4.9859999999999998},"last":0.40899999999999997}]},{"osd":1,"last update":"Wed Nov 8 05:30:03 2023","interfaces":[{"interface":"back","average":{"1min":0.28499999999999998,"5min":0.65100000000000002,"15min":0.47099999999999997},"min":{"1min":0.20100000000000001,"5min":0.153,"15min":0.153},"max":{"1min":0.39800000000000002,"5min":23.242999999999999,"15min":23.242999999999999},"last":0.35999999999999999},{"interface":"front","average":{"1min":0.29499999999999998,"5min":0.63100000000000001,"15min":0.45900000000000002},"min":{"1min":0.224,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":0.436,"5min":23.251000000000001,"15min":23.251000000000001},"last":0.38500000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738798,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1339228,"kb_used_data":398304,"kb_used_omap":291,"kb_used_meta":936476,"kb_avail":93032612,"statfs":{"total":96636764160,"available":95265394688,"internally_reserved":0,"allocated":407863296,"data_stored":405103900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":298462,"internal_metadata":958951970},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.436,"15min":0.36299999999999999},"min":{"1min":0.19700000000000001,"5min":0.14699999999999999,"15min":0.13700000000000001},"max":{"1min":3.222,"5min":6.5499999999999998,"15min":6.9989999999999997},"last":0.245},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.46300000000000002,"15min":0.42299999999999999},"min":{"1min":0.17799999999999999,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":5.0030000000000001,"5min":6.0629999999999997,"15min":9.6869999999999994},"last":0.28599999999999998}]},{"osd":2,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.51600000000000001,"15min":0.47099999999999997},"min":{"1min":0.21099999999999999,"5min":0.153,"15min":0.153},"max":{"1min":5.7439999999999998,"5min":5.8280000000000003,"15min":9.7639999999999993},"last":0.252},{"interface":"front","average":{"1min":0.497,"5min":0.56200000000000006,"15min":0.42299999999999999},"min":{"1min":0.20399999999999999,"5min":0.187,"15min":0.16800000000000001},"max":{"1min":3.9319999999999999,"5min":8.8040000000000003,"15min":8.8040000000000003},"last":0.29899999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738831,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":650100,"kb_used_data":5356,"kb_used_omap":214,"kb_used_meta":644521,"kb_avail":93721740,"statfs":{"total":96636764160,"available":95971061760,"internally_reserved":0,"allocated":5484544,"data_stored":2698616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":219335,"internal_metadata":659990329},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:56 2023","interfaces":[{"interface":"back","average":{"1min":0.30399999999999999,"5min":0.34699999999999998,"15min":0.36499999999999999},"min":{"1min":0.23999999999999999,"5min":0.14199999999999999,"15min":0.14199999999999999},"max":{"1min":0.42399999999999999,"5min":1.595,"15min":7.4249999999999998},"last":0.17100000000000001},{"interface":"front","average":{"1min":0.35499999999999998,"5min":0.40899999999999997,"15min":0.376},"min":{"1min":0.23999999999999999,"5min":0.19600000000000001,"15min":0.183},"max":{"1min":0.46999999999999997,"5min":2.6859999999999999,"15min":2.6859999999999999},"last":0.23000000000000001}]},{"osd":2,"last update":"Wed Nov 8 05:29:57 2023","interfaces":[{"interface":"back","average":{"1min":0.28199999999999997,"5min":0.36099999999999999,"15min":0.373},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":0.34399999999999997,"5min":2.2650000000000001,"15min":5.8209999999999997},"last":0.24099999999999999},{"interface":"front","average":{"1min":0.28599999999999998,"5min":0.31,"15min":0.35799999999999998},"min":{"1min":0.17199999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.35699999999999998,"5min":1.137,"15min":4.2350000000000003},"last":0.223}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-08T05:30:58.806 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-11-08T05:30:58.807 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 459, 'reported_epoch': 458, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:30:44.490644+0000', 'last_change': '2023-11-08T05:28:13.557071+0000', 'last_active': '2023-11-08T05:30:44.490644+0000', 'last_peered': '2023-11-08T05:30:44.490644+0000', 'last_clean': '2023-11-08T05:28:01.458439+0000', 'last_became_active': '2023-11-08T05:28:13.557071+0000', 'last_became_peered': '2023-11-08T05:28:13.557071+0000', 'last_unstale': '2023-11-08T05:30:44.490644+0000', 'last_undegraded': '2023-11-08T05:30:44.490644+0000', 'last_fullsized': '2023-11-08T05:28:13.547025+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:30:58.807 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-11-08T05:30:58.807 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 11, 'reported_epoch': 459, 'state': 'active+clean+remapped', 'last_fresh': '2023-11-08T05:30:45.794760+0000', 'last_change': '2023-11-08T05:30:45.794760+0000', 'last_active': '2023-11-08T05:30:45.794760+0000', 'last_peered': '2023-11-08T05:30:45.794760+0000', 'last_clean': '2023-11-08T05:30:45.794760+0000', 'last_became_active': '2023-11-08T05:30:45.794398+0000', 'last_became_peered': '2023-11-08T05:30:45.794398+0000', 'last_unstale': '2023-11-08T05:30:45.794760+0000', 'last_undegraded': '2023-11-08T05:30:45.794760+0000', 'last_fullsized': '2023-11-08T05:30:45.794760+0000', 'mapping_epoch': 458, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 459, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2], 'acting': [2, 1], 'avail_no_missing': ['2', '1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2023-11-08T05:30:58.808 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.1 is not active+clean 2023-11-08T05:30:58.808 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.1', 'version': "0'0", 'reported_seq': 465, 'reported_epoch': 459, 'state': 'active+clean+remapped', 'last_fresh': '2023-11-08T05:30:45.795085+0000', 'last_change': '2023-11-08T05:30:45.795085+0000', 'last_active': '2023-11-08T05:30:45.795085+0000', 'last_peered': '2023-11-08T05:30:45.795085+0000', 'last_clean': '2023-11-08T05:30:45.795085+0000', 'last_became_active': '2023-11-08T05:30:45.794743+0000', 'last_became_peered': '2023-11-08T05:30:45.794743+0000', 'last_unstale': '2023-11-08T05:30:45.795085+0000', 'last_undegraded': '2023-11-08T05:30:45.795085+0000', 'last_fullsized': '2023-11-08T05:30:45.795085+0000', 'mapping_epoch': 458, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 459, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2], 'acting': [2, 1], 'avail_no_missing': ['2', '1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2023-11-08T05:30:58.808 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-11-08T05:30:58.809 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "0'0", 'reported_seq': 460, 'reported_epoch': 458, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:30:44.490523+0000', 'last_change': '2023-11-08T05:28:13.545468+0000', 'last_active': '2023-11-08T05:30:44.490523+0000', 'last_peered': '2023-11-08T05:30:44.490523+0000', 'last_clean': '2023-11-08T05:28:01.456723+0000', 'last_became_active': '2023-11-08T05:28:13.545468+0000', 'last_became_peered': '2023-11-08T05:28:13.545468+0000', 'last_unstale': '2023-11-08T05:30:44.490523+0000', 'last_undegraded': '2023-11-08T05:30:44.490523+0000', 'last_fullsized': '2023-11-08T05:28:13.537207+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 9, 'last_epoch_clean': 10, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:45.528796+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:45.528796+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:45.528796+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:30:58.809 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.0 is not active+clean 2023-11-08T05:30:58.809 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.0', 'version': "0'0", 'reported_seq': 465, 'reported_epoch': 458, 'state': 'active+clean+remapped', 'last_fresh': '2023-11-08T05:30:44.491907+0000', 'last_change': '2023-11-08T05:29:44.395350+0000', 'last_active': '2023-11-08T05:30:44.491907+0000', 'last_peered': '2023-11-08T05:30:44.491907+0000', 'last_clean': '2023-11-08T05:30:44.491907+0000', 'last_became_active': '2023-11-08T05:29:44.395012+0000', 'last_became_peered': '2023-11-08T05:29:44.395012+0000', 'last_unstale': '2023-11-08T05:30:44.491907+0000', 'last_undegraded': '2023-11-08T05:30:44.491907+0000', 'last_fullsized': '2023-11-08T05:30:44.491907+0000', 'mapping_epoch': 455, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 456, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2], 'acting': [2, 1], 'avail_no_missing': ['2', '1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2023-11-08T05:30:58.810 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-11-08T05:30:58.810 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 459, 'reported_epoch': 458, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:30:44.490629+0000', 'last_change': '2023-11-08T05:28:13.563357+0000', 'last_active': '2023-11-08T05:30:44.490629+0000', 'last_peered': '2023-11-08T05:30:44.490629+0000', 'last_clean': '2023-11-08T05:28:01.457691+0000', 'last_became_active': '2023-11-08T05:28:13.563357+0000', 'last_became_peered': '2023-11-08T05:28:13.563357+0000', 'last_unstale': '2023-11-08T05:30:44.490629+0000', 'last_undegraded': '2023-11-08T05:30:44.490629+0000', 'last_fullsized': '2023-11-08T05:28:13.541614+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:30:58.810 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-11-08T05:30:58.811 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 459, 'reported_epoch': 458, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:30:44.490691+0000', 'last_change': '2023-11-08T05:28:13.550672+0000', 'last_active': '2023-11-08T05:30:44.490691+0000', 'last_peered': '2023-11-08T05:30:44.490691+0000', 'last_clean': '2023-11-08T05:28:01.457904+0000', 'last_became_active': '2023-11-08T05:28:13.550672+0000', 'last_became_peered': '2023-11-08T05:28:13.550672+0000', 'last_unstale': '2023-11-08T05:30:44.490691+0000', 'last_undegraded': '2023-11-08T05:30:44.490691+0000', 'last_fullsized': '2023-11-08T05:28:13.539595+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:30:58.811 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-11-08T05:30:58.811 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 459, 'reported_epoch': 458, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:30:44.490805+0000', 'last_change': '2023-11-08T05:28:13.557968+0000', 'last_active': '2023-11-08T05:30:44.490805+0000', 'last_peered': '2023-11-08T05:30:44.490805+0000', 'last_clean': '2023-11-08T05:28:01.458433+0000', 'last_became_active': '2023-11-08T05:28:13.557968+0000', 'last_became_peered': '2023-11-08T05:28:13.557968+0000', 'last_unstale': '2023-11-08T05:30:44.490805+0000', 'last_undegraded': '2023-11-08T05:30:44.490805+0000', 'last_fullsized': '2023-11-08T05:28:13.542832+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:30:58.812 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-11-08T05:30:59.158 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T05:30:59.158 INFO:teuthology.orchestra.run.smithi087.stdout:{"fsid":"ab4cbc06-449d-497d-b513-220d9db2b590","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 5 pgs undersized","count":5},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2301,"monmap":{"epoch":1,"min_mon_release_name":"pacific","num_mons":1},"osdmap":{"epoch":459,"num_osds":3,"num_up_osds":2,"osd_up_since":1699421292,"num_in_osds":3,"osd_in_since":1699419158,"num_remapped_pgs":3},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean+remapped","count":3},{"state_name":"active+clean","count":1}],"num_pgs":9,"num_pools":2,"num_objects":0,"data_bytes":0,"bytes_used":2697592832,"bytes_avail":287212699648,"bytes_total":289910292480},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":41,"modified":"2023-11-08T05:29:32.044544+0000","services":{}},"progress_events":{}} 2023-11-08T05:30:59.171 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-11-08T05:30:59.171 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-11-08T05:30:59.451 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T05:30:59.453 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2023-11-08T05:30:59.464 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":1603,"stamp":"2023-11-08T05:30:58.062280+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":10,"acting":13,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":109,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2634368,"kb_used_data":409012,"kb_used_omap":780,"kb_used_meta":2220403,"kb_avail":280481152,"statfs":{"total":289910292480,"available":287212699648,"internally_reserved":0,"allocated":418828288,"data_stored":410500163,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":799447,"internal_metadata":2273692969},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-536782902,"num_objects":-164,"num_object_clones":0,"num_object_copies":-328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-115,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-164,"num_whiteouts":0,"num_read":-122056,"num_read_kb":-39011942,"num_write":-413642,"num_write_kb":-151705339,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-8,"num_bytes_recovered":-16922624,"num_keys_recovered":-2,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.424817"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490644+0000","last_change":"2023-11-08T05:28:13.557071+0000","last_active":"2023-11-08T05:30:44.490644+0000","last_peered":"2023-11-08T05:30:44.490644+0000","last_clean":"2023-11-08T05:28:01.458439+0000","last_became_active":"2023-11-08T05:28:13.557071+0000","last_became_peered":"2023-11-08T05:28:13.557071+0000","last_unstale":"2023-11-08T05:30:44.490644+0000","last_undegraded":"2023-11-08T05:30:44.490644+0000","last_fullsized":"2023-11-08T05:28:13.547025+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":459,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:45.794760+0000","last_change":"2023-11-08T05:30:45.794760+0000","last_active":"2023-11-08T05:30:45.794760+0000","last_peered":"2023-11-08T05:30:45.794760+0000","last_clean":"2023-11-08T05:30:45.794760+0000","last_became_active":"2023-11-08T05:30:45.794398+0000","last_became_peered":"2023-11-08T05:30:45.794398+0000","last_unstale":"2023-11-08T05:30:45.794760+0000","last_undegraded":"2023-11-08T05:30:45.794760+0000","last_fullsized":"2023-11-08T05:30:45.794760+0000","mapping_epoch":458,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":459,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":465,"reported_epoch":459,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:45.795085+0000","last_change":"2023-11-08T05:30:45.795085+0000","last_active":"2023-11-08T05:30:45.795085+0000","last_peered":"2023-11-08T05:30:45.795085+0000","last_clean":"2023-11-08T05:30:45.795085+0000","last_became_active":"2023-11-08T05:30:45.794743+0000","last_became_peered":"2023-11-08T05:30:45.794743+0000","last_unstale":"2023-11-08T05:30:45.795085+0000","last_undegraded":"2023-11-08T05:30:45.795085+0000","last_fullsized":"2023-11-08T05:30:45.795085+0000","mapping_epoch":458,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":459,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":454,"reported_epoch":458,"state":"active+clean","last_fresh":"2023-11-08T05:30:44.490618+0000","last_change":"2023-11-08T04:52:47.548597+0000","last_active":"2023-11-08T05:30:44.490618+0000","last_peered":"2023-11-08T05:30:44.490618+0000","last_clean":"2023-11-08T05:30:44.490618+0000","last_became_active":"2023-11-08T04:52:47.548294+0000","last_became_peered":"2023-11-08T04:52:47.548294+0000","last_unstale":"2023-11-08T05:30:44.490618+0000","last_undegraded":"2023-11-08T05:30:44.490618+0000","last_fullsized":"2023-11-08T05:30:44.490618+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":460,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490523+0000","last_change":"2023-11-08T05:28:13.545468+0000","last_active":"2023-11-08T05:30:44.490523+0000","last_peered":"2023-11-08T05:30:44.490523+0000","last_clean":"2023-11-08T05:28:01.456723+0000","last_became_active":"2023-11-08T05:28:13.545468+0000","last_became_peered":"2023-11-08T05:28:13.545468+0000","last_unstale":"2023-11-08T05:30:44.490523+0000","last_undegraded":"2023-11-08T05:30:44.490523+0000","last_fullsized":"2023-11-08T05:28:13.537207+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_clean_scrub_stamp":"2023-11-08T04:52:45.528796+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":465,"reported_epoch":458,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:44.491907+0000","last_change":"2023-11-08T05:29:44.395350+0000","last_active":"2023-11-08T05:30:44.491907+0000","last_peered":"2023-11-08T05:30:44.491907+0000","last_clean":"2023-11-08T05:30:44.491907+0000","last_became_active":"2023-11-08T05:29:44.395012+0000","last_became_peered":"2023-11-08T05:29:44.395012+0000","last_unstale":"2023-11-08T05:30:44.491907+0000","last_undegraded":"2023-11-08T05:30:44.491907+0000","last_fullsized":"2023-11-08T05:30:44.491907+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490629+0000","last_change":"2023-11-08T05:28:13.563357+0000","last_active":"2023-11-08T05:30:44.490629+0000","last_peered":"2023-11-08T05:30:44.490629+0000","last_clean":"2023-11-08T05:28:01.457691+0000","last_became_active":"2023-11-08T05:28:13.563357+0000","last_became_peered":"2023-11-08T05:28:13.563357+0000","last_unstale":"2023-11-08T05:30:44.490629+0000","last_undegraded":"2023-11-08T05:30:44.490629+0000","last_fullsized":"2023-11-08T05:28:13.541614+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490691+0000","last_change":"2023-11-08T05:28:13.550672+0000","last_active":"2023-11-08T05:30:44.490691+0000","last_peered":"2023-11-08T05:30:44.490691+0000","last_clean":"2023-11-08T05:28:01.457904+0000","last_became_active":"2023-11-08T05:28:13.550672+0000","last_became_peered":"2023-11-08T05:28:13.550672+0000","last_unstale":"2023-11-08T05:30:44.490691+0000","last_undegraded":"2023-11-08T05:30:44.490691+0000","last_fullsized":"2023-11-08T05:28:13.539595+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490805+0000","last_change":"2023-11-08T05:28:13.557968+0000","last_active":"2023-11-08T05:30:44.490805+0000","last_peered":"2023-11-08T05:30:44.490805+0000","last_clean":"2023-11-08T05:28:01.458433+0000","last_became_active":"2023-11-08T05:28:13.557968+0000","last_became_peered":"2023-11-08T05:28:13.557968+0000","last_unstale":"2023-11-08T05:30:44.490805+0000","last_undegraded":"2023-11-08T05:30:44.490805+0000","last_fullsized":"2023-11-08T05:28:13.542832+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":1,"acting":1,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":12,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738829,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":645040,"kb_used_data":5352,"kb_used_omap":275,"kb_used_meta":639404,"kb_avail":93726800,"statfs":{"total":96636764160,"available":95976243200,"internally_reserved":0,"allocated":5480448,"data_stored":2697647,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":281650,"internal_metadata":654750670},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:59 2023","interfaces":[{"interface":"back","average":{"1min":0.30499999999999999,"5min":0.439,"15min":0.36099999999999999},"min":{"1min":0.17399999999999999,"5min":0.14499999999999999,"15min":0.14499999999999999},"max":{"1min":0.60099999999999998,"5min":4.9379999999999997,"15min":4.9379999999999997},"last":0.379},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.49399999999999999,"15min":0.42399999999999999},"min":{"1min":0.221,"5min":0.19600000000000001,"15min":0.17000000000000001},"max":{"1min":0.63200000000000001,"5min":4.9859999999999998,"15min":4.9859999999999998},"last":0.40899999999999997}]},{"osd":1,"last update":"Wed Nov 8 05:30:03 2023","interfaces":[{"interface":"back","average":{"1min":0.28499999999999998,"5min":0.65100000000000002,"15min":0.47099999999999997},"min":{"1min":0.20100000000000001,"5min":0.153,"15min":0.153},"max":{"1min":0.39800000000000002,"5min":23.242999999999999,"15min":23.242999999999999},"last":0.35999999999999999},{"interface":"front","average":{"1min":0.29499999999999998,"5min":0.63100000000000001,"15min":0.45900000000000002},"min":{"1min":0.224,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":0.436,"5min":23.251000000000001,"15min":23.251000000000001},"last":0.38500000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738798,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1339228,"kb_used_data":398304,"kb_used_omap":291,"kb_used_meta":936476,"kb_avail":93032612,"statfs":{"total":96636764160,"available":95265394688,"internally_reserved":0,"allocated":407863296,"data_stored":405103900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":298462,"internal_metadata":958951970},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.436,"15min":0.36299999999999999},"min":{"1min":0.19700000000000001,"5min":0.14699999999999999,"15min":0.13700000000000001},"max":{"1min":3.222,"5min":6.5499999999999998,"15min":6.9989999999999997},"last":0.245},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.46300000000000002,"15min":0.42299999999999999},"min":{"1min":0.17799999999999999,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":5.0030000000000001,"5min":6.0629999999999997,"15min":9.6869999999999994},"last":0.28599999999999998}]},{"osd":2,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.51600000000000001,"15min":0.47099999999999997},"min":{"1min":0.21099999999999999,"5min":0.153,"15min":0.153},"max":{"1min":5.7439999999999998,"5min":5.8280000000000003,"15min":9.7639999999999993},"last":0.252},{"interface":"front","average":{"1min":0.497,"5min":0.56200000000000006,"15min":0.42299999999999999},"min":{"1min":0.20399999999999999,"5min":0.187,"15min":0.16800000000000001},"max":{"1min":3.9319999999999999,"5min":8.8040000000000003,"15min":8.8040000000000003},"last":0.29899999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738831,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":650100,"kb_used_data":5356,"kb_used_omap":214,"kb_used_meta":644521,"kb_avail":93721740,"statfs":{"total":96636764160,"available":95971061760,"internally_reserved":0,"allocated":5484544,"data_stored":2698616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":219335,"internal_metadata":659990329},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:56 2023","interfaces":[{"interface":"back","average":{"1min":0.30399999999999999,"5min":0.34699999999999998,"15min":0.36499999999999999},"min":{"1min":0.23999999999999999,"5min":0.14199999999999999,"15min":0.14199999999999999},"max":{"1min":0.42399999999999999,"5min":1.595,"15min":7.4249999999999998},"last":0.17100000000000001},{"interface":"front","average":{"1min":0.35499999999999998,"5min":0.40899999999999997,"15min":0.376},"min":{"1min":0.23999999999999999,"5min":0.19600000000000001,"15min":0.183},"max":{"1min":0.46999999999999997,"5min":2.6859999999999999,"15min":2.6859999999999999},"last":0.23000000000000001}]},{"osd":2,"last update":"Wed Nov 8 05:29:57 2023","interfaces":[{"interface":"back","average":{"1min":0.28199999999999997,"5min":0.36099999999999999,"15min":0.373},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":0.34399999999999997,"5min":2.2650000000000001,"15min":5.8209999999999997},"last":0.24099999999999999},{"interface":"front","average":{"1min":0.28599999999999998,"5min":0.31,"15min":0.35799999999999998},"min":{"1min":0.17199999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.35699999999999998,"5min":1.137,"15min":4.2350000000000003},"last":0.223}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-08T05:31:00.167 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~162s 2023-11-08T05:31:02.466 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-11-08T05:31:02.764 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T05:31:02.766 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2023-11-08T05:31:02.777 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":1605,"stamp":"2023-11-08T05:31:02.063077+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":10,"acting":13,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":109,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2634368,"kb_used_data":409012,"kb_used_omap":780,"kb_used_meta":2220403,"kb_avail":280481152,"statfs":{"total":289910292480,"available":287212699648,"internally_reserved":0,"allocated":418828288,"data_stored":410500163,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":799447,"internal_metadata":2273692969},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-536782902,"num_objects":-164,"num_object_clones":0,"num_object_copies":-328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-115,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-164,"num_whiteouts":0,"num_read":-122056,"num_read_kb":-39011942,"num_write":-413642,"num_write_kb":-151705339,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-8,"num_bytes_recovered":-16922624,"num_keys_recovered":-2,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.424817"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490644+0000","last_change":"2023-11-08T05:28:13.557071+0000","last_active":"2023-11-08T05:30:44.490644+0000","last_peered":"2023-11-08T05:30:44.490644+0000","last_clean":"2023-11-08T05:28:01.458439+0000","last_became_active":"2023-11-08T05:28:13.557071+0000","last_became_peered":"2023-11-08T05:28:13.557071+0000","last_unstale":"2023-11-08T05:30:44.490644+0000","last_undegraded":"2023-11-08T05:30:44.490644+0000","last_fullsized":"2023-11-08T05:28:13.547025+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":459,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:45.794760+0000","last_change":"2023-11-08T05:30:45.794760+0000","last_active":"2023-11-08T05:30:45.794760+0000","last_peered":"2023-11-08T05:30:45.794760+0000","last_clean":"2023-11-08T05:30:45.794760+0000","last_became_active":"2023-11-08T05:30:45.794398+0000","last_became_peered":"2023-11-08T05:30:45.794398+0000","last_unstale":"2023-11-08T05:30:45.794760+0000","last_undegraded":"2023-11-08T05:30:45.794760+0000","last_fullsized":"2023-11-08T05:30:45.794760+0000","mapping_epoch":458,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":459,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":465,"reported_epoch":459,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:45.795085+0000","last_change":"2023-11-08T05:30:45.795085+0000","last_active":"2023-11-08T05:30:45.795085+0000","last_peered":"2023-11-08T05:30:45.795085+0000","last_clean":"2023-11-08T05:30:45.795085+0000","last_became_active":"2023-11-08T05:30:45.794743+0000","last_became_peered":"2023-11-08T05:30:45.794743+0000","last_unstale":"2023-11-08T05:30:45.795085+0000","last_undegraded":"2023-11-08T05:30:45.795085+0000","last_fullsized":"2023-11-08T05:30:45.795085+0000","mapping_epoch":458,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":459,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":454,"reported_epoch":458,"state":"active+clean","last_fresh":"2023-11-08T05:30:44.490618+0000","last_change":"2023-11-08T04:52:47.548597+0000","last_active":"2023-11-08T05:30:44.490618+0000","last_peered":"2023-11-08T05:30:44.490618+0000","last_clean":"2023-11-08T05:30:44.490618+0000","last_became_active":"2023-11-08T04:52:47.548294+0000","last_became_peered":"2023-11-08T04:52:47.548294+0000","last_unstale":"2023-11-08T05:30:44.490618+0000","last_undegraded":"2023-11-08T05:30:44.490618+0000","last_fullsized":"2023-11-08T05:30:44.490618+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":460,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490523+0000","last_change":"2023-11-08T05:28:13.545468+0000","last_active":"2023-11-08T05:30:44.490523+0000","last_peered":"2023-11-08T05:30:44.490523+0000","last_clean":"2023-11-08T05:28:01.456723+0000","last_became_active":"2023-11-08T05:28:13.545468+0000","last_became_peered":"2023-11-08T05:28:13.545468+0000","last_unstale":"2023-11-08T05:30:44.490523+0000","last_undegraded":"2023-11-08T05:30:44.490523+0000","last_fullsized":"2023-11-08T05:28:13.537207+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_clean_scrub_stamp":"2023-11-08T04:52:45.528796+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":465,"reported_epoch":458,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:44.491907+0000","last_change":"2023-11-08T05:29:44.395350+0000","last_active":"2023-11-08T05:30:44.491907+0000","last_peered":"2023-11-08T05:30:44.491907+0000","last_clean":"2023-11-08T05:30:44.491907+0000","last_became_active":"2023-11-08T05:29:44.395012+0000","last_became_peered":"2023-11-08T05:29:44.395012+0000","last_unstale":"2023-11-08T05:30:44.491907+0000","last_undegraded":"2023-11-08T05:30:44.491907+0000","last_fullsized":"2023-11-08T05:30:44.491907+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490629+0000","last_change":"2023-11-08T05:28:13.563357+0000","last_active":"2023-11-08T05:30:44.490629+0000","last_peered":"2023-11-08T05:30:44.490629+0000","last_clean":"2023-11-08T05:28:01.457691+0000","last_became_active":"2023-11-08T05:28:13.563357+0000","last_became_peered":"2023-11-08T05:28:13.563357+0000","last_unstale":"2023-11-08T05:30:44.490629+0000","last_undegraded":"2023-11-08T05:30:44.490629+0000","last_fullsized":"2023-11-08T05:28:13.541614+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490691+0000","last_change":"2023-11-08T05:28:13.550672+0000","last_active":"2023-11-08T05:30:44.490691+0000","last_peered":"2023-11-08T05:30:44.490691+0000","last_clean":"2023-11-08T05:28:01.457904+0000","last_became_active":"2023-11-08T05:28:13.550672+0000","last_became_peered":"2023-11-08T05:28:13.550672+0000","last_unstale":"2023-11-08T05:30:44.490691+0000","last_undegraded":"2023-11-08T05:30:44.490691+0000","last_fullsized":"2023-11-08T05:28:13.539595+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490805+0000","last_change":"2023-11-08T05:28:13.557968+0000","last_active":"2023-11-08T05:30:44.490805+0000","last_peered":"2023-11-08T05:30:44.490805+0000","last_clean":"2023-11-08T05:28:01.458433+0000","last_became_active":"2023-11-08T05:28:13.557968+0000","last_became_peered":"2023-11-08T05:28:13.557968+0000","last_unstale":"2023-11-08T05:30:44.490805+0000","last_undegraded":"2023-11-08T05:30:44.490805+0000","last_fullsized":"2023-11-08T05:28:13.542832+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":1,"acting":1,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":12,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738830,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":645040,"kb_used_data":5352,"kb_used_omap":275,"kb_used_meta":639404,"kb_avail":93726800,"statfs":{"total":96636764160,"available":95976243200,"internally_reserved":0,"allocated":5480448,"data_stored":2697647,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":281650,"internal_metadata":654750670},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:59 2023","interfaces":[{"interface":"back","average":{"1min":0.30499999999999999,"5min":0.439,"15min":0.36099999999999999},"min":{"1min":0.17399999999999999,"5min":0.14499999999999999,"15min":0.14499999999999999},"max":{"1min":0.60099999999999998,"5min":4.9379999999999997,"15min":4.9379999999999997},"last":0.379},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.49399999999999999,"15min":0.42399999999999999},"min":{"1min":0.221,"5min":0.19600000000000001,"15min":0.17000000000000001},"max":{"1min":0.63200000000000001,"5min":4.9859999999999998,"15min":4.9859999999999998},"last":0.40899999999999997}]},{"osd":1,"last update":"Wed Nov 8 05:30:03 2023","interfaces":[{"interface":"back","average":{"1min":0.28499999999999998,"5min":0.65100000000000002,"15min":0.47099999999999997},"min":{"1min":0.20100000000000001,"5min":0.153,"15min":0.153},"max":{"1min":0.39800000000000002,"5min":23.242999999999999,"15min":23.242999999999999},"last":0.25},{"interface":"front","average":{"1min":0.29499999999999998,"5min":0.63100000000000001,"15min":0.45900000000000002},"min":{"1min":0.224,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":0.436,"5min":23.251000000000001,"15min":23.251000000000001},"last":0.27800000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738798,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1339228,"kb_used_data":398304,"kb_used_omap":291,"kb_used_meta":936476,"kb_avail":93032612,"statfs":{"total":96636764160,"available":95265394688,"internally_reserved":0,"allocated":407863296,"data_stored":405103900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":298462,"internal_metadata":958951970},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.436,"15min":0.36299999999999999},"min":{"1min":0.19700000000000001,"5min":0.14699999999999999,"15min":0.13700000000000001},"max":{"1min":3.222,"5min":6.5499999999999998,"15min":6.9989999999999997},"last":0.245},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.46300000000000002,"15min":0.42299999999999999},"min":{"1min":0.17799999999999999,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":5.0030000000000001,"5min":6.0629999999999997,"15min":9.6869999999999994},"last":0.28599999999999998}]},{"osd":2,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.51600000000000001,"15min":0.47099999999999997},"min":{"1min":0.21099999999999999,"5min":0.153,"15min":0.153},"max":{"1min":5.7439999999999998,"5min":5.8280000000000003,"15min":9.7639999999999993},"last":0.252},{"interface":"front","average":{"1min":0.497,"5min":0.56200000000000006,"15min":0.42299999999999999},"min":{"1min":0.20399999999999999,"5min":0.187,"15min":0.16800000000000001},"max":{"1min":3.9319999999999999,"5min":8.8040000000000003,"15min":8.8040000000000003},"last":0.29899999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738832,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":650100,"kb_used_data":5356,"kb_used_omap":214,"kb_used_meta":644521,"kb_avail":93721740,"statfs":{"total":96636764160,"available":95971061760,"internally_reserved":0,"allocated":5484544,"data_stored":2698616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":219335,"internal_metadata":659990329},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:56 2023","interfaces":[{"interface":"back","average":{"1min":0.30399999999999999,"5min":0.34699999999999998,"15min":0.36499999999999999},"min":{"1min":0.23999999999999999,"5min":0.14199999999999999,"15min":0.14199999999999999},"max":{"1min":0.42399999999999999,"5min":1.595,"15min":7.4249999999999998},"last":0.17100000000000001},{"interface":"front","average":{"1min":0.35499999999999998,"5min":0.40899999999999997,"15min":0.376},"min":{"1min":0.23999999999999999,"5min":0.19600000000000001,"15min":0.183},"max":{"1min":0.46999999999999997,"5min":2.6859999999999999,"15min":2.6859999999999999},"last":0.23000000000000001}]},{"osd":2,"last update":"Wed Nov 8 05:30:57 2023","interfaces":[{"interface":"back","average":{"1min":0.28599999999999998,"5min":0.34699999999999998,"15min":0.36899999999999999},"min":{"1min":0.19700000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":0.41399999999999998,"5min":2.2650000000000001,"15min":5.8209999999999997},"last":0.20899999999999999},{"interface":"front","average":{"1min":0.30599999999999999,"5min":0.313,"15min":0.35799999999999998},"min":{"1min":0.17799999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.435,"5min":1.137,"15min":4.2350000000000003},"last":0.23000000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-08T05:31:02.779 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-11-08T05:31:03.062 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T05:31:03.064 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2023-11-08T05:31:03.075 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":1605,"stamp":"2023-11-08T05:31:02.063077+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":10,"acting":13,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":109,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2634368,"kb_used_data":409012,"kb_used_omap":780,"kb_used_meta":2220403,"kb_avail":280481152,"statfs":{"total":289910292480,"available":287212699648,"internally_reserved":0,"allocated":418828288,"data_stored":410500163,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":799447,"internal_metadata":2273692969},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-536782902,"num_objects":-164,"num_object_clones":0,"num_object_copies":-328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-115,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-164,"num_whiteouts":0,"num_read":-122056,"num_read_kb":-39011942,"num_write":-413642,"num_write_kb":-151705339,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-8,"num_bytes_recovered":-16922624,"num_keys_recovered":-2,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.424817"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490644+0000","last_change":"2023-11-08T05:28:13.557071+0000","last_active":"2023-11-08T05:30:44.490644+0000","last_peered":"2023-11-08T05:30:44.490644+0000","last_clean":"2023-11-08T05:28:01.458439+0000","last_became_active":"2023-11-08T05:28:13.557071+0000","last_became_peered":"2023-11-08T05:28:13.557071+0000","last_unstale":"2023-11-08T05:30:44.490644+0000","last_undegraded":"2023-11-08T05:30:44.490644+0000","last_fullsized":"2023-11-08T05:28:13.547025+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":459,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:45.794760+0000","last_change":"2023-11-08T05:30:45.794760+0000","last_active":"2023-11-08T05:30:45.794760+0000","last_peered":"2023-11-08T05:30:45.794760+0000","last_clean":"2023-11-08T05:30:45.794760+0000","last_became_active":"2023-11-08T05:30:45.794398+0000","last_became_peered":"2023-11-08T05:30:45.794398+0000","last_unstale":"2023-11-08T05:30:45.794760+0000","last_undegraded":"2023-11-08T05:30:45.794760+0000","last_fullsized":"2023-11-08T05:30:45.794760+0000","mapping_epoch":458,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":459,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":465,"reported_epoch":459,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:45.795085+0000","last_change":"2023-11-08T05:30:45.795085+0000","last_active":"2023-11-08T05:30:45.795085+0000","last_peered":"2023-11-08T05:30:45.795085+0000","last_clean":"2023-11-08T05:30:45.795085+0000","last_became_active":"2023-11-08T05:30:45.794743+0000","last_became_peered":"2023-11-08T05:30:45.794743+0000","last_unstale":"2023-11-08T05:30:45.795085+0000","last_undegraded":"2023-11-08T05:30:45.795085+0000","last_fullsized":"2023-11-08T05:30:45.795085+0000","mapping_epoch":458,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":459,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":454,"reported_epoch":458,"state":"active+clean","last_fresh":"2023-11-08T05:30:44.490618+0000","last_change":"2023-11-08T04:52:47.548597+0000","last_active":"2023-11-08T05:30:44.490618+0000","last_peered":"2023-11-08T05:30:44.490618+0000","last_clean":"2023-11-08T05:30:44.490618+0000","last_became_active":"2023-11-08T04:52:47.548294+0000","last_became_peered":"2023-11-08T04:52:47.548294+0000","last_unstale":"2023-11-08T05:30:44.490618+0000","last_undegraded":"2023-11-08T05:30:44.490618+0000","last_fullsized":"2023-11-08T05:30:44.490618+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":460,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490523+0000","last_change":"2023-11-08T05:28:13.545468+0000","last_active":"2023-11-08T05:30:44.490523+0000","last_peered":"2023-11-08T05:30:44.490523+0000","last_clean":"2023-11-08T05:28:01.456723+0000","last_became_active":"2023-11-08T05:28:13.545468+0000","last_became_peered":"2023-11-08T05:28:13.545468+0000","last_unstale":"2023-11-08T05:30:44.490523+0000","last_undegraded":"2023-11-08T05:30:44.490523+0000","last_fullsized":"2023-11-08T05:28:13.537207+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_clean_scrub_stamp":"2023-11-08T04:52:45.528796+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":465,"reported_epoch":458,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:44.491907+0000","last_change":"2023-11-08T05:29:44.395350+0000","last_active":"2023-11-08T05:30:44.491907+0000","last_peered":"2023-11-08T05:30:44.491907+0000","last_clean":"2023-11-08T05:30:44.491907+0000","last_became_active":"2023-11-08T05:29:44.395012+0000","last_became_peered":"2023-11-08T05:29:44.395012+0000","last_unstale":"2023-11-08T05:30:44.491907+0000","last_undegraded":"2023-11-08T05:30:44.491907+0000","last_fullsized":"2023-11-08T05:30:44.491907+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490629+0000","last_change":"2023-11-08T05:28:13.563357+0000","last_active":"2023-11-08T05:30:44.490629+0000","last_peered":"2023-11-08T05:30:44.490629+0000","last_clean":"2023-11-08T05:28:01.457691+0000","last_became_active":"2023-11-08T05:28:13.563357+0000","last_became_peered":"2023-11-08T05:28:13.563357+0000","last_unstale":"2023-11-08T05:30:44.490629+0000","last_undegraded":"2023-11-08T05:30:44.490629+0000","last_fullsized":"2023-11-08T05:28:13.541614+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490691+0000","last_change":"2023-11-08T05:28:13.550672+0000","last_active":"2023-11-08T05:30:44.490691+0000","last_peered":"2023-11-08T05:30:44.490691+0000","last_clean":"2023-11-08T05:28:01.457904+0000","last_became_active":"2023-11-08T05:28:13.550672+0000","last_became_peered":"2023-11-08T05:28:13.550672+0000","last_unstale":"2023-11-08T05:30:44.490691+0000","last_undegraded":"2023-11-08T05:30:44.490691+0000","last_fullsized":"2023-11-08T05:28:13.539595+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490805+0000","last_change":"2023-11-08T05:28:13.557968+0000","last_active":"2023-11-08T05:30:44.490805+0000","last_peered":"2023-11-08T05:30:44.490805+0000","last_clean":"2023-11-08T05:28:01.458433+0000","last_became_active":"2023-11-08T05:28:13.557968+0000","last_became_peered":"2023-11-08T05:28:13.557968+0000","last_unstale":"2023-11-08T05:30:44.490805+0000","last_undegraded":"2023-11-08T05:30:44.490805+0000","last_fullsized":"2023-11-08T05:28:13.542832+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":1,"acting":1,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":12,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738830,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":645040,"kb_used_data":5352,"kb_used_omap":275,"kb_used_meta":639404,"kb_avail":93726800,"statfs":{"total":96636764160,"available":95976243200,"internally_reserved":0,"allocated":5480448,"data_stored":2697647,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":281650,"internal_metadata":654750670},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:59 2023","interfaces":[{"interface":"back","average":{"1min":0.30499999999999999,"5min":0.439,"15min":0.36099999999999999},"min":{"1min":0.17399999999999999,"5min":0.14499999999999999,"15min":0.14499999999999999},"max":{"1min":0.60099999999999998,"5min":4.9379999999999997,"15min":4.9379999999999997},"last":0.379},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.49399999999999999,"15min":0.42399999999999999},"min":{"1min":0.221,"5min":0.19600000000000001,"15min":0.17000000000000001},"max":{"1min":0.63200000000000001,"5min":4.9859999999999998,"15min":4.9859999999999998},"last":0.40899999999999997}]},{"osd":1,"last update":"Wed Nov 8 05:30:03 2023","interfaces":[{"interface":"back","average":{"1min":0.28499999999999998,"5min":0.65100000000000002,"15min":0.47099999999999997},"min":{"1min":0.20100000000000001,"5min":0.153,"15min":0.153},"max":{"1min":0.39800000000000002,"5min":23.242999999999999,"15min":23.242999999999999},"last":0.25},{"interface":"front","average":{"1min":0.29499999999999998,"5min":0.63100000000000001,"15min":0.45900000000000002},"min":{"1min":0.224,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":0.436,"5min":23.251000000000001,"15min":23.251000000000001},"last":0.27800000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738798,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1339228,"kb_used_data":398304,"kb_used_omap":291,"kb_used_meta":936476,"kb_avail":93032612,"statfs":{"total":96636764160,"available":95265394688,"internally_reserved":0,"allocated":407863296,"data_stored":405103900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":298462,"internal_metadata":958951970},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.436,"15min":0.36299999999999999},"min":{"1min":0.19700000000000001,"5min":0.14699999999999999,"15min":0.13700000000000001},"max":{"1min":3.222,"5min":6.5499999999999998,"15min":6.9989999999999997},"last":0.245},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.46300000000000002,"15min":0.42299999999999999},"min":{"1min":0.17799999999999999,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":5.0030000000000001,"5min":6.0629999999999997,"15min":9.6869999999999994},"last":0.28599999999999998}]},{"osd":2,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.51600000000000001,"15min":0.47099999999999997},"min":{"1min":0.21099999999999999,"5min":0.153,"15min":0.153},"max":{"1min":5.7439999999999998,"5min":5.8280000000000003,"15min":9.7639999999999993},"last":0.252},{"interface":"front","average":{"1min":0.497,"5min":0.56200000000000006,"15min":0.42299999999999999},"min":{"1min":0.20399999999999999,"5min":0.187,"15min":0.16800000000000001},"max":{"1min":3.9319999999999999,"5min":8.8040000000000003,"15min":8.8040000000000003},"last":0.29899999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738832,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":650100,"kb_used_data":5356,"kb_used_omap":214,"kb_used_meta":644521,"kb_avail":93721740,"statfs":{"total":96636764160,"available":95971061760,"internally_reserved":0,"allocated":5484544,"data_stored":2698616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":219335,"internal_metadata":659990329},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:56 2023","interfaces":[{"interface":"back","average":{"1min":0.30399999999999999,"5min":0.34699999999999998,"15min":0.36499999999999999},"min":{"1min":0.23999999999999999,"5min":0.14199999999999999,"15min":0.14199999999999999},"max":{"1min":0.42399999999999999,"5min":1.595,"15min":7.4249999999999998},"last":0.17100000000000001},{"interface":"front","average":{"1min":0.35499999999999998,"5min":0.40899999999999997,"15min":0.376},"min":{"1min":0.23999999999999999,"5min":0.19600000000000001,"15min":0.183},"max":{"1min":0.46999999999999997,"5min":2.6859999999999999,"15min":2.6859999999999999},"last":0.23000000000000001}]},{"osd":2,"last update":"Wed Nov 8 05:30:57 2023","interfaces":[{"interface":"back","average":{"1min":0.28599999999999998,"5min":0.34699999999999998,"15min":0.36899999999999999},"min":{"1min":0.19700000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":0.41399999999999998,"5min":2.2650000000000001,"15min":5.8209999999999997},"last":0.20899999999999999},{"interface":"front","average":{"1min":0.30599999999999999,"5min":0.313,"15min":0.35799999999999998},"min":{"1min":0.17799999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.435,"5min":1.137,"15min":4.2350000000000003},"last":0.23000000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-08T05:31:03.076 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-11-08T05:31:03.076 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 459, 'reported_epoch': 458, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:30:44.490644+0000', 'last_change': '2023-11-08T05:28:13.557071+0000', 'last_active': '2023-11-08T05:30:44.490644+0000', 'last_peered': '2023-11-08T05:30:44.490644+0000', 'last_clean': '2023-11-08T05:28:01.458439+0000', 'last_became_active': '2023-11-08T05:28:13.557071+0000', 'last_became_peered': '2023-11-08T05:28:13.557071+0000', 'last_unstale': '2023-11-08T05:30:44.490644+0000', 'last_undegraded': '2023-11-08T05:30:44.490644+0000', 'last_fullsized': '2023-11-08T05:28:13.547025+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:31:03.076 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-11-08T05:31:03.077 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 11, 'reported_epoch': 459, 'state': 'active+clean+remapped', 'last_fresh': '2023-11-08T05:30:45.794760+0000', 'last_change': '2023-11-08T05:30:45.794760+0000', 'last_active': '2023-11-08T05:30:45.794760+0000', 'last_peered': '2023-11-08T05:30:45.794760+0000', 'last_clean': '2023-11-08T05:30:45.794760+0000', 'last_became_active': '2023-11-08T05:30:45.794398+0000', 'last_became_peered': '2023-11-08T05:30:45.794398+0000', 'last_unstale': '2023-11-08T05:30:45.794760+0000', 'last_undegraded': '2023-11-08T05:30:45.794760+0000', 'last_fullsized': '2023-11-08T05:30:45.794760+0000', 'mapping_epoch': 458, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 459, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2], 'acting': [2, 1], 'avail_no_missing': ['2', '1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2023-11-08T05:31:03.077 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.1 is not active+clean 2023-11-08T05:31:03.077 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.1', 'version': "0'0", 'reported_seq': 465, 'reported_epoch': 459, 'state': 'active+clean+remapped', 'last_fresh': '2023-11-08T05:30:45.795085+0000', 'last_change': '2023-11-08T05:30:45.795085+0000', 'last_active': '2023-11-08T05:30:45.795085+0000', 'last_peered': '2023-11-08T05:30:45.795085+0000', 'last_clean': '2023-11-08T05:30:45.795085+0000', 'last_became_active': '2023-11-08T05:30:45.794743+0000', 'last_became_peered': '2023-11-08T05:30:45.794743+0000', 'last_unstale': '2023-11-08T05:30:45.795085+0000', 'last_undegraded': '2023-11-08T05:30:45.795085+0000', 'last_fullsized': '2023-11-08T05:30:45.795085+0000', 'mapping_epoch': 458, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 459, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2], 'acting': [2, 1], 'avail_no_missing': ['2', '1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2023-11-08T05:31:03.078 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-11-08T05:31:03.078 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "0'0", 'reported_seq': 460, 'reported_epoch': 458, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:30:44.490523+0000', 'last_change': '2023-11-08T05:28:13.545468+0000', 'last_active': '2023-11-08T05:30:44.490523+0000', 'last_peered': '2023-11-08T05:30:44.490523+0000', 'last_clean': '2023-11-08T05:28:01.456723+0000', 'last_became_active': '2023-11-08T05:28:13.545468+0000', 'last_became_peered': '2023-11-08T05:28:13.545468+0000', 'last_unstale': '2023-11-08T05:30:44.490523+0000', 'last_undegraded': '2023-11-08T05:30:44.490523+0000', 'last_fullsized': '2023-11-08T05:28:13.537207+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 9, 'last_epoch_clean': 10, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:45.528796+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:45.528796+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:45.528796+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:31:03.078 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.0 is not active+clean 2023-11-08T05:31:03.078 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.0', 'version': "0'0", 'reported_seq': 465, 'reported_epoch': 458, 'state': 'active+clean+remapped', 'last_fresh': '2023-11-08T05:30:44.491907+0000', 'last_change': '2023-11-08T05:29:44.395350+0000', 'last_active': '2023-11-08T05:30:44.491907+0000', 'last_peered': '2023-11-08T05:30:44.491907+0000', 'last_clean': '2023-11-08T05:30:44.491907+0000', 'last_became_active': '2023-11-08T05:29:44.395012+0000', 'last_became_peered': '2023-11-08T05:29:44.395012+0000', 'last_unstale': '2023-11-08T05:30:44.491907+0000', 'last_undegraded': '2023-11-08T05:30:44.491907+0000', 'last_fullsized': '2023-11-08T05:30:44.491907+0000', 'mapping_epoch': 455, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 456, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2], 'acting': [2, 1], 'avail_no_missing': ['2', '1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2023-11-08T05:31:03.079 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-11-08T05:31:03.079 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 459, 'reported_epoch': 458, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:30:44.490629+0000', 'last_change': '2023-11-08T05:28:13.563357+0000', 'last_active': '2023-11-08T05:30:44.490629+0000', 'last_peered': '2023-11-08T05:30:44.490629+0000', 'last_clean': '2023-11-08T05:28:01.457691+0000', 'last_became_active': '2023-11-08T05:28:13.563357+0000', 'last_became_peered': '2023-11-08T05:28:13.563357+0000', 'last_unstale': '2023-11-08T05:30:44.490629+0000', 'last_undegraded': '2023-11-08T05:30:44.490629+0000', 'last_fullsized': '2023-11-08T05:28:13.541614+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:31:03.079 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-11-08T05:31:03.080 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 459, 'reported_epoch': 458, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:30:44.490691+0000', 'last_change': '2023-11-08T05:28:13.550672+0000', 'last_active': '2023-11-08T05:30:44.490691+0000', 'last_peered': '2023-11-08T05:30:44.490691+0000', 'last_clean': '2023-11-08T05:28:01.457904+0000', 'last_became_active': '2023-11-08T05:28:13.550672+0000', 'last_became_peered': '2023-11-08T05:28:13.550672+0000', 'last_unstale': '2023-11-08T05:30:44.490691+0000', 'last_undegraded': '2023-11-08T05:30:44.490691+0000', 'last_fullsized': '2023-11-08T05:28:13.539595+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:31:03.080 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-11-08T05:31:03.080 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 459, 'reported_epoch': 458, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:30:44.490805+0000', 'last_change': '2023-11-08T05:28:13.557968+0000', 'last_active': '2023-11-08T05:30:44.490805+0000', 'last_peered': '2023-11-08T05:30:44.490805+0000', 'last_clean': '2023-11-08T05:28:01.458433+0000', 'last_became_active': '2023-11-08T05:28:13.557968+0000', 'last_became_peered': '2023-11-08T05:28:13.557968+0000', 'last_unstale': '2023-11-08T05:30:44.490805+0000', 'last_undegraded': '2023-11-08T05:30:44.490805+0000', 'last_fullsized': '2023-11-08T05:28:13.542832+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:31:03.081 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-11-08T05:31:03.432 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T05:31:03.432 INFO:teuthology.orchestra.run.smithi087.stdout:{"fsid":"ab4cbc06-449d-497d-b513-220d9db2b590","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 5 pgs undersized","count":5},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2306,"monmap":{"epoch":1,"min_mon_release_name":"pacific","num_mons":1},"osdmap":{"epoch":459,"num_osds":3,"num_up_osds":2,"osd_up_since":1699421292,"num_in_osds":3,"osd_in_since":1699419158,"num_remapped_pgs":3},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean+remapped","count":3},{"state_name":"active+clean","count":1}],"num_pgs":9,"num_pools":2,"num_objects":0,"data_bytes":0,"bytes_used":2697592832,"bytes_avail":287212699648,"bytes_total":289910292480},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":41,"modified":"2023-11-08T05:29:32.044544+0000","services":{}},"progress_events":{}} 2023-11-08T05:31:03.446 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-11-08T05:31:03.446 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-11-08T05:31:03.731 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T05:31:03.732 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2023-11-08T05:31:03.743 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":1605,"stamp":"2023-11-08T05:31:02.063077+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":10,"acting":13,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":109,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2634368,"kb_used_data":409012,"kb_used_omap":780,"kb_used_meta":2220403,"kb_avail":280481152,"statfs":{"total":289910292480,"available":287212699648,"internally_reserved":0,"allocated":418828288,"data_stored":410500163,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":799447,"internal_metadata":2273692969},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-536782902,"num_objects":-164,"num_object_clones":0,"num_object_copies":-328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-115,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-164,"num_whiteouts":0,"num_read":-122056,"num_read_kb":-39011942,"num_write":-413642,"num_write_kb":-151705339,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-8,"num_bytes_recovered":-16922624,"num_keys_recovered":-2,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.424817"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490644+0000","last_change":"2023-11-08T05:28:13.557071+0000","last_active":"2023-11-08T05:30:44.490644+0000","last_peered":"2023-11-08T05:30:44.490644+0000","last_clean":"2023-11-08T05:28:01.458439+0000","last_became_active":"2023-11-08T05:28:13.557071+0000","last_became_peered":"2023-11-08T05:28:13.557071+0000","last_unstale":"2023-11-08T05:30:44.490644+0000","last_undegraded":"2023-11-08T05:30:44.490644+0000","last_fullsized":"2023-11-08T05:28:13.547025+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":459,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:45.794760+0000","last_change":"2023-11-08T05:30:45.794760+0000","last_active":"2023-11-08T05:30:45.794760+0000","last_peered":"2023-11-08T05:30:45.794760+0000","last_clean":"2023-11-08T05:30:45.794760+0000","last_became_active":"2023-11-08T05:30:45.794398+0000","last_became_peered":"2023-11-08T05:30:45.794398+0000","last_unstale":"2023-11-08T05:30:45.794760+0000","last_undegraded":"2023-11-08T05:30:45.794760+0000","last_fullsized":"2023-11-08T05:30:45.794760+0000","mapping_epoch":458,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":459,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":465,"reported_epoch":459,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:45.795085+0000","last_change":"2023-11-08T05:30:45.795085+0000","last_active":"2023-11-08T05:30:45.795085+0000","last_peered":"2023-11-08T05:30:45.795085+0000","last_clean":"2023-11-08T05:30:45.795085+0000","last_became_active":"2023-11-08T05:30:45.794743+0000","last_became_peered":"2023-11-08T05:30:45.794743+0000","last_unstale":"2023-11-08T05:30:45.795085+0000","last_undegraded":"2023-11-08T05:30:45.795085+0000","last_fullsized":"2023-11-08T05:30:45.795085+0000","mapping_epoch":458,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":459,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":454,"reported_epoch":458,"state":"active+clean","last_fresh":"2023-11-08T05:30:44.490618+0000","last_change":"2023-11-08T04:52:47.548597+0000","last_active":"2023-11-08T05:30:44.490618+0000","last_peered":"2023-11-08T05:30:44.490618+0000","last_clean":"2023-11-08T05:30:44.490618+0000","last_became_active":"2023-11-08T04:52:47.548294+0000","last_became_peered":"2023-11-08T04:52:47.548294+0000","last_unstale":"2023-11-08T05:30:44.490618+0000","last_undegraded":"2023-11-08T05:30:44.490618+0000","last_fullsized":"2023-11-08T05:30:44.490618+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":460,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490523+0000","last_change":"2023-11-08T05:28:13.545468+0000","last_active":"2023-11-08T05:30:44.490523+0000","last_peered":"2023-11-08T05:30:44.490523+0000","last_clean":"2023-11-08T05:28:01.456723+0000","last_became_active":"2023-11-08T05:28:13.545468+0000","last_became_peered":"2023-11-08T05:28:13.545468+0000","last_unstale":"2023-11-08T05:30:44.490523+0000","last_undegraded":"2023-11-08T05:30:44.490523+0000","last_fullsized":"2023-11-08T05:28:13.537207+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_clean_scrub_stamp":"2023-11-08T04:52:45.528796+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":465,"reported_epoch":458,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:44.491907+0000","last_change":"2023-11-08T05:29:44.395350+0000","last_active":"2023-11-08T05:30:44.491907+0000","last_peered":"2023-11-08T05:30:44.491907+0000","last_clean":"2023-11-08T05:30:44.491907+0000","last_became_active":"2023-11-08T05:29:44.395012+0000","last_became_peered":"2023-11-08T05:29:44.395012+0000","last_unstale":"2023-11-08T05:30:44.491907+0000","last_undegraded":"2023-11-08T05:30:44.491907+0000","last_fullsized":"2023-11-08T05:30:44.491907+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490629+0000","last_change":"2023-11-08T05:28:13.563357+0000","last_active":"2023-11-08T05:30:44.490629+0000","last_peered":"2023-11-08T05:30:44.490629+0000","last_clean":"2023-11-08T05:28:01.457691+0000","last_became_active":"2023-11-08T05:28:13.563357+0000","last_became_peered":"2023-11-08T05:28:13.563357+0000","last_unstale":"2023-11-08T05:30:44.490629+0000","last_undegraded":"2023-11-08T05:30:44.490629+0000","last_fullsized":"2023-11-08T05:28:13.541614+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490691+0000","last_change":"2023-11-08T05:28:13.550672+0000","last_active":"2023-11-08T05:30:44.490691+0000","last_peered":"2023-11-08T05:30:44.490691+0000","last_clean":"2023-11-08T05:28:01.457904+0000","last_became_active":"2023-11-08T05:28:13.550672+0000","last_became_peered":"2023-11-08T05:28:13.550672+0000","last_unstale":"2023-11-08T05:30:44.490691+0000","last_undegraded":"2023-11-08T05:30:44.490691+0000","last_fullsized":"2023-11-08T05:28:13.539595+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490805+0000","last_change":"2023-11-08T05:28:13.557968+0000","last_active":"2023-11-08T05:30:44.490805+0000","last_peered":"2023-11-08T05:30:44.490805+0000","last_clean":"2023-11-08T05:28:01.458433+0000","last_became_active":"2023-11-08T05:28:13.557968+0000","last_became_peered":"2023-11-08T05:28:13.557968+0000","last_unstale":"2023-11-08T05:30:44.490805+0000","last_undegraded":"2023-11-08T05:30:44.490805+0000","last_fullsized":"2023-11-08T05:28:13.542832+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":1,"acting":1,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":12,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738830,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":645040,"kb_used_data":5352,"kb_used_omap":275,"kb_used_meta":639404,"kb_avail":93726800,"statfs":{"total":96636764160,"available":95976243200,"internally_reserved":0,"allocated":5480448,"data_stored":2697647,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":281650,"internal_metadata":654750670},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:59 2023","interfaces":[{"interface":"back","average":{"1min":0.30499999999999999,"5min":0.439,"15min":0.36099999999999999},"min":{"1min":0.17399999999999999,"5min":0.14499999999999999,"15min":0.14499999999999999},"max":{"1min":0.60099999999999998,"5min":4.9379999999999997,"15min":4.9379999999999997},"last":0.379},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.49399999999999999,"15min":0.42399999999999999},"min":{"1min":0.221,"5min":0.19600000000000001,"15min":0.17000000000000001},"max":{"1min":0.63200000000000001,"5min":4.9859999999999998,"15min":4.9859999999999998},"last":0.40899999999999997}]},{"osd":1,"last update":"Wed Nov 8 05:30:03 2023","interfaces":[{"interface":"back","average":{"1min":0.28499999999999998,"5min":0.65100000000000002,"15min":0.47099999999999997},"min":{"1min":0.20100000000000001,"5min":0.153,"15min":0.153},"max":{"1min":0.39800000000000002,"5min":23.242999999999999,"15min":23.242999999999999},"last":0.25},{"interface":"front","average":{"1min":0.29499999999999998,"5min":0.63100000000000001,"15min":0.45900000000000002},"min":{"1min":0.224,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":0.436,"5min":23.251000000000001,"15min":23.251000000000001},"last":0.27800000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738798,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1339228,"kb_used_data":398304,"kb_used_omap":291,"kb_used_meta":936476,"kb_avail":93032612,"statfs":{"total":96636764160,"available":95265394688,"internally_reserved":0,"allocated":407863296,"data_stored":405103900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":298462,"internal_metadata":958951970},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.436,"15min":0.36299999999999999},"min":{"1min":0.19700000000000001,"5min":0.14699999999999999,"15min":0.13700000000000001},"max":{"1min":3.222,"5min":6.5499999999999998,"15min":6.9989999999999997},"last":0.245},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.46300000000000002,"15min":0.42299999999999999},"min":{"1min":0.17799999999999999,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":5.0030000000000001,"5min":6.0629999999999997,"15min":9.6869999999999994},"last":0.28599999999999998}]},{"osd":2,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.51600000000000001,"15min":0.47099999999999997},"min":{"1min":0.21099999999999999,"5min":0.153,"15min":0.153},"max":{"1min":5.7439999999999998,"5min":5.8280000000000003,"15min":9.7639999999999993},"last":0.252},{"interface":"front","average":{"1min":0.497,"5min":0.56200000000000006,"15min":0.42299999999999999},"min":{"1min":0.20399999999999999,"5min":0.187,"15min":0.16800000000000001},"max":{"1min":3.9319999999999999,"5min":8.8040000000000003,"15min":8.8040000000000003},"last":0.29899999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738832,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":650100,"kb_used_data":5356,"kb_used_omap":214,"kb_used_meta":644521,"kb_avail":93721740,"statfs":{"total":96636764160,"available":95971061760,"internally_reserved":0,"allocated":5484544,"data_stored":2698616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":219335,"internal_metadata":659990329},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:56 2023","interfaces":[{"interface":"back","average":{"1min":0.30399999999999999,"5min":0.34699999999999998,"15min":0.36499999999999999},"min":{"1min":0.23999999999999999,"5min":0.14199999999999999,"15min":0.14199999999999999},"max":{"1min":0.42399999999999999,"5min":1.595,"15min":7.4249999999999998},"last":0.17100000000000001},{"interface":"front","average":{"1min":0.35499999999999998,"5min":0.40899999999999997,"15min":0.376},"min":{"1min":0.23999999999999999,"5min":0.19600000000000001,"15min":0.183},"max":{"1min":0.46999999999999997,"5min":2.6859999999999999,"15min":2.6859999999999999},"last":0.23000000000000001}]},{"osd":2,"last update":"Wed Nov 8 05:30:57 2023","interfaces":[{"interface":"back","average":{"1min":0.28599999999999998,"5min":0.34699999999999998,"15min":0.36899999999999999},"min":{"1min":0.19700000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":0.41399999999999998,"5min":2.2650000000000001,"15min":5.8209999999999997},"last":0.20899999999999999},{"interface":"front","average":{"1min":0.30599999999999999,"5min":0.313,"15min":0.35799999999999998},"min":{"1min":0.17799999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.435,"5min":1.137,"15min":4.2350000000000003},"last":0.23000000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-08T05:31:05.572 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~168s 2023-11-08T05:31:06.745 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-11-08T05:31:07.043 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T05:31:07.045 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2023-11-08T05:31:07.055 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":1607,"stamp":"2023-11-08T05:31:06.063865+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":10,"acting":13,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":109,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2634368,"kb_used_data":409012,"kb_used_omap":780,"kb_used_meta":2220403,"kb_avail":280481152,"statfs":{"total":289910292480,"available":287212699648,"internally_reserved":0,"allocated":418828288,"data_stored":410500163,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":799447,"internal_metadata":2273692969},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-536782902,"num_objects":-164,"num_object_clones":0,"num_object_copies":-328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-115,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-164,"num_whiteouts":0,"num_read":-122056,"num_read_kb":-39011942,"num_write":-413642,"num_write_kb":-151705339,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-8,"num_bytes_recovered":-16922624,"num_keys_recovered":-2,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.424817"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490644+0000","last_change":"2023-11-08T05:28:13.557071+0000","last_active":"2023-11-08T05:30:44.490644+0000","last_peered":"2023-11-08T05:30:44.490644+0000","last_clean":"2023-11-08T05:28:01.458439+0000","last_became_active":"2023-11-08T05:28:13.557071+0000","last_became_peered":"2023-11-08T05:28:13.557071+0000","last_unstale":"2023-11-08T05:30:44.490644+0000","last_undegraded":"2023-11-08T05:30:44.490644+0000","last_fullsized":"2023-11-08T05:28:13.547025+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":459,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:45.794760+0000","last_change":"2023-11-08T05:30:45.794760+0000","last_active":"2023-11-08T05:30:45.794760+0000","last_peered":"2023-11-08T05:30:45.794760+0000","last_clean":"2023-11-08T05:30:45.794760+0000","last_became_active":"2023-11-08T05:30:45.794398+0000","last_became_peered":"2023-11-08T05:30:45.794398+0000","last_unstale":"2023-11-08T05:30:45.794760+0000","last_undegraded":"2023-11-08T05:30:45.794760+0000","last_fullsized":"2023-11-08T05:30:45.794760+0000","mapping_epoch":458,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":459,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":465,"reported_epoch":459,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:45.795085+0000","last_change":"2023-11-08T05:30:45.795085+0000","last_active":"2023-11-08T05:30:45.795085+0000","last_peered":"2023-11-08T05:30:45.795085+0000","last_clean":"2023-11-08T05:30:45.795085+0000","last_became_active":"2023-11-08T05:30:45.794743+0000","last_became_peered":"2023-11-08T05:30:45.794743+0000","last_unstale":"2023-11-08T05:30:45.795085+0000","last_undegraded":"2023-11-08T05:30:45.795085+0000","last_fullsized":"2023-11-08T05:30:45.795085+0000","mapping_epoch":458,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":459,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":454,"reported_epoch":458,"state":"active+clean","last_fresh":"2023-11-08T05:30:44.490618+0000","last_change":"2023-11-08T04:52:47.548597+0000","last_active":"2023-11-08T05:30:44.490618+0000","last_peered":"2023-11-08T05:30:44.490618+0000","last_clean":"2023-11-08T05:30:44.490618+0000","last_became_active":"2023-11-08T04:52:47.548294+0000","last_became_peered":"2023-11-08T04:52:47.548294+0000","last_unstale":"2023-11-08T05:30:44.490618+0000","last_undegraded":"2023-11-08T05:30:44.490618+0000","last_fullsized":"2023-11-08T05:30:44.490618+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":460,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490523+0000","last_change":"2023-11-08T05:28:13.545468+0000","last_active":"2023-11-08T05:30:44.490523+0000","last_peered":"2023-11-08T05:30:44.490523+0000","last_clean":"2023-11-08T05:28:01.456723+0000","last_became_active":"2023-11-08T05:28:13.545468+0000","last_became_peered":"2023-11-08T05:28:13.545468+0000","last_unstale":"2023-11-08T05:30:44.490523+0000","last_undegraded":"2023-11-08T05:30:44.490523+0000","last_fullsized":"2023-11-08T05:28:13.537207+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_clean_scrub_stamp":"2023-11-08T04:52:45.528796+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":465,"reported_epoch":458,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:44.491907+0000","last_change":"2023-11-08T05:29:44.395350+0000","last_active":"2023-11-08T05:30:44.491907+0000","last_peered":"2023-11-08T05:30:44.491907+0000","last_clean":"2023-11-08T05:30:44.491907+0000","last_became_active":"2023-11-08T05:29:44.395012+0000","last_became_peered":"2023-11-08T05:29:44.395012+0000","last_unstale":"2023-11-08T05:30:44.491907+0000","last_undegraded":"2023-11-08T05:30:44.491907+0000","last_fullsized":"2023-11-08T05:30:44.491907+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490629+0000","last_change":"2023-11-08T05:28:13.563357+0000","last_active":"2023-11-08T05:30:44.490629+0000","last_peered":"2023-11-08T05:30:44.490629+0000","last_clean":"2023-11-08T05:28:01.457691+0000","last_became_active":"2023-11-08T05:28:13.563357+0000","last_became_peered":"2023-11-08T05:28:13.563357+0000","last_unstale":"2023-11-08T05:30:44.490629+0000","last_undegraded":"2023-11-08T05:30:44.490629+0000","last_fullsized":"2023-11-08T05:28:13.541614+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490691+0000","last_change":"2023-11-08T05:28:13.550672+0000","last_active":"2023-11-08T05:30:44.490691+0000","last_peered":"2023-11-08T05:30:44.490691+0000","last_clean":"2023-11-08T05:28:01.457904+0000","last_became_active":"2023-11-08T05:28:13.550672+0000","last_became_peered":"2023-11-08T05:28:13.550672+0000","last_unstale":"2023-11-08T05:30:44.490691+0000","last_undegraded":"2023-11-08T05:30:44.490691+0000","last_fullsized":"2023-11-08T05:28:13.539595+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490805+0000","last_change":"2023-11-08T05:28:13.557968+0000","last_active":"2023-11-08T05:30:44.490805+0000","last_peered":"2023-11-08T05:30:44.490805+0000","last_clean":"2023-11-08T05:28:01.458433+0000","last_became_active":"2023-11-08T05:28:13.557968+0000","last_became_peered":"2023-11-08T05:28:13.557968+0000","last_unstale":"2023-11-08T05:30:44.490805+0000","last_undegraded":"2023-11-08T05:30:44.490805+0000","last_fullsized":"2023-11-08T05:28:13.542832+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":1,"acting":1,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":12,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738831,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":645040,"kb_used_data":5352,"kb_used_omap":275,"kb_used_meta":639404,"kb_avail":93726800,"statfs":{"total":96636764160,"available":95976243200,"internally_reserved":0,"allocated":5480448,"data_stored":2697647,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":281650,"internal_metadata":654750670},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:59 2023","interfaces":[{"interface":"back","average":{"1min":0.30499999999999999,"5min":0.439,"15min":0.36099999999999999},"min":{"1min":0.17399999999999999,"5min":0.14499999999999999,"15min":0.14499999999999999},"max":{"1min":0.60099999999999998,"5min":4.9379999999999997,"15min":4.9379999999999997},"last":0.379},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.49399999999999999,"15min":0.42399999999999999},"min":{"1min":0.221,"5min":0.19600000000000001,"15min":0.17000000000000001},"max":{"1min":0.63200000000000001,"5min":4.9859999999999998,"15min":4.9859999999999998},"last":0.40899999999999997}]},{"osd":1,"last update":"Wed Nov 8 05:30:03 2023","interfaces":[{"interface":"back","average":{"1min":0.28499999999999998,"5min":0.65100000000000002,"15min":0.47099999999999997},"min":{"1min":0.20100000000000001,"5min":0.153,"15min":0.153},"max":{"1min":0.39800000000000002,"5min":23.242999999999999,"15min":23.242999999999999},"last":0.23699999999999999},{"interface":"front","average":{"1min":0.29499999999999998,"5min":0.63100000000000001,"15min":0.45900000000000002},"min":{"1min":0.224,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":0.436,"5min":23.251000000000001,"15min":23.251000000000001},"last":0.27100000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738798,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1339228,"kb_used_data":398304,"kb_used_omap":291,"kb_used_meta":936476,"kb_avail":93032612,"statfs":{"total":96636764160,"available":95265394688,"internally_reserved":0,"allocated":407863296,"data_stored":405103900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":298462,"internal_metadata":958951970},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.436,"15min":0.36299999999999999},"min":{"1min":0.19700000000000001,"5min":0.14699999999999999,"15min":0.13700000000000001},"max":{"1min":3.222,"5min":6.5499999999999998,"15min":6.9989999999999997},"last":0.245},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.46300000000000002,"15min":0.42299999999999999},"min":{"1min":0.17799999999999999,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":5.0030000000000001,"5min":6.0629999999999997,"15min":9.6869999999999994},"last":0.28599999999999998}]},{"osd":2,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.51600000000000001,"15min":0.47099999999999997},"min":{"1min":0.21099999999999999,"5min":0.153,"15min":0.153},"max":{"1min":5.7439999999999998,"5min":5.8280000000000003,"15min":9.7639999999999993},"last":0.252},{"interface":"front","average":{"1min":0.497,"5min":0.56200000000000006,"15min":0.42299999999999999},"min":{"1min":0.20399999999999999,"5min":0.187,"15min":0.16800000000000001},"max":{"1min":3.9319999999999999,"5min":8.8040000000000003,"15min":8.8040000000000003},"last":0.29899999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738833,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":650100,"kb_used_data":5356,"kb_used_omap":214,"kb_used_meta":644521,"kb_avail":93721740,"statfs":{"total":96636764160,"available":95971061760,"internally_reserved":0,"allocated":5484544,"data_stored":2698616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":219335,"internal_metadata":659990329},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:56 2023","interfaces":[{"interface":"back","average":{"1min":0.30399999999999999,"5min":0.34699999999999998,"15min":0.36499999999999999},"min":{"1min":0.23999999999999999,"5min":0.14199999999999999,"15min":0.14199999999999999},"max":{"1min":0.42399999999999999,"5min":1.595,"15min":7.4249999999999998},"last":0.17100000000000001},{"interface":"front","average":{"1min":0.35499999999999998,"5min":0.40899999999999997,"15min":0.376},"min":{"1min":0.23999999999999999,"5min":0.19600000000000001,"15min":0.183},"max":{"1min":0.46999999999999997,"5min":2.6859999999999999,"15min":2.6859999999999999},"last":0.23000000000000001}]},{"osd":2,"last update":"Wed Nov 8 05:30:57 2023","interfaces":[{"interface":"back","average":{"1min":0.28599999999999998,"5min":0.34699999999999998,"15min":0.36899999999999999},"min":{"1min":0.19700000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":0.41399999999999998,"5min":2.2650000000000001,"15min":5.8209999999999997},"last":0.21099999999999999},{"interface":"front","average":{"1min":0.30599999999999999,"5min":0.313,"15min":0.35799999999999998},"min":{"1min":0.17799999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.435,"5min":1.137,"15min":4.2350000000000003},"last":0.223}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-08T05:31:07.057 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-11-08T05:31:07.338 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T05:31:07.340 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2023-11-08T05:31:07.351 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":1607,"stamp":"2023-11-08T05:31:06.063865+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":10,"acting":13,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":109,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2634368,"kb_used_data":409012,"kb_used_omap":780,"kb_used_meta":2220403,"kb_avail":280481152,"statfs":{"total":289910292480,"available":287212699648,"internally_reserved":0,"allocated":418828288,"data_stored":410500163,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":799447,"internal_metadata":2273692969},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-536782902,"num_objects":-164,"num_object_clones":0,"num_object_copies":-328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-115,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-164,"num_whiteouts":0,"num_read":-122056,"num_read_kb":-39011942,"num_write":-413642,"num_write_kb":-151705339,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-8,"num_bytes_recovered":-16922624,"num_keys_recovered":-2,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.424817"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490644+0000","last_change":"2023-11-08T05:28:13.557071+0000","last_active":"2023-11-08T05:30:44.490644+0000","last_peered":"2023-11-08T05:30:44.490644+0000","last_clean":"2023-11-08T05:28:01.458439+0000","last_became_active":"2023-11-08T05:28:13.557071+0000","last_became_peered":"2023-11-08T05:28:13.557071+0000","last_unstale":"2023-11-08T05:30:44.490644+0000","last_undegraded":"2023-11-08T05:30:44.490644+0000","last_fullsized":"2023-11-08T05:28:13.547025+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":459,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:45.794760+0000","last_change":"2023-11-08T05:30:45.794760+0000","last_active":"2023-11-08T05:30:45.794760+0000","last_peered":"2023-11-08T05:30:45.794760+0000","last_clean":"2023-11-08T05:30:45.794760+0000","last_became_active":"2023-11-08T05:30:45.794398+0000","last_became_peered":"2023-11-08T05:30:45.794398+0000","last_unstale":"2023-11-08T05:30:45.794760+0000","last_undegraded":"2023-11-08T05:30:45.794760+0000","last_fullsized":"2023-11-08T05:30:45.794760+0000","mapping_epoch":458,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":459,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":465,"reported_epoch":459,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:45.795085+0000","last_change":"2023-11-08T05:30:45.795085+0000","last_active":"2023-11-08T05:30:45.795085+0000","last_peered":"2023-11-08T05:30:45.795085+0000","last_clean":"2023-11-08T05:30:45.795085+0000","last_became_active":"2023-11-08T05:30:45.794743+0000","last_became_peered":"2023-11-08T05:30:45.794743+0000","last_unstale":"2023-11-08T05:30:45.795085+0000","last_undegraded":"2023-11-08T05:30:45.795085+0000","last_fullsized":"2023-11-08T05:30:45.795085+0000","mapping_epoch":458,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":459,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":454,"reported_epoch":458,"state":"active+clean","last_fresh":"2023-11-08T05:30:44.490618+0000","last_change":"2023-11-08T04:52:47.548597+0000","last_active":"2023-11-08T05:30:44.490618+0000","last_peered":"2023-11-08T05:30:44.490618+0000","last_clean":"2023-11-08T05:30:44.490618+0000","last_became_active":"2023-11-08T04:52:47.548294+0000","last_became_peered":"2023-11-08T04:52:47.548294+0000","last_unstale":"2023-11-08T05:30:44.490618+0000","last_undegraded":"2023-11-08T05:30:44.490618+0000","last_fullsized":"2023-11-08T05:30:44.490618+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":460,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490523+0000","last_change":"2023-11-08T05:28:13.545468+0000","last_active":"2023-11-08T05:30:44.490523+0000","last_peered":"2023-11-08T05:30:44.490523+0000","last_clean":"2023-11-08T05:28:01.456723+0000","last_became_active":"2023-11-08T05:28:13.545468+0000","last_became_peered":"2023-11-08T05:28:13.545468+0000","last_unstale":"2023-11-08T05:30:44.490523+0000","last_undegraded":"2023-11-08T05:30:44.490523+0000","last_fullsized":"2023-11-08T05:28:13.537207+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_clean_scrub_stamp":"2023-11-08T04:52:45.528796+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":465,"reported_epoch":458,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:44.491907+0000","last_change":"2023-11-08T05:29:44.395350+0000","last_active":"2023-11-08T05:30:44.491907+0000","last_peered":"2023-11-08T05:30:44.491907+0000","last_clean":"2023-11-08T05:30:44.491907+0000","last_became_active":"2023-11-08T05:29:44.395012+0000","last_became_peered":"2023-11-08T05:29:44.395012+0000","last_unstale":"2023-11-08T05:30:44.491907+0000","last_undegraded":"2023-11-08T05:30:44.491907+0000","last_fullsized":"2023-11-08T05:30:44.491907+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490629+0000","last_change":"2023-11-08T05:28:13.563357+0000","last_active":"2023-11-08T05:30:44.490629+0000","last_peered":"2023-11-08T05:30:44.490629+0000","last_clean":"2023-11-08T05:28:01.457691+0000","last_became_active":"2023-11-08T05:28:13.563357+0000","last_became_peered":"2023-11-08T05:28:13.563357+0000","last_unstale":"2023-11-08T05:30:44.490629+0000","last_undegraded":"2023-11-08T05:30:44.490629+0000","last_fullsized":"2023-11-08T05:28:13.541614+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490691+0000","last_change":"2023-11-08T05:28:13.550672+0000","last_active":"2023-11-08T05:30:44.490691+0000","last_peered":"2023-11-08T05:30:44.490691+0000","last_clean":"2023-11-08T05:28:01.457904+0000","last_became_active":"2023-11-08T05:28:13.550672+0000","last_became_peered":"2023-11-08T05:28:13.550672+0000","last_unstale":"2023-11-08T05:30:44.490691+0000","last_undegraded":"2023-11-08T05:30:44.490691+0000","last_fullsized":"2023-11-08T05:28:13.539595+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490805+0000","last_change":"2023-11-08T05:28:13.557968+0000","last_active":"2023-11-08T05:30:44.490805+0000","last_peered":"2023-11-08T05:30:44.490805+0000","last_clean":"2023-11-08T05:28:01.458433+0000","last_became_active":"2023-11-08T05:28:13.557968+0000","last_became_peered":"2023-11-08T05:28:13.557968+0000","last_unstale":"2023-11-08T05:30:44.490805+0000","last_undegraded":"2023-11-08T05:30:44.490805+0000","last_fullsized":"2023-11-08T05:28:13.542832+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":1,"acting":1,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":12,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738831,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":645040,"kb_used_data":5352,"kb_used_omap":275,"kb_used_meta":639404,"kb_avail":93726800,"statfs":{"total":96636764160,"available":95976243200,"internally_reserved":0,"allocated":5480448,"data_stored":2697647,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":281650,"internal_metadata":654750670},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:59 2023","interfaces":[{"interface":"back","average":{"1min":0.30499999999999999,"5min":0.439,"15min":0.36099999999999999},"min":{"1min":0.17399999999999999,"5min":0.14499999999999999,"15min":0.14499999999999999},"max":{"1min":0.60099999999999998,"5min":4.9379999999999997,"15min":4.9379999999999997},"last":0.379},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.49399999999999999,"15min":0.42399999999999999},"min":{"1min":0.221,"5min":0.19600000000000001,"15min":0.17000000000000001},"max":{"1min":0.63200000000000001,"5min":4.9859999999999998,"15min":4.9859999999999998},"last":0.40899999999999997}]},{"osd":1,"last update":"Wed Nov 8 05:30:03 2023","interfaces":[{"interface":"back","average":{"1min":0.28499999999999998,"5min":0.65100000000000002,"15min":0.47099999999999997},"min":{"1min":0.20100000000000001,"5min":0.153,"15min":0.153},"max":{"1min":0.39800000000000002,"5min":23.242999999999999,"15min":23.242999999999999},"last":0.23699999999999999},{"interface":"front","average":{"1min":0.29499999999999998,"5min":0.63100000000000001,"15min":0.45900000000000002},"min":{"1min":0.224,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":0.436,"5min":23.251000000000001,"15min":23.251000000000001},"last":0.27100000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738798,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1339228,"kb_used_data":398304,"kb_used_omap":291,"kb_used_meta":936476,"kb_avail":93032612,"statfs":{"total":96636764160,"available":95265394688,"internally_reserved":0,"allocated":407863296,"data_stored":405103900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":298462,"internal_metadata":958951970},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.436,"15min":0.36299999999999999},"min":{"1min":0.19700000000000001,"5min":0.14699999999999999,"15min":0.13700000000000001},"max":{"1min":3.222,"5min":6.5499999999999998,"15min":6.9989999999999997},"last":0.245},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.46300000000000002,"15min":0.42299999999999999},"min":{"1min":0.17799999999999999,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":5.0030000000000001,"5min":6.0629999999999997,"15min":9.6869999999999994},"last":0.28599999999999998}]},{"osd":2,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.51600000000000001,"15min":0.47099999999999997},"min":{"1min":0.21099999999999999,"5min":0.153,"15min":0.153},"max":{"1min":5.7439999999999998,"5min":5.8280000000000003,"15min":9.7639999999999993},"last":0.252},{"interface":"front","average":{"1min":0.497,"5min":0.56200000000000006,"15min":0.42299999999999999},"min":{"1min":0.20399999999999999,"5min":0.187,"15min":0.16800000000000001},"max":{"1min":3.9319999999999999,"5min":8.8040000000000003,"15min":8.8040000000000003},"last":0.29899999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738833,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":650100,"kb_used_data":5356,"kb_used_omap":214,"kb_used_meta":644521,"kb_avail":93721740,"statfs":{"total":96636764160,"available":95971061760,"internally_reserved":0,"allocated":5484544,"data_stored":2698616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":219335,"internal_metadata":659990329},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:56 2023","interfaces":[{"interface":"back","average":{"1min":0.30399999999999999,"5min":0.34699999999999998,"15min":0.36499999999999999},"min":{"1min":0.23999999999999999,"5min":0.14199999999999999,"15min":0.14199999999999999},"max":{"1min":0.42399999999999999,"5min":1.595,"15min":7.4249999999999998},"last":0.17100000000000001},{"interface":"front","average":{"1min":0.35499999999999998,"5min":0.40899999999999997,"15min":0.376},"min":{"1min":0.23999999999999999,"5min":0.19600000000000001,"15min":0.183},"max":{"1min":0.46999999999999997,"5min":2.6859999999999999,"15min":2.6859999999999999},"last":0.23000000000000001}]},{"osd":2,"last update":"Wed Nov 8 05:30:57 2023","interfaces":[{"interface":"back","average":{"1min":0.28599999999999998,"5min":0.34699999999999998,"15min":0.36899999999999999},"min":{"1min":0.19700000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":0.41399999999999998,"5min":2.2650000000000001,"15min":5.8209999999999997},"last":0.21099999999999999},{"interface":"front","average":{"1min":0.30599999999999999,"5min":0.313,"15min":0.35799999999999998},"min":{"1min":0.17799999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.435,"5min":1.137,"15min":4.2350000000000003},"last":0.223}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-08T05:31:07.352 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-11-08T05:31:07.353 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 459, 'reported_epoch': 458, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:30:44.490644+0000', 'last_change': '2023-11-08T05:28:13.557071+0000', 'last_active': '2023-11-08T05:30:44.490644+0000', 'last_peered': '2023-11-08T05:30:44.490644+0000', 'last_clean': '2023-11-08T05:28:01.458439+0000', 'last_became_active': '2023-11-08T05:28:13.557071+0000', 'last_became_peered': '2023-11-08T05:28:13.557071+0000', 'last_unstale': '2023-11-08T05:30:44.490644+0000', 'last_undegraded': '2023-11-08T05:30:44.490644+0000', 'last_fullsized': '2023-11-08T05:28:13.547025+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:31:07.353 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-11-08T05:31:07.353 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 11, 'reported_epoch': 459, 'state': 'active+clean+remapped', 'last_fresh': '2023-11-08T05:30:45.794760+0000', 'last_change': '2023-11-08T05:30:45.794760+0000', 'last_active': '2023-11-08T05:30:45.794760+0000', 'last_peered': '2023-11-08T05:30:45.794760+0000', 'last_clean': '2023-11-08T05:30:45.794760+0000', 'last_became_active': '2023-11-08T05:30:45.794398+0000', 'last_became_peered': '2023-11-08T05:30:45.794398+0000', 'last_unstale': '2023-11-08T05:30:45.794760+0000', 'last_undegraded': '2023-11-08T05:30:45.794760+0000', 'last_fullsized': '2023-11-08T05:30:45.794760+0000', 'mapping_epoch': 458, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 459, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2], 'acting': [2, 1], 'avail_no_missing': ['2', '1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2023-11-08T05:31:07.354 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.1 is not active+clean 2023-11-08T05:31:07.354 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.1', 'version': "0'0", 'reported_seq': 465, 'reported_epoch': 459, 'state': 'active+clean+remapped', 'last_fresh': '2023-11-08T05:30:45.795085+0000', 'last_change': '2023-11-08T05:30:45.795085+0000', 'last_active': '2023-11-08T05:30:45.795085+0000', 'last_peered': '2023-11-08T05:30:45.795085+0000', 'last_clean': '2023-11-08T05:30:45.795085+0000', 'last_became_active': '2023-11-08T05:30:45.794743+0000', 'last_became_peered': '2023-11-08T05:30:45.794743+0000', 'last_unstale': '2023-11-08T05:30:45.795085+0000', 'last_undegraded': '2023-11-08T05:30:45.795085+0000', 'last_fullsized': '2023-11-08T05:30:45.795085+0000', 'mapping_epoch': 458, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 459, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2], 'acting': [2, 1], 'avail_no_missing': ['2', '1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2023-11-08T05:31:07.354 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-11-08T05:31:07.355 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "0'0", 'reported_seq': 460, 'reported_epoch': 458, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:30:44.490523+0000', 'last_change': '2023-11-08T05:28:13.545468+0000', 'last_active': '2023-11-08T05:30:44.490523+0000', 'last_peered': '2023-11-08T05:30:44.490523+0000', 'last_clean': '2023-11-08T05:28:01.456723+0000', 'last_became_active': '2023-11-08T05:28:13.545468+0000', 'last_became_peered': '2023-11-08T05:28:13.545468+0000', 'last_unstale': '2023-11-08T05:30:44.490523+0000', 'last_undegraded': '2023-11-08T05:30:44.490523+0000', 'last_fullsized': '2023-11-08T05:28:13.537207+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 9, 'last_epoch_clean': 10, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:45.528796+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:45.528796+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:45.528796+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:31:07.355 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.0 is not active+clean 2023-11-08T05:31:07.355 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.0', 'version': "0'0", 'reported_seq': 465, 'reported_epoch': 458, 'state': 'active+clean+remapped', 'last_fresh': '2023-11-08T05:30:44.491907+0000', 'last_change': '2023-11-08T05:29:44.395350+0000', 'last_active': '2023-11-08T05:30:44.491907+0000', 'last_peered': '2023-11-08T05:30:44.491907+0000', 'last_clean': '2023-11-08T05:30:44.491907+0000', 'last_became_active': '2023-11-08T05:29:44.395012+0000', 'last_became_peered': '2023-11-08T05:29:44.395012+0000', 'last_unstale': '2023-11-08T05:30:44.491907+0000', 'last_undegraded': '2023-11-08T05:30:44.491907+0000', 'last_fullsized': '2023-11-08T05:30:44.491907+0000', 'mapping_epoch': 455, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 456, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2], 'acting': [2, 1], 'avail_no_missing': ['2', '1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2023-11-08T05:31:07.356 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-11-08T05:31:07.356 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 459, 'reported_epoch': 458, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:30:44.490629+0000', 'last_change': '2023-11-08T05:28:13.563357+0000', 'last_active': '2023-11-08T05:30:44.490629+0000', 'last_peered': '2023-11-08T05:30:44.490629+0000', 'last_clean': '2023-11-08T05:28:01.457691+0000', 'last_became_active': '2023-11-08T05:28:13.563357+0000', 'last_became_peered': '2023-11-08T05:28:13.563357+0000', 'last_unstale': '2023-11-08T05:30:44.490629+0000', 'last_undegraded': '2023-11-08T05:30:44.490629+0000', 'last_fullsized': '2023-11-08T05:28:13.541614+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:31:07.356 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-11-08T05:31:07.356 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 459, 'reported_epoch': 458, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:30:44.490691+0000', 'last_change': '2023-11-08T05:28:13.550672+0000', 'last_active': '2023-11-08T05:30:44.490691+0000', 'last_peered': '2023-11-08T05:30:44.490691+0000', 'last_clean': '2023-11-08T05:28:01.457904+0000', 'last_became_active': '2023-11-08T05:28:13.550672+0000', 'last_became_peered': '2023-11-08T05:28:13.550672+0000', 'last_unstale': '2023-11-08T05:30:44.490691+0000', 'last_undegraded': '2023-11-08T05:30:44.490691+0000', 'last_fullsized': '2023-11-08T05:28:13.539595+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:31:07.357 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-11-08T05:31:07.357 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 459, 'reported_epoch': 458, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:30:44.490805+0000', 'last_change': '2023-11-08T05:28:13.557968+0000', 'last_active': '2023-11-08T05:30:44.490805+0000', 'last_peered': '2023-11-08T05:30:44.490805+0000', 'last_clean': '2023-11-08T05:28:01.458433+0000', 'last_became_active': '2023-11-08T05:28:13.557968+0000', 'last_became_peered': '2023-11-08T05:28:13.557968+0000', 'last_unstale': '2023-11-08T05:30:44.490805+0000', 'last_undegraded': '2023-11-08T05:30:44.490805+0000', 'last_fullsized': '2023-11-08T05:28:13.542832+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:31:07.357 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-11-08T05:31:07.704 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T05:31:07.705 INFO:teuthology.orchestra.run.smithi087.stdout:{"fsid":"ab4cbc06-449d-497d-b513-220d9db2b590","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 5 pgs undersized","count":5},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2310,"monmap":{"epoch":1,"min_mon_release_name":"pacific","num_mons":1},"osdmap":{"epoch":459,"num_osds":3,"num_up_osds":2,"osd_up_since":1699421292,"num_in_osds":3,"osd_in_since":1699419158,"num_remapped_pgs":3},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean+remapped","count":3},{"state_name":"active+clean","count":1}],"num_pgs":9,"num_pools":2,"num_objects":0,"data_bytes":0,"bytes_used":2697592832,"bytes_avail":287212699648,"bytes_total":289910292480},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":41,"modified":"2023-11-08T05:29:32.044544+0000","services":{}},"progress_events":{}} 2023-11-08T05:31:07.717 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-11-08T05:31:07.718 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-11-08T05:31:08.002 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T05:31:08.003 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2023-11-08T05:31:08.014 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":1607,"stamp":"2023-11-08T05:31:06.063865+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":10,"acting":13,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":109,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2634368,"kb_used_data":409012,"kb_used_omap":780,"kb_used_meta":2220403,"kb_avail":280481152,"statfs":{"total":289910292480,"available":287212699648,"internally_reserved":0,"allocated":418828288,"data_stored":410500163,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":799447,"internal_metadata":2273692969},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-536782902,"num_objects":-164,"num_object_clones":0,"num_object_copies":-328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-115,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-164,"num_whiteouts":0,"num_read":-122056,"num_read_kb":-39011942,"num_write":-413642,"num_write_kb":-151705339,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-8,"num_bytes_recovered":-16922624,"num_keys_recovered":-2,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.424817"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490644+0000","last_change":"2023-11-08T05:28:13.557071+0000","last_active":"2023-11-08T05:30:44.490644+0000","last_peered":"2023-11-08T05:30:44.490644+0000","last_clean":"2023-11-08T05:28:01.458439+0000","last_became_active":"2023-11-08T05:28:13.557071+0000","last_became_peered":"2023-11-08T05:28:13.557071+0000","last_unstale":"2023-11-08T05:30:44.490644+0000","last_undegraded":"2023-11-08T05:30:44.490644+0000","last_fullsized":"2023-11-08T05:28:13.547025+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":459,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:45.794760+0000","last_change":"2023-11-08T05:30:45.794760+0000","last_active":"2023-11-08T05:30:45.794760+0000","last_peered":"2023-11-08T05:30:45.794760+0000","last_clean":"2023-11-08T05:30:45.794760+0000","last_became_active":"2023-11-08T05:30:45.794398+0000","last_became_peered":"2023-11-08T05:30:45.794398+0000","last_unstale":"2023-11-08T05:30:45.794760+0000","last_undegraded":"2023-11-08T05:30:45.794760+0000","last_fullsized":"2023-11-08T05:30:45.794760+0000","mapping_epoch":458,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":459,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":465,"reported_epoch":459,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:45.795085+0000","last_change":"2023-11-08T05:30:45.795085+0000","last_active":"2023-11-08T05:30:45.795085+0000","last_peered":"2023-11-08T05:30:45.795085+0000","last_clean":"2023-11-08T05:30:45.795085+0000","last_became_active":"2023-11-08T05:30:45.794743+0000","last_became_peered":"2023-11-08T05:30:45.794743+0000","last_unstale":"2023-11-08T05:30:45.795085+0000","last_undegraded":"2023-11-08T05:30:45.795085+0000","last_fullsized":"2023-11-08T05:30:45.795085+0000","mapping_epoch":458,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":459,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":454,"reported_epoch":458,"state":"active+clean","last_fresh":"2023-11-08T05:30:44.490618+0000","last_change":"2023-11-08T04:52:47.548597+0000","last_active":"2023-11-08T05:30:44.490618+0000","last_peered":"2023-11-08T05:30:44.490618+0000","last_clean":"2023-11-08T05:30:44.490618+0000","last_became_active":"2023-11-08T04:52:47.548294+0000","last_became_peered":"2023-11-08T04:52:47.548294+0000","last_unstale":"2023-11-08T05:30:44.490618+0000","last_undegraded":"2023-11-08T05:30:44.490618+0000","last_fullsized":"2023-11-08T05:30:44.490618+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":460,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490523+0000","last_change":"2023-11-08T05:28:13.545468+0000","last_active":"2023-11-08T05:30:44.490523+0000","last_peered":"2023-11-08T05:30:44.490523+0000","last_clean":"2023-11-08T05:28:01.456723+0000","last_became_active":"2023-11-08T05:28:13.545468+0000","last_became_peered":"2023-11-08T05:28:13.545468+0000","last_unstale":"2023-11-08T05:30:44.490523+0000","last_undegraded":"2023-11-08T05:30:44.490523+0000","last_fullsized":"2023-11-08T05:28:13.537207+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_clean_scrub_stamp":"2023-11-08T04:52:45.528796+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":465,"reported_epoch":458,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:44.491907+0000","last_change":"2023-11-08T05:29:44.395350+0000","last_active":"2023-11-08T05:30:44.491907+0000","last_peered":"2023-11-08T05:30:44.491907+0000","last_clean":"2023-11-08T05:30:44.491907+0000","last_became_active":"2023-11-08T05:29:44.395012+0000","last_became_peered":"2023-11-08T05:29:44.395012+0000","last_unstale":"2023-11-08T05:30:44.491907+0000","last_undegraded":"2023-11-08T05:30:44.491907+0000","last_fullsized":"2023-11-08T05:30:44.491907+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490629+0000","last_change":"2023-11-08T05:28:13.563357+0000","last_active":"2023-11-08T05:30:44.490629+0000","last_peered":"2023-11-08T05:30:44.490629+0000","last_clean":"2023-11-08T05:28:01.457691+0000","last_became_active":"2023-11-08T05:28:13.563357+0000","last_became_peered":"2023-11-08T05:28:13.563357+0000","last_unstale":"2023-11-08T05:30:44.490629+0000","last_undegraded":"2023-11-08T05:30:44.490629+0000","last_fullsized":"2023-11-08T05:28:13.541614+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490691+0000","last_change":"2023-11-08T05:28:13.550672+0000","last_active":"2023-11-08T05:30:44.490691+0000","last_peered":"2023-11-08T05:30:44.490691+0000","last_clean":"2023-11-08T05:28:01.457904+0000","last_became_active":"2023-11-08T05:28:13.550672+0000","last_became_peered":"2023-11-08T05:28:13.550672+0000","last_unstale":"2023-11-08T05:30:44.490691+0000","last_undegraded":"2023-11-08T05:30:44.490691+0000","last_fullsized":"2023-11-08T05:28:13.539595+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490805+0000","last_change":"2023-11-08T05:28:13.557968+0000","last_active":"2023-11-08T05:30:44.490805+0000","last_peered":"2023-11-08T05:30:44.490805+0000","last_clean":"2023-11-08T05:28:01.458433+0000","last_became_active":"2023-11-08T05:28:13.557968+0000","last_became_peered":"2023-11-08T05:28:13.557968+0000","last_unstale":"2023-11-08T05:30:44.490805+0000","last_undegraded":"2023-11-08T05:30:44.490805+0000","last_fullsized":"2023-11-08T05:28:13.542832+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":1,"acting":1,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":12,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738831,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":645040,"kb_used_data":5352,"kb_used_omap":275,"kb_used_meta":639404,"kb_avail":93726800,"statfs":{"total":96636764160,"available":95976243200,"internally_reserved":0,"allocated":5480448,"data_stored":2697647,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":281650,"internal_metadata":654750670},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:59 2023","interfaces":[{"interface":"back","average":{"1min":0.30499999999999999,"5min":0.439,"15min":0.36099999999999999},"min":{"1min":0.17399999999999999,"5min":0.14499999999999999,"15min":0.14499999999999999},"max":{"1min":0.60099999999999998,"5min":4.9379999999999997,"15min":4.9379999999999997},"last":0.379},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.49399999999999999,"15min":0.42399999999999999},"min":{"1min":0.221,"5min":0.19600000000000001,"15min":0.17000000000000001},"max":{"1min":0.63200000000000001,"5min":4.9859999999999998,"15min":4.9859999999999998},"last":0.40899999999999997}]},{"osd":1,"last update":"Wed Nov 8 05:30:03 2023","interfaces":[{"interface":"back","average":{"1min":0.28499999999999998,"5min":0.65100000000000002,"15min":0.47099999999999997},"min":{"1min":0.20100000000000001,"5min":0.153,"15min":0.153},"max":{"1min":0.39800000000000002,"5min":23.242999999999999,"15min":23.242999999999999},"last":0.23699999999999999},{"interface":"front","average":{"1min":0.29499999999999998,"5min":0.63100000000000001,"15min":0.45900000000000002},"min":{"1min":0.224,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":0.436,"5min":23.251000000000001,"15min":23.251000000000001},"last":0.27100000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738798,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1339228,"kb_used_data":398304,"kb_used_omap":291,"kb_used_meta":936476,"kb_avail":93032612,"statfs":{"total":96636764160,"available":95265394688,"internally_reserved":0,"allocated":407863296,"data_stored":405103900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":298462,"internal_metadata":958951970},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.436,"15min":0.36299999999999999},"min":{"1min":0.19700000000000001,"5min":0.14699999999999999,"15min":0.13700000000000001},"max":{"1min":3.222,"5min":6.5499999999999998,"15min":6.9989999999999997},"last":0.245},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.46300000000000002,"15min":0.42299999999999999},"min":{"1min":0.17799999999999999,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":5.0030000000000001,"5min":6.0629999999999997,"15min":9.6869999999999994},"last":0.28599999999999998}]},{"osd":2,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.51600000000000001,"15min":0.47099999999999997},"min":{"1min":0.21099999999999999,"5min":0.153,"15min":0.153},"max":{"1min":5.7439999999999998,"5min":5.8280000000000003,"15min":9.7639999999999993},"last":0.252},{"interface":"front","average":{"1min":0.497,"5min":0.56200000000000006,"15min":0.42299999999999999},"min":{"1min":0.20399999999999999,"5min":0.187,"15min":0.16800000000000001},"max":{"1min":3.9319999999999999,"5min":8.8040000000000003,"15min":8.8040000000000003},"last":0.29899999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738833,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":650100,"kb_used_data":5356,"kb_used_omap":214,"kb_used_meta":644521,"kb_avail":93721740,"statfs":{"total":96636764160,"available":95971061760,"internally_reserved":0,"allocated":5484544,"data_stored":2698616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":219335,"internal_metadata":659990329},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:56 2023","interfaces":[{"interface":"back","average":{"1min":0.30399999999999999,"5min":0.34699999999999998,"15min":0.36499999999999999},"min":{"1min":0.23999999999999999,"5min":0.14199999999999999,"15min":0.14199999999999999},"max":{"1min":0.42399999999999999,"5min":1.595,"15min":7.4249999999999998},"last":0.17100000000000001},{"interface":"front","average":{"1min":0.35499999999999998,"5min":0.40899999999999997,"15min":0.376},"min":{"1min":0.23999999999999999,"5min":0.19600000000000001,"15min":0.183},"max":{"1min":0.46999999999999997,"5min":2.6859999999999999,"15min":2.6859999999999999},"last":0.23000000000000001}]},{"osd":2,"last update":"Wed Nov 8 05:30:57 2023","interfaces":[{"interface":"back","average":{"1min":0.28599999999999998,"5min":0.34699999999999998,"15min":0.36899999999999999},"min":{"1min":0.19700000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":0.41399999999999998,"5min":2.2650000000000001,"15min":5.8209999999999997},"last":0.21099999999999999},{"interface":"front","average":{"1min":0.30599999999999999,"5min":0.313,"15min":0.35799999999999998},"min":{"1min":0.17799999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.435,"5min":1.137,"15min":4.2350000000000003},"last":0.223}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-08T05:31:10.977 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~173s 2023-11-08T05:31:11.017 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-11-08T05:31:11.311 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T05:31:11.313 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2023-11-08T05:31:11.324 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":1609,"stamp":"2023-11-08T05:31:10.064636+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":10,"acting":13,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":109,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2634368,"kb_used_data":409012,"kb_used_omap":780,"kb_used_meta":2220403,"kb_avail":280481152,"statfs":{"total":289910292480,"available":287212699648,"internally_reserved":0,"allocated":418828288,"data_stored":410500163,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":799447,"internal_metadata":2273692969},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-536782902,"num_objects":-164,"num_object_clones":0,"num_object_copies":-328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-115,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-164,"num_whiteouts":0,"num_read":-122056,"num_read_kb":-39011942,"num_write":-413642,"num_write_kb":-151705339,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-8,"num_bytes_recovered":-16922624,"num_keys_recovered":-2,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.424817"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490644+0000","last_change":"2023-11-08T05:28:13.557071+0000","last_active":"2023-11-08T05:30:44.490644+0000","last_peered":"2023-11-08T05:30:44.490644+0000","last_clean":"2023-11-08T05:28:01.458439+0000","last_became_active":"2023-11-08T05:28:13.557071+0000","last_became_peered":"2023-11-08T05:28:13.557071+0000","last_unstale":"2023-11-08T05:30:44.490644+0000","last_undegraded":"2023-11-08T05:30:44.490644+0000","last_fullsized":"2023-11-08T05:28:13.547025+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":459,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:45.794760+0000","last_change":"2023-11-08T05:30:45.794760+0000","last_active":"2023-11-08T05:30:45.794760+0000","last_peered":"2023-11-08T05:30:45.794760+0000","last_clean":"2023-11-08T05:30:45.794760+0000","last_became_active":"2023-11-08T05:30:45.794398+0000","last_became_peered":"2023-11-08T05:30:45.794398+0000","last_unstale":"2023-11-08T05:30:45.794760+0000","last_undegraded":"2023-11-08T05:30:45.794760+0000","last_fullsized":"2023-11-08T05:30:45.794760+0000","mapping_epoch":458,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":459,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":465,"reported_epoch":459,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:45.795085+0000","last_change":"2023-11-08T05:30:45.795085+0000","last_active":"2023-11-08T05:30:45.795085+0000","last_peered":"2023-11-08T05:30:45.795085+0000","last_clean":"2023-11-08T05:30:45.795085+0000","last_became_active":"2023-11-08T05:30:45.794743+0000","last_became_peered":"2023-11-08T05:30:45.794743+0000","last_unstale":"2023-11-08T05:30:45.795085+0000","last_undegraded":"2023-11-08T05:30:45.795085+0000","last_fullsized":"2023-11-08T05:30:45.795085+0000","mapping_epoch":458,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":459,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":454,"reported_epoch":458,"state":"active+clean","last_fresh":"2023-11-08T05:30:44.490618+0000","last_change":"2023-11-08T04:52:47.548597+0000","last_active":"2023-11-08T05:30:44.490618+0000","last_peered":"2023-11-08T05:30:44.490618+0000","last_clean":"2023-11-08T05:30:44.490618+0000","last_became_active":"2023-11-08T04:52:47.548294+0000","last_became_peered":"2023-11-08T04:52:47.548294+0000","last_unstale":"2023-11-08T05:30:44.490618+0000","last_undegraded":"2023-11-08T05:30:44.490618+0000","last_fullsized":"2023-11-08T05:30:44.490618+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":460,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490523+0000","last_change":"2023-11-08T05:28:13.545468+0000","last_active":"2023-11-08T05:30:44.490523+0000","last_peered":"2023-11-08T05:30:44.490523+0000","last_clean":"2023-11-08T05:28:01.456723+0000","last_became_active":"2023-11-08T05:28:13.545468+0000","last_became_peered":"2023-11-08T05:28:13.545468+0000","last_unstale":"2023-11-08T05:30:44.490523+0000","last_undegraded":"2023-11-08T05:30:44.490523+0000","last_fullsized":"2023-11-08T05:28:13.537207+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_clean_scrub_stamp":"2023-11-08T04:52:45.528796+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":465,"reported_epoch":458,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:44.491907+0000","last_change":"2023-11-08T05:29:44.395350+0000","last_active":"2023-11-08T05:30:44.491907+0000","last_peered":"2023-11-08T05:30:44.491907+0000","last_clean":"2023-11-08T05:30:44.491907+0000","last_became_active":"2023-11-08T05:29:44.395012+0000","last_became_peered":"2023-11-08T05:29:44.395012+0000","last_unstale":"2023-11-08T05:30:44.491907+0000","last_undegraded":"2023-11-08T05:30:44.491907+0000","last_fullsized":"2023-11-08T05:30:44.491907+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490629+0000","last_change":"2023-11-08T05:28:13.563357+0000","last_active":"2023-11-08T05:30:44.490629+0000","last_peered":"2023-11-08T05:30:44.490629+0000","last_clean":"2023-11-08T05:28:01.457691+0000","last_became_active":"2023-11-08T05:28:13.563357+0000","last_became_peered":"2023-11-08T05:28:13.563357+0000","last_unstale":"2023-11-08T05:30:44.490629+0000","last_undegraded":"2023-11-08T05:30:44.490629+0000","last_fullsized":"2023-11-08T05:28:13.541614+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490691+0000","last_change":"2023-11-08T05:28:13.550672+0000","last_active":"2023-11-08T05:30:44.490691+0000","last_peered":"2023-11-08T05:30:44.490691+0000","last_clean":"2023-11-08T05:28:01.457904+0000","last_became_active":"2023-11-08T05:28:13.550672+0000","last_became_peered":"2023-11-08T05:28:13.550672+0000","last_unstale":"2023-11-08T05:30:44.490691+0000","last_undegraded":"2023-11-08T05:30:44.490691+0000","last_fullsized":"2023-11-08T05:28:13.539595+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490805+0000","last_change":"2023-11-08T05:28:13.557968+0000","last_active":"2023-11-08T05:30:44.490805+0000","last_peered":"2023-11-08T05:30:44.490805+0000","last_clean":"2023-11-08T05:28:01.458433+0000","last_became_active":"2023-11-08T05:28:13.557968+0000","last_became_peered":"2023-11-08T05:28:13.557968+0000","last_unstale":"2023-11-08T05:30:44.490805+0000","last_undegraded":"2023-11-08T05:30:44.490805+0000","last_fullsized":"2023-11-08T05:28:13.542832+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":1,"acting":1,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":12,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738832,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":645040,"kb_used_data":5352,"kb_used_omap":275,"kb_used_meta":639404,"kb_avail":93726800,"statfs":{"total":96636764160,"available":95976243200,"internally_reserved":0,"allocated":5480448,"data_stored":2697647,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":281650,"internal_metadata":654750670},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:59 2023","interfaces":[{"interface":"back","average":{"1min":0.30499999999999999,"5min":0.439,"15min":0.36099999999999999},"min":{"1min":0.17399999999999999,"5min":0.14499999999999999,"15min":0.14499999999999999},"max":{"1min":0.60099999999999998,"5min":4.9379999999999997,"15min":4.9379999999999997},"last":0.379},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.49399999999999999,"15min":0.42399999999999999},"min":{"1min":0.221,"5min":0.19600000000000001,"15min":0.17000000000000001},"max":{"1min":0.63200000000000001,"5min":4.9859999999999998,"15min":4.9859999999999998},"last":0.40899999999999997}]},{"osd":1,"last update":"Wed Nov 8 05:31:06 2023","interfaces":[{"interface":"back","average":{"1min":0.27500000000000002,"5min":0.40300000000000002,"15min":0.46999999999999997},"min":{"1min":0.20899999999999999,"5min":0.16,"15min":0.153},"max":{"1min":0.41099999999999998,"5min":4.0830000000000002,"15min":23.242999999999999},"last":0.29499999999999998},{"interface":"front","average":{"1min":0.29899999999999999,"5min":0.40100000000000002,"15min":0.46000000000000002},"min":{"1min":0.23799999999999999,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":0.45300000000000001,"5min":3.5590000000000002,"15min":23.251000000000001},"last":0.27600000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738798,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1339228,"kb_used_data":398304,"kb_used_omap":291,"kb_used_meta":936476,"kb_avail":93032612,"statfs":{"total":96636764160,"available":95265394688,"internally_reserved":0,"allocated":407863296,"data_stored":405103900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":298462,"internal_metadata":958951970},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.436,"15min":0.36299999999999999},"min":{"1min":0.19700000000000001,"5min":0.14699999999999999,"15min":0.13700000000000001},"max":{"1min":3.222,"5min":6.5499999999999998,"15min":6.9989999999999997},"last":0.245},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.46300000000000002,"15min":0.42299999999999999},"min":{"1min":0.17799999999999999,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":5.0030000000000001,"5min":6.0629999999999997,"15min":9.6869999999999994},"last":0.28599999999999998}]},{"osd":2,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.51600000000000001,"15min":0.47099999999999997},"min":{"1min":0.21099999999999999,"5min":0.153,"15min":0.153},"max":{"1min":5.7439999999999998,"5min":5.8280000000000003,"15min":9.7639999999999993},"last":0.252},{"interface":"front","average":{"1min":0.497,"5min":0.56200000000000006,"15min":0.42299999999999999},"min":{"1min":0.20399999999999999,"5min":0.187,"15min":0.16800000000000001},"max":{"1min":3.9319999999999999,"5min":8.8040000000000003,"15min":8.8040000000000003},"last":0.29899999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738833,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":650100,"kb_used_data":5356,"kb_used_omap":214,"kb_used_meta":644521,"kb_avail":93721740,"statfs":{"total":96636764160,"available":95971061760,"internally_reserved":0,"allocated":5484544,"data_stored":2698616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":219335,"internal_metadata":659990329},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:56 2023","interfaces":[{"interface":"back","average":{"1min":0.30399999999999999,"5min":0.34699999999999998,"15min":0.36499999999999999},"min":{"1min":0.23999999999999999,"5min":0.14199999999999999,"15min":0.14199999999999999},"max":{"1min":0.42399999999999999,"5min":1.595,"15min":7.4249999999999998},"last":0.17100000000000001},{"interface":"front","average":{"1min":0.35499999999999998,"5min":0.40899999999999997,"15min":0.376},"min":{"1min":0.23999999999999999,"5min":0.19600000000000001,"15min":0.183},"max":{"1min":0.46999999999999997,"5min":2.6859999999999999,"15min":2.6859999999999999},"last":0.23000000000000001}]},{"osd":2,"last update":"Wed Nov 8 05:30:57 2023","interfaces":[{"interface":"back","average":{"1min":0.28599999999999998,"5min":0.34699999999999998,"15min":0.36899999999999999},"min":{"1min":0.19700000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":0.41399999999999998,"5min":2.2650000000000001,"15min":5.8209999999999997},"last":0.21099999999999999},{"interface":"front","average":{"1min":0.30599999999999999,"5min":0.313,"15min":0.35799999999999998},"min":{"1min":0.17799999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.435,"5min":1.137,"15min":4.2350000000000003},"last":0.223}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-08T05:31:11.325 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-11-08T05:31:11.607 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T05:31:11.608 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2023-11-08T05:31:11.619 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":1609,"stamp":"2023-11-08T05:31:10.064636+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":10,"acting":13,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":109,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2634368,"kb_used_data":409012,"kb_used_omap":780,"kb_used_meta":2220403,"kb_avail":280481152,"statfs":{"total":289910292480,"available":287212699648,"internally_reserved":0,"allocated":418828288,"data_stored":410500163,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":799447,"internal_metadata":2273692969},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-536782902,"num_objects":-164,"num_object_clones":0,"num_object_copies":-328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-115,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-164,"num_whiteouts":0,"num_read":-122056,"num_read_kb":-39011942,"num_write":-413642,"num_write_kb":-151705339,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-8,"num_bytes_recovered":-16922624,"num_keys_recovered":-2,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.424817"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490644+0000","last_change":"2023-11-08T05:28:13.557071+0000","last_active":"2023-11-08T05:30:44.490644+0000","last_peered":"2023-11-08T05:30:44.490644+0000","last_clean":"2023-11-08T05:28:01.458439+0000","last_became_active":"2023-11-08T05:28:13.557071+0000","last_became_peered":"2023-11-08T05:28:13.557071+0000","last_unstale":"2023-11-08T05:30:44.490644+0000","last_undegraded":"2023-11-08T05:30:44.490644+0000","last_fullsized":"2023-11-08T05:28:13.547025+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":459,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:45.794760+0000","last_change":"2023-11-08T05:30:45.794760+0000","last_active":"2023-11-08T05:30:45.794760+0000","last_peered":"2023-11-08T05:30:45.794760+0000","last_clean":"2023-11-08T05:30:45.794760+0000","last_became_active":"2023-11-08T05:30:45.794398+0000","last_became_peered":"2023-11-08T05:30:45.794398+0000","last_unstale":"2023-11-08T05:30:45.794760+0000","last_undegraded":"2023-11-08T05:30:45.794760+0000","last_fullsized":"2023-11-08T05:30:45.794760+0000","mapping_epoch":458,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":459,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":465,"reported_epoch":459,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:45.795085+0000","last_change":"2023-11-08T05:30:45.795085+0000","last_active":"2023-11-08T05:30:45.795085+0000","last_peered":"2023-11-08T05:30:45.795085+0000","last_clean":"2023-11-08T05:30:45.795085+0000","last_became_active":"2023-11-08T05:30:45.794743+0000","last_became_peered":"2023-11-08T05:30:45.794743+0000","last_unstale":"2023-11-08T05:30:45.795085+0000","last_undegraded":"2023-11-08T05:30:45.795085+0000","last_fullsized":"2023-11-08T05:30:45.795085+0000","mapping_epoch":458,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":459,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":454,"reported_epoch":458,"state":"active+clean","last_fresh":"2023-11-08T05:30:44.490618+0000","last_change":"2023-11-08T04:52:47.548597+0000","last_active":"2023-11-08T05:30:44.490618+0000","last_peered":"2023-11-08T05:30:44.490618+0000","last_clean":"2023-11-08T05:30:44.490618+0000","last_became_active":"2023-11-08T04:52:47.548294+0000","last_became_peered":"2023-11-08T04:52:47.548294+0000","last_unstale":"2023-11-08T05:30:44.490618+0000","last_undegraded":"2023-11-08T05:30:44.490618+0000","last_fullsized":"2023-11-08T05:30:44.490618+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":460,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490523+0000","last_change":"2023-11-08T05:28:13.545468+0000","last_active":"2023-11-08T05:30:44.490523+0000","last_peered":"2023-11-08T05:30:44.490523+0000","last_clean":"2023-11-08T05:28:01.456723+0000","last_became_active":"2023-11-08T05:28:13.545468+0000","last_became_peered":"2023-11-08T05:28:13.545468+0000","last_unstale":"2023-11-08T05:30:44.490523+0000","last_undegraded":"2023-11-08T05:30:44.490523+0000","last_fullsized":"2023-11-08T05:28:13.537207+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_clean_scrub_stamp":"2023-11-08T04:52:45.528796+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":465,"reported_epoch":458,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:44.491907+0000","last_change":"2023-11-08T05:29:44.395350+0000","last_active":"2023-11-08T05:30:44.491907+0000","last_peered":"2023-11-08T05:30:44.491907+0000","last_clean":"2023-11-08T05:30:44.491907+0000","last_became_active":"2023-11-08T05:29:44.395012+0000","last_became_peered":"2023-11-08T05:29:44.395012+0000","last_unstale":"2023-11-08T05:30:44.491907+0000","last_undegraded":"2023-11-08T05:30:44.491907+0000","last_fullsized":"2023-11-08T05:30:44.491907+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490629+0000","last_change":"2023-11-08T05:28:13.563357+0000","last_active":"2023-11-08T05:30:44.490629+0000","last_peered":"2023-11-08T05:30:44.490629+0000","last_clean":"2023-11-08T05:28:01.457691+0000","last_became_active":"2023-11-08T05:28:13.563357+0000","last_became_peered":"2023-11-08T05:28:13.563357+0000","last_unstale":"2023-11-08T05:30:44.490629+0000","last_undegraded":"2023-11-08T05:30:44.490629+0000","last_fullsized":"2023-11-08T05:28:13.541614+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490691+0000","last_change":"2023-11-08T05:28:13.550672+0000","last_active":"2023-11-08T05:30:44.490691+0000","last_peered":"2023-11-08T05:30:44.490691+0000","last_clean":"2023-11-08T05:28:01.457904+0000","last_became_active":"2023-11-08T05:28:13.550672+0000","last_became_peered":"2023-11-08T05:28:13.550672+0000","last_unstale":"2023-11-08T05:30:44.490691+0000","last_undegraded":"2023-11-08T05:30:44.490691+0000","last_fullsized":"2023-11-08T05:28:13.539595+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490805+0000","last_change":"2023-11-08T05:28:13.557968+0000","last_active":"2023-11-08T05:30:44.490805+0000","last_peered":"2023-11-08T05:30:44.490805+0000","last_clean":"2023-11-08T05:28:01.458433+0000","last_became_active":"2023-11-08T05:28:13.557968+0000","last_became_peered":"2023-11-08T05:28:13.557968+0000","last_unstale":"2023-11-08T05:30:44.490805+0000","last_undegraded":"2023-11-08T05:30:44.490805+0000","last_fullsized":"2023-11-08T05:28:13.542832+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":1,"acting":1,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":12,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738832,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":645040,"kb_used_data":5352,"kb_used_omap":275,"kb_used_meta":639404,"kb_avail":93726800,"statfs":{"total":96636764160,"available":95976243200,"internally_reserved":0,"allocated":5480448,"data_stored":2697647,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":281650,"internal_metadata":654750670},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:59 2023","interfaces":[{"interface":"back","average":{"1min":0.30499999999999999,"5min":0.439,"15min":0.36099999999999999},"min":{"1min":0.17399999999999999,"5min":0.14499999999999999,"15min":0.14499999999999999},"max":{"1min":0.60099999999999998,"5min":4.9379999999999997,"15min":4.9379999999999997},"last":0.379},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.49399999999999999,"15min":0.42399999999999999},"min":{"1min":0.221,"5min":0.19600000000000001,"15min":0.17000000000000001},"max":{"1min":0.63200000000000001,"5min":4.9859999999999998,"15min":4.9859999999999998},"last":0.40899999999999997}]},{"osd":1,"last update":"Wed Nov 8 05:31:06 2023","interfaces":[{"interface":"back","average":{"1min":0.27500000000000002,"5min":0.40300000000000002,"15min":0.46999999999999997},"min":{"1min":0.20899999999999999,"5min":0.16,"15min":0.153},"max":{"1min":0.41099999999999998,"5min":4.0830000000000002,"15min":23.242999999999999},"last":0.29499999999999998},{"interface":"front","average":{"1min":0.29899999999999999,"5min":0.40100000000000002,"15min":0.46000000000000002},"min":{"1min":0.23799999999999999,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":0.45300000000000001,"5min":3.5590000000000002,"15min":23.251000000000001},"last":0.27600000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738798,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1339228,"kb_used_data":398304,"kb_used_omap":291,"kb_used_meta":936476,"kb_avail":93032612,"statfs":{"total":96636764160,"available":95265394688,"internally_reserved":0,"allocated":407863296,"data_stored":405103900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":298462,"internal_metadata":958951970},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.436,"15min":0.36299999999999999},"min":{"1min":0.19700000000000001,"5min":0.14699999999999999,"15min":0.13700000000000001},"max":{"1min":3.222,"5min":6.5499999999999998,"15min":6.9989999999999997},"last":0.245},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.46300000000000002,"15min":0.42299999999999999},"min":{"1min":0.17799999999999999,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":5.0030000000000001,"5min":6.0629999999999997,"15min":9.6869999999999994},"last":0.28599999999999998}]},{"osd":2,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.51600000000000001,"15min":0.47099999999999997},"min":{"1min":0.21099999999999999,"5min":0.153,"15min":0.153},"max":{"1min":5.7439999999999998,"5min":5.8280000000000003,"15min":9.7639999999999993},"last":0.252},{"interface":"front","average":{"1min":0.497,"5min":0.56200000000000006,"15min":0.42299999999999999},"min":{"1min":0.20399999999999999,"5min":0.187,"15min":0.16800000000000001},"max":{"1min":3.9319999999999999,"5min":8.8040000000000003,"15min":8.8040000000000003},"last":0.29899999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738833,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":650100,"kb_used_data":5356,"kb_used_omap":214,"kb_used_meta":644521,"kb_avail":93721740,"statfs":{"total":96636764160,"available":95971061760,"internally_reserved":0,"allocated":5484544,"data_stored":2698616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":219335,"internal_metadata":659990329},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:56 2023","interfaces":[{"interface":"back","average":{"1min":0.30399999999999999,"5min":0.34699999999999998,"15min":0.36499999999999999},"min":{"1min":0.23999999999999999,"5min":0.14199999999999999,"15min":0.14199999999999999},"max":{"1min":0.42399999999999999,"5min":1.595,"15min":7.4249999999999998},"last":0.17100000000000001},{"interface":"front","average":{"1min":0.35499999999999998,"5min":0.40899999999999997,"15min":0.376},"min":{"1min":0.23999999999999999,"5min":0.19600000000000001,"15min":0.183},"max":{"1min":0.46999999999999997,"5min":2.6859999999999999,"15min":2.6859999999999999},"last":0.23000000000000001}]},{"osd":2,"last update":"Wed Nov 8 05:30:57 2023","interfaces":[{"interface":"back","average":{"1min":0.28599999999999998,"5min":0.34699999999999998,"15min":0.36899999999999999},"min":{"1min":0.19700000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":0.41399999999999998,"5min":2.2650000000000001,"15min":5.8209999999999997},"last":0.21099999999999999},{"interface":"front","average":{"1min":0.30599999999999999,"5min":0.313,"15min":0.35799999999999998},"min":{"1min":0.17799999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.435,"5min":1.137,"15min":4.2350000000000003},"last":0.223}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-08T05:31:11.620 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-11-08T05:31:11.621 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 459, 'reported_epoch': 458, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:30:44.490644+0000', 'last_change': '2023-11-08T05:28:13.557071+0000', 'last_active': '2023-11-08T05:30:44.490644+0000', 'last_peered': '2023-11-08T05:30:44.490644+0000', 'last_clean': '2023-11-08T05:28:01.458439+0000', 'last_became_active': '2023-11-08T05:28:13.557071+0000', 'last_became_peered': '2023-11-08T05:28:13.557071+0000', 'last_unstale': '2023-11-08T05:30:44.490644+0000', 'last_undegraded': '2023-11-08T05:30:44.490644+0000', 'last_fullsized': '2023-11-08T05:28:13.547025+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:31:11.621 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-11-08T05:31:11.621 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 11, 'reported_epoch': 459, 'state': 'active+clean+remapped', 'last_fresh': '2023-11-08T05:30:45.794760+0000', 'last_change': '2023-11-08T05:30:45.794760+0000', 'last_active': '2023-11-08T05:30:45.794760+0000', 'last_peered': '2023-11-08T05:30:45.794760+0000', 'last_clean': '2023-11-08T05:30:45.794760+0000', 'last_became_active': '2023-11-08T05:30:45.794398+0000', 'last_became_peered': '2023-11-08T05:30:45.794398+0000', 'last_unstale': '2023-11-08T05:30:45.794760+0000', 'last_undegraded': '2023-11-08T05:30:45.794760+0000', 'last_fullsized': '2023-11-08T05:30:45.794760+0000', 'mapping_epoch': 458, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 459, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2], 'acting': [2, 1], 'avail_no_missing': ['2', '1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2023-11-08T05:31:11.622 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.1 is not active+clean 2023-11-08T05:31:11.622 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.1', 'version': "0'0", 'reported_seq': 465, 'reported_epoch': 459, 'state': 'active+clean+remapped', 'last_fresh': '2023-11-08T05:30:45.795085+0000', 'last_change': '2023-11-08T05:30:45.795085+0000', 'last_active': '2023-11-08T05:30:45.795085+0000', 'last_peered': '2023-11-08T05:30:45.795085+0000', 'last_clean': '2023-11-08T05:30:45.795085+0000', 'last_became_active': '2023-11-08T05:30:45.794743+0000', 'last_became_peered': '2023-11-08T05:30:45.794743+0000', 'last_unstale': '2023-11-08T05:30:45.795085+0000', 'last_undegraded': '2023-11-08T05:30:45.795085+0000', 'last_fullsized': '2023-11-08T05:30:45.795085+0000', 'mapping_epoch': 458, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 459, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2], 'acting': [2, 1], 'avail_no_missing': ['2', '1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2023-11-08T05:31:11.622 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-11-08T05:31:11.622 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "0'0", 'reported_seq': 460, 'reported_epoch': 458, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:30:44.490523+0000', 'last_change': '2023-11-08T05:28:13.545468+0000', 'last_active': '2023-11-08T05:30:44.490523+0000', 'last_peered': '2023-11-08T05:30:44.490523+0000', 'last_clean': '2023-11-08T05:28:01.456723+0000', 'last_became_active': '2023-11-08T05:28:13.545468+0000', 'last_became_peered': '2023-11-08T05:28:13.545468+0000', 'last_unstale': '2023-11-08T05:30:44.490523+0000', 'last_undegraded': '2023-11-08T05:30:44.490523+0000', 'last_fullsized': '2023-11-08T05:28:13.537207+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 9, 'last_epoch_clean': 10, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:45.528796+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:45.528796+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:45.528796+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:31:11.623 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.0 is not active+clean 2023-11-08T05:31:11.623 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.0', 'version': "0'0", 'reported_seq': 465, 'reported_epoch': 458, 'state': 'active+clean+remapped', 'last_fresh': '2023-11-08T05:30:44.491907+0000', 'last_change': '2023-11-08T05:29:44.395350+0000', 'last_active': '2023-11-08T05:30:44.491907+0000', 'last_peered': '2023-11-08T05:30:44.491907+0000', 'last_clean': '2023-11-08T05:30:44.491907+0000', 'last_became_active': '2023-11-08T05:29:44.395012+0000', 'last_became_peered': '2023-11-08T05:29:44.395012+0000', 'last_unstale': '2023-11-08T05:30:44.491907+0000', 'last_undegraded': '2023-11-08T05:30:44.491907+0000', 'last_fullsized': '2023-11-08T05:30:44.491907+0000', 'mapping_epoch': 455, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 456, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2], 'acting': [2, 1], 'avail_no_missing': ['2', '1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2023-11-08T05:31:11.623 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-11-08T05:31:11.624 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 459, 'reported_epoch': 458, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:30:44.490629+0000', 'last_change': '2023-11-08T05:28:13.563357+0000', 'last_active': '2023-11-08T05:30:44.490629+0000', 'last_peered': '2023-11-08T05:30:44.490629+0000', 'last_clean': '2023-11-08T05:28:01.457691+0000', 'last_became_active': '2023-11-08T05:28:13.563357+0000', 'last_became_peered': '2023-11-08T05:28:13.563357+0000', 'last_unstale': '2023-11-08T05:30:44.490629+0000', 'last_undegraded': '2023-11-08T05:30:44.490629+0000', 'last_fullsized': '2023-11-08T05:28:13.541614+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:31:11.624 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-11-08T05:31:11.624 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 459, 'reported_epoch': 458, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:30:44.490691+0000', 'last_change': '2023-11-08T05:28:13.550672+0000', 'last_active': '2023-11-08T05:30:44.490691+0000', 'last_peered': '2023-11-08T05:30:44.490691+0000', 'last_clean': '2023-11-08T05:28:01.457904+0000', 'last_became_active': '2023-11-08T05:28:13.550672+0000', 'last_became_peered': '2023-11-08T05:28:13.550672+0000', 'last_unstale': '2023-11-08T05:30:44.490691+0000', 'last_undegraded': '2023-11-08T05:30:44.490691+0000', 'last_fullsized': '2023-11-08T05:28:13.539595+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:31:11.625 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-11-08T05:31:11.625 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 459, 'reported_epoch': 458, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:30:44.490805+0000', 'last_change': '2023-11-08T05:28:13.557968+0000', 'last_active': '2023-11-08T05:30:44.490805+0000', 'last_peered': '2023-11-08T05:30:44.490805+0000', 'last_clean': '2023-11-08T05:28:01.458433+0000', 'last_became_active': '2023-11-08T05:28:13.557968+0000', 'last_became_peered': '2023-11-08T05:28:13.557968+0000', 'last_unstale': '2023-11-08T05:30:44.490805+0000', 'last_undegraded': '2023-11-08T05:30:44.490805+0000', 'last_fullsized': '2023-11-08T05:28:13.542832+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:31:11.626 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-11-08T05:31:11.971 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T05:31:11.972 INFO:teuthology.orchestra.run.smithi087.stdout:{"fsid":"ab4cbc06-449d-497d-b513-220d9db2b590","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 5 pgs undersized","count":5},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2314,"monmap":{"epoch":1,"min_mon_release_name":"pacific","num_mons":1},"osdmap":{"epoch":459,"num_osds":3,"num_up_osds":2,"osd_up_since":1699421292,"num_in_osds":3,"osd_in_since":1699419158,"num_remapped_pgs":3},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean+remapped","count":3},{"state_name":"active+clean","count":1}],"num_pgs":9,"num_pools":2,"num_objects":0,"data_bytes":0,"bytes_used":2697592832,"bytes_avail":287212699648,"bytes_total":289910292480},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":41,"modified":"2023-11-08T05:29:32.044544+0000","services":{}},"progress_events":{}} 2023-11-08T05:31:11.985 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-11-08T05:31:11.986 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-11-08T05:31:12.269 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T05:31:12.271 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2023-11-08T05:31:12.282 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":1610,"stamp":"2023-11-08T05:31:12.065123+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":10,"acting":13,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":109,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2634368,"kb_used_data":409012,"kb_used_omap":780,"kb_used_meta":2220403,"kb_avail":280481152,"statfs":{"total":289910292480,"available":287212699648,"internally_reserved":0,"allocated":418828288,"data_stored":410500163,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":799447,"internal_metadata":2273692969},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-536782902,"num_objects":-164,"num_object_clones":0,"num_object_copies":-328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-115,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-164,"num_whiteouts":0,"num_read":-122056,"num_read_kb":-39011942,"num_write":-413642,"num_write_kb":-151705339,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-8,"num_bytes_recovered":-16922624,"num_keys_recovered":-2,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.424817"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490644+0000","last_change":"2023-11-08T05:28:13.557071+0000","last_active":"2023-11-08T05:30:44.490644+0000","last_peered":"2023-11-08T05:30:44.490644+0000","last_clean":"2023-11-08T05:28:01.458439+0000","last_became_active":"2023-11-08T05:28:13.557071+0000","last_became_peered":"2023-11-08T05:28:13.557071+0000","last_unstale":"2023-11-08T05:30:44.490644+0000","last_undegraded":"2023-11-08T05:30:44.490644+0000","last_fullsized":"2023-11-08T05:28:13.547025+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":459,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:45.794760+0000","last_change":"2023-11-08T05:30:45.794760+0000","last_active":"2023-11-08T05:30:45.794760+0000","last_peered":"2023-11-08T05:30:45.794760+0000","last_clean":"2023-11-08T05:30:45.794760+0000","last_became_active":"2023-11-08T05:30:45.794398+0000","last_became_peered":"2023-11-08T05:30:45.794398+0000","last_unstale":"2023-11-08T05:30:45.794760+0000","last_undegraded":"2023-11-08T05:30:45.794760+0000","last_fullsized":"2023-11-08T05:30:45.794760+0000","mapping_epoch":458,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":459,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":465,"reported_epoch":459,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:45.795085+0000","last_change":"2023-11-08T05:30:45.795085+0000","last_active":"2023-11-08T05:30:45.795085+0000","last_peered":"2023-11-08T05:30:45.795085+0000","last_clean":"2023-11-08T05:30:45.795085+0000","last_became_active":"2023-11-08T05:30:45.794743+0000","last_became_peered":"2023-11-08T05:30:45.794743+0000","last_unstale":"2023-11-08T05:30:45.795085+0000","last_undegraded":"2023-11-08T05:30:45.795085+0000","last_fullsized":"2023-11-08T05:30:45.795085+0000","mapping_epoch":458,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":459,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":454,"reported_epoch":458,"state":"active+clean","last_fresh":"2023-11-08T05:30:44.490618+0000","last_change":"2023-11-08T04:52:47.548597+0000","last_active":"2023-11-08T05:30:44.490618+0000","last_peered":"2023-11-08T05:30:44.490618+0000","last_clean":"2023-11-08T05:30:44.490618+0000","last_became_active":"2023-11-08T04:52:47.548294+0000","last_became_peered":"2023-11-08T04:52:47.548294+0000","last_unstale":"2023-11-08T05:30:44.490618+0000","last_undegraded":"2023-11-08T05:30:44.490618+0000","last_fullsized":"2023-11-08T05:30:44.490618+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":460,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490523+0000","last_change":"2023-11-08T05:28:13.545468+0000","last_active":"2023-11-08T05:30:44.490523+0000","last_peered":"2023-11-08T05:30:44.490523+0000","last_clean":"2023-11-08T05:28:01.456723+0000","last_became_active":"2023-11-08T05:28:13.545468+0000","last_became_peered":"2023-11-08T05:28:13.545468+0000","last_unstale":"2023-11-08T05:30:44.490523+0000","last_undegraded":"2023-11-08T05:30:44.490523+0000","last_fullsized":"2023-11-08T05:28:13.537207+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_clean_scrub_stamp":"2023-11-08T04:52:45.528796+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":465,"reported_epoch":458,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:44.491907+0000","last_change":"2023-11-08T05:29:44.395350+0000","last_active":"2023-11-08T05:30:44.491907+0000","last_peered":"2023-11-08T05:30:44.491907+0000","last_clean":"2023-11-08T05:30:44.491907+0000","last_became_active":"2023-11-08T05:29:44.395012+0000","last_became_peered":"2023-11-08T05:29:44.395012+0000","last_unstale":"2023-11-08T05:30:44.491907+0000","last_undegraded":"2023-11-08T05:30:44.491907+0000","last_fullsized":"2023-11-08T05:30:44.491907+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490629+0000","last_change":"2023-11-08T05:28:13.563357+0000","last_active":"2023-11-08T05:30:44.490629+0000","last_peered":"2023-11-08T05:30:44.490629+0000","last_clean":"2023-11-08T05:28:01.457691+0000","last_became_active":"2023-11-08T05:28:13.563357+0000","last_became_peered":"2023-11-08T05:28:13.563357+0000","last_unstale":"2023-11-08T05:30:44.490629+0000","last_undegraded":"2023-11-08T05:30:44.490629+0000","last_fullsized":"2023-11-08T05:28:13.541614+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490691+0000","last_change":"2023-11-08T05:28:13.550672+0000","last_active":"2023-11-08T05:30:44.490691+0000","last_peered":"2023-11-08T05:30:44.490691+0000","last_clean":"2023-11-08T05:28:01.457904+0000","last_became_active":"2023-11-08T05:28:13.550672+0000","last_became_peered":"2023-11-08T05:28:13.550672+0000","last_unstale":"2023-11-08T05:30:44.490691+0000","last_undegraded":"2023-11-08T05:30:44.490691+0000","last_fullsized":"2023-11-08T05:28:13.539595+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490805+0000","last_change":"2023-11-08T05:28:13.557968+0000","last_active":"2023-11-08T05:30:44.490805+0000","last_peered":"2023-11-08T05:30:44.490805+0000","last_clean":"2023-11-08T05:28:01.458433+0000","last_became_active":"2023-11-08T05:28:13.557968+0000","last_became_peered":"2023-11-08T05:28:13.557968+0000","last_unstale":"2023-11-08T05:30:44.490805+0000","last_undegraded":"2023-11-08T05:30:44.490805+0000","last_fullsized":"2023-11-08T05:28:13.542832+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":1,"acting":1,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":12,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738832,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":645040,"kb_used_data":5352,"kb_used_omap":275,"kb_used_meta":639404,"kb_avail":93726800,"statfs":{"total":96636764160,"available":95976243200,"internally_reserved":0,"allocated":5480448,"data_stored":2697647,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":281650,"internal_metadata":654750670},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:59 2023","interfaces":[{"interface":"back","average":{"1min":0.30499999999999999,"5min":0.439,"15min":0.36099999999999999},"min":{"1min":0.17399999999999999,"5min":0.14499999999999999,"15min":0.14499999999999999},"max":{"1min":0.60099999999999998,"5min":4.9379999999999997,"15min":4.9379999999999997},"last":0.379},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.49399999999999999,"15min":0.42399999999999999},"min":{"1min":0.221,"5min":0.19600000000000001,"15min":0.17000000000000001},"max":{"1min":0.63200000000000001,"5min":4.9859999999999998,"15min":4.9859999999999998},"last":0.40899999999999997}]},{"osd":1,"last update":"Wed Nov 8 05:31:06 2023","interfaces":[{"interface":"back","average":{"1min":0.27500000000000002,"5min":0.40300000000000002,"15min":0.46999999999999997},"min":{"1min":0.20899999999999999,"5min":0.16,"15min":0.153},"max":{"1min":0.41099999999999998,"5min":4.0830000000000002,"15min":23.242999999999999},"last":0.29499999999999998},{"interface":"front","average":{"1min":0.29899999999999999,"5min":0.40100000000000002,"15min":0.46000000000000002},"min":{"1min":0.23799999999999999,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":0.45300000000000001,"5min":3.5590000000000002,"15min":23.251000000000001},"last":0.27600000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738798,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1339228,"kb_used_data":398304,"kb_used_omap":291,"kb_used_meta":936476,"kb_avail":93032612,"statfs":{"total":96636764160,"available":95265394688,"internally_reserved":0,"allocated":407863296,"data_stored":405103900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":298462,"internal_metadata":958951970},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.436,"15min":0.36299999999999999},"min":{"1min":0.19700000000000001,"5min":0.14699999999999999,"15min":0.13700000000000001},"max":{"1min":3.222,"5min":6.5499999999999998,"15min":6.9989999999999997},"last":0.245},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.46300000000000002,"15min":0.42299999999999999},"min":{"1min":0.17799999999999999,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":5.0030000000000001,"5min":6.0629999999999997,"15min":9.6869999999999994},"last":0.28599999999999998}]},{"osd":2,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.51600000000000001,"15min":0.47099999999999997},"min":{"1min":0.21099999999999999,"5min":0.153,"15min":0.153},"max":{"1min":5.7439999999999998,"5min":5.8280000000000003,"15min":9.7639999999999993},"last":0.252},{"interface":"front","average":{"1min":0.497,"5min":0.56200000000000006,"15min":0.42299999999999999},"min":{"1min":0.20399999999999999,"5min":0.187,"15min":0.16800000000000001},"max":{"1min":3.9319999999999999,"5min":8.8040000000000003,"15min":8.8040000000000003},"last":0.29899999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738834,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":650100,"kb_used_data":5356,"kb_used_omap":214,"kb_used_meta":644521,"kb_avail":93721740,"statfs":{"total":96636764160,"available":95971061760,"internally_reserved":0,"allocated":5484544,"data_stored":2698616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":219335,"internal_metadata":659990329},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:56 2023","interfaces":[{"interface":"back","average":{"1min":0.30399999999999999,"5min":0.34699999999999998,"15min":0.36499999999999999},"min":{"1min":0.23999999999999999,"5min":0.14199999999999999,"15min":0.14199999999999999},"max":{"1min":0.42399999999999999,"5min":1.595,"15min":7.4249999999999998},"last":0.17100000000000001},{"interface":"front","average":{"1min":0.35499999999999998,"5min":0.40899999999999997,"15min":0.376},"min":{"1min":0.23999999999999999,"5min":0.19600000000000001,"15min":0.183},"max":{"1min":0.46999999999999997,"5min":2.6859999999999999,"15min":2.6859999999999999},"last":0.23000000000000001}]},{"osd":2,"last update":"Wed Nov 8 05:30:57 2023","interfaces":[{"interface":"back","average":{"1min":0.28599999999999998,"5min":0.34699999999999998,"15min":0.36899999999999999},"min":{"1min":0.19700000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":0.41399999999999998,"5min":2.2650000000000001,"15min":5.8209999999999997},"last":0.27700000000000002},{"interface":"front","average":{"1min":0.30599999999999999,"5min":0.313,"15min":0.35799999999999998},"min":{"1min":0.17799999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.435,"5min":1.137,"15min":4.2350000000000003},"last":0.252}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-08T05:31:15.284 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-11-08T05:31:15.579 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T05:31:15.581 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2023-11-08T05:31:15.591 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":1611,"stamp":"2023-11-08T05:31:14.065513+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":10,"acting":13,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":109,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2634368,"kb_used_data":409012,"kb_used_omap":780,"kb_used_meta":2220403,"kb_avail":280481152,"statfs":{"total":289910292480,"available":287212699648,"internally_reserved":0,"allocated":418828288,"data_stored":410500163,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":799447,"internal_metadata":2273692969},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-536782902,"num_objects":-164,"num_object_clones":0,"num_object_copies":-328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-115,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-164,"num_whiteouts":0,"num_read":-122056,"num_read_kb":-39011942,"num_write":-413642,"num_write_kb":-151705339,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-8,"num_bytes_recovered":-16922624,"num_keys_recovered":-2,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.424817"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490644+0000","last_change":"2023-11-08T05:28:13.557071+0000","last_active":"2023-11-08T05:30:44.490644+0000","last_peered":"2023-11-08T05:30:44.490644+0000","last_clean":"2023-11-08T05:28:01.458439+0000","last_became_active":"2023-11-08T05:28:13.557071+0000","last_became_peered":"2023-11-08T05:28:13.557071+0000","last_unstale":"2023-11-08T05:30:44.490644+0000","last_undegraded":"2023-11-08T05:30:44.490644+0000","last_fullsized":"2023-11-08T05:28:13.547025+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":459,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:45.794760+0000","last_change":"2023-11-08T05:30:45.794760+0000","last_active":"2023-11-08T05:30:45.794760+0000","last_peered":"2023-11-08T05:30:45.794760+0000","last_clean":"2023-11-08T05:30:45.794760+0000","last_became_active":"2023-11-08T05:30:45.794398+0000","last_became_peered":"2023-11-08T05:30:45.794398+0000","last_unstale":"2023-11-08T05:30:45.794760+0000","last_undegraded":"2023-11-08T05:30:45.794760+0000","last_fullsized":"2023-11-08T05:30:45.794760+0000","mapping_epoch":458,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":459,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":465,"reported_epoch":459,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:45.795085+0000","last_change":"2023-11-08T05:30:45.795085+0000","last_active":"2023-11-08T05:30:45.795085+0000","last_peered":"2023-11-08T05:30:45.795085+0000","last_clean":"2023-11-08T05:30:45.795085+0000","last_became_active":"2023-11-08T05:30:45.794743+0000","last_became_peered":"2023-11-08T05:30:45.794743+0000","last_unstale":"2023-11-08T05:30:45.795085+0000","last_undegraded":"2023-11-08T05:30:45.795085+0000","last_fullsized":"2023-11-08T05:30:45.795085+0000","mapping_epoch":458,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":459,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":454,"reported_epoch":458,"state":"active+clean","last_fresh":"2023-11-08T05:30:44.490618+0000","last_change":"2023-11-08T04:52:47.548597+0000","last_active":"2023-11-08T05:30:44.490618+0000","last_peered":"2023-11-08T05:30:44.490618+0000","last_clean":"2023-11-08T05:30:44.490618+0000","last_became_active":"2023-11-08T04:52:47.548294+0000","last_became_peered":"2023-11-08T04:52:47.548294+0000","last_unstale":"2023-11-08T05:30:44.490618+0000","last_undegraded":"2023-11-08T05:30:44.490618+0000","last_fullsized":"2023-11-08T05:30:44.490618+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":460,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490523+0000","last_change":"2023-11-08T05:28:13.545468+0000","last_active":"2023-11-08T05:30:44.490523+0000","last_peered":"2023-11-08T05:30:44.490523+0000","last_clean":"2023-11-08T05:28:01.456723+0000","last_became_active":"2023-11-08T05:28:13.545468+0000","last_became_peered":"2023-11-08T05:28:13.545468+0000","last_unstale":"2023-11-08T05:30:44.490523+0000","last_undegraded":"2023-11-08T05:30:44.490523+0000","last_fullsized":"2023-11-08T05:28:13.537207+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_clean_scrub_stamp":"2023-11-08T04:52:45.528796+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":465,"reported_epoch":458,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:44.491907+0000","last_change":"2023-11-08T05:29:44.395350+0000","last_active":"2023-11-08T05:30:44.491907+0000","last_peered":"2023-11-08T05:30:44.491907+0000","last_clean":"2023-11-08T05:30:44.491907+0000","last_became_active":"2023-11-08T05:29:44.395012+0000","last_became_peered":"2023-11-08T05:29:44.395012+0000","last_unstale":"2023-11-08T05:30:44.491907+0000","last_undegraded":"2023-11-08T05:30:44.491907+0000","last_fullsized":"2023-11-08T05:30:44.491907+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490629+0000","last_change":"2023-11-08T05:28:13.563357+0000","last_active":"2023-11-08T05:30:44.490629+0000","last_peered":"2023-11-08T05:30:44.490629+0000","last_clean":"2023-11-08T05:28:01.457691+0000","last_became_active":"2023-11-08T05:28:13.563357+0000","last_became_peered":"2023-11-08T05:28:13.563357+0000","last_unstale":"2023-11-08T05:30:44.490629+0000","last_undegraded":"2023-11-08T05:30:44.490629+0000","last_fullsized":"2023-11-08T05:28:13.541614+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490691+0000","last_change":"2023-11-08T05:28:13.550672+0000","last_active":"2023-11-08T05:30:44.490691+0000","last_peered":"2023-11-08T05:30:44.490691+0000","last_clean":"2023-11-08T05:28:01.457904+0000","last_became_active":"2023-11-08T05:28:13.550672+0000","last_became_peered":"2023-11-08T05:28:13.550672+0000","last_unstale":"2023-11-08T05:30:44.490691+0000","last_undegraded":"2023-11-08T05:30:44.490691+0000","last_fullsized":"2023-11-08T05:28:13.539595+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490805+0000","last_change":"2023-11-08T05:28:13.557968+0000","last_active":"2023-11-08T05:30:44.490805+0000","last_peered":"2023-11-08T05:30:44.490805+0000","last_clean":"2023-11-08T05:28:01.458433+0000","last_became_active":"2023-11-08T05:28:13.557968+0000","last_became_peered":"2023-11-08T05:28:13.557968+0000","last_unstale":"2023-11-08T05:30:44.490805+0000","last_undegraded":"2023-11-08T05:30:44.490805+0000","last_fullsized":"2023-11-08T05:28:13.542832+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":1,"acting":1,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":12,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738832,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":645040,"kb_used_data":5352,"kb_used_omap":275,"kb_used_meta":639404,"kb_avail":93726800,"statfs":{"total":96636764160,"available":95976243200,"internally_reserved":0,"allocated":5480448,"data_stored":2697647,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":281650,"internal_metadata":654750670},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:59 2023","interfaces":[{"interface":"back","average":{"1min":0.30499999999999999,"5min":0.439,"15min":0.36099999999999999},"min":{"1min":0.17399999999999999,"5min":0.14499999999999999,"15min":0.14499999999999999},"max":{"1min":0.60099999999999998,"5min":4.9379999999999997,"15min":4.9379999999999997},"last":0.379},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.49399999999999999,"15min":0.42399999999999999},"min":{"1min":0.221,"5min":0.19600000000000001,"15min":0.17000000000000001},"max":{"1min":0.63200000000000001,"5min":4.9859999999999998,"15min":4.9859999999999998},"last":0.40899999999999997}]},{"osd":1,"last update":"Wed Nov 8 05:31:06 2023","interfaces":[{"interface":"back","average":{"1min":0.27500000000000002,"5min":0.40300000000000002,"15min":0.46999999999999997},"min":{"1min":0.20899999999999999,"5min":0.16,"15min":0.153},"max":{"1min":0.41099999999999998,"5min":4.0830000000000002,"15min":23.242999999999999},"last":0.29499999999999998},{"interface":"front","average":{"1min":0.29899999999999999,"5min":0.40100000000000002,"15min":0.46000000000000002},"min":{"1min":0.23799999999999999,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":0.45300000000000001,"5min":3.5590000000000002,"15min":23.251000000000001},"last":0.27600000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738798,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1339228,"kb_used_data":398304,"kb_used_omap":291,"kb_used_meta":936476,"kb_avail":93032612,"statfs":{"total":96636764160,"available":95265394688,"internally_reserved":0,"allocated":407863296,"data_stored":405103900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":298462,"internal_metadata":958951970},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.436,"15min":0.36299999999999999},"min":{"1min":0.19700000000000001,"5min":0.14699999999999999,"15min":0.13700000000000001},"max":{"1min":3.222,"5min":6.5499999999999998,"15min":6.9989999999999997},"last":0.245},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.46300000000000002,"15min":0.42299999999999999},"min":{"1min":0.17799999999999999,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":5.0030000000000001,"5min":6.0629999999999997,"15min":9.6869999999999994},"last":0.28599999999999998}]},{"osd":2,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.51600000000000001,"15min":0.47099999999999997},"min":{"1min":0.21099999999999999,"5min":0.153,"15min":0.153},"max":{"1min":5.7439999999999998,"5min":5.8280000000000003,"15min":9.7639999999999993},"last":0.252},{"interface":"front","average":{"1min":0.497,"5min":0.56200000000000006,"15min":0.42299999999999999},"min":{"1min":0.20399999999999999,"5min":0.187,"15min":0.16800000000000001},"max":{"1min":3.9319999999999999,"5min":8.8040000000000003,"15min":8.8040000000000003},"last":0.29899999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738834,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":650100,"kb_used_data":5356,"kb_used_omap":214,"kb_used_meta":644521,"kb_avail":93721740,"statfs":{"total":96636764160,"available":95971061760,"internally_reserved":0,"allocated":5484544,"data_stored":2698616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":219335,"internal_metadata":659990329},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:56 2023","interfaces":[{"interface":"back","average":{"1min":0.30399999999999999,"5min":0.34699999999999998,"15min":0.36499999999999999},"min":{"1min":0.23999999999999999,"5min":0.14199999999999999,"15min":0.14199999999999999},"max":{"1min":0.42399999999999999,"5min":1.595,"15min":7.4249999999999998},"last":0.17100000000000001},{"interface":"front","average":{"1min":0.35499999999999998,"5min":0.40899999999999997,"15min":0.376},"min":{"1min":0.23999999999999999,"5min":0.19600000000000001,"15min":0.183},"max":{"1min":0.46999999999999997,"5min":2.6859999999999999,"15min":2.6859999999999999},"last":0.23000000000000001}]},{"osd":2,"last update":"Wed Nov 8 05:30:57 2023","interfaces":[{"interface":"back","average":{"1min":0.28599999999999998,"5min":0.34699999999999998,"15min":0.36899999999999999},"min":{"1min":0.19700000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":0.41399999999999998,"5min":2.2650000000000001,"15min":5.8209999999999997},"last":0.27700000000000002},{"interface":"front","average":{"1min":0.30599999999999999,"5min":0.313,"15min":0.35799999999999998},"min":{"1min":0.17799999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.435,"5min":1.137,"15min":4.2350000000000003},"last":0.252}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-08T05:31:15.593 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-11-08T05:31:15.877 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T05:31:15.879 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2023-11-08T05:31:15.890 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":1611,"stamp":"2023-11-08T05:31:14.065513+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":10,"acting":13,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":109,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2634368,"kb_used_data":409012,"kb_used_omap":780,"kb_used_meta":2220403,"kb_avail":280481152,"statfs":{"total":289910292480,"available":287212699648,"internally_reserved":0,"allocated":418828288,"data_stored":410500163,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":799447,"internal_metadata":2273692969},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-536782902,"num_objects":-164,"num_object_clones":0,"num_object_copies":-328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-115,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-164,"num_whiteouts":0,"num_read":-122056,"num_read_kb":-39011942,"num_write":-413642,"num_write_kb":-151705339,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-8,"num_bytes_recovered":-16922624,"num_keys_recovered":-2,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.424817"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490644+0000","last_change":"2023-11-08T05:28:13.557071+0000","last_active":"2023-11-08T05:30:44.490644+0000","last_peered":"2023-11-08T05:30:44.490644+0000","last_clean":"2023-11-08T05:28:01.458439+0000","last_became_active":"2023-11-08T05:28:13.557071+0000","last_became_peered":"2023-11-08T05:28:13.557071+0000","last_unstale":"2023-11-08T05:30:44.490644+0000","last_undegraded":"2023-11-08T05:30:44.490644+0000","last_fullsized":"2023-11-08T05:28:13.547025+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":459,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:45.794760+0000","last_change":"2023-11-08T05:30:45.794760+0000","last_active":"2023-11-08T05:30:45.794760+0000","last_peered":"2023-11-08T05:30:45.794760+0000","last_clean":"2023-11-08T05:30:45.794760+0000","last_became_active":"2023-11-08T05:30:45.794398+0000","last_became_peered":"2023-11-08T05:30:45.794398+0000","last_unstale":"2023-11-08T05:30:45.794760+0000","last_undegraded":"2023-11-08T05:30:45.794760+0000","last_fullsized":"2023-11-08T05:30:45.794760+0000","mapping_epoch":458,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":459,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":465,"reported_epoch":459,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:45.795085+0000","last_change":"2023-11-08T05:30:45.795085+0000","last_active":"2023-11-08T05:30:45.795085+0000","last_peered":"2023-11-08T05:30:45.795085+0000","last_clean":"2023-11-08T05:30:45.795085+0000","last_became_active":"2023-11-08T05:30:45.794743+0000","last_became_peered":"2023-11-08T05:30:45.794743+0000","last_unstale":"2023-11-08T05:30:45.795085+0000","last_undegraded":"2023-11-08T05:30:45.795085+0000","last_fullsized":"2023-11-08T05:30:45.795085+0000","mapping_epoch":458,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":459,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":454,"reported_epoch":458,"state":"active+clean","last_fresh":"2023-11-08T05:30:44.490618+0000","last_change":"2023-11-08T04:52:47.548597+0000","last_active":"2023-11-08T05:30:44.490618+0000","last_peered":"2023-11-08T05:30:44.490618+0000","last_clean":"2023-11-08T05:30:44.490618+0000","last_became_active":"2023-11-08T04:52:47.548294+0000","last_became_peered":"2023-11-08T04:52:47.548294+0000","last_unstale":"2023-11-08T05:30:44.490618+0000","last_undegraded":"2023-11-08T05:30:44.490618+0000","last_fullsized":"2023-11-08T05:30:44.490618+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":460,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490523+0000","last_change":"2023-11-08T05:28:13.545468+0000","last_active":"2023-11-08T05:30:44.490523+0000","last_peered":"2023-11-08T05:30:44.490523+0000","last_clean":"2023-11-08T05:28:01.456723+0000","last_became_active":"2023-11-08T05:28:13.545468+0000","last_became_peered":"2023-11-08T05:28:13.545468+0000","last_unstale":"2023-11-08T05:30:44.490523+0000","last_undegraded":"2023-11-08T05:30:44.490523+0000","last_fullsized":"2023-11-08T05:28:13.537207+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_clean_scrub_stamp":"2023-11-08T04:52:45.528796+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":465,"reported_epoch":458,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:44.491907+0000","last_change":"2023-11-08T05:29:44.395350+0000","last_active":"2023-11-08T05:30:44.491907+0000","last_peered":"2023-11-08T05:30:44.491907+0000","last_clean":"2023-11-08T05:30:44.491907+0000","last_became_active":"2023-11-08T05:29:44.395012+0000","last_became_peered":"2023-11-08T05:29:44.395012+0000","last_unstale":"2023-11-08T05:30:44.491907+0000","last_undegraded":"2023-11-08T05:30:44.491907+0000","last_fullsized":"2023-11-08T05:30:44.491907+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490629+0000","last_change":"2023-11-08T05:28:13.563357+0000","last_active":"2023-11-08T05:30:44.490629+0000","last_peered":"2023-11-08T05:30:44.490629+0000","last_clean":"2023-11-08T05:28:01.457691+0000","last_became_active":"2023-11-08T05:28:13.563357+0000","last_became_peered":"2023-11-08T05:28:13.563357+0000","last_unstale":"2023-11-08T05:30:44.490629+0000","last_undegraded":"2023-11-08T05:30:44.490629+0000","last_fullsized":"2023-11-08T05:28:13.541614+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490691+0000","last_change":"2023-11-08T05:28:13.550672+0000","last_active":"2023-11-08T05:30:44.490691+0000","last_peered":"2023-11-08T05:30:44.490691+0000","last_clean":"2023-11-08T05:28:01.457904+0000","last_became_active":"2023-11-08T05:28:13.550672+0000","last_became_peered":"2023-11-08T05:28:13.550672+0000","last_unstale":"2023-11-08T05:30:44.490691+0000","last_undegraded":"2023-11-08T05:30:44.490691+0000","last_fullsized":"2023-11-08T05:28:13.539595+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490805+0000","last_change":"2023-11-08T05:28:13.557968+0000","last_active":"2023-11-08T05:30:44.490805+0000","last_peered":"2023-11-08T05:30:44.490805+0000","last_clean":"2023-11-08T05:28:01.458433+0000","last_became_active":"2023-11-08T05:28:13.557968+0000","last_became_peered":"2023-11-08T05:28:13.557968+0000","last_unstale":"2023-11-08T05:30:44.490805+0000","last_undegraded":"2023-11-08T05:30:44.490805+0000","last_fullsized":"2023-11-08T05:28:13.542832+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":1,"acting":1,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":12,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738832,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":645040,"kb_used_data":5352,"kb_used_omap":275,"kb_used_meta":639404,"kb_avail":93726800,"statfs":{"total":96636764160,"available":95976243200,"internally_reserved":0,"allocated":5480448,"data_stored":2697647,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":281650,"internal_metadata":654750670},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:59 2023","interfaces":[{"interface":"back","average":{"1min":0.30499999999999999,"5min":0.439,"15min":0.36099999999999999},"min":{"1min":0.17399999999999999,"5min":0.14499999999999999,"15min":0.14499999999999999},"max":{"1min":0.60099999999999998,"5min":4.9379999999999997,"15min":4.9379999999999997},"last":0.379},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.49399999999999999,"15min":0.42399999999999999},"min":{"1min":0.221,"5min":0.19600000000000001,"15min":0.17000000000000001},"max":{"1min":0.63200000000000001,"5min":4.9859999999999998,"15min":4.9859999999999998},"last":0.40899999999999997}]},{"osd":1,"last update":"Wed Nov 8 05:31:06 2023","interfaces":[{"interface":"back","average":{"1min":0.27500000000000002,"5min":0.40300000000000002,"15min":0.46999999999999997},"min":{"1min":0.20899999999999999,"5min":0.16,"15min":0.153},"max":{"1min":0.41099999999999998,"5min":4.0830000000000002,"15min":23.242999999999999},"last":0.29499999999999998},{"interface":"front","average":{"1min":0.29899999999999999,"5min":0.40100000000000002,"15min":0.46000000000000002},"min":{"1min":0.23799999999999999,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":0.45300000000000001,"5min":3.5590000000000002,"15min":23.251000000000001},"last":0.27600000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738798,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1339228,"kb_used_data":398304,"kb_used_omap":291,"kb_used_meta":936476,"kb_avail":93032612,"statfs":{"total":96636764160,"available":95265394688,"internally_reserved":0,"allocated":407863296,"data_stored":405103900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":298462,"internal_metadata":958951970},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.436,"15min":0.36299999999999999},"min":{"1min":0.19700000000000001,"5min":0.14699999999999999,"15min":0.13700000000000001},"max":{"1min":3.222,"5min":6.5499999999999998,"15min":6.9989999999999997},"last":0.245},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.46300000000000002,"15min":0.42299999999999999},"min":{"1min":0.17799999999999999,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":5.0030000000000001,"5min":6.0629999999999997,"15min":9.6869999999999994},"last":0.28599999999999998}]},{"osd":2,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.51600000000000001,"15min":0.47099999999999997},"min":{"1min":0.21099999999999999,"5min":0.153,"15min":0.153},"max":{"1min":5.7439999999999998,"5min":5.8280000000000003,"15min":9.7639999999999993},"last":0.252},{"interface":"front","average":{"1min":0.497,"5min":0.56200000000000006,"15min":0.42299999999999999},"min":{"1min":0.20399999999999999,"5min":0.187,"15min":0.16800000000000001},"max":{"1min":3.9319999999999999,"5min":8.8040000000000003,"15min":8.8040000000000003},"last":0.29899999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738834,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":650100,"kb_used_data":5356,"kb_used_omap":214,"kb_used_meta":644521,"kb_avail":93721740,"statfs":{"total":96636764160,"available":95971061760,"internally_reserved":0,"allocated":5484544,"data_stored":2698616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":219335,"internal_metadata":659990329},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:56 2023","interfaces":[{"interface":"back","average":{"1min":0.30399999999999999,"5min":0.34699999999999998,"15min":0.36499999999999999},"min":{"1min":0.23999999999999999,"5min":0.14199999999999999,"15min":0.14199999999999999},"max":{"1min":0.42399999999999999,"5min":1.595,"15min":7.4249999999999998},"last":0.17100000000000001},{"interface":"front","average":{"1min":0.35499999999999998,"5min":0.40899999999999997,"15min":0.376},"min":{"1min":0.23999999999999999,"5min":0.19600000000000001,"15min":0.183},"max":{"1min":0.46999999999999997,"5min":2.6859999999999999,"15min":2.6859999999999999},"last":0.23000000000000001}]},{"osd":2,"last update":"Wed Nov 8 05:30:57 2023","interfaces":[{"interface":"back","average":{"1min":0.28599999999999998,"5min":0.34699999999999998,"15min":0.36899999999999999},"min":{"1min":0.19700000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":0.41399999999999998,"5min":2.2650000000000001,"15min":5.8209999999999997},"last":0.27700000000000002},{"interface":"front","average":{"1min":0.30599999999999999,"5min":0.313,"15min":0.35799999999999998},"min":{"1min":0.17799999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.435,"5min":1.137,"15min":4.2350000000000003},"last":0.252}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-08T05:31:15.892 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-11-08T05:31:15.893 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 459, 'reported_epoch': 458, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:30:44.490644+0000', 'last_change': '2023-11-08T05:28:13.557071+0000', 'last_active': '2023-11-08T05:30:44.490644+0000', 'last_peered': '2023-11-08T05:30:44.490644+0000', 'last_clean': '2023-11-08T05:28:01.458439+0000', 'last_became_active': '2023-11-08T05:28:13.557071+0000', 'last_became_peered': '2023-11-08T05:28:13.557071+0000', 'last_unstale': '2023-11-08T05:30:44.490644+0000', 'last_undegraded': '2023-11-08T05:30:44.490644+0000', 'last_fullsized': '2023-11-08T05:28:13.547025+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:31:15.893 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-11-08T05:31:15.893 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 11, 'reported_epoch': 459, 'state': 'active+clean+remapped', 'last_fresh': '2023-11-08T05:30:45.794760+0000', 'last_change': '2023-11-08T05:30:45.794760+0000', 'last_active': '2023-11-08T05:30:45.794760+0000', 'last_peered': '2023-11-08T05:30:45.794760+0000', 'last_clean': '2023-11-08T05:30:45.794760+0000', 'last_became_active': '2023-11-08T05:30:45.794398+0000', 'last_became_peered': '2023-11-08T05:30:45.794398+0000', 'last_unstale': '2023-11-08T05:30:45.794760+0000', 'last_undegraded': '2023-11-08T05:30:45.794760+0000', 'last_fullsized': '2023-11-08T05:30:45.794760+0000', 'mapping_epoch': 458, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 459, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2], 'acting': [2, 1], 'avail_no_missing': ['2', '1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2023-11-08T05:31:15.894 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.1 is not active+clean 2023-11-08T05:31:15.894 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.1', 'version': "0'0", 'reported_seq': 465, 'reported_epoch': 459, 'state': 'active+clean+remapped', 'last_fresh': '2023-11-08T05:30:45.795085+0000', 'last_change': '2023-11-08T05:30:45.795085+0000', 'last_active': '2023-11-08T05:30:45.795085+0000', 'last_peered': '2023-11-08T05:30:45.795085+0000', 'last_clean': '2023-11-08T05:30:45.795085+0000', 'last_became_active': '2023-11-08T05:30:45.794743+0000', 'last_became_peered': '2023-11-08T05:30:45.794743+0000', 'last_unstale': '2023-11-08T05:30:45.795085+0000', 'last_undegraded': '2023-11-08T05:30:45.795085+0000', 'last_fullsized': '2023-11-08T05:30:45.795085+0000', 'mapping_epoch': 458, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 459, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2], 'acting': [2, 1], 'avail_no_missing': ['2', '1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2023-11-08T05:31:15.894 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-11-08T05:31:15.894 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "0'0", 'reported_seq': 460, 'reported_epoch': 458, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:30:44.490523+0000', 'last_change': '2023-11-08T05:28:13.545468+0000', 'last_active': '2023-11-08T05:30:44.490523+0000', 'last_peered': '2023-11-08T05:30:44.490523+0000', 'last_clean': '2023-11-08T05:28:01.456723+0000', 'last_became_active': '2023-11-08T05:28:13.545468+0000', 'last_became_peered': '2023-11-08T05:28:13.545468+0000', 'last_unstale': '2023-11-08T05:30:44.490523+0000', 'last_undegraded': '2023-11-08T05:30:44.490523+0000', 'last_fullsized': '2023-11-08T05:28:13.537207+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 9, 'last_epoch_clean': 10, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:45.528796+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:45.528796+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:45.528796+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:31:15.895 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.0 is not active+clean 2023-11-08T05:31:15.895 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.0', 'version': "0'0", 'reported_seq': 465, 'reported_epoch': 458, 'state': 'active+clean+remapped', 'last_fresh': '2023-11-08T05:30:44.491907+0000', 'last_change': '2023-11-08T05:29:44.395350+0000', 'last_active': '2023-11-08T05:30:44.491907+0000', 'last_peered': '2023-11-08T05:30:44.491907+0000', 'last_clean': '2023-11-08T05:30:44.491907+0000', 'last_became_active': '2023-11-08T05:29:44.395012+0000', 'last_became_peered': '2023-11-08T05:29:44.395012+0000', 'last_unstale': '2023-11-08T05:30:44.491907+0000', 'last_undegraded': '2023-11-08T05:30:44.491907+0000', 'last_fullsized': '2023-11-08T05:30:44.491907+0000', 'mapping_epoch': 455, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 456, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2], 'acting': [2, 1], 'avail_no_missing': ['2', '1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2023-11-08T05:31:15.895 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-11-08T05:31:15.896 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 459, 'reported_epoch': 458, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:30:44.490629+0000', 'last_change': '2023-11-08T05:28:13.563357+0000', 'last_active': '2023-11-08T05:30:44.490629+0000', 'last_peered': '2023-11-08T05:30:44.490629+0000', 'last_clean': '2023-11-08T05:28:01.457691+0000', 'last_became_active': '2023-11-08T05:28:13.563357+0000', 'last_became_peered': '2023-11-08T05:28:13.563357+0000', 'last_unstale': '2023-11-08T05:30:44.490629+0000', 'last_undegraded': '2023-11-08T05:30:44.490629+0000', 'last_fullsized': '2023-11-08T05:28:13.541614+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:31:15.896 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-11-08T05:31:15.896 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 459, 'reported_epoch': 458, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:30:44.490691+0000', 'last_change': '2023-11-08T05:28:13.550672+0000', 'last_active': '2023-11-08T05:30:44.490691+0000', 'last_peered': '2023-11-08T05:30:44.490691+0000', 'last_clean': '2023-11-08T05:28:01.457904+0000', 'last_became_active': '2023-11-08T05:28:13.550672+0000', 'last_became_peered': '2023-11-08T05:28:13.550672+0000', 'last_unstale': '2023-11-08T05:30:44.490691+0000', 'last_undegraded': '2023-11-08T05:30:44.490691+0000', 'last_fullsized': '2023-11-08T05:28:13.539595+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:31:15.897 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-11-08T05:31:15.897 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 459, 'reported_epoch': 458, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:30:44.490805+0000', 'last_change': '2023-11-08T05:28:13.557968+0000', 'last_active': '2023-11-08T05:30:44.490805+0000', 'last_peered': '2023-11-08T05:30:44.490805+0000', 'last_clean': '2023-11-08T05:28:01.458433+0000', 'last_became_active': '2023-11-08T05:28:13.557968+0000', 'last_became_peered': '2023-11-08T05:28:13.557968+0000', 'last_unstale': '2023-11-08T05:30:44.490805+0000', 'last_undegraded': '2023-11-08T05:30:44.490805+0000', 'last_fullsized': '2023-11-08T05:28:13.542832+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:31:15.897 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-11-08T05:31:16.243 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T05:31:16.244 INFO:teuthology.orchestra.run.smithi087.stdout:{"fsid":"ab4cbc06-449d-497d-b513-220d9db2b590","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 5 pgs undersized","count":5},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2318,"monmap":{"epoch":1,"min_mon_release_name":"pacific","num_mons":1},"osdmap":{"epoch":459,"num_osds":3,"num_up_osds":2,"osd_up_since":1699421292,"num_in_osds":3,"osd_in_since":1699419158,"num_remapped_pgs":3},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean+remapped","count":3},{"state_name":"active+clean","count":1}],"num_pgs":9,"num_pools":2,"num_objects":0,"data_bytes":0,"bytes_used":2697592832,"bytes_avail":287212699648,"bytes_total":289910292480},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":41,"modified":"2023-11-08T05:29:32.044544+0000","services":{}},"progress_events":{}} 2023-11-08T05:31:16.257 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-11-08T05:31:16.257 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-11-08T05:31:16.380 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~178s 2023-11-08T05:31:16.540 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T05:31:16.542 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2023-11-08T05:31:16.553 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":1612,"stamp":"2023-11-08T05:31:16.065894+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":10,"acting":13,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":109,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2634368,"kb_used_data":409012,"kb_used_omap":780,"kb_used_meta":2220403,"kb_avail":280481152,"statfs":{"total":289910292480,"available":287212699648,"internally_reserved":0,"allocated":418828288,"data_stored":410500163,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":799447,"internal_metadata":2273692969},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-536782902,"num_objects":-164,"num_object_clones":0,"num_object_copies":-328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-115,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-164,"num_whiteouts":0,"num_read":-122056,"num_read_kb":-39011942,"num_write":-413642,"num_write_kb":-151705339,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-8,"num_bytes_recovered":-16922624,"num_keys_recovered":-2,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.424817"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490644+0000","last_change":"2023-11-08T05:28:13.557071+0000","last_active":"2023-11-08T05:30:44.490644+0000","last_peered":"2023-11-08T05:30:44.490644+0000","last_clean":"2023-11-08T05:28:01.458439+0000","last_became_active":"2023-11-08T05:28:13.557071+0000","last_became_peered":"2023-11-08T05:28:13.557071+0000","last_unstale":"2023-11-08T05:30:44.490644+0000","last_undegraded":"2023-11-08T05:30:44.490644+0000","last_fullsized":"2023-11-08T05:28:13.547025+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":459,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:45.794760+0000","last_change":"2023-11-08T05:30:45.794760+0000","last_active":"2023-11-08T05:30:45.794760+0000","last_peered":"2023-11-08T05:30:45.794760+0000","last_clean":"2023-11-08T05:30:45.794760+0000","last_became_active":"2023-11-08T05:30:45.794398+0000","last_became_peered":"2023-11-08T05:30:45.794398+0000","last_unstale":"2023-11-08T05:30:45.794760+0000","last_undegraded":"2023-11-08T05:30:45.794760+0000","last_fullsized":"2023-11-08T05:30:45.794760+0000","mapping_epoch":458,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":459,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":465,"reported_epoch":459,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:45.795085+0000","last_change":"2023-11-08T05:30:45.795085+0000","last_active":"2023-11-08T05:30:45.795085+0000","last_peered":"2023-11-08T05:30:45.795085+0000","last_clean":"2023-11-08T05:30:45.795085+0000","last_became_active":"2023-11-08T05:30:45.794743+0000","last_became_peered":"2023-11-08T05:30:45.794743+0000","last_unstale":"2023-11-08T05:30:45.795085+0000","last_undegraded":"2023-11-08T05:30:45.795085+0000","last_fullsized":"2023-11-08T05:30:45.795085+0000","mapping_epoch":458,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":459,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":454,"reported_epoch":458,"state":"active+clean","last_fresh":"2023-11-08T05:30:44.490618+0000","last_change":"2023-11-08T04:52:47.548597+0000","last_active":"2023-11-08T05:30:44.490618+0000","last_peered":"2023-11-08T05:30:44.490618+0000","last_clean":"2023-11-08T05:30:44.490618+0000","last_became_active":"2023-11-08T04:52:47.548294+0000","last_became_peered":"2023-11-08T04:52:47.548294+0000","last_unstale":"2023-11-08T05:30:44.490618+0000","last_undegraded":"2023-11-08T05:30:44.490618+0000","last_fullsized":"2023-11-08T05:30:44.490618+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":460,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490523+0000","last_change":"2023-11-08T05:28:13.545468+0000","last_active":"2023-11-08T05:30:44.490523+0000","last_peered":"2023-11-08T05:30:44.490523+0000","last_clean":"2023-11-08T05:28:01.456723+0000","last_became_active":"2023-11-08T05:28:13.545468+0000","last_became_peered":"2023-11-08T05:28:13.545468+0000","last_unstale":"2023-11-08T05:30:44.490523+0000","last_undegraded":"2023-11-08T05:30:44.490523+0000","last_fullsized":"2023-11-08T05:28:13.537207+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_clean_scrub_stamp":"2023-11-08T04:52:45.528796+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":465,"reported_epoch":458,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:44.491907+0000","last_change":"2023-11-08T05:29:44.395350+0000","last_active":"2023-11-08T05:30:44.491907+0000","last_peered":"2023-11-08T05:30:44.491907+0000","last_clean":"2023-11-08T05:30:44.491907+0000","last_became_active":"2023-11-08T05:29:44.395012+0000","last_became_peered":"2023-11-08T05:29:44.395012+0000","last_unstale":"2023-11-08T05:30:44.491907+0000","last_undegraded":"2023-11-08T05:30:44.491907+0000","last_fullsized":"2023-11-08T05:30:44.491907+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490629+0000","last_change":"2023-11-08T05:28:13.563357+0000","last_active":"2023-11-08T05:30:44.490629+0000","last_peered":"2023-11-08T05:30:44.490629+0000","last_clean":"2023-11-08T05:28:01.457691+0000","last_became_active":"2023-11-08T05:28:13.563357+0000","last_became_peered":"2023-11-08T05:28:13.563357+0000","last_unstale":"2023-11-08T05:30:44.490629+0000","last_undegraded":"2023-11-08T05:30:44.490629+0000","last_fullsized":"2023-11-08T05:28:13.541614+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490691+0000","last_change":"2023-11-08T05:28:13.550672+0000","last_active":"2023-11-08T05:30:44.490691+0000","last_peered":"2023-11-08T05:30:44.490691+0000","last_clean":"2023-11-08T05:28:01.457904+0000","last_became_active":"2023-11-08T05:28:13.550672+0000","last_became_peered":"2023-11-08T05:28:13.550672+0000","last_unstale":"2023-11-08T05:30:44.490691+0000","last_undegraded":"2023-11-08T05:30:44.490691+0000","last_fullsized":"2023-11-08T05:28:13.539595+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490805+0000","last_change":"2023-11-08T05:28:13.557968+0000","last_active":"2023-11-08T05:30:44.490805+0000","last_peered":"2023-11-08T05:30:44.490805+0000","last_clean":"2023-11-08T05:28:01.458433+0000","last_became_active":"2023-11-08T05:28:13.557968+0000","last_became_peered":"2023-11-08T05:28:13.557968+0000","last_unstale":"2023-11-08T05:30:44.490805+0000","last_undegraded":"2023-11-08T05:30:44.490805+0000","last_fullsized":"2023-11-08T05:28:13.542832+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":1,"acting":1,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":12,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738833,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":645040,"kb_used_data":5352,"kb_used_omap":275,"kb_used_meta":639404,"kb_avail":93726800,"statfs":{"total":96636764160,"available":95976243200,"internally_reserved":0,"allocated":5480448,"data_stored":2697647,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":281650,"internal_metadata":654750670},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:59 2023","interfaces":[{"interface":"back","average":{"1min":0.30499999999999999,"5min":0.439,"15min":0.36099999999999999},"min":{"1min":0.17399999999999999,"5min":0.14499999999999999,"15min":0.14499999999999999},"max":{"1min":0.60099999999999998,"5min":4.9379999999999997,"15min":4.9379999999999997},"last":0.379},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.49399999999999999,"15min":0.42399999999999999},"min":{"1min":0.221,"5min":0.19600000000000001,"15min":0.17000000000000001},"max":{"1min":0.63200000000000001,"5min":4.9859999999999998,"15min":4.9859999999999998},"last":0.40899999999999997}]},{"osd":1,"last update":"Wed Nov 8 05:31:06 2023","interfaces":[{"interface":"back","average":{"1min":0.27500000000000002,"5min":0.40300000000000002,"15min":0.46999999999999997},"min":{"1min":0.20899999999999999,"5min":0.16,"15min":0.153},"max":{"1min":0.41099999999999998,"5min":4.0830000000000002,"15min":23.242999999999999},"last":0.27000000000000002},{"interface":"front","average":{"1min":0.29899999999999999,"5min":0.40100000000000002,"15min":0.46000000000000002},"min":{"1min":0.23799999999999999,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":0.45300000000000001,"5min":3.5590000000000002,"15min":23.251000000000001},"last":0.30399999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738798,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1339228,"kb_used_data":398304,"kb_used_omap":291,"kb_used_meta":936476,"kb_avail":93032612,"statfs":{"total":96636764160,"available":95265394688,"internally_reserved":0,"allocated":407863296,"data_stored":405103900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":298462,"internal_metadata":958951970},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.436,"15min":0.36299999999999999},"min":{"1min":0.19700000000000001,"5min":0.14699999999999999,"15min":0.13700000000000001},"max":{"1min":3.222,"5min":6.5499999999999998,"15min":6.9989999999999997},"last":0.245},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.46300000000000002,"15min":0.42299999999999999},"min":{"1min":0.17799999999999999,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":5.0030000000000001,"5min":6.0629999999999997,"15min":9.6869999999999994},"last":0.28599999999999998}]},{"osd":2,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.51600000000000001,"15min":0.47099999999999997},"min":{"1min":0.21099999999999999,"5min":0.153,"15min":0.153},"max":{"1min":5.7439999999999998,"5min":5.8280000000000003,"15min":9.7639999999999993},"last":0.252},{"interface":"front","average":{"1min":0.497,"5min":0.56200000000000006,"15min":0.42299999999999999},"min":{"1min":0.20399999999999999,"5min":0.187,"15min":0.16800000000000001},"max":{"1min":3.9319999999999999,"5min":8.8040000000000003,"15min":8.8040000000000003},"last":0.29899999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738835,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":650100,"kb_used_data":5356,"kb_used_omap":214,"kb_used_meta":644521,"kb_avail":93721740,"statfs":{"total":96636764160,"available":95971061760,"internally_reserved":0,"allocated":5484544,"data_stored":2698616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":219335,"internal_metadata":659990329},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:56 2023","interfaces":[{"interface":"back","average":{"1min":0.30399999999999999,"5min":0.34699999999999998,"15min":0.36499999999999999},"min":{"1min":0.23999999999999999,"5min":0.14199999999999999,"15min":0.14199999999999999},"max":{"1min":0.42399999999999999,"5min":1.595,"15min":7.4249999999999998},"last":0.17100000000000001},{"interface":"front","average":{"1min":0.35499999999999998,"5min":0.40899999999999997,"15min":0.376},"min":{"1min":0.23999999999999999,"5min":0.19600000000000001,"15min":0.183},"max":{"1min":0.46999999999999997,"5min":2.6859999999999999,"15min":2.6859999999999999},"last":0.23000000000000001}]},{"osd":2,"last update":"Wed Nov 8 05:30:57 2023","interfaces":[{"interface":"back","average":{"1min":0.28599999999999998,"5min":0.34699999999999998,"15min":0.36899999999999999},"min":{"1min":0.19700000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":0.41399999999999998,"5min":2.2650000000000001,"15min":5.8209999999999997},"last":0.26000000000000001},{"interface":"front","average":{"1min":0.30599999999999999,"5min":0.313,"15min":0.35799999999999998},"min":{"1min":0.17799999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.435,"5min":1.137,"15min":4.2350000000000003},"last":0.29199999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-08T05:31:19.555 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-11-08T05:31:19.854 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T05:31:19.855 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2023-11-08T05:31:19.866 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":1613,"stamp":"2023-11-08T05:31:18.066383+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":10,"acting":13,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":109,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2634368,"kb_used_data":409012,"kb_used_omap":780,"kb_used_meta":2220403,"kb_avail":280481152,"statfs":{"total":289910292480,"available":287212699648,"internally_reserved":0,"allocated":418828288,"data_stored":410500163,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":799447,"internal_metadata":2273692969},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-536782902,"num_objects":-164,"num_object_clones":0,"num_object_copies":-328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-115,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-164,"num_whiteouts":0,"num_read":-122056,"num_read_kb":-39011942,"num_write":-413642,"num_write_kb":-151705339,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-8,"num_bytes_recovered":-16922624,"num_keys_recovered":-2,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.424817"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490644+0000","last_change":"2023-11-08T05:28:13.557071+0000","last_active":"2023-11-08T05:30:44.490644+0000","last_peered":"2023-11-08T05:30:44.490644+0000","last_clean":"2023-11-08T05:28:01.458439+0000","last_became_active":"2023-11-08T05:28:13.557071+0000","last_became_peered":"2023-11-08T05:28:13.557071+0000","last_unstale":"2023-11-08T05:30:44.490644+0000","last_undegraded":"2023-11-08T05:30:44.490644+0000","last_fullsized":"2023-11-08T05:28:13.547025+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":459,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:45.794760+0000","last_change":"2023-11-08T05:30:45.794760+0000","last_active":"2023-11-08T05:30:45.794760+0000","last_peered":"2023-11-08T05:30:45.794760+0000","last_clean":"2023-11-08T05:30:45.794760+0000","last_became_active":"2023-11-08T05:30:45.794398+0000","last_became_peered":"2023-11-08T05:30:45.794398+0000","last_unstale":"2023-11-08T05:30:45.794760+0000","last_undegraded":"2023-11-08T05:30:45.794760+0000","last_fullsized":"2023-11-08T05:30:45.794760+0000","mapping_epoch":458,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":459,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":465,"reported_epoch":459,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:45.795085+0000","last_change":"2023-11-08T05:30:45.795085+0000","last_active":"2023-11-08T05:30:45.795085+0000","last_peered":"2023-11-08T05:30:45.795085+0000","last_clean":"2023-11-08T05:30:45.795085+0000","last_became_active":"2023-11-08T05:30:45.794743+0000","last_became_peered":"2023-11-08T05:30:45.794743+0000","last_unstale":"2023-11-08T05:30:45.795085+0000","last_undegraded":"2023-11-08T05:30:45.795085+0000","last_fullsized":"2023-11-08T05:30:45.795085+0000","mapping_epoch":458,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":459,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":454,"reported_epoch":458,"state":"active+clean","last_fresh":"2023-11-08T05:30:44.490618+0000","last_change":"2023-11-08T04:52:47.548597+0000","last_active":"2023-11-08T05:30:44.490618+0000","last_peered":"2023-11-08T05:30:44.490618+0000","last_clean":"2023-11-08T05:30:44.490618+0000","last_became_active":"2023-11-08T04:52:47.548294+0000","last_became_peered":"2023-11-08T04:52:47.548294+0000","last_unstale":"2023-11-08T05:30:44.490618+0000","last_undegraded":"2023-11-08T05:30:44.490618+0000","last_fullsized":"2023-11-08T05:30:44.490618+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":460,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490523+0000","last_change":"2023-11-08T05:28:13.545468+0000","last_active":"2023-11-08T05:30:44.490523+0000","last_peered":"2023-11-08T05:30:44.490523+0000","last_clean":"2023-11-08T05:28:01.456723+0000","last_became_active":"2023-11-08T05:28:13.545468+0000","last_became_peered":"2023-11-08T05:28:13.545468+0000","last_unstale":"2023-11-08T05:30:44.490523+0000","last_undegraded":"2023-11-08T05:30:44.490523+0000","last_fullsized":"2023-11-08T05:28:13.537207+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_clean_scrub_stamp":"2023-11-08T04:52:45.528796+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":465,"reported_epoch":458,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:44.491907+0000","last_change":"2023-11-08T05:29:44.395350+0000","last_active":"2023-11-08T05:30:44.491907+0000","last_peered":"2023-11-08T05:30:44.491907+0000","last_clean":"2023-11-08T05:30:44.491907+0000","last_became_active":"2023-11-08T05:29:44.395012+0000","last_became_peered":"2023-11-08T05:29:44.395012+0000","last_unstale":"2023-11-08T05:30:44.491907+0000","last_undegraded":"2023-11-08T05:30:44.491907+0000","last_fullsized":"2023-11-08T05:30:44.491907+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490629+0000","last_change":"2023-11-08T05:28:13.563357+0000","last_active":"2023-11-08T05:30:44.490629+0000","last_peered":"2023-11-08T05:30:44.490629+0000","last_clean":"2023-11-08T05:28:01.457691+0000","last_became_active":"2023-11-08T05:28:13.563357+0000","last_became_peered":"2023-11-08T05:28:13.563357+0000","last_unstale":"2023-11-08T05:30:44.490629+0000","last_undegraded":"2023-11-08T05:30:44.490629+0000","last_fullsized":"2023-11-08T05:28:13.541614+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490691+0000","last_change":"2023-11-08T05:28:13.550672+0000","last_active":"2023-11-08T05:30:44.490691+0000","last_peered":"2023-11-08T05:30:44.490691+0000","last_clean":"2023-11-08T05:28:01.457904+0000","last_became_active":"2023-11-08T05:28:13.550672+0000","last_became_peered":"2023-11-08T05:28:13.550672+0000","last_unstale":"2023-11-08T05:30:44.490691+0000","last_undegraded":"2023-11-08T05:30:44.490691+0000","last_fullsized":"2023-11-08T05:28:13.539595+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490805+0000","last_change":"2023-11-08T05:28:13.557968+0000","last_active":"2023-11-08T05:30:44.490805+0000","last_peered":"2023-11-08T05:30:44.490805+0000","last_clean":"2023-11-08T05:28:01.458433+0000","last_became_active":"2023-11-08T05:28:13.557968+0000","last_became_peered":"2023-11-08T05:28:13.557968+0000","last_unstale":"2023-11-08T05:30:44.490805+0000","last_undegraded":"2023-11-08T05:30:44.490805+0000","last_fullsized":"2023-11-08T05:28:13.542832+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":1,"acting":1,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":12,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738833,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":645040,"kb_used_data":5352,"kb_used_omap":275,"kb_used_meta":639404,"kb_avail":93726800,"statfs":{"total":96636764160,"available":95976243200,"internally_reserved":0,"allocated":5480448,"data_stored":2697647,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":281650,"internal_metadata":654750670},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:59 2023","interfaces":[{"interface":"back","average":{"1min":0.30499999999999999,"5min":0.439,"15min":0.36099999999999999},"min":{"1min":0.17399999999999999,"5min":0.14499999999999999,"15min":0.14499999999999999},"max":{"1min":0.60099999999999998,"5min":4.9379999999999997,"15min":4.9379999999999997},"last":0.379},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.49399999999999999,"15min":0.42399999999999999},"min":{"1min":0.221,"5min":0.19600000000000001,"15min":0.17000000000000001},"max":{"1min":0.63200000000000001,"5min":4.9859999999999998,"15min":4.9859999999999998},"last":0.40899999999999997}]},{"osd":1,"last update":"Wed Nov 8 05:31:06 2023","interfaces":[{"interface":"back","average":{"1min":0.27500000000000002,"5min":0.40300000000000002,"15min":0.46999999999999997},"min":{"1min":0.20899999999999999,"5min":0.16,"15min":0.153},"max":{"1min":0.41099999999999998,"5min":4.0830000000000002,"15min":23.242999999999999},"last":0.27000000000000002},{"interface":"front","average":{"1min":0.29899999999999999,"5min":0.40100000000000002,"15min":0.46000000000000002},"min":{"1min":0.23799999999999999,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":0.45300000000000001,"5min":3.5590000000000002,"15min":23.251000000000001},"last":0.30399999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738798,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1339228,"kb_used_data":398304,"kb_used_omap":291,"kb_used_meta":936476,"kb_avail":93032612,"statfs":{"total":96636764160,"available":95265394688,"internally_reserved":0,"allocated":407863296,"data_stored":405103900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":298462,"internal_metadata":958951970},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.436,"15min":0.36299999999999999},"min":{"1min":0.19700000000000001,"5min":0.14699999999999999,"15min":0.13700000000000001},"max":{"1min":3.222,"5min":6.5499999999999998,"15min":6.9989999999999997},"last":0.245},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.46300000000000002,"15min":0.42299999999999999},"min":{"1min":0.17799999999999999,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":5.0030000000000001,"5min":6.0629999999999997,"15min":9.6869999999999994},"last":0.28599999999999998}]},{"osd":2,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.51600000000000001,"15min":0.47099999999999997},"min":{"1min":0.21099999999999999,"5min":0.153,"15min":0.153},"max":{"1min":5.7439999999999998,"5min":5.8280000000000003,"15min":9.7639999999999993},"last":0.252},{"interface":"front","average":{"1min":0.497,"5min":0.56200000000000006,"15min":0.42299999999999999},"min":{"1min":0.20399999999999999,"5min":0.187,"15min":0.16800000000000001},"max":{"1min":3.9319999999999999,"5min":8.8040000000000003,"15min":8.8040000000000003},"last":0.29899999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738835,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":650100,"kb_used_data":5356,"kb_used_omap":214,"kb_used_meta":644521,"kb_avail":93721740,"statfs":{"total":96636764160,"available":95971061760,"internally_reserved":0,"allocated":5484544,"data_stored":2698616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":219335,"internal_metadata":659990329},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:56 2023","interfaces":[{"interface":"back","average":{"1min":0.30399999999999999,"5min":0.34699999999999998,"15min":0.36499999999999999},"min":{"1min":0.23999999999999999,"5min":0.14199999999999999,"15min":0.14199999999999999},"max":{"1min":0.42399999999999999,"5min":1.595,"15min":7.4249999999999998},"last":0.17100000000000001},{"interface":"front","average":{"1min":0.35499999999999998,"5min":0.40899999999999997,"15min":0.376},"min":{"1min":0.23999999999999999,"5min":0.19600000000000001,"15min":0.183},"max":{"1min":0.46999999999999997,"5min":2.6859999999999999,"15min":2.6859999999999999},"last":0.23000000000000001}]},{"osd":2,"last update":"Wed Nov 8 05:30:57 2023","interfaces":[{"interface":"back","average":{"1min":0.28599999999999998,"5min":0.34699999999999998,"15min":0.36899999999999999},"min":{"1min":0.19700000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":0.41399999999999998,"5min":2.2650000000000001,"15min":5.8209999999999997},"last":0.26000000000000001},{"interface":"front","average":{"1min":0.30599999999999999,"5min":0.313,"15min":0.35799999999999998},"min":{"1min":0.17799999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.435,"5min":1.137,"15min":4.2350000000000003},"last":0.29199999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-08T05:31:19.868 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-11-08T05:31:20.150 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T05:31:20.152 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2023-11-08T05:31:20.163 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":1614,"stamp":"2023-11-08T05:31:20.066870+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":10,"acting":13,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":109,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2634368,"kb_used_data":409012,"kb_used_omap":780,"kb_used_meta":2220403,"kb_avail":280481152,"statfs":{"total":289910292480,"available":287212699648,"internally_reserved":0,"allocated":418828288,"data_stored":410500163,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":799447,"internal_metadata":2273692969},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-536782902,"num_objects":-164,"num_object_clones":0,"num_object_copies":-328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-115,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-164,"num_whiteouts":0,"num_read":-122056,"num_read_kb":-39011942,"num_write":-413642,"num_write_kb":-151705339,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-8,"num_bytes_recovered":-16922624,"num_keys_recovered":-2,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.424817"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490644+0000","last_change":"2023-11-08T05:28:13.557071+0000","last_active":"2023-11-08T05:30:44.490644+0000","last_peered":"2023-11-08T05:30:44.490644+0000","last_clean":"2023-11-08T05:28:01.458439+0000","last_became_active":"2023-11-08T05:28:13.557071+0000","last_became_peered":"2023-11-08T05:28:13.557071+0000","last_unstale":"2023-11-08T05:30:44.490644+0000","last_undegraded":"2023-11-08T05:30:44.490644+0000","last_fullsized":"2023-11-08T05:28:13.547025+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":459,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:45.794760+0000","last_change":"2023-11-08T05:30:45.794760+0000","last_active":"2023-11-08T05:30:45.794760+0000","last_peered":"2023-11-08T05:30:45.794760+0000","last_clean":"2023-11-08T05:30:45.794760+0000","last_became_active":"2023-11-08T05:30:45.794398+0000","last_became_peered":"2023-11-08T05:30:45.794398+0000","last_unstale":"2023-11-08T05:30:45.794760+0000","last_undegraded":"2023-11-08T05:30:45.794760+0000","last_fullsized":"2023-11-08T05:30:45.794760+0000","mapping_epoch":458,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":459,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":465,"reported_epoch":459,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:45.795085+0000","last_change":"2023-11-08T05:30:45.795085+0000","last_active":"2023-11-08T05:30:45.795085+0000","last_peered":"2023-11-08T05:30:45.795085+0000","last_clean":"2023-11-08T05:30:45.795085+0000","last_became_active":"2023-11-08T05:30:45.794743+0000","last_became_peered":"2023-11-08T05:30:45.794743+0000","last_unstale":"2023-11-08T05:30:45.795085+0000","last_undegraded":"2023-11-08T05:30:45.795085+0000","last_fullsized":"2023-11-08T05:30:45.795085+0000","mapping_epoch":458,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":459,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":454,"reported_epoch":458,"state":"active+clean","last_fresh":"2023-11-08T05:30:44.490618+0000","last_change":"2023-11-08T04:52:47.548597+0000","last_active":"2023-11-08T05:30:44.490618+0000","last_peered":"2023-11-08T05:30:44.490618+0000","last_clean":"2023-11-08T05:30:44.490618+0000","last_became_active":"2023-11-08T04:52:47.548294+0000","last_became_peered":"2023-11-08T04:52:47.548294+0000","last_unstale":"2023-11-08T05:30:44.490618+0000","last_undegraded":"2023-11-08T05:30:44.490618+0000","last_fullsized":"2023-11-08T05:30:44.490618+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":460,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490523+0000","last_change":"2023-11-08T05:28:13.545468+0000","last_active":"2023-11-08T05:30:44.490523+0000","last_peered":"2023-11-08T05:30:44.490523+0000","last_clean":"2023-11-08T05:28:01.456723+0000","last_became_active":"2023-11-08T05:28:13.545468+0000","last_became_peered":"2023-11-08T05:28:13.545468+0000","last_unstale":"2023-11-08T05:30:44.490523+0000","last_undegraded":"2023-11-08T05:30:44.490523+0000","last_fullsized":"2023-11-08T05:28:13.537207+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_clean_scrub_stamp":"2023-11-08T04:52:45.528796+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":465,"reported_epoch":458,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:44.491907+0000","last_change":"2023-11-08T05:29:44.395350+0000","last_active":"2023-11-08T05:30:44.491907+0000","last_peered":"2023-11-08T05:30:44.491907+0000","last_clean":"2023-11-08T05:30:44.491907+0000","last_became_active":"2023-11-08T05:29:44.395012+0000","last_became_peered":"2023-11-08T05:29:44.395012+0000","last_unstale":"2023-11-08T05:30:44.491907+0000","last_undegraded":"2023-11-08T05:30:44.491907+0000","last_fullsized":"2023-11-08T05:30:44.491907+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490629+0000","last_change":"2023-11-08T05:28:13.563357+0000","last_active":"2023-11-08T05:30:44.490629+0000","last_peered":"2023-11-08T05:30:44.490629+0000","last_clean":"2023-11-08T05:28:01.457691+0000","last_became_active":"2023-11-08T05:28:13.563357+0000","last_became_peered":"2023-11-08T05:28:13.563357+0000","last_unstale":"2023-11-08T05:30:44.490629+0000","last_undegraded":"2023-11-08T05:30:44.490629+0000","last_fullsized":"2023-11-08T05:28:13.541614+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490691+0000","last_change":"2023-11-08T05:28:13.550672+0000","last_active":"2023-11-08T05:30:44.490691+0000","last_peered":"2023-11-08T05:30:44.490691+0000","last_clean":"2023-11-08T05:28:01.457904+0000","last_became_active":"2023-11-08T05:28:13.550672+0000","last_became_peered":"2023-11-08T05:28:13.550672+0000","last_unstale":"2023-11-08T05:30:44.490691+0000","last_undegraded":"2023-11-08T05:30:44.490691+0000","last_fullsized":"2023-11-08T05:28:13.539595+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490805+0000","last_change":"2023-11-08T05:28:13.557968+0000","last_active":"2023-11-08T05:30:44.490805+0000","last_peered":"2023-11-08T05:30:44.490805+0000","last_clean":"2023-11-08T05:28:01.458433+0000","last_became_active":"2023-11-08T05:28:13.557968+0000","last_became_peered":"2023-11-08T05:28:13.557968+0000","last_unstale":"2023-11-08T05:30:44.490805+0000","last_undegraded":"2023-11-08T05:30:44.490805+0000","last_fullsized":"2023-11-08T05:28:13.542832+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":1,"acting":1,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":12,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738834,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":645040,"kb_used_data":5352,"kb_used_omap":275,"kb_used_meta":639404,"kb_avail":93726800,"statfs":{"total":96636764160,"available":95976243200,"internally_reserved":0,"allocated":5480448,"data_stored":2697647,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":281650,"internal_metadata":654750670},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:59 2023","interfaces":[{"interface":"back","average":{"1min":0.30499999999999999,"5min":0.439,"15min":0.36099999999999999},"min":{"1min":0.17399999999999999,"5min":0.14499999999999999,"15min":0.14499999999999999},"max":{"1min":0.60099999999999998,"5min":4.9379999999999997,"15min":4.9379999999999997},"last":0.379},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.49399999999999999,"15min":0.42399999999999999},"min":{"1min":0.221,"5min":0.19600000000000001,"15min":0.17000000000000001},"max":{"1min":0.63200000000000001,"5min":4.9859999999999998,"15min":4.9859999999999998},"last":0.40899999999999997}]},{"osd":1,"last update":"Wed Nov 8 05:31:06 2023","interfaces":[{"interface":"back","average":{"1min":0.27500000000000002,"5min":0.40300000000000002,"15min":0.46999999999999997},"min":{"1min":0.20899999999999999,"5min":0.16,"15min":0.153},"max":{"1min":0.41099999999999998,"5min":4.0830000000000002,"15min":23.242999999999999},"last":0.26600000000000001},{"interface":"front","average":{"1min":0.29899999999999999,"5min":0.40100000000000002,"15min":0.46000000000000002},"min":{"1min":0.23799999999999999,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":0.45300000000000001,"5min":3.5590000000000002,"15min":23.251000000000001},"last":0.23200000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738798,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1339228,"kb_used_data":398304,"kb_used_omap":291,"kb_used_meta":936476,"kb_avail":93032612,"statfs":{"total":96636764160,"available":95265394688,"internally_reserved":0,"allocated":407863296,"data_stored":405103900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":298462,"internal_metadata":958951970},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.436,"15min":0.36299999999999999},"min":{"1min":0.19700000000000001,"5min":0.14699999999999999,"15min":0.13700000000000001},"max":{"1min":3.222,"5min":6.5499999999999998,"15min":6.9989999999999997},"last":0.245},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.46300000000000002,"15min":0.42299999999999999},"min":{"1min":0.17799999999999999,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":5.0030000000000001,"5min":6.0629999999999997,"15min":9.6869999999999994},"last":0.28599999999999998}]},{"osd":2,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.51600000000000001,"15min":0.47099999999999997},"min":{"1min":0.21099999999999999,"5min":0.153,"15min":0.153},"max":{"1min":5.7439999999999998,"5min":5.8280000000000003,"15min":9.7639999999999993},"last":0.252},{"interface":"front","average":{"1min":0.497,"5min":0.56200000000000006,"15min":0.42299999999999999},"min":{"1min":0.20399999999999999,"5min":0.187,"15min":0.16800000000000001},"max":{"1min":3.9319999999999999,"5min":8.8040000000000003,"15min":8.8040000000000003},"last":0.29899999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738835,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":650100,"kb_used_data":5356,"kb_used_omap":214,"kb_used_meta":644521,"kb_avail":93721740,"statfs":{"total":96636764160,"available":95971061760,"internally_reserved":0,"allocated":5484544,"data_stored":2698616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":219335,"internal_metadata":659990329},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:56 2023","interfaces":[{"interface":"back","average":{"1min":0.30399999999999999,"5min":0.34699999999999998,"15min":0.36499999999999999},"min":{"1min":0.23999999999999999,"5min":0.14199999999999999,"15min":0.14199999999999999},"max":{"1min":0.42399999999999999,"5min":1.595,"15min":7.4249999999999998},"last":0.17100000000000001},{"interface":"front","average":{"1min":0.35499999999999998,"5min":0.40899999999999997,"15min":0.376},"min":{"1min":0.23999999999999999,"5min":0.19600000000000001,"15min":0.183},"max":{"1min":0.46999999999999997,"5min":2.6859999999999999,"15min":2.6859999999999999},"last":0.23000000000000001}]},{"osd":2,"last update":"Wed Nov 8 05:30:57 2023","interfaces":[{"interface":"back","average":{"1min":0.28599999999999998,"5min":0.34699999999999998,"15min":0.36899999999999999},"min":{"1min":0.19700000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":0.41399999999999998,"5min":2.2650000000000001,"15min":5.8209999999999997},"last":0.26000000000000001},{"interface":"front","average":{"1min":0.30599999999999999,"5min":0.313,"15min":0.35799999999999998},"min":{"1min":0.17799999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.435,"5min":1.137,"15min":4.2350000000000003},"last":0.29199999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-08T05:31:20.165 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-11-08T05:31:20.166 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 459, 'reported_epoch': 458, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:30:44.490644+0000', 'last_change': '2023-11-08T05:28:13.557071+0000', 'last_active': '2023-11-08T05:30:44.490644+0000', 'last_peered': '2023-11-08T05:30:44.490644+0000', 'last_clean': '2023-11-08T05:28:01.458439+0000', 'last_became_active': '2023-11-08T05:28:13.557071+0000', 'last_became_peered': '2023-11-08T05:28:13.557071+0000', 'last_unstale': '2023-11-08T05:30:44.490644+0000', 'last_undegraded': '2023-11-08T05:30:44.490644+0000', 'last_fullsized': '2023-11-08T05:28:13.547025+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:31:20.166 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-11-08T05:31:20.167 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 11, 'reported_epoch': 459, 'state': 'active+clean+remapped', 'last_fresh': '2023-11-08T05:30:45.794760+0000', 'last_change': '2023-11-08T05:30:45.794760+0000', 'last_active': '2023-11-08T05:30:45.794760+0000', 'last_peered': '2023-11-08T05:30:45.794760+0000', 'last_clean': '2023-11-08T05:30:45.794760+0000', 'last_became_active': '2023-11-08T05:30:45.794398+0000', 'last_became_peered': '2023-11-08T05:30:45.794398+0000', 'last_unstale': '2023-11-08T05:30:45.794760+0000', 'last_undegraded': '2023-11-08T05:30:45.794760+0000', 'last_fullsized': '2023-11-08T05:30:45.794760+0000', 'mapping_epoch': 458, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 459, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2], 'acting': [2, 1], 'avail_no_missing': ['2', '1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2023-11-08T05:31:20.167 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.1 is not active+clean 2023-11-08T05:31:20.168 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.1', 'version': "0'0", 'reported_seq': 465, 'reported_epoch': 459, 'state': 'active+clean+remapped', 'last_fresh': '2023-11-08T05:30:45.795085+0000', 'last_change': '2023-11-08T05:30:45.795085+0000', 'last_active': '2023-11-08T05:30:45.795085+0000', 'last_peered': '2023-11-08T05:30:45.795085+0000', 'last_clean': '2023-11-08T05:30:45.795085+0000', 'last_became_active': '2023-11-08T05:30:45.794743+0000', 'last_became_peered': '2023-11-08T05:30:45.794743+0000', 'last_unstale': '2023-11-08T05:30:45.795085+0000', 'last_undegraded': '2023-11-08T05:30:45.795085+0000', 'last_fullsized': '2023-11-08T05:30:45.795085+0000', 'mapping_epoch': 458, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 459, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2], 'acting': [2, 1], 'avail_no_missing': ['2', '1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2023-11-08T05:31:20.168 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-11-08T05:31:20.169 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "0'0", 'reported_seq': 460, 'reported_epoch': 458, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:30:44.490523+0000', 'last_change': '2023-11-08T05:28:13.545468+0000', 'last_active': '2023-11-08T05:30:44.490523+0000', 'last_peered': '2023-11-08T05:30:44.490523+0000', 'last_clean': '2023-11-08T05:28:01.456723+0000', 'last_became_active': '2023-11-08T05:28:13.545468+0000', 'last_became_peered': '2023-11-08T05:28:13.545468+0000', 'last_unstale': '2023-11-08T05:30:44.490523+0000', 'last_undegraded': '2023-11-08T05:30:44.490523+0000', 'last_fullsized': '2023-11-08T05:28:13.537207+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 9, 'last_epoch_clean': 10, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:45.528796+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:45.528796+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:45.528796+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:31:20.169 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.0 is not active+clean 2023-11-08T05:31:20.170 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.0', 'version': "0'0", 'reported_seq': 465, 'reported_epoch': 458, 'state': 'active+clean+remapped', 'last_fresh': '2023-11-08T05:30:44.491907+0000', 'last_change': '2023-11-08T05:29:44.395350+0000', 'last_active': '2023-11-08T05:30:44.491907+0000', 'last_peered': '2023-11-08T05:30:44.491907+0000', 'last_clean': '2023-11-08T05:30:44.491907+0000', 'last_became_active': '2023-11-08T05:29:44.395012+0000', 'last_became_peered': '2023-11-08T05:29:44.395012+0000', 'last_unstale': '2023-11-08T05:30:44.491907+0000', 'last_undegraded': '2023-11-08T05:30:44.491907+0000', 'last_fullsized': '2023-11-08T05:30:44.491907+0000', 'mapping_epoch': 455, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 456, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2], 'acting': [2, 1], 'avail_no_missing': ['2', '1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2023-11-08T05:31:20.171 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-11-08T05:31:20.171 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 459, 'reported_epoch': 458, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:30:44.490629+0000', 'last_change': '2023-11-08T05:28:13.563357+0000', 'last_active': '2023-11-08T05:30:44.490629+0000', 'last_peered': '2023-11-08T05:30:44.490629+0000', 'last_clean': '2023-11-08T05:28:01.457691+0000', 'last_became_active': '2023-11-08T05:28:13.563357+0000', 'last_became_peered': '2023-11-08T05:28:13.563357+0000', 'last_unstale': '2023-11-08T05:30:44.490629+0000', 'last_undegraded': '2023-11-08T05:30:44.490629+0000', 'last_fullsized': '2023-11-08T05:28:13.541614+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:31:20.171 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-11-08T05:31:20.172 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 459, 'reported_epoch': 458, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:30:44.490691+0000', 'last_change': '2023-11-08T05:28:13.550672+0000', 'last_active': '2023-11-08T05:30:44.490691+0000', 'last_peered': '2023-11-08T05:30:44.490691+0000', 'last_clean': '2023-11-08T05:28:01.457904+0000', 'last_became_active': '2023-11-08T05:28:13.550672+0000', 'last_became_peered': '2023-11-08T05:28:13.550672+0000', 'last_unstale': '2023-11-08T05:30:44.490691+0000', 'last_undegraded': '2023-11-08T05:30:44.490691+0000', 'last_fullsized': '2023-11-08T05:28:13.539595+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:31:20.172 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-11-08T05:31:20.172 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 459, 'reported_epoch': 458, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:30:44.490805+0000', 'last_change': '2023-11-08T05:28:13.557968+0000', 'last_active': '2023-11-08T05:30:44.490805+0000', 'last_peered': '2023-11-08T05:30:44.490805+0000', 'last_clean': '2023-11-08T05:28:01.458433+0000', 'last_became_active': '2023-11-08T05:28:13.557968+0000', 'last_became_peered': '2023-11-08T05:28:13.557968+0000', 'last_unstale': '2023-11-08T05:30:44.490805+0000', 'last_undegraded': '2023-11-08T05:30:44.490805+0000', 'last_fullsized': '2023-11-08T05:28:13.542832+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:31:20.173 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-11-08T05:31:20.516 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T05:31:20.517 INFO:teuthology.orchestra.run.smithi087.stdout:{"fsid":"ab4cbc06-449d-497d-b513-220d9db2b590","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 5 pgs undersized","count":5},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2323,"monmap":{"epoch":1,"min_mon_release_name":"pacific","num_mons":1},"osdmap":{"epoch":459,"num_osds":3,"num_up_osds":2,"osd_up_since":1699421292,"num_in_osds":3,"osd_in_since":1699419158,"num_remapped_pgs":3},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean+remapped","count":3},{"state_name":"active+clean","count":1}],"num_pgs":9,"num_pools":2,"num_objects":0,"data_bytes":0,"bytes_used":2697592832,"bytes_avail":287212699648,"bytes_total":289910292480},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":41,"modified":"2023-11-08T05:29:32.044544+0000","services":{}},"progress_events":{}} 2023-11-08T05:31:20.530 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-11-08T05:31:20.530 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-11-08T05:31:20.810 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T05:31:20.811 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2023-11-08T05:31:20.823 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":1614,"stamp":"2023-11-08T05:31:20.066870+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":10,"acting":13,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":109,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2634368,"kb_used_data":409012,"kb_used_omap":780,"kb_used_meta":2220403,"kb_avail":280481152,"statfs":{"total":289910292480,"available":287212699648,"internally_reserved":0,"allocated":418828288,"data_stored":410500163,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":799447,"internal_metadata":2273692969},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-536782902,"num_objects":-164,"num_object_clones":0,"num_object_copies":-328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-115,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-164,"num_whiteouts":0,"num_read":-122056,"num_read_kb":-39011942,"num_write":-413642,"num_write_kb":-151705339,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-8,"num_bytes_recovered":-16922624,"num_keys_recovered":-2,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.424817"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490644+0000","last_change":"2023-11-08T05:28:13.557071+0000","last_active":"2023-11-08T05:30:44.490644+0000","last_peered":"2023-11-08T05:30:44.490644+0000","last_clean":"2023-11-08T05:28:01.458439+0000","last_became_active":"2023-11-08T05:28:13.557071+0000","last_became_peered":"2023-11-08T05:28:13.557071+0000","last_unstale":"2023-11-08T05:30:44.490644+0000","last_undegraded":"2023-11-08T05:30:44.490644+0000","last_fullsized":"2023-11-08T05:28:13.547025+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":459,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:45.794760+0000","last_change":"2023-11-08T05:30:45.794760+0000","last_active":"2023-11-08T05:30:45.794760+0000","last_peered":"2023-11-08T05:30:45.794760+0000","last_clean":"2023-11-08T05:30:45.794760+0000","last_became_active":"2023-11-08T05:30:45.794398+0000","last_became_peered":"2023-11-08T05:30:45.794398+0000","last_unstale":"2023-11-08T05:30:45.794760+0000","last_undegraded":"2023-11-08T05:30:45.794760+0000","last_fullsized":"2023-11-08T05:30:45.794760+0000","mapping_epoch":458,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":459,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":465,"reported_epoch":459,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:45.795085+0000","last_change":"2023-11-08T05:30:45.795085+0000","last_active":"2023-11-08T05:30:45.795085+0000","last_peered":"2023-11-08T05:30:45.795085+0000","last_clean":"2023-11-08T05:30:45.795085+0000","last_became_active":"2023-11-08T05:30:45.794743+0000","last_became_peered":"2023-11-08T05:30:45.794743+0000","last_unstale":"2023-11-08T05:30:45.795085+0000","last_undegraded":"2023-11-08T05:30:45.795085+0000","last_fullsized":"2023-11-08T05:30:45.795085+0000","mapping_epoch":458,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":459,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":454,"reported_epoch":458,"state":"active+clean","last_fresh":"2023-11-08T05:30:44.490618+0000","last_change":"2023-11-08T04:52:47.548597+0000","last_active":"2023-11-08T05:30:44.490618+0000","last_peered":"2023-11-08T05:30:44.490618+0000","last_clean":"2023-11-08T05:30:44.490618+0000","last_became_active":"2023-11-08T04:52:47.548294+0000","last_became_peered":"2023-11-08T04:52:47.548294+0000","last_unstale":"2023-11-08T05:30:44.490618+0000","last_undegraded":"2023-11-08T05:30:44.490618+0000","last_fullsized":"2023-11-08T05:30:44.490618+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":460,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490523+0000","last_change":"2023-11-08T05:28:13.545468+0000","last_active":"2023-11-08T05:30:44.490523+0000","last_peered":"2023-11-08T05:30:44.490523+0000","last_clean":"2023-11-08T05:28:01.456723+0000","last_became_active":"2023-11-08T05:28:13.545468+0000","last_became_peered":"2023-11-08T05:28:13.545468+0000","last_unstale":"2023-11-08T05:30:44.490523+0000","last_undegraded":"2023-11-08T05:30:44.490523+0000","last_fullsized":"2023-11-08T05:28:13.537207+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_clean_scrub_stamp":"2023-11-08T04:52:45.528796+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":465,"reported_epoch":458,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:44.491907+0000","last_change":"2023-11-08T05:29:44.395350+0000","last_active":"2023-11-08T05:30:44.491907+0000","last_peered":"2023-11-08T05:30:44.491907+0000","last_clean":"2023-11-08T05:30:44.491907+0000","last_became_active":"2023-11-08T05:29:44.395012+0000","last_became_peered":"2023-11-08T05:29:44.395012+0000","last_unstale":"2023-11-08T05:30:44.491907+0000","last_undegraded":"2023-11-08T05:30:44.491907+0000","last_fullsized":"2023-11-08T05:30:44.491907+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490629+0000","last_change":"2023-11-08T05:28:13.563357+0000","last_active":"2023-11-08T05:30:44.490629+0000","last_peered":"2023-11-08T05:30:44.490629+0000","last_clean":"2023-11-08T05:28:01.457691+0000","last_became_active":"2023-11-08T05:28:13.563357+0000","last_became_peered":"2023-11-08T05:28:13.563357+0000","last_unstale":"2023-11-08T05:30:44.490629+0000","last_undegraded":"2023-11-08T05:30:44.490629+0000","last_fullsized":"2023-11-08T05:28:13.541614+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490691+0000","last_change":"2023-11-08T05:28:13.550672+0000","last_active":"2023-11-08T05:30:44.490691+0000","last_peered":"2023-11-08T05:30:44.490691+0000","last_clean":"2023-11-08T05:28:01.457904+0000","last_became_active":"2023-11-08T05:28:13.550672+0000","last_became_peered":"2023-11-08T05:28:13.550672+0000","last_unstale":"2023-11-08T05:30:44.490691+0000","last_undegraded":"2023-11-08T05:30:44.490691+0000","last_fullsized":"2023-11-08T05:28:13.539595+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490805+0000","last_change":"2023-11-08T05:28:13.557968+0000","last_active":"2023-11-08T05:30:44.490805+0000","last_peered":"2023-11-08T05:30:44.490805+0000","last_clean":"2023-11-08T05:28:01.458433+0000","last_became_active":"2023-11-08T05:28:13.557968+0000","last_became_peered":"2023-11-08T05:28:13.557968+0000","last_unstale":"2023-11-08T05:30:44.490805+0000","last_undegraded":"2023-11-08T05:30:44.490805+0000","last_fullsized":"2023-11-08T05:28:13.542832+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":1,"acting":1,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":12,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738834,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":645040,"kb_used_data":5352,"kb_used_omap":275,"kb_used_meta":639404,"kb_avail":93726800,"statfs":{"total":96636764160,"available":95976243200,"internally_reserved":0,"allocated":5480448,"data_stored":2697647,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":281650,"internal_metadata":654750670},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:59 2023","interfaces":[{"interface":"back","average":{"1min":0.30499999999999999,"5min":0.439,"15min":0.36099999999999999},"min":{"1min":0.17399999999999999,"5min":0.14499999999999999,"15min":0.14499999999999999},"max":{"1min":0.60099999999999998,"5min":4.9379999999999997,"15min":4.9379999999999997},"last":0.379},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.49399999999999999,"15min":0.42399999999999999},"min":{"1min":0.221,"5min":0.19600000000000001,"15min":0.17000000000000001},"max":{"1min":0.63200000000000001,"5min":4.9859999999999998,"15min":4.9859999999999998},"last":0.40899999999999997}]},{"osd":1,"last update":"Wed Nov 8 05:31:06 2023","interfaces":[{"interface":"back","average":{"1min":0.27500000000000002,"5min":0.40300000000000002,"15min":0.46999999999999997},"min":{"1min":0.20899999999999999,"5min":0.16,"15min":0.153},"max":{"1min":0.41099999999999998,"5min":4.0830000000000002,"15min":23.242999999999999},"last":0.26600000000000001},{"interface":"front","average":{"1min":0.29899999999999999,"5min":0.40100000000000002,"15min":0.46000000000000002},"min":{"1min":0.23799999999999999,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":0.45300000000000001,"5min":3.5590000000000002,"15min":23.251000000000001},"last":0.23200000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738798,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1339228,"kb_used_data":398304,"kb_used_omap":291,"kb_used_meta":936476,"kb_avail":93032612,"statfs":{"total":96636764160,"available":95265394688,"internally_reserved":0,"allocated":407863296,"data_stored":405103900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":298462,"internal_metadata":958951970},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.436,"15min":0.36299999999999999},"min":{"1min":0.19700000000000001,"5min":0.14699999999999999,"15min":0.13700000000000001},"max":{"1min":3.222,"5min":6.5499999999999998,"15min":6.9989999999999997},"last":0.245},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.46300000000000002,"15min":0.42299999999999999},"min":{"1min":0.17799999999999999,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":5.0030000000000001,"5min":6.0629999999999997,"15min":9.6869999999999994},"last":0.28599999999999998}]},{"osd":2,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.51600000000000001,"15min":0.47099999999999997},"min":{"1min":0.21099999999999999,"5min":0.153,"15min":0.153},"max":{"1min":5.7439999999999998,"5min":5.8280000000000003,"15min":9.7639999999999993},"last":0.252},{"interface":"front","average":{"1min":0.497,"5min":0.56200000000000006,"15min":0.42299999999999999},"min":{"1min":0.20399999999999999,"5min":0.187,"15min":0.16800000000000001},"max":{"1min":3.9319999999999999,"5min":8.8040000000000003,"15min":8.8040000000000003},"last":0.29899999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738835,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":650100,"kb_used_data":5356,"kb_used_omap":214,"kb_used_meta":644521,"kb_avail":93721740,"statfs":{"total":96636764160,"available":95971061760,"internally_reserved":0,"allocated":5484544,"data_stored":2698616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":219335,"internal_metadata":659990329},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:56 2023","interfaces":[{"interface":"back","average":{"1min":0.30399999999999999,"5min":0.34699999999999998,"15min":0.36499999999999999},"min":{"1min":0.23999999999999999,"5min":0.14199999999999999,"15min":0.14199999999999999},"max":{"1min":0.42399999999999999,"5min":1.595,"15min":7.4249999999999998},"last":0.17100000000000001},{"interface":"front","average":{"1min":0.35499999999999998,"5min":0.40899999999999997,"15min":0.376},"min":{"1min":0.23999999999999999,"5min":0.19600000000000001,"15min":0.183},"max":{"1min":0.46999999999999997,"5min":2.6859999999999999,"15min":2.6859999999999999},"last":0.23000000000000001}]},{"osd":2,"last update":"Wed Nov 8 05:30:57 2023","interfaces":[{"interface":"back","average":{"1min":0.28599999999999998,"5min":0.34699999999999998,"15min":0.36899999999999999},"min":{"1min":0.19700000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":0.41399999999999998,"5min":2.2650000000000001,"15min":5.8209999999999997},"last":0.26000000000000001},{"interface":"front","average":{"1min":0.30599999999999999,"5min":0.313,"15min":0.35799999999999998},"min":{"1min":0.17799999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.435,"5min":1.137,"15min":4.2350000000000003},"last":0.29199999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-08T05:31:21.784 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~184s 2023-11-08T05:31:23.825 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-11-08T05:31:24.124 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T05:31:24.126 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2023-11-08T05:31:24.137 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":1616,"stamp":"2023-11-08T05:31:24.067620+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":10,"acting":13,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":109,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2634368,"kb_used_data":409012,"kb_used_omap":780,"kb_used_meta":2220403,"kb_avail":280481152,"statfs":{"total":289910292480,"available":287212699648,"internally_reserved":0,"allocated":418828288,"data_stored":410500163,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":799447,"internal_metadata":2273692969},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-536782902,"num_objects":-164,"num_object_clones":0,"num_object_copies":-328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-115,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-164,"num_whiteouts":0,"num_read":-122056,"num_read_kb":-39011942,"num_write":-413642,"num_write_kb":-151705339,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-8,"num_bytes_recovered":-16922624,"num_keys_recovered":-2,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.424817"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490644+0000","last_change":"2023-11-08T05:28:13.557071+0000","last_active":"2023-11-08T05:30:44.490644+0000","last_peered":"2023-11-08T05:30:44.490644+0000","last_clean":"2023-11-08T05:28:01.458439+0000","last_became_active":"2023-11-08T05:28:13.557071+0000","last_became_peered":"2023-11-08T05:28:13.557071+0000","last_unstale":"2023-11-08T05:30:44.490644+0000","last_undegraded":"2023-11-08T05:30:44.490644+0000","last_fullsized":"2023-11-08T05:28:13.547025+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":459,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:45.794760+0000","last_change":"2023-11-08T05:30:45.794760+0000","last_active":"2023-11-08T05:30:45.794760+0000","last_peered":"2023-11-08T05:30:45.794760+0000","last_clean":"2023-11-08T05:30:45.794760+0000","last_became_active":"2023-11-08T05:30:45.794398+0000","last_became_peered":"2023-11-08T05:30:45.794398+0000","last_unstale":"2023-11-08T05:30:45.794760+0000","last_undegraded":"2023-11-08T05:30:45.794760+0000","last_fullsized":"2023-11-08T05:30:45.794760+0000","mapping_epoch":458,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":459,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":465,"reported_epoch":459,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:45.795085+0000","last_change":"2023-11-08T05:30:45.795085+0000","last_active":"2023-11-08T05:30:45.795085+0000","last_peered":"2023-11-08T05:30:45.795085+0000","last_clean":"2023-11-08T05:30:45.795085+0000","last_became_active":"2023-11-08T05:30:45.794743+0000","last_became_peered":"2023-11-08T05:30:45.794743+0000","last_unstale":"2023-11-08T05:30:45.795085+0000","last_undegraded":"2023-11-08T05:30:45.795085+0000","last_fullsized":"2023-11-08T05:30:45.795085+0000","mapping_epoch":458,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":459,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":454,"reported_epoch":458,"state":"active+clean","last_fresh":"2023-11-08T05:30:44.490618+0000","last_change":"2023-11-08T04:52:47.548597+0000","last_active":"2023-11-08T05:30:44.490618+0000","last_peered":"2023-11-08T05:30:44.490618+0000","last_clean":"2023-11-08T05:30:44.490618+0000","last_became_active":"2023-11-08T04:52:47.548294+0000","last_became_peered":"2023-11-08T04:52:47.548294+0000","last_unstale":"2023-11-08T05:30:44.490618+0000","last_undegraded":"2023-11-08T05:30:44.490618+0000","last_fullsized":"2023-11-08T05:30:44.490618+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":460,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490523+0000","last_change":"2023-11-08T05:28:13.545468+0000","last_active":"2023-11-08T05:30:44.490523+0000","last_peered":"2023-11-08T05:30:44.490523+0000","last_clean":"2023-11-08T05:28:01.456723+0000","last_became_active":"2023-11-08T05:28:13.545468+0000","last_became_peered":"2023-11-08T05:28:13.545468+0000","last_unstale":"2023-11-08T05:30:44.490523+0000","last_undegraded":"2023-11-08T05:30:44.490523+0000","last_fullsized":"2023-11-08T05:28:13.537207+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_clean_scrub_stamp":"2023-11-08T04:52:45.528796+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":465,"reported_epoch":458,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:44.491907+0000","last_change":"2023-11-08T05:29:44.395350+0000","last_active":"2023-11-08T05:30:44.491907+0000","last_peered":"2023-11-08T05:30:44.491907+0000","last_clean":"2023-11-08T05:30:44.491907+0000","last_became_active":"2023-11-08T05:29:44.395012+0000","last_became_peered":"2023-11-08T05:29:44.395012+0000","last_unstale":"2023-11-08T05:30:44.491907+0000","last_undegraded":"2023-11-08T05:30:44.491907+0000","last_fullsized":"2023-11-08T05:30:44.491907+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490629+0000","last_change":"2023-11-08T05:28:13.563357+0000","last_active":"2023-11-08T05:30:44.490629+0000","last_peered":"2023-11-08T05:30:44.490629+0000","last_clean":"2023-11-08T05:28:01.457691+0000","last_became_active":"2023-11-08T05:28:13.563357+0000","last_became_peered":"2023-11-08T05:28:13.563357+0000","last_unstale":"2023-11-08T05:30:44.490629+0000","last_undegraded":"2023-11-08T05:30:44.490629+0000","last_fullsized":"2023-11-08T05:28:13.541614+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490691+0000","last_change":"2023-11-08T05:28:13.550672+0000","last_active":"2023-11-08T05:30:44.490691+0000","last_peered":"2023-11-08T05:30:44.490691+0000","last_clean":"2023-11-08T05:28:01.457904+0000","last_became_active":"2023-11-08T05:28:13.550672+0000","last_became_peered":"2023-11-08T05:28:13.550672+0000","last_unstale":"2023-11-08T05:30:44.490691+0000","last_undegraded":"2023-11-08T05:30:44.490691+0000","last_fullsized":"2023-11-08T05:28:13.539595+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490805+0000","last_change":"2023-11-08T05:28:13.557968+0000","last_active":"2023-11-08T05:30:44.490805+0000","last_peered":"2023-11-08T05:30:44.490805+0000","last_clean":"2023-11-08T05:28:01.458433+0000","last_became_active":"2023-11-08T05:28:13.557968+0000","last_became_peered":"2023-11-08T05:28:13.557968+0000","last_unstale":"2023-11-08T05:30:44.490805+0000","last_undegraded":"2023-11-08T05:30:44.490805+0000","last_fullsized":"2023-11-08T05:28:13.542832+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":1,"acting":1,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":12,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738834,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":645040,"kb_used_data":5352,"kb_used_omap":275,"kb_used_meta":639404,"kb_avail":93726800,"statfs":{"total":96636764160,"available":95976243200,"internally_reserved":0,"allocated":5480448,"data_stored":2697647,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":281650,"internal_metadata":654750670},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:59 2023","interfaces":[{"interface":"back","average":{"1min":0.30499999999999999,"5min":0.439,"15min":0.36099999999999999},"min":{"1min":0.17399999999999999,"5min":0.14499999999999999,"15min":0.14499999999999999},"max":{"1min":0.60099999999999998,"5min":4.9379999999999997,"15min":4.9379999999999997},"last":0.379},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.49399999999999999,"15min":0.42399999999999999},"min":{"1min":0.221,"5min":0.19600000000000001,"15min":0.17000000000000001},"max":{"1min":0.63200000000000001,"5min":4.9859999999999998,"15min":4.9859999999999998},"last":0.40899999999999997}]},{"osd":1,"last update":"Wed Nov 8 05:31:06 2023","interfaces":[{"interface":"back","average":{"1min":0.27500000000000002,"5min":0.40300000000000002,"15min":0.46999999999999997},"min":{"1min":0.20899999999999999,"5min":0.16,"15min":0.153},"max":{"1min":0.41099999999999998,"5min":4.0830000000000002,"15min":23.242999999999999},"last":0.26600000000000001},{"interface":"front","average":{"1min":0.29899999999999999,"5min":0.40100000000000002,"15min":0.46000000000000002},"min":{"1min":0.23799999999999999,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":0.45300000000000001,"5min":3.5590000000000002,"15min":23.251000000000001},"last":0.23200000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738798,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1339228,"kb_used_data":398304,"kb_used_omap":291,"kb_used_meta":936476,"kb_avail":93032612,"statfs":{"total":96636764160,"available":95265394688,"internally_reserved":0,"allocated":407863296,"data_stored":405103900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":298462,"internal_metadata":958951970},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.436,"15min":0.36299999999999999},"min":{"1min":0.19700000000000001,"5min":0.14699999999999999,"15min":0.13700000000000001},"max":{"1min":3.222,"5min":6.5499999999999998,"15min":6.9989999999999997},"last":0.245},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.46300000000000002,"15min":0.42299999999999999},"min":{"1min":0.17799999999999999,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":5.0030000000000001,"5min":6.0629999999999997,"15min":9.6869999999999994},"last":0.28599999999999998}]},{"osd":2,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.51600000000000001,"15min":0.47099999999999997},"min":{"1min":0.21099999999999999,"5min":0.153,"15min":0.153},"max":{"1min":5.7439999999999998,"5min":5.8280000000000003,"15min":9.7639999999999993},"last":0.252},{"interface":"front","average":{"1min":0.497,"5min":0.56200000000000006,"15min":0.42299999999999999},"min":{"1min":0.20399999999999999,"5min":0.187,"15min":0.16800000000000001},"max":{"1min":3.9319999999999999,"5min":8.8040000000000003,"15min":8.8040000000000003},"last":0.29899999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738836,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":650100,"kb_used_data":5356,"kb_used_omap":214,"kb_used_meta":644521,"kb_avail":93721740,"statfs":{"total":96636764160,"available":95971061760,"internally_reserved":0,"allocated":5484544,"data_stored":2698616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":219335,"internal_metadata":659990329},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:56 2023","interfaces":[{"interface":"back","average":{"1min":0.30399999999999999,"5min":0.34699999999999998,"15min":0.36499999999999999},"min":{"1min":0.23999999999999999,"5min":0.14199999999999999,"15min":0.14199999999999999},"max":{"1min":0.42399999999999999,"5min":1.595,"15min":7.4249999999999998},"last":0.17100000000000001},{"interface":"front","average":{"1min":0.35499999999999998,"5min":0.40899999999999997,"15min":0.376},"min":{"1min":0.23999999999999999,"5min":0.19600000000000001,"15min":0.183},"max":{"1min":0.46999999999999997,"5min":2.6859999999999999,"15min":2.6859999999999999},"last":0.23000000000000001}]},{"osd":2,"last update":"Wed Nov 8 05:30:57 2023","interfaces":[{"interface":"back","average":{"1min":0.28599999999999998,"5min":0.34699999999999998,"15min":0.36899999999999999},"min":{"1min":0.19700000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":0.41399999999999998,"5min":2.2650000000000001,"15min":5.8209999999999997},"last":0.29899999999999999},{"interface":"front","average":{"1min":0.30599999999999999,"5min":0.313,"15min":0.35799999999999998},"min":{"1min":0.17799999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.435,"5min":1.137,"15min":4.2350000000000003},"last":0.32300000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-08T05:31:24.138 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-11-08T05:31:24.421 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T05:31:24.422 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2023-11-08T05:31:24.433 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":1616,"stamp":"2023-11-08T05:31:24.067620+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":10,"acting":13,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":109,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2634368,"kb_used_data":409012,"kb_used_omap":780,"kb_used_meta":2220403,"kb_avail":280481152,"statfs":{"total":289910292480,"available":287212699648,"internally_reserved":0,"allocated":418828288,"data_stored":410500163,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":799447,"internal_metadata":2273692969},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-536782902,"num_objects":-164,"num_object_clones":0,"num_object_copies":-328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-115,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-164,"num_whiteouts":0,"num_read":-122056,"num_read_kb":-39011942,"num_write":-413642,"num_write_kb":-151705339,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-8,"num_bytes_recovered":-16922624,"num_keys_recovered":-2,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.424817"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490644+0000","last_change":"2023-11-08T05:28:13.557071+0000","last_active":"2023-11-08T05:30:44.490644+0000","last_peered":"2023-11-08T05:30:44.490644+0000","last_clean":"2023-11-08T05:28:01.458439+0000","last_became_active":"2023-11-08T05:28:13.557071+0000","last_became_peered":"2023-11-08T05:28:13.557071+0000","last_unstale":"2023-11-08T05:30:44.490644+0000","last_undegraded":"2023-11-08T05:30:44.490644+0000","last_fullsized":"2023-11-08T05:28:13.547025+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":459,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:45.794760+0000","last_change":"2023-11-08T05:30:45.794760+0000","last_active":"2023-11-08T05:30:45.794760+0000","last_peered":"2023-11-08T05:30:45.794760+0000","last_clean":"2023-11-08T05:30:45.794760+0000","last_became_active":"2023-11-08T05:30:45.794398+0000","last_became_peered":"2023-11-08T05:30:45.794398+0000","last_unstale":"2023-11-08T05:30:45.794760+0000","last_undegraded":"2023-11-08T05:30:45.794760+0000","last_fullsized":"2023-11-08T05:30:45.794760+0000","mapping_epoch":458,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":459,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":465,"reported_epoch":459,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:45.795085+0000","last_change":"2023-11-08T05:30:45.795085+0000","last_active":"2023-11-08T05:30:45.795085+0000","last_peered":"2023-11-08T05:30:45.795085+0000","last_clean":"2023-11-08T05:30:45.795085+0000","last_became_active":"2023-11-08T05:30:45.794743+0000","last_became_peered":"2023-11-08T05:30:45.794743+0000","last_unstale":"2023-11-08T05:30:45.795085+0000","last_undegraded":"2023-11-08T05:30:45.795085+0000","last_fullsized":"2023-11-08T05:30:45.795085+0000","mapping_epoch":458,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":459,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":454,"reported_epoch":458,"state":"active+clean","last_fresh":"2023-11-08T05:30:44.490618+0000","last_change":"2023-11-08T04:52:47.548597+0000","last_active":"2023-11-08T05:30:44.490618+0000","last_peered":"2023-11-08T05:30:44.490618+0000","last_clean":"2023-11-08T05:30:44.490618+0000","last_became_active":"2023-11-08T04:52:47.548294+0000","last_became_peered":"2023-11-08T04:52:47.548294+0000","last_unstale":"2023-11-08T05:30:44.490618+0000","last_undegraded":"2023-11-08T05:30:44.490618+0000","last_fullsized":"2023-11-08T05:30:44.490618+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":460,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490523+0000","last_change":"2023-11-08T05:28:13.545468+0000","last_active":"2023-11-08T05:30:44.490523+0000","last_peered":"2023-11-08T05:30:44.490523+0000","last_clean":"2023-11-08T05:28:01.456723+0000","last_became_active":"2023-11-08T05:28:13.545468+0000","last_became_peered":"2023-11-08T05:28:13.545468+0000","last_unstale":"2023-11-08T05:30:44.490523+0000","last_undegraded":"2023-11-08T05:30:44.490523+0000","last_fullsized":"2023-11-08T05:28:13.537207+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_clean_scrub_stamp":"2023-11-08T04:52:45.528796+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":465,"reported_epoch":458,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:44.491907+0000","last_change":"2023-11-08T05:29:44.395350+0000","last_active":"2023-11-08T05:30:44.491907+0000","last_peered":"2023-11-08T05:30:44.491907+0000","last_clean":"2023-11-08T05:30:44.491907+0000","last_became_active":"2023-11-08T05:29:44.395012+0000","last_became_peered":"2023-11-08T05:29:44.395012+0000","last_unstale":"2023-11-08T05:30:44.491907+0000","last_undegraded":"2023-11-08T05:30:44.491907+0000","last_fullsized":"2023-11-08T05:30:44.491907+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490629+0000","last_change":"2023-11-08T05:28:13.563357+0000","last_active":"2023-11-08T05:30:44.490629+0000","last_peered":"2023-11-08T05:30:44.490629+0000","last_clean":"2023-11-08T05:28:01.457691+0000","last_became_active":"2023-11-08T05:28:13.563357+0000","last_became_peered":"2023-11-08T05:28:13.563357+0000","last_unstale":"2023-11-08T05:30:44.490629+0000","last_undegraded":"2023-11-08T05:30:44.490629+0000","last_fullsized":"2023-11-08T05:28:13.541614+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490691+0000","last_change":"2023-11-08T05:28:13.550672+0000","last_active":"2023-11-08T05:30:44.490691+0000","last_peered":"2023-11-08T05:30:44.490691+0000","last_clean":"2023-11-08T05:28:01.457904+0000","last_became_active":"2023-11-08T05:28:13.550672+0000","last_became_peered":"2023-11-08T05:28:13.550672+0000","last_unstale":"2023-11-08T05:30:44.490691+0000","last_undegraded":"2023-11-08T05:30:44.490691+0000","last_fullsized":"2023-11-08T05:28:13.539595+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490805+0000","last_change":"2023-11-08T05:28:13.557968+0000","last_active":"2023-11-08T05:30:44.490805+0000","last_peered":"2023-11-08T05:30:44.490805+0000","last_clean":"2023-11-08T05:28:01.458433+0000","last_became_active":"2023-11-08T05:28:13.557968+0000","last_became_peered":"2023-11-08T05:28:13.557968+0000","last_unstale":"2023-11-08T05:30:44.490805+0000","last_undegraded":"2023-11-08T05:30:44.490805+0000","last_fullsized":"2023-11-08T05:28:13.542832+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":1,"acting":1,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":12,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738834,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":645040,"kb_used_data":5352,"kb_used_omap":275,"kb_used_meta":639404,"kb_avail":93726800,"statfs":{"total":96636764160,"available":95976243200,"internally_reserved":0,"allocated":5480448,"data_stored":2697647,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":281650,"internal_metadata":654750670},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:59 2023","interfaces":[{"interface":"back","average":{"1min":0.30499999999999999,"5min":0.439,"15min":0.36099999999999999},"min":{"1min":0.17399999999999999,"5min":0.14499999999999999,"15min":0.14499999999999999},"max":{"1min":0.60099999999999998,"5min":4.9379999999999997,"15min":4.9379999999999997},"last":0.379},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.49399999999999999,"15min":0.42399999999999999},"min":{"1min":0.221,"5min":0.19600000000000001,"15min":0.17000000000000001},"max":{"1min":0.63200000000000001,"5min":4.9859999999999998,"15min":4.9859999999999998},"last":0.40899999999999997}]},{"osd":1,"last update":"Wed Nov 8 05:31:06 2023","interfaces":[{"interface":"back","average":{"1min":0.27500000000000002,"5min":0.40300000000000002,"15min":0.46999999999999997},"min":{"1min":0.20899999999999999,"5min":0.16,"15min":0.153},"max":{"1min":0.41099999999999998,"5min":4.0830000000000002,"15min":23.242999999999999},"last":0.26600000000000001},{"interface":"front","average":{"1min":0.29899999999999999,"5min":0.40100000000000002,"15min":0.46000000000000002},"min":{"1min":0.23799999999999999,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":0.45300000000000001,"5min":3.5590000000000002,"15min":23.251000000000001},"last":0.23200000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738798,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1339228,"kb_used_data":398304,"kb_used_omap":291,"kb_used_meta":936476,"kb_avail":93032612,"statfs":{"total":96636764160,"available":95265394688,"internally_reserved":0,"allocated":407863296,"data_stored":405103900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":298462,"internal_metadata":958951970},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.436,"15min":0.36299999999999999},"min":{"1min":0.19700000000000001,"5min":0.14699999999999999,"15min":0.13700000000000001},"max":{"1min":3.222,"5min":6.5499999999999998,"15min":6.9989999999999997},"last":0.245},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.46300000000000002,"15min":0.42299999999999999},"min":{"1min":0.17799999999999999,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":5.0030000000000001,"5min":6.0629999999999997,"15min":9.6869999999999994},"last":0.28599999999999998}]},{"osd":2,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.51600000000000001,"15min":0.47099999999999997},"min":{"1min":0.21099999999999999,"5min":0.153,"15min":0.153},"max":{"1min":5.7439999999999998,"5min":5.8280000000000003,"15min":9.7639999999999993},"last":0.252},{"interface":"front","average":{"1min":0.497,"5min":0.56200000000000006,"15min":0.42299999999999999},"min":{"1min":0.20399999999999999,"5min":0.187,"15min":0.16800000000000001},"max":{"1min":3.9319999999999999,"5min":8.8040000000000003,"15min":8.8040000000000003},"last":0.29899999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738836,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":650100,"kb_used_data":5356,"kb_used_omap":214,"kb_used_meta":644521,"kb_avail":93721740,"statfs":{"total":96636764160,"available":95971061760,"internally_reserved":0,"allocated":5484544,"data_stored":2698616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":219335,"internal_metadata":659990329},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:56 2023","interfaces":[{"interface":"back","average":{"1min":0.30399999999999999,"5min":0.34699999999999998,"15min":0.36499999999999999},"min":{"1min":0.23999999999999999,"5min":0.14199999999999999,"15min":0.14199999999999999},"max":{"1min":0.42399999999999999,"5min":1.595,"15min":7.4249999999999998},"last":0.17100000000000001},{"interface":"front","average":{"1min":0.35499999999999998,"5min":0.40899999999999997,"15min":0.376},"min":{"1min":0.23999999999999999,"5min":0.19600000000000001,"15min":0.183},"max":{"1min":0.46999999999999997,"5min":2.6859999999999999,"15min":2.6859999999999999},"last":0.23000000000000001}]},{"osd":2,"last update":"Wed Nov 8 05:30:57 2023","interfaces":[{"interface":"back","average":{"1min":0.28599999999999998,"5min":0.34699999999999998,"15min":0.36899999999999999},"min":{"1min":0.19700000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":0.41399999999999998,"5min":2.2650000000000001,"15min":5.8209999999999997},"last":0.29899999999999999},{"interface":"front","average":{"1min":0.30599999999999999,"5min":0.313,"15min":0.35799999999999998},"min":{"1min":0.17799999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.435,"5min":1.137,"15min":4.2350000000000003},"last":0.32300000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-08T05:31:24.435 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-11-08T05:31:24.435 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 459, 'reported_epoch': 458, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:30:44.490644+0000', 'last_change': '2023-11-08T05:28:13.557071+0000', 'last_active': '2023-11-08T05:30:44.490644+0000', 'last_peered': '2023-11-08T05:30:44.490644+0000', 'last_clean': '2023-11-08T05:28:01.458439+0000', 'last_became_active': '2023-11-08T05:28:13.557071+0000', 'last_became_peered': '2023-11-08T05:28:13.557071+0000', 'last_unstale': '2023-11-08T05:30:44.490644+0000', 'last_undegraded': '2023-11-08T05:30:44.490644+0000', 'last_fullsized': '2023-11-08T05:28:13.547025+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:31:24.435 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-11-08T05:31:24.436 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 11, 'reported_epoch': 459, 'state': 'active+clean+remapped', 'last_fresh': '2023-11-08T05:30:45.794760+0000', 'last_change': '2023-11-08T05:30:45.794760+0000', 'last_active': '2023-11-08T05:30:45.794760+0000', 'last_peered': '2023-11-08T05:30:45.794760+0000', 'last_clean': '2023-11-08T05:30:45.794760+0000', 'last_became_active': '2023-11-08T05:30:45.794398+0000', 'last_became_peered': '2023-11-08T05:30:45.794398+0000', 'last_unstale': '2023-11-08T05:30:45.794760+0000', 'last_undegraded': '2023-11-08T05:30:45.794760+0000', 'last_fullsized': '2023-11-08T05:30:45.794760+0000', 'mapping_epoch': 458, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 459, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2], 'acting': [2, 1], 'avail_no_missing': ['2', '1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2023-11-08T05:31:24.436 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.1 is not active+clean 2023-11-08T05:31:24.436 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.1', 'version': "0'0", 'reported_seq': 465, 'reported_epoch': 459, 'state': 'active+clean+remapped', 'last_fresh': '2023-11-08T05:30:45.795085+0000', 'last_change': '2023-11-08T05:30:45.795085+0000', 'last_active': '2023-11-08T05:30:45.795085+0000', 'last_peered': '2023-11-08T05:30:45.795085+0000', 'last_clean': '2023-11-08T05:30:45.795085+0000', 'last_became_active': '2023-11-08T05:30:45.794743+0000', 'last_became_peered': '2023-11-08T05:30:45.794743+0000', 'last_unstale': '2023-11-08T05:30:45.795085+0000', 'last_undegraded': '2023-11-08T05:30:45.795085+0000', 'last_fullsized': '2023-11-08T05:30:45.795085+0000', 'mapping_epoch': 458, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 459, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2], 'acting': [2, 1], 'avail_no_missing': ['2', '1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2023-11-08T05:31:24.437 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-11-08T05:31:24.437 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "0'0", 'reported_seq': 460, 'reported_epoch': 458, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:30:44.490523+0000', 'last_change': '2023-11-08T05:28:13.545468+0000', 'last_active': '2023-11-08T05:30:44.490523+0000', 'last_peered': '2023-11-08T05:30:44.490523+0000', 'last_clean': '2023-11-08T05:28:01.456723+0000', 'last_became_active': '2023-11-08T05:28:13.545468+0000', 'last_became_peered': '2023-11-08T05:28:13.545468+0000', 'last_unstale': '2023-11-08T05:30:44.490523+0000', 'last_undegraded': '2023-11-08T05:30:44.490523+0000', 'last_fullsized': '2023-11-08T05:28:13.537207+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 9, 'last_epoch_clean': 10, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:45.528796+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:45.528796+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:45.528796+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:31:24.438 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.0 is not active+clean 2023-11-08T05:31:24.438 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.0', 'version': "0'0", 'reported_seq': 465, 'reported_epoch': 458, 'state': 'active+clean+remapped', 'last_fresh': '2023-11-08T05:30:44.491907+0000', 'last_change': '2023-11-08T05:29:44.395350+0000', 'last_active': '2023-11-08T05:30:44.491907+0000', 'last_peered': '2023-11-08T05:30:44.491907+0000', 'last_clean': '2023-11-08T05:30:44.491907+0000', 'last_became_active': '2023-11-08T05:29:44.395012+0000', 'last_became_peered': '2023-11-08T05:29:44.395012+0000', 'last_unstale': '2023-11-08T05:30:44.491907+0000', 'last_undegraded': '2023-11-08T05:30:44.491907+0000', 'last_fullsized': '2023-11-08T05:30:44.491907+0000', 'mapping_epoch': 455, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 456, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2], 'acting': [2, 1], 'avail_no_missing': ['2', '1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2023-11-08T05:31:24.438 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-11-08T05:31:24.439 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 459, 'reported_epoch': 458, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:30:44.490629+0000', 'last_change': '2023-11-08T05:28:13.563357+0000', 'last_active': '2023-11-08T05:30:44.490629+0000', 'last_peered': '2023-11-08T05:30:44.490629+0000', 'last_clean': '2023-11-08T05:28:01.457691+0000', 'last_became_active': '2023-11-08T05:28:13.563357+0000', 'last_became_peered': '2023-11-08T05:28:13.563357+0000', 'last_unstale': '2023-11-08T05:30:44.490629+0000', 'last_undegraded': '2023-11-08T05:30:44.490629+0000', 'last_fullsized': '2023-11-08T05:28:13.541614+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:31:24.439 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-11-08T05:31:24.439 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 459, 'reported_epoch': 458, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:30:44.490691+0000', 'last_change': '2023-11-08T05:28:13.550672+0000', 'last_active': '2023-11-08T05:30:44.490691+0000', 'last_peered': '2023-11-08T05:30:44.490691+0000', 'last_clean': '2023-11-08T05:28:01.457904+0000', 'last_became_active': '2023-11-08T05:28:13.550672+0000', 'last_became_peered': '2023-11-08T05:28:13.550672+0000', 'last_unstale': '2023-11-08T05:30:44.490691+0000', 'last_undegraded': '2023-11-08T05:30:44.490691+0000', 'last_fullsized': '2023-11-08T05:28:13.539595+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:31:24.440 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-11-08T05:31:24.440 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 459, 'reported_epoch': 458, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:30:44.490805+0000', 'last_change': '2023-11-08T05:28:13.557968+0000', 'last_active': '2023-11-08T05:30:44.490805+0000', 'last_peered': '2023-11-08T05:30:44.490805+0000', 'last_clean': '2023-11-08T05:28:01.458433+0000', 'last_became_active': '2023-11-08T05:28:13.557968+0000', 'last_became_peered': '2023-11-08T05:28:13.557968+0000', 'last_unstale': '2023-11-08T05:30:44.490805+0000', 'last_undegraded': '2023-11-08T05:30:44.490805+0000', 'last_fullsized': '2023-11-08T05:28:13.542832+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:31:24.440 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-11-08T05:31:24.787 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T05:31:24.788 INFO:teuthology.orchestra.run.smithi087.stdout:{"fsid":"ab4cbc06-449d-497d-b513-220d9db2b590","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 5 pgs undersized","count":5},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2327,"monmap":{"epoch":1,"min_mon_release_name":"pacific","num_mons":1},"osdmap":{"epoch":459,"num_osds":3,"num_up_osds":2,"osd_up_since":1699421292,"num_in_osds":3,"osd_in_since":1699419158,"num_remapped_pgs":3},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean+remapped","count":3},{"state_name":"active+clean","count":1}],"num_pgs":9,"num_pools":2,"num_objects":0,"data_bytes":0,"bytes_used":2697592832,"bytes_avail":287212699648,"bytes_total":289910292480},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":41,"modified":"2023-11-08T05:29:32.044544+0000","services":{}},"progress_events":{}} 2023-11-08T05:31:24.800 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-11-08T05:31:24.801 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-11-08T05:31:25.084 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T05:31:25.086 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2023-11-08T05:31:25.097 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":1616,"stamp":"2023-11-08T05:31:24.067620+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":10,"acting":13,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":109,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2634368,"kb_used_data":409012,"kb_used_omap":780,"kb_used_meta":2220403,"kb_avail":280481152,"statfs":{"total":289910292480,"available":287212699648,"internally_reserved":0,"allocated":418828288,"data_stored":410500163,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":799447,"internal_metadata":2273692969},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-536782902,"num_objects":-164,"num_object_clones":0,"num_object_copies":-328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-115,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-164,"num_whiteouts":0,"num_read":-122056,"num_read_kb":-39011942,"num_write":-413642,"num_write_kb":-151705339,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-8,"num_bytes_recovered":-16922624,"num_keys_recovered":-2,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.424817"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490644+0000","last_change":"2023-11-08T05:28:13.557071+0000","last_active":"2023-11-08T05:30:44.490644+0000","last_peered":"2023-11-08T05:30:44.490644+0000","last_clean":"2023-11-08T05:28:01.458439+0000","last_became_active":"2023-11-08T05:28:13.557071+0000","last_became_peered":"2023-11-08T05:28:13.557071+0000","last_unstale":"2023-11-08T05:30:44.490644+0000","last_undegraded":"2023-11-08T05:30:44.490644+0000","last_fullsized":"2023-11-08T05:28:13.547025+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":459,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:45.794760+0000","last_change":"2023-11-08T05:30:45.794760+0000","last_active":"2023-11-08T05:30:45.794760+0000","last_peered":"2023-11-08T05:30:45.794760+0000","last_clean":"2023-11-08T05:30:45.794760+0000","last_became_active":"2023-11-08T05:30:45.794398+0000","last_became_peered":"2023-11-08T05:30:45.794398+0000","last_unstale":"2023-11-08T05:30:45.794760+0000","last_undegraded":"2023-11-08T05:30:45.794760+0000","last_fullsized":"2023-11-08T05:30:45.794760+0000","mapping_epoch":458,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":459,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":465,"reported_epoch":459,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:45.795085+0000","last_change":"2023-11-08T05:30:45.795085+0000","last_active":"2023-11-08T05:30:45.795085+0000","last_peered":"2023-11-08T05:30:45.795085+0000","last_clean":"2023-11-08T05:30:45.795085+0000","last_became_active":"2023-11-08T05:30:45.794743+0000","last_became_peered":"2023-11-08T05:30:45.794743+0000","last_unstale":"2023-11-08T05:30:45.795085+0000","last_undegraded":"2023-11-08T05:30:45.795085+0000","last_fullsized":"2023-11-08T05:30:45.795085+0000","mapping_epoch":458,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":459,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":454,"reported_epoch":458,"state":"active+clean","last_fresh":"2023-11-08T05:30:44.490618+0000","last_change":"2023-11-08T04:52:47.548597+0000","last_active":"2023-11-08T05:30:44.490618+0000","last_peered":"2023-11-08T05:30:44.490618+0000","last_clean":"2023-11-08T05:30:44.490618+0000","last_became_active":"2023-11-08T04:52:47.548294+0000","last_became_peered":"2023-11-08T04:52:47.548294+0000","last_unstale":"2023-11-08T05:30:44.490618+0000","last_undegraded":"2023-11-08T05:30:44.490618+0000","last_fullsized":"2023-11-08T05:30:44.490618+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":460,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490523+0000","last_change":"2023-11-08T05:28:13.545468+0000","last_active":"2023-11-08T05:30:44.490523+0000","last_peered":"2023-11-08T05:30:44.490523+0000","last_clean":"2023-11-08T05:28:01.456723+0000","last_became_active":"2023-11-08T05:28:13.545468+0000","last_became_peered":"2023-11-08T05:28:13.545468+0000","last_unstale":"2023-11-08T05:30:44.490523+0000","last_undegraded":"2023-11-08T05:30:44.490523+0000","last_fullsized":"2023-11-08T05:28:13.537207+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_clean_scrub_stamp":"2023-11-08T04:52:45.528796+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":465,"reported_epoch":458,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:44.491907+0000","last_change":"2023-11-08T05:29:44.395350+0000","last_active":"2023-11-08T05:30:44.491907+0000","last_peered":"2023-11-08T05:30:44.491907+0000","last_clean":"2023-11-08T05:30:44.491907+0000","last_became_active":"2023-11-08T05:29:44.395012+0000","last_became_peered":"2023-11-08T05:29:44.395012+0000","last_unstale":"2023-11-08T05:30:44.491907+0000","last_undegraded":"2023-11-08T05:30:44.491907+0000","last_fullsized":"2023-11-08T05:30:44.491907+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490629+0000","last_change":"2023-11-08T05:28:13.563357+0000","last_active":"2023-11-08T05:30:44.490629+0000","last_peered":"2023-11-08T05:30:44.490629+0000","last_clean":"2023-11-08T05:28:01.457691+0000","last_became_active":"2023-11-08T05:28:13.563357+0000","last_became_peered":"2023-11-08T05:28:13.563357+0000","last_unstale":"2023-11-08T05:30:44.490629+0000","last_undegraded":"2023-11-08T05:30:44.490629+0000","last_fullsized":"2023-11-08T05:28:13.541614+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490691+0000","last_change":"2023-11-08T05:28:13.550672+0000","last_active":"2023-11-08T05:30:44.490691+0000","last_peered":"2023-11-08T05:30:44.490691+0000","last_clean":"2023-11-08T05:28:01.457904+0000","last_became_active":"2023-11-08T05:28:13.550672+0000","last_became_peered":"2023-11-08T05:28:13.550672+0000","last_unstale":"2023-11-08T05:30:44.490691+0000","last_undegraded":"2023-11-08T05:30:44.490691+0000","last_fullsized":"2023-11-08T05:28:13.539595+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490805+0000","last_change":"2023-11-08T05:28:13.557968+0000","last_active":"2023-11-08T05:30:44.490805+0000","last_peered":"2023-11-08T05:30:44.490805+0000","last_clean":"2023-11-08T05:28:01.458433+0000","last_became_active":"2023-11-08T05:28:13.557968+0000","last_became_peered":"2023-11-08T05:28:13.557968+0000","last_unstale":"2023-11-08T05:30:44.490805+0000","last_undegraded":"2023-11-08T05:30:44.490805+0000","last_fullsized":"2023-11-08T05:28:13.542832+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":1,"acting":1,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":12,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738834,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":645040,"kb_used_data":5352,"kb_used_omap":275,"kb_used_meta":639404,"kb_avail":93726800,"statfs":{"total":96636764160,"available":95976243200,"internally_reserved":0,"allocated":5480448,"data_stored":2697647,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":281650,"internal_metadata":654750670},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:59 2023","interfaces":[{"interface":"back","average":{"1min":0.30499999999999999,"5min":0.439,"15min":0.36099999999999999},"min":{"1min":0.17399999999999999,"5min":0.14499999999999999,"15min":0.14499999999999999},"max":{"1min":0.60099999999999998,"5min":4.9379999999999997,"15min":4.9379999999999997},"last":0.379},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.49399999999999999,"15min":0.42399999999999999},"min":{"1min":0.221,"5min":0.19600000000000001,"15min":0.17000000000000001},"max":{"1min":0.63200000000000001,"5min":4.9859999999999998,"15min":4.9859999999999998},"last":0.40899999999999997}]},{"osd":1,"last update":"Wed Nov 8 05:31:06 2023","interfaces":[{"interface":"back","average":{"1min":0.27500000000000002,"5min":0.40300000000000002,"15min":0.46999999999999997},"min":{"1min":0.20899999999999999,"5min":0.16,"15min":0.153},"max":{"1min":0.41099999999999998,"5min":4.0830000000000002,"15min":23.242999999999999},"last":0.26600000000000001},{"interface":"front","average":{"1min":0.29899999999999999,"5min":0.40100000000000002,"15min":0.46000000000000002},"min":{"1min":0.23799999999999999,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":0.45300000000000001,"5min":3.5590000000000002,"15min":23.251000000000001},"last":0.23200000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738798,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1339228,"kb_used_data":398304,"kb_used_omap":291,"kb_used_meta":936476,"kb_avail":93032612,"statfs":{"total":96636764160,"available":95265394688,"internally_reserved":0,"allocated":407863296,"data_stored":405103900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":298462,"internal_metadata":958951970},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.436,"15min":0.36299999999999999},"min":{"1min":0.19700000000000001,"5min":0.14699999999999999,"15min":0.13700000000000001},"max":{"1min":3.222,"5min":6.5499999999999998,"15min":6.9989999999999997},"last":0.245},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.46300000000000002,"15min":0.42299999999999999},"min":{"1min":0.17799999999999999,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":5.0030000000000001,"5min":6.0629999999999997,"15min":9.6869999999999994},"last":0.28599999999999998}]},{"osd":2,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.51600000000000001,"15min":0.47099999999999997},"min":{"1min":0.21099999999999999,"5min":0.153,"15min":0.153},"max":{"1min":5.7439999999999998,"5min":5.8280000000000003,"15min":9.7639999999999993},"last":0.252},{"interface":"front","average":{"1min":0.497,"5min":0.56200000000000006,"15min":0.42299999999999999},"min":{"1min":0.20399999999999999,"5min":0.187,"15min":0.16800000000000001},"max":{"1min":3.9319999999999999,"5min":8.8040000000000003,"15min":8.8040000000000003},"last":0.29899999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738836,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":650100,"kb_used_data":5356,"kb_used_omap":214,"kb_used_meta":644521,"kb_avail":93721740,"statfs":{"total":96636764160,"available":95971061760,"internally_reserved":0,"allocated":5484544,"data_stored":2698616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":219335,"internal_metadata":659990329},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:56 2023","interfaces":[{"interface":"back","average":{"1min":0.30399999999999999,"5min":0.34699999999999998,"15min":0.36499999999999999},"min":{"1min":0.23999999999999999,"5min":0.14199999999999999,"15min":0.14199999999999999},"max":{"1min":0.42399999999999999,"5min":1.595,"15min":7.4249999999999998},"last":0.17100000000000001},{"interface":"front","average":{"1min":0.35499999999999998,"5min":0.40899999999999997,"15min":0.376},"min":{"1min":0.23999999999999999,"5min":0.19600000000000001,"15min":0.183},"max":{"1min":0.46999999999999997,"5min":2.6859999999999999,"15min":2.6859999999999999},"last":0.23000000000000001}]},{"osd":2,"last update":"Wed Nov 8 05:30:57 2023","interfaces":[{"interface":"back","average":{"1min":0.28599999999999998,"5min":0.34699999999999998,"15min":0.36899999999999999},"min":{"1min":0.19700000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":0.41399999999999998,"5min":2.2650000000000001,"15min":5.8209999999999997},"last":0.29899999999999999},{"interface":"front","average":{"1min":0.30599999999999999,"5min":0.313,"15min":0.35799999999999998},"min":{"1min":0.17799999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.435,"5min":1.137,"15min":4.2350000000000003},"last":0.32300000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-08T05:31:27.188 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~189s 2023-11-08T05:31:28.100 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-11-08T05:31:28.393 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T05:31:28.395 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2023-11-08T05:31:28.406 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":1618,"stamp":"2023-11-08T05:31:28.068516+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":10,"acting":13,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":109,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2634368,"kb_used_data":409012,"kb_used_omap":780,"kb_used_meta":2220403,"kb_avail":280481152,"statfs":{"total":289910292480,"available":287212699648,"internally_reserved":0,"allocated":418828288,"data_stored":410500163,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":799447,"internal_metadata":2273692969},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-536782902,"num_objects":-164,"num_object_clones":0,"num_object_copies":-328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-115,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-164,"num_whiteouts":0,"num_read":-122056,"num_read_kb":-39011942,"num_write":-413642,"num_write_kb":-151705339,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-8,"num_bytes_recovered":-16922624,"num_keys_recovered":-2,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.424817"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490644+0000","last_change":"2023-11-08T05:28:13.557071+0000","last_active":"2023-11-08T05:30:44.490644+0000","last_peered":"2023-11-08T05:30:44.490644+0000","last_clean":"2023-11-08T05:28:01.458439+0000","last_became_active":"2023-11-08T05:28:13.557071+0000","last_became_peered":"2023-11-08T05:28:13.557071+0000","last_unstale":"2023-11-08T05:30:44.490644+0000","last_undegraded":"2023-11-08T05:30:44.490644+0000","last_fullsized":"2023-11-08T05:28:13.547025+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":459,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:45.794760+0000","last_change":"2023-11-08T05:30:45.794760+0000","last_active":"2023-11-08T05:30:45.794760+0000","last_peered":"2023-11-08T05:30:45.794760+0000","last_clean":"2023-11-08T05:30:45.794760+0000","last_became_active":"2023-11-08T05:30:45.794398+0000","last_became_peered":"2023-11-08T05:30:45.794398+0000","last_unstale":"2023-11-08T05:30:45.794760+0000","last_undegraded":"2023-11-08T05:30:45.794760+0000","last_fullsized":"2023-11-08T05:30:45.794760+0000","mapping_epoch":458,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":459,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":465,"reported_epoch":459,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:45.795085+0000","last_change":"2023-11-08T05:30:45.795085+0000","last_active":"2023-11-08T05:30:45.795085+0000","last_peered":"2023-11-08T05:30:45.795085+0000","last_clean":"2023-11-08T05:30:45.795085+0000","last_became_active":"2023-11-08T05:30:45.794743+0000","last_became_peered":"2023-11-08T05:30:45.794743+0000","last_unstale":"2023-11-08T05:30:45.795085+0000","last_undegraded":"2023-11-08T05:30:45.795085+0000","last_fullsized":"2023-11-08T05:30:45.795085+0000","mapping_epoch":458,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":459,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":454,"reported_epoch":458,"state":"active+clean","last_fresh":"2023-11-08T05:30:44.490618+0000","last_change":"2023-11-08T04:52:47.548597+0000","last_active":"2023-11-08T05:30:44.490618+0000","last_peered":"2023-11-08T05:30:44.490618+0000","last_clean":"2023-11-08T05:30:44.490618+0000","last_became_active":"2023-11-08T04:52:47.548294+0000","last_became_peered":"2023-11-08T04:52:47.548294+0000","last_unstale":"2023-11-08T05:30:44.490618+0000","last_undegraded":"2023-11-08T05:30:44.490618+0000","last_fullsized":"2023-11-08T05:30:44.490618+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":460,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490523+0000","last_change":"2023-11-08T05:28:13.545468+0000","last_active":"2023-11-08T05:30:44.490523+0000","last_peered":"2023-11-08T05:30:44.490523+0000","last_clean":"2023-11-08T05:28:01.456723+0000","last_became_active":"2023-11-08T05:28:13.545468+0000","last_became_peered":"2023-11-08T05:28:13.545468+0000","last_unstale":"2023-11-08T05:30:44.490523+0000","last_undegraded":"2023-11-08T05:30:44.490523+0000","last_fullsized":"2023-11-08T05:28:13.537207+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_clean_scrub_stamp":"2023-11-08T04:52:45.528796+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":465,"reported_epoch":458,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:44.491907+0000","last_change":"2023-11-08T05:29:44.395350+0000","last_active":"2023-11-08T05:30:44.491907+0000","last_peered":"2023-11-08T05:30:44.491907+0000","last_clean":"2023-11-08T05:30:44.491907+0000","last_became_active":"2023-11-08T05:29:44.395012+0000","last_became_peered":"2023-11-08T05:29:44.395012+0000","last_unstale":"2023-11-08T05:30:44.491907+0000","last_undegraded":"2023-11-08T05:30:44.491907+0000","last_fullsized":"2023-11-08T05:30:44.491907+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490629+0000","last_change":"2023-11-08T05:28:13.563357+0000","last_active":"2023-11-08T05:30:44.490629+0000","last_peered":"2023-11-08T05:30:44.490629+0000","last_clean":"2023-11-08T05:28:01.457691+0000","last_became_active":"2023-11-08T05:28:13.563357+0000","last_became_peered":"2023-11-08T05:28:13.563357+0000","last_unstale":"2023-11-08T05:30:44.490629+0000","last_undegraded":"2023-11-08T05:30:44.490629+0000","last_fullsized":"2023-11-08T05:28:13.541614+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490691+0000","last_change":"2023-11-08T05:28:13.550672+0000","last_active":"2023-11-08T05:30:44.490691+0000","last_peered":"2023-11-08T05:30:44.490691+0000","last_clean":"2023-11-08T05:28:01.457904+0000","last_became_active":"2023-11-08T05:28:13.550672+0000","last_became_peered":"2023-11-08T05:28:13.550672+0000","last_unstale":"2023-11-08T05:30:44.490691+0000","last_undegraded":"2023-11-08T05:30:44.490691+0000","last_fullsized":"2023-11-08T05:28:13.539595+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490805+0000","last_change":"2023-11-08T05:28:13.557968+0000","last_active":"2023-11-08T05:30:44.490805+0000","last_peered":"2023-11-08T05:30:44.490805+0000","last_clean":"2023-11-08T05:28:01.458433+0000","last_became_active":"2023-11-08T05:28:13.557968+0000","last_became_peered":"2023-11-08T05:28:13.557968+0000","last_unstale":"2023-11-08T05:30:44.490805+0000","last_undegraded":"2023-11-08T05:30:44.490805+0000","last_fullsized":"2023-11-08T05:28:13.542832+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":1,"acting":1,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":12,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738835,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":645040,"kb_used_data":5352,"kb_used_omap":275,"kb_used_meta":639404,"kb_avail":93726800,"statfs":{"total":96636764160,"available":95976243200,"internally_reserved":0,"allocated":5480448,"data_stored":2697647,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":281650,"internal_metadata":654750670},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:59 2023","interfaces":[{"interface":"back","average":{"1min":0.30499999999999999,"5min":0.439,"15min":0.36099999999999999},"min":{"1min":0.17399999999999999,"5min":0.14499999999999999,"15min":0.14499999999999999},"max":{"1min":0.60099999999999998,"5min":4.9379999999999997,"15min":4.9379999999999997},"last":0.379},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.49399999999999999,"15min":0.42399999999999999},"min":{"1min":0.221,"5min":0.19600000000000001,"15min":0.17000000000000001},"max":{"1min":0.63200000000000001,"5min":4.9859999999999998,"15min":4.9859999999999998},"last":0.40899999999999997}]},{"osd":1,"last update":"Wed Nov 8 05:31:06 2023","interfaces":[{"interface":"back","average":{"1min":0.27500000000000002,"5min":0.40300000000000002,"15min":0.46999999999999997},"min":{"1min":0.20899999999999999,"5min":0.16,"15min":0.153},"max":{"1min":0.41099999999999998,"5min":4.0830000000000002,"15min":23.242999999999999},"last":0.19800000000000001},{"interface":"front","average":{"1min":0.29899999999999999,"5min":0.40100000000000002,"15min":0.46000000000000002},"min":{"1min":0.23799999999999999,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":0.45300000000000001,"5min":3.5590000000000002,"15min":23.251000000000001},"last":0.20999999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738798,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1339228,"kb_used_data":398304,"kb_used_omap":291,"kb_used_meta":936476,"kb_avail":93032612,"statfs":{"total":96636764160,"available":95265394688,"internally_reserved":0,"allocated":407863296,"data_stored":405103900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":298462,"internal_metadata":958951970},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.436,"15min":0.36299999999999999},"min":{"1min":0.19700000000000001,"5min":0.14699999999999999,"15min":0.13700000000000001},"max":{"1min":3.222,"5min":6.5499999999999998,"15min":6.9989999999999997},"last":0.245},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.46300000000000002,"15min":0.42299999999999999},"min":{"1min":0.17799999999999999,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":5.0030000000000001,"5min":6.0629999999999997,"15min":9.6869999999999994},"last":0.28599999999999998}]},{"osd":2,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.51600000000000001,"15min":0.47099999999999997},"min":{"1min":0.21099999999999999,"5min":0.153,"15min":0.153},"max":{"1min":5.7439999999999998,"5min":5.8280000000000003,"15min":9.7639999999999993},"last":0.252},{"interface":"front","average":{"1min":0.497,"5min":0.56200000000000006,"15min":0.42299999999999999},"min":{"1min":0.20399999999999999,"5min":0.187,"15min":0.16800000000000001},"max":{"1min":3.9319999999999999,"5min":8.8040000000000003,"15min":8.8040000000000003},"last":0.29899999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738837,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":650100,"kb_used_data":5356,"kb_used_omap":214,"kb_used_meta":644521,"kb_avail":93721740,"statfs":{"total":96636764160,"available":95971061760,"internally_reserved":0,"allocated":5484544,"data_stored":2698616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":219335,"internal_metadata":659990329},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:56 2023","interfaces":[{"interface":"back","average":{"1min":0.30399999999999999,"5min":0.34699999999999998,"15min":0.36499999999999999},"min":{"1min":0.23999999999999999,"5min":0.14199999999999999,"15min":0.14199999999999999},"max":{"1min":0.42399999999999999,"5min":1.595,"15min":7.4249999999999998},"last":0.17100000000000001},{"interface":"front","average":{"1min":0.35499999999999998,"5min":0.40899999999999997,"15min":0.376},"min":{"1min":0.23999999999999999,"5min":0.19600000000000001,"15min":0.183},"max":{"1min":0.46999999999999997,"5min":2.6859999999999999,"15min":2.6859999999999999},"last":0.23000000000000001}]},{"osd":2,"last update":"Wed Nov 8 05:30:57 2023","interfaces":[{"interface":"back","average":{"1min":0.28599999999999998,"5min":0.34699999999999998,"15min":0.36899999999999999},"min":{"1min":0.19700000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":0.41399999999999998,"5min":2.2650000000000001,"15min":5.8209999999999997},"last":0.251},{"interface":"front","average":{"1min":0.30599999999999999,"5min":0.313,"15min":0.35799999999999998},"min":{"1min":0.17799999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.435,"5min":1.137,"15min":4.2350000000000003},"last":0.28000000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-08T05:31:28.407 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-11-08T05:31:28.692 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T05:31:28.694 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2023-11-08T05:31:28.705 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":1618,"stamp":"2023-11-08T05:31:28.068516+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":10,"acting":13,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":109,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2634368,"kb_used_data":409012,"kb_used_omap":780,"kb_used_meta":2220403,"kb_avail":280481152,"statfs":{"total":289910292480,"available":287212699648,"internally_reserved":0,"allocated":418828288,"data_stored":410500163,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":799447,"internal_metadata":2273692969},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-536782902,"num_objects":-164,"num_object_clones":0,"num_object_copies":-328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-115,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-164,"num_whiteouts":0,"num_read":-122056,"num_read_kb":-39011942,"num_write":-413642,"num_write_kb":-151705339,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-8,"num_bytes_recovered":-16922624,"num_keys_recovered":-2,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.424817"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490644+0000","last_change":"2023-11-08T05:28:13.557071+0000","last_active":"2023-11-08T05:30:44.490644+0000","last_peered":"2023-11-08T05:30:44.490644+0000","last_clean":"2023-11-08T05:28:01.458439+0000","last_became_active":"2023-11-08T05:28:13.557071+0000","last_became_peered":"2023-11-08T05:28:13.557071+0000","last_unstale":"2023-11-08T05:30:44.490644+0000","last_undegraded":"2023-11-08T05:30:44.490644+0000","last_fullsized":"2023-11-08T05:28:13.547025+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":459,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:45.794760+0000","last_change":"2023-11-08T05:30:45.794760+0000","last_active":"2023-11-08T05:30:45.794760+0000","last_peered":"2023-11-08T05:30:45.794760+0000","last_clean":"2023-11-08T05:30:45.794760+0000","last_became_active":"2023-11-08T05:30:45.794398+0000","last_became_peered":"2023-11-08T05:30:45.794398+0000","last_unstale":"2023-11-08T05:30:45.794760+0000","last_undegraded":"2023-11-08T05:30:45.794760+0000","last_fullsized":"2023-11-08T05:30:45.794760+0000","mapping_epoch":458,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":459,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":465,"reported_epoch":459,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:45.795085+0000","last_change":"2023-11-08T05:30:45.795085+0000","last_active":"2023-11-08T05:30:45.795085+0000","last_peered":"2023-11-08T05:30:45.795085+0000","last_clean":"2023-11-08T05:30:45.795085+0000","last_became_active":"2023-11-08T05:30:45.794743+0000","last_became_peered":"2023-11-08T05:30:45.794743+0000","last_unstale":"2023-11-08T05:30:45.795085+0000","last_undegraded":"2023-11-08T05:30:45.795085+0000","last_fullsized":"2023-11-08T05:30:45.795085+0000","mapping_epoch":458,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":459,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":454,"reported_epoch":458,"state":"active+clean","last_fresh":"2023-11-08T05:30:44.490618+0000","last_change":"2023-11-08T04:52:47.548597+0000","last_active":"2023-11-08T05:30:44.490618+0000","last_peered":"2023-11-08T05:30:44.490618+0000","last_clean":"2023-11-08T05:30:44.490618+0000","last_became_active":"2023-11-08T04:52:47.548294+0000","last_became_peered":"2023-11-08T04:52:47.548294+0000","last_unstale":"2023-11-08T05:30:44.490618+0000","last_undegraded":"2023-11-08T05:30:44.490618+0000","last_fullsized":"2023-11-08T05:30:44.490618+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":460,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490523+0000","last_change":"2023-11-08T05:28:13.545468+0000","last_active":"2023-11-08T05:30:44.490523+0000","last_peered":"2023-11-08T05:30:44.490523+0000","last_clean":"2023-11-08T05:28:01.456723+0000","last_became_active":"2023-11-08T05:28:13.545468+0000","last_became_peered":"2023-11-08T05:28:13.545468+0000","last_unstale":"2023-11-08T05:30:44.490523+0000","last_undegraded":"2023-11-08T05:30:44.490523+0000","last_fullsized":"2023-11-08T05:28:13.537207+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_clean_scrub_stamp":"2023-11-08T04:52:45.528796+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":465,"reported_epoch":458,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:44.491907+0000","last_change":"2023-11-08T05:29:44.395350+0000","last_active":"2023-11-08T05:30:44.491907+0000","last_peered":"2023-11-08T05:30:44.491907+0000","last_clean":"2023-11-08T05:30:44.491907+0000","last_became_active":"2023-11-08T05:29:44.395012+0000","last_became_peered":"2023-11-08T05:29:44.395012+0000","last_unstale":"2023-11-08T05:30:44.491907+0000","last_undegraded":"2023-11-08T05:30:44.491907+0000","last_fullsized":"2023-11-08T05:30:44.491907+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490629+0000","last_change":"2023-11-08T05:28:13.563357+0000","last_active":"2023-11-08T05:30:44.490629+0000","last_peered":"2023-11-08T05:30:44.490629+0000","last_clean":"2023-11-08T05:28:01.457691+0000","last_became_active":"2023-11-08T05:28:13.563357+0000","last_became_peered":"2023-11-08T05:28:13.563357+0000","last_unstale":"2023-11-08T05:30:44.490629+0000","last_undegraded":"2023-11-08T05:30:44.490629+0000","last_fullsized":"2023-11-08T05:28:13.541614+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490691+0000","last_change":"2023-11-08T05:28:13.550672+0000","last_active":"2023-11-08T05:30:44.490691+0000","last_peered":"2023-11-08T05:30:44.490691+0000","last_clean":"2023-11-08T05:28:01.457904+0000","last_became_active":"2023-11-08T05:28:13.550672+0000","last_became_peered":"2023-11-08T05:28:13.550672+0000","last_unstale":"2023-11-08T05:30:44.490691+0000","last_undegraded":"2023-11-08T05:30:44.490691+0000","last_fullsized":"2023-11-08T05:28:13.539595+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490805+0000","last_change":"2023-11-08T05:28:13.557968+0000","last_active":"2023-11-08T05:30:44.490805+0000","last_peered":"2023-11-08T05:30:44.490805+0000","last_clean":"2023-11-08T05:28:01.458433+0000","last_became_active":"2023-11-08T05:28:13.557968+0000","last_became_peered":"2023-11-08T05:28:13.557968+0000","last_unstale":"2023-11-08T05:30:44.490805+0000","last_undegraded":"2023-11-08T05:30:44.490805+0000","last_fullsized":"2023-11-08T05:28:13.542832+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":1,"acting":1,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":12,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738835,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":645040,"kb_used_data":5352,"kb_used_omap":275,"kb_used_meta":639404,"kb_avail":93726800,"statfs":{"total":96636764160,"available":95976243200,"internally_reserved":0,"allocated":5480448,"data_stored":2697647,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":281650,"internal_metadata":654750670},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:59 2023","interfaces":[{"interface":"back","average":{"1min":0.30499999999999999,"5min":0.439,"15min":0.36099999999999999},"min":{"1min":0.17399999999999999,"5min":0.14499999999999999,"15min":0.14499999999999999},"max":{"1min":0.60099999999999998,"5min":4.9379999999999997,"15min":4.9379999999999997},"last":0.379},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.49399999999999999,"15min":0.42399999999999999},"min":{"1min":0.221,"5min":0.19600000000000001,"15min":0.17000000000000001},"max":{"1min":0.63200000000000001,"5min":4.9859999999999998,"15min":4.9859999999999998},"last":0.40899999999999997}]},{"osd":1,"last update":"Wed Nov 8 05:31:06 2023","interfaces":[{"interface":"back","average":{"1min":0.27500000000000002,"5min":0.40300000000000002,"15min":0.46999999999999997},"min":{"1min":0.20899999999999999,"5min":0.16,"15min":0.153},"max":{"1min":0.41099999999999998,"5min":4.0830000000000002,"15min":23.242999999999999},"last":0.19800000000000001},{"interface":"front","average":{"1min":0.29899999999999999,"5min":0.40100000000000002,"15min":0.46000000000000002},"min":{"1min":0.23799999999999999,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":0.45300000000000001,"5min":3.5590000000000002,"15min":23.251000000000001},"last":0.20999999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738798,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1339228,"kb_used_data":398304,"kb_used_omap":291,"kb_used_meta":936476,"kb_avail":93032612,"statfs":{"total":96636764160,"available":95265394688,"internally_reserved":0,"allocated":407863296,"data_stored":405103900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":298462,"internal_metadata":958951970},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.436,"15min":0.36299999999999999},"min":{"1min":0.19700000000000001,"5min":0.14699999999999999,"15min":0.13700000000000001},"max":{"1min":3.222,"5min":6.5499999999999998,"15min":6.9989999999999997},"last":0.245},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.46300000000000002,"15min":0.42299999999999999},"min":{"1min":0.17799999999999999,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":5.0030000000000001,"5min":6.0629999999999997,"15min":9.6869999999999994},"last":0.28599999999999998}]},{"osd":2,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.51600000000000001,"15min":0.47099999999999997},"min":{"1min":0.21099999999999999,"5min":0.153,"15min":0.153},"max":{"1min":5.7439999999999998,"5min":5.8280000000000003,"15min":9.7639999999999993},"last":0.252},{"interface":"front","average":{"1min":0.497,"5min":0.56200000000000006,"15min":0.42299999999999999},"min":{"1min":0.20399999999999999,"5min":0.187,"15min":0.16800000000000001},"max":{"1min":3.9319999999999999,"5min":8.8040000000000003,"15min":8.8040000000000003},"last":0.29899999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738837,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":650100,"kb_used_data":5356,"kb_used_omap":214,"kb_used_meta":644521,"kb_avail":93721740,"statfs":{"total":96636764160,"available":95971061760,"internally_reserved":0,"allocated":5484544,"data_stored":2698616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":219335,"internal_metadata":659990329},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:56 2023","interfaces":[{"interface":"back","average":{"1min":0.30399999999999999,"5min":0.34699999999999998,"15min":0.36499999999999999},"min":{"1min":0.23999999999999999,"5min":0.14199999999999999,"15min":0.14199999999999999},"max":{"1min":0.42399999999999999,"5min":1.595,"15min":7.4249999999999998},"last":0.17100000000000001},{"interface":"front","average":{"1min":0.35499999999999998,"5min":0.40899999999999997,"15min":0.376},"min":{"1min":0.23999999999999999,"5min":0.19600000000000001,"15min":0.183},"max":{"1min":0.46999999999999997,"5min":2.6859999999999999,"15min":2.6859999999999999},"last":0.23000000000000001}]},{"osd":2,"last update":"Wed Nov 8 05:30:57 2023","interfaces":[{"interface":"back","average":{"1min":0.28599999999999998,"5min":0.34699999999999998,"15min":0.36899999999999999},"min":{"1min":0.19700000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":0.41399999999999998,"5min":2.2650000000000001,"15min":5.8209999999999997},"last":0.251},{"interface":"front","average":{"1min":0.30599999999999999,"5min":0.313,"15min":0.35799999999999998},"min":{"1min":0.17799999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.435,"5min":1.137,"15min":4.2350000000000003},"last":0.28000000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-08T05:31:28.706 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-11-08T05:31:28.706 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 459, 'reported_epoch': 458, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:30:44.490644+0000', 'last_change': '2023-11-08T05:28:13.557071+0000', 'last_active': '2023-11-08T05:30:44.490644+0000', 'last_peered': '2023-11-08T05:30:44.490644+0000', 'last_clean': '2023-11-08T05:28:01.458439+0000', 'last_became_active': '2023-11-08T05:28:13.557071+0000', 'last_became_peered': '2023-11-08T05:28:13.557071+0000', 'last_unstale': '2023-11-08T05:30:44.490644+0000', 'last_undegraded': '2023-11-08T05:30:44.490644+0000', 'last_fullsized': '2023-11-08T05:28:13.547025+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:31:28.707 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-11-08T05:31:28.707 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 11, 'reported_epoch': 459, 'state': 'active+clean+remapped', 'last_fresh': '2023-11-08T05:30:45.794760+0000', 'last_change': '2023-11-08T05:30:45.794760+0000', 'last_active': '2023-11-08T05:30:45.794760+0000', 'last_peered': '2023-11-08T05:30:45.794760+0000', 'last_clean': '2023-11-08T05:30:45.794760+0000', 'last_became_active': '2023-11-08T05:30:45.794398+0000', 'last_became_peered': '2023-11-08T05:30:45.794398+0000', 'last_unstale': '2023-11-08T05:30:45.794760+0000', 'last_undegraded': '2023-11-08T05:30:45.794760+0000', 'last_fullsized': '2023-11-08T05:30:45.794760+0000', 'mapping_epoch': 458, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 459, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2], 'acting': [2, 1], 'avail_no_missing': ['2', '1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2023-11-08T05:31:28.707 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.1 is not active+clean 2023-11-08T05:31:28.708 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.1', 'version': "0'0", 'reported_seq': 465, 'reported_epoch': 459, 'state': 'active+clean+remapped', 'last_fresh': '2023-11-08T05:30:45.795085+0000', 'last_change': '2023-11-08T05:30:45.795085+0000', 'last_active': '2023-11-08T05:30:45.795085+0000', 'last_peered': '2023-11-08T05:30:45.795085+0000', 'last_clean': '2023-11-08T05:30:45.795085+0000', 'last_became_active': '2023-11-08T05:30:45.794743+0000', 'last_became_peered': '2023-11-08T05:30:45.794743+0000', 'last_unstale': '2023-11-08T05:30:45.795085+0000', 'last_undegraded': '2023-11-08T05:30:45.795085+0000', 'last_fullsized': '2023-11-08T05:30:45.795085+0000', 'mapping_epoch': 458, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 459, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2], 'acting': [2, 1], 'avail_no_missing': ['2', '1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2023-11-08T05:31:28.708 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-11-08T05:31:28.708 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "0'0", 'reported_seq': 460, 'reported_epoch': 458, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:30:44.490523+0000', 'last_change': '2023-11-08T05:28:13.545468+0000', 'last_active': '2023-11-08T05:30:44.490523+0000', 'last_peered': '2023-11-08T05:30:44.490523+0000', 'last_clean': '2023-11-08T05:28:01.456723+0000', 'last_became_active': '2023-11-08T05:28:13.545468+0000', 'last_became_peered': '2023-11-08T05:28:13.545468+0000', 'last_unstale': '2023-11-08T05:30:44.490523+0000', 'last_undegraded': '2023-11-08T05:30:44.490523+0000', 'last_fullsized': '2023-11-08T05:28:13.537207+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 9, 'last_epoch_clean': 10, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:45.528796+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:45.528796+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:45.528796+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:31:28.709 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.0 is not active+clean 2023-11-08T05:31:28.709 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.0', 'version': "0'0", 'reported_seq': 465, 'reported_epoch': 458, 'state': 'active+clean+remapped', 'last_fresh': '2023-11-08T05:30:44.491907+0000', 'last_change': '2023-11-08T05:29:44.395350+0000', 'last_active': '2023-11-08T05:30:44.491907+0000', 'last_peered': '2023-11-08T05:30:44.491907+0000', 'last_clean': '2023-11-08T05:30:44.491907+0000', 'last_became_active': '2023-11-08T05:29:44.395012+0000', 'last_became_peered': '2023-11-08T05:29:44.395012+0000', 'last_unstale': '2023-11-08T05:30:44.491907+0000', 'last_undegraded': '2023-11-08T05:30:44.491907+0000', 'last_fullsized': '2023-11-08T05:30:44.491907+0000', 'mapping_epoch': 455, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 456, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2], 'acting': [2, 1], 'avail_no_missing': ['2', '1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2023-11-08T05:31:28.709 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-11-08T05:31:28.709 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 459, 'reported_epoch': 458, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:30:44.490629+0000', 'last_change': '2023-11-08T05:28:13.563357+0000', 'last_active': '2023-11-08T05:30:44.490629+0000', 'last_peered': '2023-11-08T05:30:44.490629+0000', 'last_clean': '2023-11-08T05:28:01.457691+0000', 'last_became_active': '2023-11-08T05:28:13.563357+0000', 'last_became_peered': '2023-11-08T05:28:13.563357+0000', 'last_unstale': '2023-11-08T05:30:44.490629+0000', 'last_undegraded': '2023-11-08T05:30:44.490629+0000', 'last_fullsized': '2023-11-08T05:28:13.541614+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:31:28.710 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-11-08T05:31:28.710 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 459, 'reported_epoch': 458, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:30:44.490691+0000', 'last_change': '2023-11-08T05:28:13.550672+0000', 'last_active': '2023-11-08T05:30:44.490691+0000', 'last_peered': '2023-11-08T05:30:44.490691+0000', 'last_clean': '2023-11-08T05:28:01.457904+0000', 'last_became_active': '2023-11-08T05:28:13.550672+0000', 'last_became_peered': '2023-11-08T05:28:13.550672+0000', 'last_unstale': '2023-11-08T05:30:44.490691+0000', 'last_undegraded': '2023-11-08T05:30:44.490691+0000', 'last_fullsized': '2023-11-08T05:28:13.539595+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:31:28.710 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-11-08T05:31:28.711 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 459, 'reported_epoch': 458, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:30:44.490805+0000', 'last_change': '2023-11-08T05:28:13.557968+0000', 'last_active': '2023-11-08T05:30:44.490805+0000', 'last_peered': '2023-11-08T05:30:44.490805+0000', 'last_clean': '2023-11-08T05:28:01.458433+0000', 'last_became_active': '2023-11-08T05:28:13.557968+0000', 'last_became_peered': '2023-11-08T05:28:13.557968+0000', 'last_unstale': '2023-11-08T05:30:44.490805+0000', 'last_undegraded': '2023-11-08T05:30:44.490805+0000', 'last_fullsized': '2023-11-08T05:28:13.542832+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:31:28.711 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-11-08T05:31:29.058 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T05:31:29.059 INFO:teuthology.orchestra.run.smithi087.stdout:{"fsid":"ab4cbc06-449d-497d-b513-220d9db2b590","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 5 pgs undersized","count":5},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2331,"monmap":{"epoch":1,"min_mon_release_name":"pacific","num_mons":1},"osdmap":{"epoch":459,"num_osds":3,"num_up_osds":2,"osd_up_since":1699421292,"num_in_osds":3,"osd_in_since":1699419158,"num_remapped_pgs":3},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean+remapped","count":3},{"state_name":"active+clean","count":1}],"num_pgs":9,"num_pools":2,"num_objects":0,"data_bytes":0,"bytes_used":2697592832,"bytes_avail":287212699648,"bytes_total":289910292480},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":41,"modified":"2023-11-08T05:29:32.044544+0000","services":{}},"progress_events":{}} 2023-11-08T05:31:29.072 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-11-08T05:31:29.072 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-11-08T05:31:29.356 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T05:31:29.357 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2023-11-08T05:31:29.369 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":1618,"stamp":"2023-11-08T05:31:28.068516+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":10,"acting":13,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":109,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2634368,"kb_used_data":409012,"kb_used_omap":780,"kb_used_meta":2220403,"kb_avail":280481152,"statfs":{"total":289910292480,"available":287212699648,"internally_reserved":0,"allocated":418828288,"data_stored":410500163,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":799447,"internal_metadata":2273692969},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-536782902,"num_objects":-164,"num_object_clones":0,"num_object_copies":-328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-115,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-164,"num_whiteouts":0,"num_read":-122056,"num_read_kb":-39011942,"num_write":-413642,"num_write_kb":-151705339,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-8,"num_bytes_recovered":-16922624,"num_keys_recovered":-2,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.424817"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490644+0000","last_change":"2023-11-08T05:28:13.557071+0000","last_active":"2023-11-08T05:30:44.490644+0000","last_peered":"2023-11-08T05:30:44.490644+0000","last_clean":"2023-11-08T05:28:01.458439+0000","last_became_active":"2023-11-08T05:28:13.557071+0000","last_became_peered":"2023-11-08T05:28:13.557071+0000","last_unstale":"2023-11-08T05:30:44.490644+0000","last_undegraded":"2023-11-08T05:30:44.490644+0000","last_fullsized":"2023-11-08T05:28:13.547025+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":459,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:45.794760+0000","last_change":"2023-11-08T05:30:45.794760+0000","last_active":"2023-11-08T05:30:45.794760+0000","last_peered":"2023-11-08T05:30:45.794760+0000","last_clean":"2023-11-08T05:30:45.794760+0000","last_became_active":"2023-11-08T05:30:45.794398+0000","last_became_peered":"2023-11-08T05:30:45.794398+0000","last_unstale":"2023-11-08T05:30:45.794760+0000","last_undegraded":"2023-11-08T05:30:45.794760+0000","last_fullsized":"2023-11-08T05:30:45.794760+0000","mapping_epoch":458,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":459,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":465,"reported_epoch":459,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:45.795085+0000","last_change":"2023-11-08T05:30:45.795085+0000","last_active":"2023-11-08T05:30:45.795085+0000","last_peered":"2023-11-08T05:30:45.795085+0000","last_clean":"2023-11-08T05:30:45.795085+0000","last_became_active":"2023-11-08T05:30:45.794743+0000","last_became_peered":"2023-11-08T05:30:45.794743+0000","last_unstale":"2023-11-08T05:30:45.795085+0000","last_undegraded":"2023-11-08T05:30:45.795085+0000","last_fullsized":"2023-11-08T05:30:45.795085+0000","mapping_epoch":458,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":459,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":454,"reported_epoch":458,"state":"active+clean","last_fresh":"2023-11-08T05:30:44.490618+0000","last_change":"2023-11-08T04:52:47.548597+0000","last_active":"2023-11-08T05:30:44.490618+0000","last_peered":"2023-11-08T05:30:44.490618+0000","last_clean":"2023-11-08T05:30:44.490618+0000","last_became_active":"2023-11-08T04:52:47.548294+0000","last_became_peered":"2023-11-08T04:52:47.548294+0000","last_unstale":"2023-11-08T05:30:44.490618+0000","last_undegraded":"2023-11-08T05:30:44.490618+0000","last_fullsized":"2023-11-08T05:30:44.490618+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":460,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490523+0000","last_change":"2023-11-08T05:28:13.545468+0000","last_active":"2023-11-08T05:30:44.490523+0000","last_peered":"2023-11-08T05:30:44.490523+0000","last_clean":"2023-11-08T05:28:01.456723+0000","last_became_active":"2023-11-08T05:28:13.545468+0000","last_became_peered":"2023-11-08T05:28:13.545468+0000","last_unstale":"2023-11-08T05:30:44.490523+0000","last_undegraded":"2023-11-08T05:30:44.490523+0000","last_fullsized":"2023-11-08T05:28:13.537207+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_clean_scrub_stamp":"2023-11-08T04:52:45.528796+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":465,"reported_epoch":458,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:44.491907+0000","last_change":"2023-11-08T05:29:44.395350+0000","last_active":"2023-11-08T05:30:44.491907+0000","last_peered":"2023-11-08T05:30:44.491907+0000","last_clean":"2023-11-08T05:30:44.491907+0000","last_became_active":"2023-11-08T05:29:44.395012+0000","last_became_peered":"2023-11-08T05:29:44.395012+0000","last_unstale":"2023-11-08T05:30:44.491907+0000","last_undegraded":"2023-11-08T05:30:44.491907+0000","last_fullsized":"2023-11-08T05:30:44.491907+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490629+0000","last_change":"2023-11-08T05:28:13.563357+0000","last_active":"2023-11-08T05:30:44.490629+0000","last_peered":"2023-11-08T05:30:44.490629+0000","last_clean":"2023-11-08T05:28:01.457691+0000","last_became_active":"2023-11-08T05:28:13.563357+0000","last_became_peered":"2023-11-08T05:28:13.563357+0000","last_unstale":"2023-11-08T05:30:44.490629+0000","last_undegraded":"2023-11-08T05:30:44.490629+0000","last_fullsized":"2023-11-08T05:28:13.541614+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490691+0000","last_change":"2023-11-08T05:28:13.550672+0000","last_active":"2023-11-08T05:30:44.490691+0000","last_peered":"2023-11-08T05:30:44.490691+0000","last_clean":"2023-11-08T05:28:01.457904+0000","last_became_active":"2023-11-08T05:28:13.550672+0000","last_became_peered":"2023-11-08T05:28:13.550672+0000","last_unstale":"2023-11-08T05:30:44.490691+0000","last_undegraded":"2023-11-08T05:30:44.490691+0000","last_fullsized":"2023-11-08T05:28:13.539595+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490805+0000","last_change":"2023-11-08T05:28:13.557968+0000","last_active":"2023-11-08T05:30:44.490805+0000","last_peered":"2023-11-08T05:30:44.490805+0000","last_clean":"2023-11-08T05:28:01.458433+0000","last_became_active":"2023-11-08T05:28:13.557968+0000","last_became_peered":"2023-11-08T05:28:13.557968+0000","last_unstale":"2023-11-08T05:30:44.490805+0000","last_undegraded":"2023-11-08T05:30:44.490805+0000","last_fullsized":"2023-11-08T05:28:13.542832+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":1,"acting":1,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":12,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738835,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":645040,"kb_used_data":5352,"kb_used_omap":275,"kb_used_meta":639404,"kb_avail":93726800,"statfs":{"total":96636764160,"available":95976243200,"internally_reserved":0,"allocated":5480448,"data_stored":2697647,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":281650,"internal_metadata":654750670},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:59 2023","interfaces":[{"interface":"back","average":{"1min":0.30499999999999999,"5min":0.439,"15min":0.36099999999999999},"min":{"1min":0.17399999999999999,"5min":0.14499999999999999,"15min":0.14499999999999999},"max":{"1min":0.60099999999999998,"5min":4.9379999999999997,"15min":4.9379999999999997},"last":0.379},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.49399999999999999,"15min":0.42399999999999999},"min":{"1min":0.221,"5min":0.19600000000000001,"15min":0.17000000000000001},"max":{"1min":0.63200000000000001,"5min":4.9859999999999998,"15min":4.9859999999999998},"last":0.40899999999999997}]},{"osd":1,"last update":"Wed Nov 8 05:31:06 2023","interfaces":[{"interface":"back","average":{"1min":0.27500000000000002,"5min":0.40300000000000002,"15min":0.46999999999999997},"min":{"1min":0.20899999999999999,"5min":0.16,"15min":0.153},"max":{"1min":0.41099999999999998,"5min":4.0830000000000002,"15min":23.242999999999999},"last":0.19800000000000001},{"interface":"front","average":{"1min":0.29899999999999999,"5min":0.40100000000000002,"15min":0.46000000000000002},"min":{"1min":0.23799999999999999,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":0.45300000000000001,"5min":3.5590000000000002,"15min":23.251000000000001},"last":0.20999999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738798,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1339228,"kb_used_data":398304,"kb_used_omap":291,"kb_used_meta":936476,"kb_avail":93032612,"statfs":{"total":96636764160,"available":95265394688,"internally_reserved":0,"allocated":407863296,"data_stored":405103900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":298462,"internal_metadata":958951970},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.436,"15min":0.36299999999999999},"min":{"1min":0.19700000000000001,"5min":0.14699999999999999,"15min":0.13700000000000001},"max":{"1min":3.222,"5min":6.5499999999999998,"15min":6.9989999999999997},"last":0.245},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.46300000000000002,"15min":0.42299999999999999},"min":{"1min":0.17799999999999999,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":5.0030000000000001,"5min":6.0629999999999997,"15min":9.6869999999999994},"last":0.28599999999999998}]},{"osd":2,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.51600000000000001,"15min":0.47099999999999997},"min":{"1min":0.21099999999999999,"5min":0.153,"15min":0.153},"max":{"1min":5.7439999999999998,"5min":5.8280000000000003,"15min":9.7639999999999993},"last":0.252},{"interface":"front","average":{"1min":0.497,"5min":0.56200000000000006,"15min":0.42299999999999999},"min":{"1min":0.20399999999999999,"5min":0.187,"15min":0.16800000000000001},"max":{"1min":3.9319999999999999,"5min":8.8040000000000003,"15min":8.8040000000000003},"last":0.29899999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738837,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":650100,"kb_used_data":5356,"kb_used_omap":214,"kb_used_meta":644521,"kb_avail":93721740,"statfs":{"total":96636764160,"available":95971061760,"internally_reserved":0,"allocated":5484544,"data_stored":2698616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":219335,"internal_metadata":659990329},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:56 2023","interfaces":[{"interface":"back","average":{"1min":0.30399999999999999,"5min":0.34699999999999998,"15min":0.36499999999999999},"min":{"1min":0.23999999999999999,"5min":0.14199999999999999,"15min":0.14199999999999999},"max":{"1min":0.42399999999999999,"5min":1.595,"15min":7.4249999999999998},"last":0.17100000000000001},{"interface":"front","average":{"1min":0.35499999999999998,"5min":0.40899999999999997,"15min":0.376},"min":{"1min":0.23999999999999999,"5min":0.19600000000000001,"15min":0.183},"max":{"1min":0.46999999999999997,"5min":2.6859999999999999,"15min":2.6859999999999999},"last":0.23000000000000001}]},{"osd":2,"last update":"Wed Nov 8 05:30:57 2023","interfaces":[{"interface":"back","average":{"1min":0.28599999999999998,"5min":0.34699999999999998,"15min":0.36899999999999999},"min":{"1min":0.19700000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":0.41399999999999998,"5min":2.2650000000000001,"15min":5.8209999999999997},"last":0.251},{"interface":"front","average":{"1min":0.30599999999999999,"5min":0.313,"15min":0.35799999999999998},"min":{"1min":0.17799999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.435,"5min":1.137,"15min":4.2350000000000003},"last":0.28000000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-08T05:31:32.371 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-11-08T05:31:32.591 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~195s 2023-11-08T05:31:32.668 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T05:31:32.669 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2023-11-08T05:31:32.681 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":1620,"stamp":"2023-11-08T05:31:32.069316+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":10,"acting":13,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":109,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2634368,"kb_used_data":409012,"kb_used_omap":780,"kb_used_meta":2220403,"kb_avail":280481152,"statfs":{"total":289910292480,"available":287212699648,"internally_reserved":0,"allocated":418828288,"data_stored":410500163,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":799447,"internal_metadata":2273692969},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-536782902,"num_objects":-164,"num_object_clones":0,"num_object_copies":-328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-115,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-164,"num_whiteouts":0,"num_read":-122056,"num_read_kb":-39011942,"num_write":-413642,"num_write_kb":-151705339,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-8,"num_bytes_recovered":-16922624,"num_keys_recovered":-2,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.424817"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490644+0000","last_change":"2023-11-08T05:28:13.557071+0000","last_active":"2023-11-08T05:30:44.490644+0000","last_peered":"2023-11-08T05:30:44.490644+0000","last_clean":"2023-11-08T05:28:01.458439+0000","last_became_active":"2023-11-08T05:28:13.557071+0000","last_became_peered":"2023-11-08T05:28:13.557071+0000","last_unstale":"2023-11-08T05:30:44.490644+0000","last_undegraded":"2023-11-08T05:30:44.490644+0000","last_fullsized":"2023-11-08T05:28:13.547025+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":459,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:45.794760+0000","last_change":"2023-11-08T05:30:45.794760+0000","last_active":"2023-11-08T05:30:45.794760+0000","last_peered":"2023-11-08T05:30:45.794760+0000","last_clean":"2023-11-08T05:30:45.794760+0000","last_became_active":"2023-11-08T05:30:45.794398+0000","last_became_peered":"2023-11-08T05:30:45.794398+0000","last_unstale":"2023-11-08T05:30:45.794760+0000","last_undegraded":"2023-11-08T05:30:45.794760+0000","last_fullsized":"2023-11-08T05:30:45.794760+0000","mapping_epoch":458,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":459,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":465,"reported_epoch":459,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:45.795085+0000","last_change":"2023-11-08T05:30:45.795085+0000","last_active":"2023-11-08T05:30:45.795085+0000","last_peered":"2023-11-08T05:30:45.795085+0000","last_clean":"2023-11-08T05:30:45.795085+0000","last_became_active":"2023-11-08T05:30:45.794743+0000","last_became_peered":"2023-11-08T05:30:45.794743+0000","last_unstale":"2023-11-08T05:30:45.795085+0000","last_undegraded":"2023-11-08T05:30:45.795085+0000","last_fullsized":"2023-11-08T05:30:45.795085+0000","mapping_epoch":458,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":459,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":454,"reported_epoch":458,"state":"active+clean","last_fresh":"2023-11-08T05:30:44.490618+0000","last_change":"2023-11-08T04:52:47.548597+0000","last_active":"2023-11-08T05:30:44.490618+0000","last_peered":"2023-11-08T05:30:44.490618+0000","last_clean":"2023-11-08T05:30:44.490618+0000","last_became_active":"2023-11-08T04:52:47.548294+0000","last_became_peered":"2023-11-08T04:52:47.548294+0000","last_unstale":"2023-11-08T05:30:44.490618+0000","last_undegraded":"2023-11-08T05:30:44.490618+0000","last_fullsized":"2023-11-08T05:30:44.490618+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":460,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490523+0000","last_change":"2023-11-08T05:28:13.545468+0000","last_active":"2023-11-08T05:30:44.490523+0000","last_peered":"2023-11-08T05:30:44.490523+0000","last_clean":"2023-11-08T05:28:01.456723+0000","last_became_active":"2023-11-08T05:28:13.545468+0000","last_became_peered":"2023-11-08T05:28:13.545468+0000","last_unstale":"2023-11-08T05:30:44.490523+0000","last_undegraded":"2023-11-08T05:30:44.490523+0000","last_fullsized":"2023-11-08T05:28:13.537207+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_clean_scrub_stamp":"2023-11-08T04:52:45.528796+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":465,"reported_epoch":458,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:44.491907+0000","last_change":"2023-11-08T05:29:44.395350+0000","last_active":"2023-11-08T05:30:44.491907+0000","last_peered":"2023-11-08T05:30:44.491907+0000","last_clean":"2023-11-08T05:30:44.491907+0000","last_became_active":"2023-11-08T05:29:44.395012+0000","last_became_peered":"2023-11-08T05:29:44.395012+0000","last_unstale":"2023-11-08T05:30:44.491907+0000","last_undegraded":"2023-11-08T05:30:44.491907+0000","last_fullsized":"2023-11-08T05:30:44.491907+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490629+0000","last_change":"2023-11-08T05:28:13.563357+0000","last_active":"2023-11-08T05:30:44.490629+0000","last_peered":"2023-11-08T05:30:44.490629+0000","last_clean":"2023-11-08T05:28:01.457691+0000","last_became_active":"2023-11-08T05:28:13.563357+0000","last_became_peered":"2023-11-08T05:28:13.563357+0000","last_unstale":"2023-11-08T05:30:44.490629+0000","last_undegraded":"2023-11-08T05:30:44.490629+0000","last_fullsized":"2023-11-08T05:28:13.541614+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490691+0000","last_change":"2023-11-08T05:28:13.550672+0000","last_active":"2023-11-08T05:30:44.490691+0000","last_peered":"2023-11-08T05:30:44.490691+0000","last_clean":"2023-11-08T05:28:01.457904+0000","last_became_active":"2023-11-08T05:28:13.550672+0000","last_became_peered":"2023-11-08T05:28:13.550672+0000","last_unstale":"2023-11-08T05:30:44.490691+0000","last_undegraded":"2023-11-08T05:30:44.490691+0000","last_fullsized":"2023-11-08T05:28:13.539595+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490805+0000","last_change":"2023-11-08T05:28:13.557968+0000","last_active":"2023-11-08T05:30:44.490805+0000","last_peered":"2023-11-08T05:30:44.490805+0000","last_clean":"2023-11-08T05:28:01.458433+0000","last_became_active":"2023-11-08T05:28:13.557968+0000","last_became_peered":"2023-11-08T05:28:13.557968+0000","last_unstale":"2023-11-08T05:30:44.490805+0000","last_undegraded":"2023-11-08T05:30:44.490805+0000","last_fullsized":"2023-11-08T05:28:13.542832+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":1,"acting":1,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":12,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738836,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":645040,"kb_used_data":5352,"kb_used_omap":275,"kb_used_meta":639404,"kb_avail":93726800,"statfs":{"total":96636764160,"available":95976243200,"internally_reserved":0,"allocated":5480448,"data_stored":2697647,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":281650,"internal_metadata":654750670},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:59 2023","interfaces":[{"interface":"back","average":{"1min":0.30499999999999999,"5min":0.439,"15min":0.36099999999999999},"min":{"1min":0.17399999999999999,"5min":0.14499999999999999,"15min":0.14499999999999999},"max":{"1min":0.60099999999999998,"5min":4.9379999999999997,"15min":4.9379999999999997},"last":0.379},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.49399999999999999,"15min":0.42399999999999999},"min":{"1min":0.221,"5min":0.19600000000000001,"15min":0.17000000000000001},"max":{"1min":0.63200000000000001,"5min":4.9859999999999998,"15min":4.9859999999999998},"last":0.40899999999999997}]},{"osd":1,"last update":"Wed Nov 8 05:31:06 2023","interfaces":[{"interface":"back","average":{"1min":0.27500000000000002,"5min":0.40300000000000002,"15min":0.46999999999999997},"min":{"1min":0.20899999999999999,"5min":0.16,"15min":0.153},"max":{"1min":0.41099999999999998,"5min":4.0830000000000002,"15min":23.242999999999999},"last":0.223},{"interface":"front","average":{"1min":0.29899999999999999,"5min":0.40100000000000002,"15min":0.46000000000000002},"min":{"1min":0.23799999999999999,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":0.45300000000000001,"5min":3.5590000000000002,"15min":23.251000000000001},"last":0.24099999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738798,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1339228,"kb_used_data":398304,"kb_used_omap":291,"kb_used_meta":936476,"kb_avail":93032612,"statfs":{"total":96636764160,"available":95265394688,"internally_reserved":0,"allocated":407863296,"data_stored":405103900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":298462,"internal_metadata":958951970},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.436,"15min":0.36299999999999999},"min":{"1min":0.19700000000000001,"5min":0.14699999999999999,"15min":0.13700000000000001},"max":{"1min":3.222,"5min":6.5499999999999998,"15min":6.9989999999999997},"last":0.245},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.46300000000000002,"15min":0.42299999999999999},"min":{"1min":0.17799999999999999,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":5.0030000000000001,"5min":6.0629999999999997,"15min":9.6869999999999994},"last":0.28599999999999998}]},{"osd":2,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.51600000000000001,"15min":0.47099999999999997},"min":{"1min":0.21099999999999999,"5min":0.153,"15min":0.153},"max":{"1min":5.7439999999999998,"5min":5.8280000000000003,"15min":9.7639999999999993},"last":0.252},{"interface":"front","average":{"1min":0.497,"5min":0.56200000000000006,"15min":0.42299999999999999},"min":{"1min":0.20399999999999999,"5min":0.187,"15min":0.16800000000000001},"max":{"1min":3.9319999999999999,"5min":8.8040000000000003,"15min":8.8040000000000003},"last":0.29899999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738838,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":650100,"kb_used_data":5356,"kb_used_omap":214,"kb_used_meta":644521,"kb_avail":93721740,"statfs":{"total":96636764160,"available":95971061760,"internally_reserved":0,"allocated":5484544,"data_stored":2698616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":219335,"internal_metadata":659990329},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:56 2023","interfaces":[{"interface":"back","average":{"1min":0.30399999999999999,"5min":0.34699999999999998,"15min":0.36499999999999999},"min":{"1min":0.23999999999999999,"5min":0.14199999999999999,"15min":0.14199999999999999},"max":{"1min":0.42399999999999999,"5min":1.595,"15min":7.4249999999999998},"last":0.17100000000000001},{"interface":"front","average":{"1min":0.35499999999999998,"5min":0.40899999999999997,"15min":0.376},"min":{"1min":0.23999999999999999,"5min":0.19600000000000001,"15min":0.183},"max":{"1min":0.46999999999999997,"5min":2.6859999999999999,"15min":2.6859999999999999},"last":0.23000000000000001}]},{"osd":2,"last update":"Wed Nov 8 05:30:57 2023","interfaces":[{"interface":"back","average":{"1min":0.28599999999999998,"5min":0.34699999999999998,"15min":0.36899999999999999},"min":{"1min":0.19700000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":0.41399999999999998,"5min":2.2650000000000001,"15min":5.8209999999999997},"last":0.20499999999999999},{"interface":"front","average":{"1min":0.30599999999999999,"5min":0.313,"15min":0.35799999999999998},"min":{"1min":0.17799999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.435,"5min":1.137,"15min":4.2350000000000003},"last":0.24199999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-08T05:31:32.682 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-11-08T05:31:32.963 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T05:31:32.965 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2023-11-08T05:31:32.976 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":1620,"stamp":"2023-11-08T05:31:32.069316+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":10,"acting":13,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":109,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2634368,"kb_used_data":409012,"kb_used_omap":780,"kb_used_meta":2220403,"kb_avail":280481152,"statfs":{"total":289910292480,"available":287212699648,"internally_reserved":0,"allocated":418828288,"data_stored":410500163,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":799447,"internal_metadata":2273692969},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-536782902,"num_objects":-164,"num_object_clones":0,"num_object_copies":-328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-115,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-164,"num_whiteouts":0,"num_read":-122056,"num_read_kb":-39011942,"num_write":-413642,"num_write_kb":-151705339,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-8,"num_bytes_recovered":-16922624,"num_keys_recovered":-2,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.424817"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490644+0000","last_change":"2023-11-08T05:28:13.557071+0000","last_active":"2023-11-08T05:30:44.490644+0000","last_peered":"2023-11-08T05:30:44.490644+0000","last_clean":"2023-11-08T05:28:01.458439+0000","last_became_active":"2023-11-08T05:28:13.557071+0000","last_became_peered":"2023-11-08T05:28:13.557071+0000","last_unstale":"2023-11-08T05:30:44.490644+0000","last_undegraded":"2023-11-08T05:30:44.490644+0000","last_fullsized":"2023-11-08T05:28:13.547025+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":459,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:45.794760+0000","last_change":"2023-11-08T05:30:45.794760+0000","last_active":"2023-11-08T05:30:45.794760+0000","last_peered":"2023-11-08T05:30:45.794760+0000","last_clean":"2023-11-08T05:30:45.794760+0000","last_became_active":"2023-11-08T05:30:45.794398+0000","last_became_peered":"2023-11-08T05:30:45.794398+0000","last_unstale":"2023-11-08T05:30:45.794760+0000","last_undegraded":"2023-11-08T05:30:45.794760+0000","last_fullsized":"2023-11-08T05:30:45.794760+0000","mapping_epoch":458,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":459,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":465,"reported_epoch":459,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:45.795085+0000","last_change":"2023-11-08T05:30:45.795085+0000","last_active":"2023-11-08T05:30:45.795085+0000","last_peered":"2023-11-08T05:30:45.795085+0000","last_clean":"2023-11-08T05:30:45.795085+0000","last_became_active":"2023-11-08T05:30:45.794743+0000","last_became_peered":"2023-11-08T05:30:45.794743+0000","last_unstale":"2023-11-08T05:30:45.795085+0000","last_undegraded":"2023-11-08T05:30:45.795085+0000","last_fullsized":"2023-11-08T05:30:45.795085+0000","mapping_epoch":458,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":459,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":454,"reported_epoch":458,"state":"active+clean","last_fresh":"2023-11-08T05:30:44.490618+0000","last_change":"2023-11-08T04:52:47.548597+0000","last_active":"2023-11-08T05:30:44.490618+0000","last_peered":"2023-11-08T05:30:44.490618+0000","last_clean":"2023-11-08T05:30:44.490618+0000","last_became_active":"2023-11-08T04:52:47.548294+0000","last_became_peered":"2023-11-08T04:52:47.548294+0000","last_unstale":"2023-11-08T05:30:44.490618+0000","last_undegraded":"2023-11-08T05:30:44.490618+0000","last_fullsized":"2023-11-08T05:30:44.490618+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":460,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490523+0000","last_change":"2023-11-08T05:28:13.545468+0000","last_active":"2023-11-08T05:30:44.490523+0000","last_peered":"2023-11-08T05:30:44.490523+0000","last_clean":"2023-11-08T05:28:01.456723+0000","last_became_active":"2023-11-08T05:28:13.545468+0000","last_became_peered":"2023-11-08T05:28:13.545468+0000","last_unstale":"2023-11-08T05:30:44.490523+0000","last_undegraded":"2023-11-08T05:30:44.490523+0000","last_fullsized":"2023-11-08T05:28:13.537207+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_clean_scrub_stamp":"2023-11-08T04:52:45.528796+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":465,"reported_epoch":458,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:44.491907+0000","last_change":"2023-11-08T05:29:44.395350+0000","last_active":"2023-11-08T05:30:44.491907+0000","last_peered":"2023-11-08T05:30:44.491907+0000","last_clean":"2023-11-08T05:30:44.491907+0000","last_became_active":"2023-11-08T05:29:44.395012+0000","last_became_peered":"2023-11-08T05:29:44.395012+0000","last_unstale":"2023-11-08T05:30:44.491907+0000","last_undegraded":"2023-11-08T05:30:44.491907+0000","last_fullsized":"2023-11-08T05:30:44.491907+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490629+0000","last_change":"2023-11-08T05:28:13.563357+0000","last_active":"2023-11-08T05:30:44.490629+0000","last_peered":"2023-11-08T05:30:44.490629+0000","last_clean":"2023-11-08T05:28:01.457691+0000","last_became_active":"2023-11-08T05:28:13.563357+0000","last_became_peered":"2023-11-08T05:28:13.563357+0000","last_unstale":"2023-11-08T05:30:44.490629+0000","last_undegraded":"2023-11-08T05:30:44.490629+0000","last_fullsized":"2023-11-08T05:28:13.541614+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490691+0000","last_change":"2023-11-08T05:28:13.550672+0000","last_active":"2023-11-08T05:30:44.490691+0000","last_peered":"2023-11-08T05:30:44.490691+0000","last_clean":"2023-11-08T05:28:01.457904+0000","last_became_active":"2023-11-08T05:28:13.550672+0000","last_became_peered":"2023-11-08T05:28:13.550672+0000","last_unstale":"2023-11-08T05:30:44.490691+0000","last_undegraded":"2023-11-08T05:30:44.490691+0000","last_fullsized":"2023-11-08T05:28:13.539595+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490805+0000","last_change":"2023-11-08T05:28:13.557968+0000","last_active":"2023-11-08T05:30:44.490805+0000","last_peered":"2023-11-08T05:30:44.490805+0000","last_clean":"2023-11-08T05:28:01.458433+0000","last_became_active":"2023-11-08T05:28:13.557968+0000","last_became_peered":"2023-11-08T05:28:13.557968+0000","last_unstale":"2023-11-08T05:30:44.490805+0000","last_undegraded":"2023-11-08T05:30:44.490805+0000","last_fullsized":"2023-11-08T05:28:13.542832+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":1,"acting":1,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":12,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738836,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":645040,"kb_used_data":5352,"kb_used_omap":275,"kb_used_meta":639404,"kb_avail":93726800,"statfs":{"total":96636764160,"available":95976243200,"internally_reserved":0,"allocated":5480448,"data_stored":2697647,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":281650,"internal_metadata":654750670},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:59 2023","interfaces":[{"interface":"back","average":{"1min":0.30499999999999999,"5min":0.439,"15min":0.36099999999999999},"min":{"1min":0.17399999999999999,"5min":0.14499999999999999,"15min":0.14499999999999999},"max":{"1min":0.60099999999999998,"5min":4.9379999999999997,"15min":4.9379999999999997},"last":0.379},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.49399999999999999,"15min":0.42399999999999999},"min":{"1min":0.221,"5min":0.19600000000000001,"15min":0.17000000000000001},"max":{"1min":0.63200000000000001,"5min":4.9859999999999998,"15min":4.9859999999999998},"last":0.40899999999999997}]},{"osd":1,"last update":"Wed Nov 8 05:31:06 2023","interfaces":[{"interface":"back","average":{"1min":0.27500000000000002,"5min":0.40300000000000002,"15min":0.46999999999999997},"min":{"1min":0.20899999999999999,"5min":0.16,"15min":0.153},"max":{"1min":0.41099999999999998,"5min":4.0830000000000002,"15min":23.242999999999999},"last":0.223},{"interface":"front","average":{"1min":0.29899999999999999,"5min":0.40100000000000002,"15min":0.46000000000000002},"min":{"1min":0.23799999999999999,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":0.45300000000000001,"5min":3.5590000000000002,"15min":23.251000000000001},"last":0.24099999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738798,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1339228,"kb_used_data":398304,"kb_used_omap":291,"kb_used_meta":936476,"kb_avail":93032612,"statfs":{"total":96636764160,"available":95265394688,"internally_reserved":0,"allocated":407863296,"data_stored":405103900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":298462,"internal_metadata":958951970},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.436,"15min":0.36299999999999999},"min":{"1min":0.19700000000000001,"5min":0.14699999999999999,"15min":0.13700000000000001},"max":{"1min":3.222,"5min":6.5499999999999998,"15min":6.9989999999999997},"last":0.245},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.46300000000000002,"15min":0.42299999999999999},"min":{"1min":0.17799999999999999,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":5.0030000000000001,"5min":6.0629999999999997,"15min":9.6869999999999994},"last":0.28599999999999998}]},{"osd":2,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.51600000000000001,"15min":0.47099999999999997},"min":{"1min":0.21099999999999999,"5min":0.153,"15min":0.153},"max":{"1min":5.7439999999999998,"5min":5.8280000000000003,"15min":9.7639999999999993},"last":0.252},{"interface":"front","average":{"1min":0.497,"5min":0.56200000000000006,"15min":0.42299999999999999},"min":{"1min":0.20399999999999999,"5min":0.187,"15min":0.16800000000000001},"max":{"1min":3.9319999999999999,"5min":8.8040000000000003,"15min":8.8040000000000003},"last":0.29899999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738838,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":650100,"kb_used_data":5356,"kb_used_omap":214,"kb_used_meta":644521,"kb_avail":93721740,"statfs":{"total":96636764160,"available":95971061760,"internally_reserved":0,"allocated":5484544,"data_stored":2698616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":219335,"internal_metadata":659990329},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:56 2023","interfaces":[{"interface":"back","average":{"1min":0.30399999999999999,"5min":0.34699999999999998,"15min":0.36499999999999999},"min":{"1min":0.23999999999999999,"5min":0.14199999999999999,"15min":0.14199999999999999},"max":{"1min":0.42399999999999999,"5min":1.595,"15min":7.4249999999999998},"last":0.17100000000000001},{"interface":"front","average":{"1min":0.35499999999999998,"5min":0.40899999999999997,"15min":0.376},"min":{"1min":0.23999999999999999,"5min":0.19600000000000001,"15min":0.183},"max":{"1min":0.46999999999999997,"5min":2.6859999999999999,"15min":2.6859999999999999},"last":0.23000000000000001}]},{"osd":2,"last update":"Wed Nov 8 05:30:57 2023","interfaces":[{"interface":"back","average":{"1min":0.28599999999999998,"5min":0.34699999999999998,"15min":0.36899999999999999},"min":{"1min":0.19700000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":0.41399999999999998,"5min":2.2650000000000001,"15min":5.8209999999999997},"last":0.20499999999999999},{"interface":"front","average":{"1min":0.30599999999999999,"5min":0.313,"15min":0.35799999999999998},"min":{"1min":0.17799999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.435,"5min":1.137,"15min":4.2350000000000003},"last":0.24199999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-08T05:31:32.977 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-11-08T05:31:32.978 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 459, 'reported_epoch': 458, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:30:44.490644+0000', 'last_change': '2023-11-08T05:28:13.557071+0000', 'last_active': '2023-11-08T05:30:44.490644+0000', 'last_peered': '2023-11-08T05:30:44.490644+0000', 'last_clean': '2023-11-08T05:28:01.458439+0000', 'last_became_active': '2023-11-08T05:28:13.557071+0000', 'last_became_peered': '2023-11-08T05:28:13.557071+0000', 'last_unstale': '2023-11-08T05:30:44.490644+0000', 'last_undegraded': '2023-11-08T05:30:44.490644+0000', 'last_fullsized': '2023-11-08T05:28:13.547025+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:31:32.978 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-11-08T05:31:32.978 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 11, 'reported_epoch': 459, 'state': 'active+clean+remapped', 'last_fresh': '2023-11-08T05:30:45.794760+0000', 'last_change': '2023-11-08T05:30:45.794760+0000', 'last_active': '2023-11-08T05:30:45.794760+0000', 'last_peered': '2023-11-08T05:30:45.794760+0000', 'last_clean': '2023-11-08T05:30:45.794760+0000', 'last_became_active': '2023-11-08T05:30:45.794398+0000', 'last_became_peered': '2023-11-08T05:30:45.794398+0000', 'last_unstale': '2023-11-08T05:30:45.794760+0000', 'last_undegraded': '2023-11-08T05:30:45.794760+0000', 'last_fullsized': '2023-11-08T05:30:45.794760+0000', 'mapping_epoch': 458, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 459, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2], 'acting': [2, 1], 'avail_no_missing': ['2', '1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2023-11-08T05:31:32.979 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.1 is not active+clean 2023-11-08T05:31:32.979 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.1', 'version': "0'0", 'reported_seq': 465, 'reported_epoch': 459, 'state': 'active+clean+remapped', 'last_fresh': '2023-11-08T05:30:45.795085+0000', 'last_change': '2023-11-08T05:30:45.795085+0000', 'last_active': '2023-11-08T05:30:45.795085+0000', 'last_peered': '2023-11-08T05:30:45.795085+0000', 'last_clean': '2023-11-08T05:30:45.795085+0000', 'last_became_active': '2023-11-08T05:30:45.794743+0000', 'last_became_peered': '2023-11-08T05:30:45.794743+0000', 'last_unstale': '2023-11-08T05:30:45.795085+0000', 'last_undegraded': '2023-11-08T05:30:45.795085+0000', 'last_fullsized': '2023-11-08T05:30:45.795085+0000', 'mapping_epoch': 458, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 459, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2], 'acting': [2, 1], 'avail_no_missing': ['2', '1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2023-11-08T05:31:32.979 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-11-08T05:31:32.980 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "0'0", 'reported_seq': 460, 'reported_epoch': 458, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:30:44.490523+0000', 'last_change': '2023-11-08T05:28:13.545468+0000', 'last_active': '2023-11-08T05:30:44.490523+0000', 'last_peered': '2023-11-08T05:30:44.490523+0000', 'last_clean': '2023-11-08T05:28:01.456723+0000', 'last_became_active': '2023-11-08T05:28:13.545468+0000', 'last_became_peered': '2023-11-08T05:28:13.545468+0000', 'last_unstale': '2023-11-08T05:30:44.490523+0000', 'last_undegraded': '2023-11-08T05:30:44.490523+0000', 'last_fullsized': '2023-11-08T05:28:13.537207+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 9, 'last_epoch_clean': 10, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:45.528796+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:45.528796+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:45.528796+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:31:32.980 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.0 is not active+clean 2023-11-08T05:31:32.980 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.0', 'version': "0'0", 'reported_seq': 465, 'reported_epoch': 458, 'state': 'active+clean+remapped', 'last_fresh': '2023-11-08T05:30:44.491907+0000', 'last_change': '2023-11-08T05:29:44.395350+0000', 'last_active': '2023-11-08T05:30:44.491907+0000', 'last_peered': '2023-11-08T05:30:44.491907+0000', 'last_clean': '2023-11-08T05:30:44.491907+0000', 'last_became_active': '2023-11-08T05:29:44.395012+0000', 'last_became_peered': '2023-11-08T05:29:44.395012+0000', 'last_unstale': '2023-11-08T05:30:44.491907+0000', 'last_undegraded': '2023-11-08T05:30:44.491907+0000', 'last_fullsized': '2023-11-08T05:30:44.491907+0000', 'mapping_epoch': 455, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 456, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2], 'acting': [2, 1], 'avail_no_missing': ['2', '1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2023-11-08T05:31:32.981 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-11-08T05:31:32.981 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 459, 'reported_epoch': 458, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:30:44.490629+0000', 'last_change': '2023-11-08T05:28:13.563357+0000', 'last_active': '2023-11-08T05:30:44.490629+0000', 'last_peered': '2023-11-08T05:30:44.490629+0000', 'last_clean': '2023-11-08T05:28:01.457691+0000', 'last_became_active': '2023-11-08T05:28:13.563357+0000', 'last_became_peered': '2023-11-08T05:28:13.563357+0000', 'last_unstale': '2023-11-08T05:30:44.490629+0000', 'last_undegraded': '2023-11-08T05:30:44.490629+0000', 'last_fullsized': '2023-11-08T05:28:13.541614+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:31:32.981 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-11-08T05:31:32.982 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 459, 'reported_epoch': 458, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:30:44.490691+0000', 'last_change': '2023-11-08T05:28:13.550672+0000', 'last_active': '2023-11-08T05:30:44.490691+0000', 'last_peered': '2023-11-08T05:30:44.490691+0000', 'last_clean': '2023-11-08T05:28:01.457904+0000', 'last_became_active': '2023-11-08T05:28:13.550672+0000', 'last_became_peered': '2023-11-08T05:28:13.550672+0000', 'last_unstale': '2023-11-08T05:30:44.490691+0000', 'last_undegraded': '2023-11-08T05:30:44.490691+0000', 'last_fullsized': '2023-11-08T05:28:13.539595+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:31:32.982 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-11-08T05:31:32.982 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 459, 'reported_epoch': 458, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:30:44.490805+0000', 'last_change': '2023-11-08T05:28:13.557968+0000', 'last_active': '2023-11-08T05:30:44.490805+0000', 'last_peered': '2023-11-08T05:30:44.490805+0000', 'last_clean': '2023-11-08T05:28:01.458433+0000', 'last_became_active': '2023-11-08T05:28:13.557968+0000', 'last_became_peered': '2023-11-08T05:28:13.557968+0000', 'last_unstale': '2023-11-08T05:30:44.490805+0000', 'last_undegraded': '2023-11-08T05:30:44.490805+0000', 'last_fullsized': '2023-11-08T05:28:13.542832+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:31:32.983 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-11-08T05:31:33.331 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T05:31:33.332 INFO:teuthology.orchestra.run.smithi087.stdout:{"fsid":"ab4cbc06-449d-497d-b513-220d9db2b590","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 5 pgs undersized","count":5},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2335,"monmap":{"epoch":1,"min_mon_release_name":"pacific","num_mons":1},"osdmap":{"epoch":459,"num_osds":3,"num_up_osds":2,"osd_up_since":1699421292,"num_in_osds":3,"osd_in_since":1699419158,"num_remapped_pgs":3},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean+remapped","count":3},{"state_name":"active+clean","count":1}],"num_pgs":9,"num_pools":2,"num_objects":0,"data_bytes":0,"bytes_used":2697592832,"bytes_avail":287212699648,"bytes_total":289910292480},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":41,"modified":"2023-11-08T05:29:32.044544+0000","services":{}},"progress_events":{}} 2023-11-08T05:31:33.345 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-11-08T05:31:33.345 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-11-08T05:31:33.632 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T05:31:33.634 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2023-11-08T05:31:33.645 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":1620,"stamp":"2023-11-08T05:31:32.069316+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":10,"acting":13,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":109,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2634368,"kb_used_data":409012,"kb_used_omap":780,"kb_used_meta":2220403,"kb_avail":280481152,"statfs":{"total":289910292480,"available":287212699648,"internally_reserved":0,"allocated":418828288,"data_stored":410500163,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":799447,"internal_metadata":2273692969},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-536782902,"num_objects":-164,"num_object_clones":0,"num_object_copies":-328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-115,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-164,"num_whiteouts":0,"num_read":-122056,"num_read_kb":-39011942,"num_write":-413642,"num_write_kb":-151705339,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-8,"num_bytes_recovered":-16922624,"num_keys_recovered":-2,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.424817"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490644+0000","last_change":"2023-11-08T05:28:13.557071+0000","last_active":"2023-11-08T05:30:44.490644+0000","last_peered":"2023-11-08T05:30:44.490644+0000","last_clean":"2023-11-08T05:28:01.458439+0000","last_became_active":"2023-11-08T05:28:13.557071+0000","last_became_peered":"2023-11-08T05:28:13.557071+0000","last_unstale":"2023-11-08T05:30:44.490644+0000","last_undegraded":"2023-11-08T05:30:44.490644+0000","last_fullsized":"2023-11-08T05:28:13.547025+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":459,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:45.794760+0000","last_change":"2023-11-08T05:30:45.794760+0000","last_active":"2023-11-08T05:30:45.794760+0000","last_peered":"2023-11-08T05:30:45.794760+0000","last_clean":"2023-11-08T05:30:45.794760+0000","last_became_active":"2023-11-08T05:30:45.794398+0000","last_became_peered":"2023-11-08T05:30:45.794398+0000","last_unstale":"2023-11-08T05:30:45.794760+0000","last_undegraded":"2023-11-08T05:30:45.794760+0000","last_fullsized":"2023-11-08T05:30:45.794760+0000","mapping_epoch":458,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":459,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":465,"reported_epoch":459,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:45.795085+0000","last_change":"2023-11-08T05:30:45.795085+0000","last_active":"2023-11-08T05:30:45.795085+0000","last_peered":"2023-11-08T05:30:45.795085+0000","last_clean":"2023-11-08T05:30:45.795085+0000","last_became_active":"2023-11-08T05:30:45.794743+0000","last_became_peered":"2023-11-08T05:30:45.794743+0000","last_unstale":"2023-11-08T05:30:45.795085+0000","last_undegraded":"2023-11-08T05:30:45.795085+0000","last_fullsized":"2023-11-08T05:30:45.795085+0000","mapping_epoch":458,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":459,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":454,"reported_epoch":458,"state":"active+clean","last_fresh":"2023-11-08T05:30:44.490618+0000","last_change":"2023-11-08T04:52:47.548597+0000","last_active":"2023-11-08T05:30:44.490618+0000","last_peered":"2023-11-08T05:30:44.490618+0000","last_clean":"2023-11-08T05:30:44.490618+0000","last_became_active":"2023-11-08T04:52:47.548294+0000","last_became_peered":"2023-11-08T04:52:47.548294+0000","last_unstale":"2023-11-08T05:30:44.490618+0000","last_undegraded":"2023-11-08T05:30:44.490618+0000","last_fullsized":"2023-11-08T05:30:44.490618+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":460,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490523+0000","last_change":"2023-11-08T05:28:13.545468+0000","last_active":"2023-11-08T05:30:44.490523+0000","last_peered":"2023-11-08T05:30:44.490523+0000","last_clean":"2023-11-08T05:28:01.456723+0000","last_became_active":"2023-11-08T05:28:13.545468+0000","last_became_peered":"2023-11-08T05:28:13.545468+0000","last_unstale":"2023-11-08T05:30:44.490523+0000","last_undegraded":"2023-11-08T05:30:44.490523+0000","last_fullsized":"2023-11-08T05:28:13.537207+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_clean_scrub_stamp":"2023-11-08T04:52:45.528796+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":465,"reported_epoch":458,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:44.491907+0000","last_change":"2023-11-08T05:29:44.395350+0000","last_active":"2023-11-08T05:30:44.491907+0000","last_peered":"2023-11-08T05:30:44.491907+0000","last_clean":"2023-11-08T05:30:44.491907+0000","last_became_active":"2023-11-08T05:29:44.395012+0000","last_became_peered":"2023-11-08T05:29:44.395012+0000","last_unstale":"2023-11-08T05:30:44.491907+0000","last_undegraded":"2023-11-08T05:30:44.491907+0000","last_fullsized":"2023-11-08T05:30:44.491907+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490629+0000","last_change":"2023-11-08T05:28:13.563357+0000","last_active":"2023-11-08T05:30:44.490629+0000","last_peered":"2023-11-08T05:30:44.490629+0000","last_clean":"2023-11-08T05:28:01.457691+0000","last_became_active":"2023-11-08T05:28:13.563357+0000","last_became_peered":"2023-11-08T05:28:13.563357+0000","last_unstale":"2023-11-08T05:30:44.490629+0000","last_undegraded":"2023-11-08T05:30:44.490629+0000","last_fullsized":"2023-11-08T05:28:13.541614+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490691+0000","last_change":"2023-11-08T05:28:13.550672+0000","last_active":"2023-11-08T05:30:44.490691+0000","last_peered":"2023-11-08T05:30:44.490691+0000","last_clean":"2023-11-08T05:28:01.457904+0000","last_became_active":"2023-11-08T05:28:13.550672+0000","last_became_peered":"2023-11-08T05:28:13.550672+0000","last_unstale":"2023-11-08T05:30:44.490691+0000","last_undegraded":"2023-11-08T05:30:44.490691+0000","last_fullsized":"2023-11-08T05:28:13.539595+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490805+0000","last_change":"2023-11-08T05:28:13.557968+0000","last_active":"2023-11-08T05:30:44.490805+0000","last_peered":"2023-11-08T05:30:44.490805+0000","last_clean":"2023-11-08T05:28:01.458433+0000","last_became_active":"2023-11-08T05:28:13.557968+0000","last_became_peered":"2023-11-08T05:28:13.557968+0000","last_unstale":"2023-11-08T05:30:44.490805+0000","last_undegraded":"2023-11-08T05:30:44.490805+0000","last_fullsized":"2023-11-08T05:28:13.542832+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":1,"acting":1,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":12,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738836,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":645040,"kb_used_data":5352,"kb_used_omap":275,"kb_used_meta":639404,"kb_avail":93726800,"statfs":{"total":96636764160,"available":95976243200,"internally_reserved":0,"allocated":5480448,"data_stored":2697647,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":281650,"internal_metadata":654750670},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:59 2023","interfaces":[{"interface":"back","average":{"1min":0.30499999999999999,"5min":0.439,"15min":0.36099999999999999},"min":{"1min":0.17399999999999999,"5min":0.14499999999999999,"15min":0.14499999999999999},"max":{"1min":0.60099999999999998,"5min":4.9379999999999997,"15min":4.9379999999999997},"last":0.379},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.49399999999999999,"15min":0.42399999999999999},"min":{"1min":0.221,"5min":0.19600000000000001,"15min":0.17000000000000001},"max":{"1min":0.63200000000000001,"5min":4.9859999999999998,"15min":4.9859999999999998},"last":0.40899999999999997}]},{"osd":1,"last update":"Wed Nov 8 05:31:06 2023","interfaces":[{"interface":"back","average":{"1min":0.27500000000000002,"5min":0.40300000000000002,"15min":0.46999999999999997},"min":{"1min":0.20899999999999999,"5min":0.16,"15min":0.153},"max":{"1min":0.41099999999999998,"5min":4.0830000000000002,"15min":23.242999999999999},"last":0.223},{"interface":"front","average":{"1min":0.29899999999999999,"5min":0.40100000000000002,"15min":0.46000000000000002},"min":{"1min":0.23799999999999999,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":0.45300000000000001,"5min":3.5590000000000002,"15min":23.251000000000001},"last":0.24099999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738798,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1339228,"kb_used_data":398304,"kb_used_omap":291,"kb_used_meta":936476,"kb_avail":93032612,"statfs":{"total":96636764160,"available":95265394688,"internally_reserved":0,"allocated":407863296,"data_stored":405103900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":298462,"internal_metadata":958951970},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.436,"15min":0.36299999999999999},"min":{"1min":0.19700000000000001,"5min":0.14699999999999999,"15min":0.13700000000000001},"max":{"1min":3.222,"5min":6.5499999999999998,"15min":6.9989999999999997},"last":0.245},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.46300000000000002,"15min":0.42299999999999999},"min":{"1min":0.17799999999999999,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":5.0030000000000001,"5min":6.0629999999999997,"15min":9.6869999999999994},"last":0.28599999999999998}]},{"osd":2,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.51600000000000001,"15min":0.47099999999999997},"min":{"1min":0.21099999999999999,"5min":0.153,"15min":0.153},"max":{"1min":5.7439999999999998,"5min":5.8280000000000003,"15min":9.7639999999999993},"last":0.252},{"interface":"front","average":{"1min":0.497,"5min":0.56200000000000006,"15min":0.42299999999999999},"min":{"1min":0.20399999999999999,"5min":0.187,"15min":0.16800000000000001},"max":{"1min":3.9319999999999999,"5min":8.8040000000000003,"15min":8.8040000000000003},"last":0.29899999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738838,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":650100,"kb_used_data":5356,"kb_used_omap":214,"kb_used_meta":644521,"kb_avail":93721740,"statfs":{"total":96636764160,"available":95971061760,"internally_reserved":0,"allocated":5484544,"data_stored":2698616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":219335,"internal_metadata":659990329},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:56 2023","interfaces":[{"interface":"back","average":{"1min":0.30399999999999999,"5min":0.34699999999999998,"15min":0.36499999999999999},"min":{"1min":0.23999999999999999,"5min":0.14199999999999999,"15min":0.14199999999999999},"max":{"1min":0.42399999999999999,"5min":1.595,"15min":7.4249999999999998},"last":0.17100000000000001},{"interface":"front","average":{"1min":0.35499999999999998,"5min":0.40899999999999997,"15min":0.376},"min":{"1min":0.23999999999999999,"5min":0.19600000000000001,"15min":0.183},"max":{"1min":0.46999999999999997,"5min":2.6859999999999999,"15min":2.6859999999999999},"last":0.23000000000000001}]},{"osd":2,"last update":"Wed Nov 8 05:30:57 2023","interfaces":[{"interface":"back","average":{"1min":0.28599999999999998,"5min":0.34699999999999998,"15min":0.36899999999999999},"min":{"1min":0.19700000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":0.41399999999999998,"5min":2.2650000000000001,"15min":5.8209999999999997},"last":0.20499999999999999},{"interface":"front","average":{"1min":0.30599999999999999,"5min":0.313,"15min":0.35799999999999998},"min":{"1min":0.17799999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.435,"5min":1.137,"15min":4.2350000000000003},"last":0.24199999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-08T05:31:36.647 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-11-08T05:31:36.945 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T05:31:36.946 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2023-11-08T05:31:36.958 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":1622,"stamp":"2023-11-08T05:31:36.070196+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":10,"acting":13,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":109,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2634368,"kb_used_data":409012,"kb_used_omap":780,"kb_used_meta":2220403,"kb_avail":280481152,"statfs":{"total":289910292480,"available":287212699648,"internally_reserved":0,"allocated":418828288,"data_stored":410500163,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":799447,"internal_metadata":2273692969},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-536782902,"num_objects":-164,"num_object_clones":0,"num_object_copies":-328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-115,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-164,"num_whiteouts":0,"num_read":-122056,"num_read_kb":-39011942,"num_write":-413642,"num_write_kb":-151705339,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-8,"num_bytes_recovered":-16922624,"num_keys_recovered":-2,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.424817"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490644+0000","last_change":"2023-11-08T05:28:13.557071+0000","last_active":"2023-11-08T05:30:44.490644+0000","last_peered":"2023-11-08T05:30:44.490644+0000","last_clean":"2023-11-08T05:28:01.458439+0000","last_became_active":"2023-11-08T05:28:13.557071+0000","last_became_peered":"2023-11-08T05:28:13.557071+0000","last_unstale":"2023-11-08T05:30:44.490644+0000","last_undegraded":"2023-11-08T05:30:44.490644+0000","last_fullsized":"2023-11-08T05:28:13.547025+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":459,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:45.794760+0000","last_change":"2023-11-08T05:30:45.794760+0000","last_active":"2023-11-08T05:30:45.794760+0000","last_peered":"2023-11-08T05:30:45.794760+0000","last_clean":"2023-11-08T05:30:45.794760+0000","last_became_active":"2023-11-08T05:30:45.794398+0000","last_became_peered":"2023-11-08T05:30:45.794398+0000","last_unstale":"2023-11-08T05:30:45.794760+0000","last_undegraded":"2023-11-08T05:30:45.794760+0000","last_fullsized":"2023-11-08T05:30:45.794760+0000","mapping_epoch":458,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":459,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":465,"reported_epoch":459,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:45.795085+0000","last_change":"2023-11-08T05:30:45.795085+0000","last_active":"2023-11-08T05:30:45.795085+0000","last_peered":"2023-11-08T05:30:45.795085+0000","last_clean":"2023-11-08T05:30:45.795085+0000","last_became_active":"2023-11-08T05:30:45.794743+0000","last_became_peered":"2023-11-08T05:30:45.794743+0000","last_unstale":"2023-11-08T05:30:45.795085+0000","last_undegraded":"2023-11-08T05:30:45.795085+0000","last_fullsized":"2023-11-08T05:30:45.795085+0000","mapping_epoch":458,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":459,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":454,"reported_epoch":458,"state":"active+clean","last_fresh":"2023-11-08T05:30:44.490618+0000","last_change":"2023-11-08T04:52:47.548597+0000","last_active":"2023-11-08T05:30:44.490618+0000","last_peered":"2023-11-08T05:30:44.490618+0000","last_clean":"2023-11-08T05:30:44.490618+0000","last_became_active":"2023-11-08T04:52:47.548294+0000","last_became_peered":"2023-11-08T04:52:47.548294+0000","last_unstale":"2023-11-08T05:30:44.490618+0000","last_undegraded":"2023-11-08T05:30:44.490618+0000","last_fullsized":"2023-11-08T05:30:44.490618+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":460,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490523+0000","last_change":"2023-11-08T05:28:13.545468+0000","last_active":"2023-11-08T05:30:44.490523+0000","last_peered":"2023-11-08T05:30:44.490523+0000","last_clean":"2023-11-08T05:28:01.456723+0000","last_became_active":"2023-11-08T05:28:13.545468+0000","last_became_peered":"2023-11-08T05:28:13.545468+0000","last_unstale":"2023-11-08T05:30:44.490523+0000","last_undegraded":"2023-11-08T05:30:44.490523+0000","last_fullsized":"2023-11-08T05:28:13.537207+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_clean_scrub_stamp":"2023-11-08T04:52:45.528796+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":465,"reported_epoch":458,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:44.491907+0000","last_change":"2023-11-08T05:29:44.395350+0000","last_active":"2023-11-08T05:30:44.491907+0000","last_peered":"2023-11-08T05:30:44.491907+0000","last_clean":"2023-11-08T05:30:44.491907+0000","last_became_active":"2023-11-08T05:29:44.395012+0000","last_became_peered":"2023-11-08T05:29:44.395012+0000","last_unstale":"2023-11-08T05:30:44.491907+0000","last_undegraded":"2023-11-08T05:30:44.491907+0000","last_fullsized":"2023-11-08T05:30:44.491907+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490629+0000","last_change":"2023-11-08T05:28:13.563357+0000","last_active":"2023-11-08T05:30:44.490629+0000","last_peered":"2023-11-08T05:30:44.490629+0000","last_clean":"2023-11-08T05:28:01.457691+0000","last_became_active":"2023-11-08T05:28:13.563357+0000","last_became_peered":"2023-11-08T05:28:13.563357+0000","last_unstale":"2023-11-08T05:30:44.490629+0000","last_undegraded":"2023-11-08T05:30:44.490629+0000","last_fullsized":"2023-11-08T05:28:13.541614+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490691+0000","last_change":"2023-11-08T05:28:13.550672+0000","last_active":"2023-11-08T05:30:44.490691+0000","last_peered":"2023-11-08T05:30:44.490691+0000","last_clean":"2023-11-08T05:28:01.457904+0000","last_became_active":"2023-11-08T05:28:13.550672+0000","last_became_peered":"2023-11-08T05:28:13.550672+0000","last_unstale":"2023-11-08T05:30:44.490691+0000","last_undegraded":"2023-11-08T05:30:44.490691+0000","last_fullsized":"2023-11-08T05:28:13.539595+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490805+0000","last_change":"2023-11-08T05:28:13.557968+0000","last_active":"2023-11-08T05:30:44.490805+0000","last_peered":"2023-11-08T05:30:44.490805+0000","last_clean":"2023-11-08T05:28:01.458433+0000","last_became_active":"2023-11-08T05:28:13.557968+0000","last_became_peered":"2023-11-08T05:28:13.557968+0000","last_unstale":"2023-11-08T05:30:44.490805+0000","last_undegraded":"2023-11-08T05:30:44.490805+0000","last_fullsized":"2023-11-08T05:28:13.542832+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":1,"acting":1,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":12,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738837,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":645040,"kb_used_data":5352,"kb_used_omap":275,"kb_used_meta":639404,"kb_avail":93726800,"statfs":{"total":96636764160,"available":95976243200,"internally_reserved":0,"allocated":5480448,"data_stored":2697647,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":281650,"internal_metadata":654750670},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:59 2023","interfaces":[{"interface":"back","average":{"1min":0.30499999999999999,"5min":0.439,"15min":0.36099999999999999},"min":{"1min":0.17399999999999999,"5min":0.14499999999999999,"15min":0.14499999999999999},"max":{"1min":0.60099999999999998,"5min":4.9379999999999997,"15min":4.9379999999999997},"last":0.379},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.49399999999999999,"15min":0.42399999999999999},"min":{"1min":0.221,"5min":0.19600000000000001,"15min":0.17000000000000001},"max":{"1min":0.63200000000000001,"5min":4.9859999999999998,"15min":4.9859999999999998},"last":0.40899999999999997}]},{"osd":1,"last update":"Wed Nov 8 05:31:06 2023","interfaces":[{"interface":"back","average":{"1min":0.27500000000000002,"5min":0.40300000000000002,"15min":0.46999999999999997},"min":{"1min":0.20899999999999999,"5min":0.16,"15min":0.153},"max":{"1min":0.41099999999999998,"5min":4.0830000000000002,"15min":23.242999999999999},"last":0.17599999999999999},{"interface":"front","average":{"1min":0.29899999999999999,"5min":0.40100000000000002,"15min":0.46000000000000002},"min":{"1min":0.23799999999999999,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":0.45300000000000001,"5min":3.5590000000000002,"15min":23.251000000000001},"last":0.20200000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738798,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1339228,"kb_used_data":398304,"kb_used_omap":291,"kb_used_meta":936476,"kb_avail":93032612,"statfs":{"total":96636764160,"available":95265394688,"internally_reserved":0,"allocated":407863296,"data_stored":405103900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":298462,"internal_metadata":958951970},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.436,"15min":0.36299999999999999},"min":{"1min":0.19700000000000001,"5min":0.14699999999999999,"15min":0.13700000000000001},"max":{"1min":3.222,"5min":6.5499999999999998,"15min":6.9989999999999997},"last":0.245},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.46300000000000002,"15min":0.42299999999999999},"min":{"1min":0.17799999999999999,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":5.0030000000000001,"5min":6.0629999999999997,"15min":9.6869999999999994},"last":0.28599999999999998}]},{"osd":2,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.51600000000000001,"15min":0.47099999999999997},"min":{"1min":0.21099999999999999,"5min":0.153,"15min":0.153},"max":{"1min":5.7439999999999998,"5min":5.8280000000000003,"15min":9.7639999999999993},"last":0.252},{"interface":"front","average":{"1min":0.497,"5min":0.56200000000000006,"15min":0.42299999999999999},"min":{"1min":0.20399999999999999,"5min":0.187,"15min":0.16800000000000001},"max":{"1min":3.9319999999999999,"5min":8.8040000000000003,"15min":8.8040000000000003},"last":0.29899999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738839,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":650100,"kb_used_data":5356,"kb_used_omap":214,"kb_used_meta":644521,"kb_avail":93721740,"statfs":{"total":96636764160,"available":95971061760,"internally_reserved":0,"allocated":5484544,"data_stored":2698616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":219335,"internal_metadata":659990329},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:56 2023","interfaces":[{"interface":"back","average":{"1min":0.30399999999999999,"5min":0.34699999999999998,"15min":0.36499999999999999},"min":{"1min":0.23999999999999999,"5min":0.14199999999999999,"15min":0.14199999999999999},"max":{"1min":0.42399999999999999,"5min":1.595,"15min":7.4249999999999998},"last":0.17100000000000001},{"interface":"front","average":{"1min":0.35499999999999998,"5min":0.40899999999999997,"15min":0.376},"min":{"1min":0.23999999999999999,"5min":0.19600000000000001,"15min":0.183},"max":{"1min":0.46999999999999997,"5min":2.6859999999999999,"15min":2.6859999999999999},"last":0.23000000000000001}]},{"osd":2,"last update":"Wed Nov 8 05:30:57 2023","interfaces":[{"interface":"back","average":{"1min":0.28599999999999998,"5min":0.34699999999999998,"15min":0.36899999999999999},"min":{"1min":0.19700000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":0.41399999999999998,"5min":2.2650000000000001,"15min":5.8209999999999997},"last":0.161},{"interface":"front","average":{"1min":0.30599999999999999,"5min":0.313,"15min":0.35799999999999998},"min":{"1min":0.17799999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.435,"5min":1.137,"15min":4.2350000000000003},"last":0.17000000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-08T05:31:36.959 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-11-08T05:31:37.239 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T05:31:37.290 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2023-11-08T05:31:37.290 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":1622,"stamp":"2023-11-08T05:31:36.070196+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":10,"acting":13,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":109,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2634368,"kb_used_data":409012,"kb_used_omap":780,"kb_used_meta":2220403,"kb_avail":280481152,"statfs":{"total":289910292480,"available":287212699648,"internally_reserved":0,"allocated":418828288,"data_stored":410500163,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":799447,"internal_metadata":2273692969},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-536782902,"num_objects":-164,"num_object_clones":0,"num_object_copies":-328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-115,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-164,"num_whiteouts":0,"num_read":-122056,"num_read_kb":-39011942,"num_write":-413642,"num_write_kb":-151705339,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-8,"num_bytes_recovered":-16922624,"num_keys_recovered":-2,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.424817"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490644+0000","last_change":"2023-11-08T05:28:13.557071+0000","last_active":"2023-11-08T05:30:44.490644+0000","last_peered":"2023-11-08T05:30:44.490644+0000","last_clean":"2023-11-08T05:28:01.458439+0000","last_became_active":"2023-11-08T05:28:13.557071+0000","last_became_peered":"2023-11-08T05:28:13.557071+0000","last_unstale":"2023-11-08T05:30:44.490644+0000","last_undegraded":"2023-11-08T05:30:44.490644+0000","last_fullsized":"2023-11-08T05:28:13.547025+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":459,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:45.794760+0000","last_change":"2023-11-08T05:30:45.794760+0000","last_active":"2023-11-08T05:30:45.794760+0000","last_peered":"2023-11-08T05:30:45.794760+0000","last_clean":"2023-11-08T05:30:45.794760+0000","last_became_active":"2023-11-08T05:30:45.794398+0000","last_became_peered":"2023-11-08T05:30:45.794398+0000","last_unstale":"2023-11-08T05:30:45.794760+0000","last_undegraded":"2023-11-08T05:30:45.794760+0000","last_fullsized":"2023-11-08T05:30:45.794760+0000","mapping_epoch":458,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":459,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":465,"reported_epoch":459,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:45.795085+0000","last_change":"2023-11-08T05:30:45.795085+0000","last_active":"2023-11-08T05:30:45.795085+0000","last_peered":"2023-11-08T05:30:45.795085+0000","last_clean":"2023-11-08T05:30:45.795085+0000","last_became_active":"2023-11-08T05:30:45.794743+0000","last_became_peered":"2023-11-08T05:30:45.794743+0000","last_unstale":"2023-11-08T05:30:45.795085+0000","last_undegraded":"2023-11-08T05:30:45.795085+0000","last_fullsized":"2023-11-08T05:30:45.795085+0000","mapping_epoch":458,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":459,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":454,"reported_epoch":458,"state":"active+clean","last_fresh":"2023-11-08T05:30:44.490618+0000","last_change":"2023-11-08T04:52:47.548597+0000","last_active":"2023-11-08T05:30:44.490618+0000","last_peered":"2023-11-08T05:30:44.490618+0000","last_clean":"2023-11-08T05:30:44.490618+0000","last_became_active":"2023-11-08T04:52:47.548294+0000","last_became_peered":"2023-11-08T04:52:47.548294+0000","last_unstale":"2023-11-08T05:30:44.490618+0000","last_undegraded":"2023-11-08T05:30:44.490618+0000","last_fullsized":"2023-11-08T05:30:44.490618+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":460,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490523+0000","last_change":"2023-11-08T05:28:13.545468+0000","last_active":"2023-11-08T05:30:44.490523+0000","last_peered":"2023-11-08T05:30:44.490523+0000","last_clean":"2023-11-08T05:28:01.456723+0000","last_became_active":"2023-11-08T05:28:13.545468+0000","last_became_peered":"2023-11-08T05:28:13.545468+0000","last_unstale":"2023-11-08T05:30:44.490523+0000","last_undegraded":"2023-11-08T05:30:44.490523+0000","last_fullsized":"2023-11-08T05:28:13.537207+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_clean_scrub_stamp":"2023-11-08T04:52:45.528796+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":465,"reported_epoch":458,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:44.491907+0000","last_change":"2023-11-08T05:29:44.395350+0000","last_active":"2023-11-08T05:30:44.491907+0000","last_peered":"2023-11-08T05:30:44.491907+0000","last_clean":"2023-11-08T05:30:44.491907+0000","last_became_active":"2023-11-08T05:29:44.395012+0000","last_became_peered":"2023-11-08T05:29:44.395012+0000","last_unstale":"2023-11-08T05:30:44.491907+0000","last_undegraded":"2023-11-08T05:30:44.491907+0000","last_fullsized":"2023-11-08T05:30:44.491907+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490629+0000","last_change":"2023-11-08T05:28:13.563357+0000","last_active":"2023-11-08T05:30:44.490629+0000","last_peered":"2023-11-08T05:30:44.490629+0000","last_clean":"2023-11-08T05:28:01.457691+0000","last_became_active":"2023-11-08T05:28:13.563357+0000","last_became_peered":"2023-11-08T05:28:13.563357+0000","last_unstale":"2023-11-08T05:30:44.490629+0000","last_undegraded":"2023-11-08T05:30:44.490629+0000","last_fullsized":"2023-11-08T05:28:13.541614+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490691+0000","last_change":"2023-11-08T05:28:13.550672+0000","last_active":"2023-11-08T05:30:44.490691+0000","last_peered":"2023-11-08T05:30:44.490691+0000","last_clean":"2023-11-08T05:28:01.457904+0000","last_became_active":"2023-11-08T05:28:13.550672+0000","last_became_peered":"2023-11-08T05:28:13.550672+0000","last_unstale":"2023-11-08T05:30:44.490691+0000","last_undegraded":"2023-11-08T05:30:44.490691+0000","last_fullsized":"2023-11-08T05:28:13.539595+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490805+0000","last_change":"2023-11-08T05:28:13.557968+0000","last_active":"2023-11-08T05:30:44.490805+0000","last_peered":"2023-11-08T05:30:44.490805+0000","last_clean":"2023-11-08T05:28:01.458433+0000","last_became_active":"2023-11-08T05:28:13.557968+0000","last_became_peered":"2023-11-08T05:28:13.557968+0000","last_unstale":"2023-11-08T05:30:44.490805+0000","last_undegraded":"2023-11-08T05:30:44.490805+0000","last_fullsized":"2023-11-08T05:28:13.542832+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":1,"acting":1,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":12,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738837,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":645040,"kb_used_data":5352,"kb_used_omap":275,"kb_used_meta":639404,"kb_avail":93726800,"statfs":{"total":96636764160,"available":95976243200,"internally_reserved":0,"allocated":5480448,"data_stored":2697647,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":281650,"internal_metadata":654750670},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:59 2023","interfaces":[{"interface":"back","average":{"1min":0.30499999999999999,"5min":0.439,"15min":0.36099999999999999},"min":{"1min":0.17399999999999999,"5min":0.14499999999999999,"15min":0.14499999999999999},"max":{"1min":0.60099999999999998,"5min":4.9379999999999997,"15min":4.9379999999999997},"last":0.379},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.49399999999999999,"15min":0.42399999999999999},"min":{"1min":0.221,"5min":0.19600000000000001,"15min":0.17000000000000001},"max":{"1min":0.63200000000000001,"5min":4.9859999999999998,"15min":4.9859999999999998},"last":0.40899999999999997}]},{"osd":1,"last update":"Wed Nov 8 05:31:06 2023","interfaces":[{"interface":"back","average":{"1min":0.27500000000000002,"5min":0.40300000000000002,"15min":0.46999999999999997},"min":{"1min":0.20899999999999999,"5min":0.16,"15min":0.153},"max":{"1min":0.41099999999999998,"5min":4.0830000000000002,"15min":23.242999999999999},"last":0.17599999999999999},{"interface":"front","average":{"1min":0.29899999999999999,"5min":0.40100000000000002,"15min":0.46000000000000002},"min":{"1min":0.23799999999999999,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":0.45300000000000001,"5min":3.5590000000000002,"15min":23.251000000000001},"last":0.20200000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738798,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1339228,"kb_used_data":398304,"kb_used_omap":291,"kb_used_meta":936476,"kb_avail":93032612,"statfs":{"total":96636764160,"available":95265394688,"internally_reserved":0,"allocated":407863296,"data_stored":405103900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":298462,"internal_metadata":958951970},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.436,"15min":0.36299999999999999},"min":{"1min":0.19700000000000001,"5min":0.14699999999999999,"15min":0.13700000000000001},"max":{"1min":3.222,"5min":6.5499999999999998,"15min":6.9989999999999997},"last":0.245},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.46300000000000002,"15min":0.42299999999999999},"min":{"1min":0.17799999999999999,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":5.0030000000000001,"5min":6.0629999999999997,"15min":9.6869999999999994},"last":0.28599999999999998}]},{"osd":2,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.51600000000000001,"15min":0.47099999999999997},"min":{"1min":0.21099999999999999,"5min":0.153,"15min":0.153},"max":{"1min":5.7439999999999998,"5min":5.8280000000000003,"15min":9.7639999999999993},"last":0.252},{"interface":"front","average":{"1min":0.497,"5min":0.56200000000000006,"15min":0.42299999999999999},"min":{"1min":0.20399999999999999,"5min":0.187,"15min":0.16800000000000001},"max":{"1min":3.9319999999999999,"5min":8.8040000000000003,"15min":8.8040000000000003},"last":0.29899999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738839,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":650100,"kb_used_data":5356,"kb_used_omap":214,"kb_used_meta":644521,"kb_avail":93721740,"statfs":{"total":96636764160,"available":95971061760,"internally_reserved":0,"allocated":5484544,"data_stored":2698616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":219335,"internal_metadata":659990329},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:56 2023","interfaces":[{"interface":"back","average":{"1min":0.30399999999999999,"5min":0.34699999999999998,"15min":0.36499999999999999},"min":{"1min":0.23999999999999999,"5min":0.14199999999999999,"15min":0.14199999999999999},"max":{"1min":0.42399999999999999,"5min":1.595,"15min":7.4249999999999998},"last":0.17100000000000001},{"interface":"front","average":{"1min":0.35499999999999998,"5min":0.40899999999999997,"15min":0.376},"min":{"1min":0.23999999999999999,"5min":0.19600000000000001,"15min":0.183},"max":{"1min":0.46999999999999997,"5min":2.6859999999999999,"15min":2.6859999999999999},"last":0.23000000000000001}]},{"osd":2,"last update":"Wed Nov 8 05:30:57 2023","interfaces":[{"interface":"back","average":{"1min":0.28599999999999998,"5min":0.34699999999999998,"15min":0.36899999999999999},"min":{"1min":0.19700000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":0.41399999999999998,"5min":2.2650000000000001,"15min":5.8209999999999997},"last":0.161},{"interface":"front","average":{"1min":0.30599999999999999,"5min":0.313,"15min":0.35799999999999998},"min":{"1min":0.17799999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.435,"5min":1.137,"15min":4.2350000000000003},"last":0.17000000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-08T05:31:37.292 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-11-08T05:31:37.292 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 459, 'reported_epoch': 458, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:30:44.490644+0000', 'last_change': '2023-11-08T05:28:13.557071+0000', 'last_active': '2023-11-08T05:30:44.490644+0000', 'last_peered': '2023-11-08T05:30:44.490644+0000', 'last_clean': '2023-11-08T05:28:01.458439+0000', 'last_became_active': '2023-11-08T05:28:13.557071+0000', 'last_became_peered': '2023-11-08T05:28:13.557071+0000', 'last_unstale': '2023-11-08T05:30:44.490644+0000', 'last_undegraded': '2023-11-08T05:30:44.490644+0000', 'last_fullsized': '2023-11-08T05:28:13.547025+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:31:37.292 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-11-08T05:31:37.292 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 11, 'reported_epoch': 459, 'state': 'active+clean+remapped', 'last_fresh': '2023-11-08T05:30:45.794760+0000', 'last_change': '2023-11-08T05:30:45.794760+0000', 'last_active': '2023-11-08T05:30:45.794760+0000', 'last_peered': '2023-11-08T05:30:45.794760+0000', 'last_clean': '2023-11-08T05:30:45.794760+0000', 'last_became_active': '2023-11-08T05:30:45.794398+0000', 'last_became_peered': '2023-11-08T05:30:45.794398+0000', 'last_unstale': '2023-11-08T05:30:45.794760+0000', 'last_undegraded': '2023-11-08T05:30:45.794760+0000', 'last_fullsized': '2023-11-08T05:30:45.794760+0000', 'mapping_epoch': 458, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 459, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2], 'acting': [2, 1], 'avail_no_missing': ['2', '1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2023-11-08T05:31:37.293 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.1 is not active+clean 2023-11-08T05:31:37.293 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.1', 'version': "0'0", 'reported_seq': 465, 'reported_epoch': 459, 'state': 'active+clean+remapped', 'last_fresh': '2023-11-08T05:30:45.795085+0000', 'last_change': '2023-11-08T05:30:45.795085+0000', 'last_active': '2023-11-08T05:30:45.795085+0000', 'last_peered': '2023-11-08T05:30:45.795085+0000', 'last_clean': '2023-11-08T05:30:45.795085+0000', 'last_became_active': '2023-11-08T05:30:45.794743+0000', 'last_became_peered': '2023-11-08T05:30:45.794743+0000', 'last_unstale': '2023-11-08T05:30:45.795085+0000', 'last_undegraded': '2023-11-08T05:30:45.795085+0000', 'last_fullsized': '2023-11-08T05:30:45.795085+0000', 'mapping_epoch': 458, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 459, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2], 'acting': [2, 1], 'avail_no_missing': ['2', '1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2023-11-08T05:31:37.293 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-11-08T05:31:37.294 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "0'0", 'reported_seq': 460, 'reported_epoch': 458, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:30:44.490523+0000', 'last_change': '2023-11-08T05:28:13.545468+0000', 'last_active': '2023-11-08T05:30:44.490523+0000', 'last_peered': '2023-11-08T05:30:44.490523+0000', 'last_clean': '2023-11-08T05:28:01.456723+0000', 'last_became_active': '2023-11-08T05:28:13.545468+0000', 'last_became_peered': '2023-11-08T05:28:13.545468+0000', 'last_unstale': '2023-11-08T05:30:44.490523+0000', 'last_undegraded': '2023-11-08T05:30:44.490523+0000', 'last_fullsized': '2023-11-08T05:28:13.537207+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 9, 'last_epoch_clean': 10, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:45.528796+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:45.528796+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:45.528796+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:31:37.294 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.0 is not active+clean 2023-11-08T05:31:37.294 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.0', 'version': "0'0", 'reported_seq': 465, 'reported_epoch': 458, 'state': 'active+clean+remapped', 'last_fresh': '2023-11-08T05:30:44.491907+0000', 'last_change': '2023-11-08T05:29:44.395350+0000', 'last_active': '2023-11-08T05:30:44.491907+0000', 'last_peered': '2023-11-08T05:30:44.491907+0000', 'last_clean': '2023-11-08T05:30:44.491907+0000', 'last_became_active': '2023-11-08T05:29:44.395012+0000', 'last_became_peered': '2023-11-08T05:29:44.395012+0000', 'last_unstale': '2023-11-08T05:30:44.491907+0000', 'last_undegraded': '2023-11-08T05:30:44.491907+0000', 'last_fullsized': '2023-11-08T05:30:44.491907+0000', 'mapping_epoch': 455, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 456, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2], 'acting': [2, 1], 'avail_no_missing': ['2', '1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2023-11-08T05:31:37.295 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-11-08T05:31:37.295 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 459, 'reported_epoch': 458, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:30:44.490629+0000', 'last_change': '2023-11-08T05:28:13.563357+0000', 'last_active': '2023-11-08T05:30:44.490629+0000', 'last_peered': '2023-11-08T05:30:44.490629+0000', 'last_clean': '2023-11-08T05:28:01.457691+0000', 'last_became_active': '2023-11-08T05:28:13.563357+0000', 'last_became_peered': '2023-11-08T05:28:13.563357+0000', 'last_unstale': '2023-11-08T05:30:44.490629+0000', 'last_undegraded': '2023-11-08T05:30:44.490629+0000', 'last_fullsized': '2023-11-08T05:28:13.541614+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:31:37.295 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-11-08T05:31:37.295 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 459, 'reported_epoch': 458, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:30:44.490691+0000', 'last_change': '2023-11-08T05:28:13.550672+0000', 'last_active': '2023-11-08T05:30:44.490691+0000', 'last_peered': '2023-11-08T05:30:44.490691+0000', 'last_clean': '2023-11-08T05:28:01.457904+0000', 'last_became_active': '2023-11-08T05:28:13.550672+0000', 'last_became_peered': '2023-11-08T05:28:13.550672+0000', 'last_unstale': '2023-11-08T05:30:44.490691+0000', 'last_undegraded': '2023-11-08T05:30:44.490691+0000', 'last_fullsized': '2023-11-08T05:28:13.539595+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:31:37.296 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-11-08T05:31:37.296 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 459, 'reported_epoch': 458, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:30:44.490805+0000', 'last_change': '2023-11-08T05:28:13.557968+0000', 'last_active': '2023-11-08T05:30:44.490805+0000', 'last_peered': '2023-11-08T05:30:44.490805+0000', 'last_clean': '2023-11-08T05:28:01.458433+0000', 'last_became_active': '2023-11-08T05:28:13.557968+0000', 'last_became_peered': '2023-11-08T05:28:13.557968+0000', 'last_unstale': '2023-11-08T05:30:44.490805+0000', 'last_undegraded': '2023-11-08T05:30:44.490805+0000', 'last_fullsized': '2023-11-08T05:28:13.542832+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:31:37.297 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-11-08T05:31:37.642 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T05:31:37.642 INFO:teuthology.orchestra.run.smithi087.stdout:{"fsid":"ab4cbc06-449d-497d-b513-220d9db2b590","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 5 pgs undersized","count":5},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2340,"monmap":{"epoch":1,"min_mon_release_name":"pacific","num_mons":1},"osdmap":{"epoch":459,"num_osds":3,"num_up_osds":2,"osd_up_since":1699421292,"num_in_osds":3,"osd_in_since":1699419158,"num_remapped_pgs":3},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean+remapped","count":3},{"state_name":"active+clean","count":1}],"num_pgs":9,"num_pools":2,"num_objects":0,"data_bytes":0,"bytes_used":2697592832,"bytes_avail":287212699648,"bytes_total":289910292480},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":41,"modified":"2023-11-08T05:29:32.044544+0000","services":{}},"progress_events":{}} 2023-11-08T05:31:37.656 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-11-08T05:31:37.656 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-11-08T05:31:37.938 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T05:31:37.940 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2023-11-08T05:31:37.951 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":1622,"stamp":"2023-11-08T05:31:36.070196+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":10,"acting":13,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":109,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2634368,"kb_used_data":409012,"kb_used_omap":780,"kb_used_meta":2220403,"kb_avail":280481152,"statfs":{"total":289910292480,"available":287212699648,"internally_reserved":0,"allocated":418828288,"data_stored":410500163,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":799447,"internal_metadata":2273692969},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-536782902,"num_objects":-164,"num_object_clones":0,"num_object_copies":-328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-115,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-164,"num_whiteouts":0,"num_read":-122056,"num_read_kb":-39011942,"num_write":-413642,"num_write_kb":-151705339,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-8,"num_bytes_recovered":-16922624,"num_keys_recovered":-2,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.424817"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490644+0000","last_change":"2023-11-08T05:28:13.557071+0000","last_active":"2023-11-08T05:30:44.490644+0000","last_peered":"2023-11-08T05:30:44.490644+0000","last_clean":"2023-11-08T05:28:01.458439+0000","last_became_active":"2023-11-08T05:28:13.557071+0000","last_became_peered":"2023-11-08T05:28:13.557071+0000","last_unstale":"2023-11-08T05:30:44.490644+0000","last_undegraded":"2023-11-08T05:30:44.490644+0000","last_fullsized":"2023-11-08T05:28:13.547025+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":459,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:45.794760+0000","last_change":"2023-11-08T05:30:45.794760+0000","last_active":"2023-11-08T05:30:45.794760+0000","last_peered":"2023-11-08T05:30:45.794760+0000","last_clean":"2023-11-08T05:30:45.794760+0000","last_became_active":"2023-11-08T05:30:45.794398+0000","last_became_peered":"2023-11-08T05:30:45.794398+0000","last_unstale":"2023-11-08T05:30:45.794760+0000","last_undegraded":"2023-11-08T05:30:45.794760+0000","last_fullsized":"2023-11-08T05:30:45.794760+0000","mapping_epoch":458,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":459,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":465,"reported_epoch":459,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:45.795085+0000","last_change":"2023-11-08T05:30:45.795085+0000","last_active":"2023-11-08T05:30:45.795085+0000","last_peered":"2023-11-08T05:30:45.795085+0000","last_clean":"2023-11-08T05:30:45.795085+0000","last_became_active":"2023-11-08T05:30:45.794743+0000","last_became_peered":"2023-11-08T05:30:45.794743+0000","last_unstale":"2023-11-08T05:30:45.795085+0000","last_undegraded":"2023-11-08T05:30:45.795085+0000","last_fullsized":"2023-11-08T05:30:45.795085+0000","mapping_epoch":458,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":459,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":454,"reported_epoch":458,"state":"active+clean","last_fresh":"2023-11-08T05:30:44.490618+0000","last_change":"2023-11-08T04:52:47.548597+0000","last_active":"2023-11-08T05:30:44.490618+0000","last_peered":"2023-11-08T05:30:44.490618+0000","last_clean":"2023-11-08T05:30:44.490618+0000","last_became_active":"2023-11-08T04:52:47.548294+0000","last_became_peered":"2023-11-08T04:52:47.548294+0000","last_unstale":"2023-11-08T05:30:44.490618+0000","last_undegraded":"2023-11-08T05:30:44.490618+0000","last_fullsized":"2023-11-08T05:30:44.490618+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":460,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490523+0000","last_change":"2023-11-08T05:28:13.545468+0000","last_active":"2023-11-08T05:30:44.490523+0000","last_peered":"2023-11-08T05:30:44.490523+0000","last_clean":"2023-11-08T05:28:01.456723+0000","last_became_active":"2023-11-08T05:28:13.545468+0000","last_became_peered":"2023-11-08T05:28:13.545468+0000","last_unstale":"2023-11-08T05:30:44.490523+0000","last_undegraded":"2023-11-08T05:30:44.490523+0000","last_fullsized":"2023-11-08T05:28:13.537207+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_clean_scrub_stamp":"2023-11-08T04:52:45.528796+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":465,"reported_epoch":458,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:44.491907+0000","last_change":"2023-11-08T05:29:44.395350+0000","last_active":"2023-11-08T05:30:44.491907+0000","last_peered":"2023-11-08T05:30:44.491907+0000","last_clean":"2023-11-08T05:30:44.491907+0000","last_became_active":"2023-11-08T05:29:44.395012+0000","last_became_peered":"2023-11-08T05:29:44.395012+0000","last_unstale":"2023-11-08T05:30:44.491907+0000","last_undegraded":"2023-11-08T05:30:44.491907+0000","last_fullsized":"2023-11-08T05:30:44.491907+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490629+0000","last_change":"2023-11-08T05:28:13.563357+0000","last_active":"2023-11-08T05:30:44.490629+0000","last_peered":"2023-11-08T05:30:44.490629+0000","last_clean":"2023-11-08T05:28:01.457691+0000","last_became_active":"2023-11-08T05:28:13.563357+0000","last_became_peered":"2023-11-08T05:28:13.563357+0000","last_unstale":"2023-11-08T05:30:44.490629+0000","last_undegraded":"2023-11-08T05:30:44.490629+0000","last_fullsized":"2023-11-08T05:28:13.541614+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490691+0000","last_change":"2023-11-08T05:28:13.550672+0000","last_active":"2023-11-08T05:30:44.490691+0000","last_peered":"2023-11-08T05:30:44.490691+0000","last_clean":"2023-11-08T05:28:01.457904+0000","last_became_active":"2023-11-08T05:28:13.550672+0000","last_became_peered":"2023-11-08T05:28:13.550672+0000","last_unstale":"2023-11-08T05:30:44.490691+0000","last_undegraded":"2023-11-08T05:30:44.490691+0000","last_fullsized":"2023-11-08T05:28:13.539595+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490805+0000","last_change":"2023-11-08T05:28:13.557968+0000","last_active":"2023-11-08T05:30:44.490805+0000","last_peered":"2023-11-08T05:30:44.490805+0000","last_clean":"2023-11-08T05:28:01.458433+0000","last_became_active":"2023-11-08T05:28:13.557968+0000","last_became_peered":"2023-11-08T05:28:13.557968+0000","last_unstale":"2023-11-08T05:30:44.490805+0000","last_undegraded":"2023-11-08T05:30:44.490805+0000","last_fullsized":"2023-11-08T05:28:13.542832+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":1,"acting":1,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":12,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738837,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":645040,"kb_used_data":5352,"kb_used_omap":275,"kb_used_meta":639404,"kb_avail":93726800,"statfs":{"total":96636764160,"available":95976243200,"internally_reserved":0,"allocated":5480448,"data_stored":2697647,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":281650,"internal_metadata":654750670},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:59 2023","interfaces":[{"interface":"back","average":{"1min":0.30499999999999999,"5min":0.439,"15min":0.36099999999999999},"min":{"1min":0.17399999999999999,"5min":0.14499999999999999,"15min":0.14499999999999999},"max":{"1min":0.60099999999999998,"5min":4.9379999999999997,"15min":4.9379999999999997},"last":0.379},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.49399999999999999,"15min":0.42399999999999999},"min":{"1min":0.221,"5min":0.19600000000000001,"15min":0.17000000000000001},"max":{"1min":0.63200000000000001,"5min":4.9859999999999998,"15min":4.9859999999999998},"last":0.40899999999999997}]},{"osd":1,"last update":"Wed Nov 8 05:31:06 2023","interfaces":[{"interface":"back","average":{"1min":0.27500000000000002,"5min":0.40300000000000002,"15min":0.46999999999999997},"min":{"1min":0.20899999999999999,"5min":0.16,"15min":0.153},"max":{"1min":0.41099999999999998,"5min":4.0830000000000002,"15min":23.242999999999999},"last":0.17599999999999999},{"interface":"front","average":{"1min":0.29899999999999999,"5min":0.40100000000000002,"15min":0.46000000000000002},"min":{"1min":0.23799999999999999,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":0.45300000000000001,"5min":3.5590000000000002,"15min":23.251000000000001},"last":0.20200000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738798,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1339228,"kb_used_data":398304,"kb_used_omap":291,"kb_used_meta":936476,"kb_avail":93032612,"statfs":{"total":96636764160,"available":95265394688,"internally_reserved":0,"allocated":407863296,"data_stored":405103900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":298462,"internal_metadata":958951970},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.436,"15min":0.36299999999999999},"min":{"1min":0.19700000000000001,"5min":0.14699999999999999,"15min":0.13700000000000001},"max":{"1min":3.222,"5min":6.5499999999999998,"15min":6.9989999999999997},"last":0.245},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.46300000000000002,"15min":0.42299999999999999},"min":{"1min":0.17799999999999999,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":5.0030000000000001,"5min":6.0629999999999997,"15min":9.6869999999999994},"last":0.28599999999999998}]},{"osd":2,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.51600000000000001,"15min":0.47099999999999997},"min":{"1min":0.21099999999999999,"5min":0.153,"15min":0.153},"max":{"1min":5.7439999999999998,"5min":5.8280000000000003,"15min":9.7639999999999993},"last":0.252},{"interface":"front","average":{"1min":0.497,"5min":0.56200000000000006,"15min":0.42299999999999999},"min":{"1min":0.20399999999999999,"5min":0.187,"15min":0.16800000000000001},"max":{"1min":3.9319999999999999,"5min":8.8040000000000003,"15min":8.8040000000000003},"last":0.29899999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738839,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":650100,"kb_used_data":5356,"kb_used_omap":214,"kb_used_meta":644521,"kb_avail":93721740,"statfs":{"total":96636764160,"available":95971061760,"internally_reserved":0,"allocated":5484544,"data_stored":2698616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":219335,"internal_metadata":659990329},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:56 2023","interfaces":[{"interface":"back","average":{"1min":0.30399999999999999,"5min":0.34699999999999998,"15min":0.36499999999999999},"min":{"1min":0.23999999999999999,"5min":0.14199999999999999,"15min":0.14199999999999999},"max":{"1min":0.42399999999999999,"5min":1.595,"15min":7.4249999999999998},"last":0.17100000000000001},{"interface":"front","average":{"1min":0.35499999999999998,"5min":0.40899999999999997,"15min":0.376},"min":{"1min":0.23999999999999999,"5min":0.19600000000000001,"15min":0.183},"max":{"1min":0.46999999999999997,"5min":2.6859999999999999,"15min":2.6859999999999999},"last":0.23000000000000001}]},{"osd":2,"last update":"Wed Nov 8 05:30:57 2023","interfaces":[{"interface":"back","average":{"1min":0.28599999999999998,"5min":0.34699999999999998,"15min":0.36899999999999999},"min":{"1min":0.19700000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":0.41399999999999998,"5min":2.2650000000000001,"15min":5.8209999999999997},"last":0.161},{"interface":"front","average":{"1min":0.30599999999999999,"5min":0.313,"15min":0.35799999999999998},"min":{"1min":0.17799999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.435,"5min":1.137,"15min":4.2350000000000003},"last":0.17000000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-08T05:31:37.996 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~200s 2023-11-08T05:31:40.953 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-11-08T05:31:41.252 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T05:31:41.254 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2023-11-08T05:31:41.265 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":1624,"stamp":"2023-11-08T05:31:40.071068+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":10,"acting":13,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":109,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2634368,"kb_used_data":409012,"kb_used_omap":780,"kb_used_meta":2220403,"kb_avail":280481152,"statfs":{"total":289910292480,"available":287212699648,"internally_reserved":0,"allocated":418828288,"data_stored":410500163,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":799447,"internal_metadata":2273692969},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-536782902,"num_objects":-164,"num_object_clones":0,"num_object_copies":-328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-115,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-164,"num_whiteouts":0,"num_read":-122056,"num_read_kb":-39011942,"num_write":-413642,"num_write_kb":-151705339,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-8,"num_bytes_recovered":-16922624,"num_keys_recovered":-2,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.424817"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490644+0000","last_change":"2023-11-08T05:28:13.557071+0000","last_active":"2023-11-08T05:30:44.490644+0000","last_peered":"2023-11-08T05:30:44.490644+0000","last_clean":"2023-11-08T05:28:01.458439+0000","last_became_active":"2023-11-08T05:28:13.557071+0000","last_became_peered":"2023-11-08T05:28:13.557071+0000","last_unstale":"2023-11-08T05:30:44.490644+0000","last_undegraded":"2023-11-08T05:30:44.490644+0000","last_fullsized":"2023-11-08T05:28:13.547025+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":459,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:45.794760+0000","last_change":"2023-11-08T05:30:45.794760+0000","last_active":"2023-11-08T05:30:45.794760+0000","last_peered":"2023-11-08T05:30:45.794760+0000","last_clean":"2023-11-08T05:30:45.794760+0000","last_became_active":"2023-11-08T05:30:45.794398+0000","last_became_peered":"2023-11-08T05:30:45.794398+0000","last_unstale":"2023-11-08T05:30:45.794760+0000","last_undegraded":"2023-11-08T05:30:45.794760+0000","last_fullsized":"2023-11-08T05:30:45.794760+0000","mapping_epoch":458,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":459,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":465,"reported_epoch":459,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:45.795085+0000","last_change":"2023-11-08T05:30:45.795085+0000","last_active":"2023-11-08T05:30:45.795085+0000","last_peered":"2023-11-08T05:30:45.795085+0000","last_clean":"2023-11-08T05:30:45.795085+0000","last_became_active":"2023-11-08T05:30:45.794743+0000","last_became_peered":"2023-11-08T05:30:45.794743+0000","last_unstale":"2023-11-08T05:30:45.795085+0000","last_undegraded":"2023-11-08T05:30:45.795085+0000","last_fullsized":"2023-11-08T05:30:45.795085+0000","mapping_epoch":458,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":459,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":454,"reported_epoch":458,"state":"active+clean","last_fresh":"2023-11-08T05:30:44.490618+0000","last_change":"2023-11-08T04:52:47.548597+0000","last_active":"2023-11-08T05:30:44.490618+0000","last_peered":"2023-11-08T05:30:44.490618+0000","last_clean":"2023-11-08T05:30:44.490618+0000","last_became_active":"2023-11-08T04:52:47.548294+0000","last_became_peered":"2023-11-08T04:52:47.548294+0000","last_unstale":"2023-11-08T05:30:44.490618+0000","last_undegraded":"2023-11-08T05:30:44.490618+0000","last_fullsized":"2023-11-08T05:30:44.490618+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":460,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490523+0000","last_change":"2023-11-08T05:28:13.545468+0000","last_active":"2023-11-08T05:30:44.490523+0000","last_peered":"2023-11-08T05:30:44.490523+0000","last_clean":"2023-11-08T05:28:01.456723+0000","last_became_active":"2023-11-08T05:28:13.545468+0000","last_became_peered":"2023-11-08T05:28:13.545468+0000","last_unstale":"2023-11-08T05:30:44.490523+0000","last_undegraded":"2023-11-08T05:30:44.490523+0000","last_fullsized":"2023-11-08T05:28:13.537207+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_clean_scrub_stamp":"2023-11-08T04:52:45.528796+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":465,"reported_epoch":458,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:44.491907+0000","last_change":"2023-11-08T05:29:44.395350+0000","last_active":"2023-11-08T05:30:44.491907+0000","last_peered":"2023-11-08T05:30:44.491907+0000","last_clean":"2023-11-08T05:30:44.491907+0000","last_became_active":"2023-11-08T05:29:44.395012+0000","last_became_peered":"2023-11-08T05:29:44.395012+0000","last_unstale":"2023-11-08T05:30:44.491907+0000","last_undegraded":"2023-11-08T05:30:44.491907+0000","last_fullsized":"2023-11-08T05:30:44.491907+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490629+0000","last_change":"2023-11-08T05:28:13.563357+0000","last_active":"2023-11-08T05:30:44.490629+0000","last_peered":"2023-11-08T05:30:44.490629+0000","last_clean":"2023-11-08T05:28:01.457691+0000","last_became_active":"2023-11-08T05:28:13.563357+0000","last_became_peered":"2023-11-08T05:28:13.563357+0000","last_unstale":"2023-11-08T05:30:44.490629+0000","last_undegraded":"2023-11-08T05:30:44.490629+0000","last_fullsized":"2023-11-08T05:28:13.541614+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490691+0000","last_change":"2023-11-08T05:28:13.550672+0000","last_active":"2023-11-08T05:30:44.490691+0000","last_peered":"2023-11-08T05:30:44.490691+0000","last_clean":"2023-11-08T05:28:01.457904+0000","last_became_active":"2023-11-08T05:28:13.550672+0000","last_became_peered":"2023-11-08T05:28:13.550672+0000","last_unstale":"2023-11-08T05:30:44.490691+0000","last_undegraded":"2023-11-08T05:30:44.490691+0000","last_fullsized":"2023-11-08T05:28:13.539595+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490805+0000","last_change":"2023-11-08T05:28:13.557968+0000","last_active":"2023-11-08T05:30:44.490805+0000","last_peered":"2023-11-08T05:30:44.490805+0000","last_clean":"2023-11-08T05:28:01.458433+0000","last_became_active":"2023-11-08T05:28:13.557968+0000","last_became_peered":"2023-11-08T05:28:13.557968+0000","last_unstale":"2023-11-08T05:30:44.490805+0000","last_undegraded":"2023-11-08T05:30:44.490805+0000","last_fullsized":"2023-11-08T05:28:13.542832+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":1,"acting":1,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":12,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738838,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":645040,"kb_used_data":5352,"kb_used_omap":275,"kb_used_meta":639404,"kb_avail":93726800,"statfs":{"total":96636764160,"available":95976243200,"internally_reserved":0,"allocated":5480448,"data_stored":2697647,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":281650,"internal_metadata":654750670},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:59 2023","interfaces":[{"interface":"back","average":{"1min":0.30499999999999999,"5min":0.439,"15min":0.36099999999999999},"min":{"1min":0.17399999999999999,"5min":0.14499999999999999,"15min":0.14499999999999999},"max":{"1min":0.60099999999999998,"5min":4.9379999999999997,"15min":4.9379999999999997},"last":0.379},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.49399999999999999,"15min":0.42399999999999999},"min":{"1min":0.221,"5min":0.19600000000000001,"15min":0.17000000000000001},"max":{"1min":0.63200000000000001,"5min":4.9859999999999998,"15min":4.9859999999999998},"last":0.40899999999999997}]},{"osd":1,"last update":"Wed Nov 8 05:31:06 2023","interfaces":[{"interface":"back","average":{"1min":0.27500000000000002,"5min":0.40300000000000002,"15min":0.46999999999999997},"min":{"1min":0.20899999999999999,"5min":0.16,"15min":0.153},"max":{"1min":0.41099999999999998,"5min":4.0830000000000002,"15min":23.242999999999999},"last":0.26300000000000001},{"interface":"front","average":{"1min":0.29899999999999999,"5min":0.40100000000000002,"15min":0.46000000000000002},"min":{"1min":0.23799999999999999,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":0.45300000000000001,"5min":3.5590000000000002,"15min":23.251000000000001},"last":0.246}]}]},{"osd":0,"up_from":8,"seq":34359738798,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1339228,"kb_used_data":398304,"kb_used_omap":291,"kb_used_meta":936476,"kb_avail":93032612,"statfs":{"total":96636764160,"available":95265394688,"internally_reserved":0,"allocated":407863296,"data_stored":405103900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":298462,"internal_metadata":958951970},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.436,"15min":0.36299999999999999},"min":{"1min":0.19700000000000001,"5min":0.14699999999999999,"15min":0.13700000000000001},"max":{"1min":3.222,"5min":6.5499999999999998,"15min":6.9989999999999997},"last":0.245},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.46300000000000002,"15min":0.42299999999999999},"min":{"1min":0.17799999999999999,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":5.0030000000000001,"5min":6.0629999999999997,"15min":9.6869999999999994},"last":0.28599999999999998}]},{"osd":2,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.51600000000000001,"15min":0.47099999999999997},"min":{"1min":0.21099999999999999,"5min":0.153,"15min":0.153},"max":{"1min":5.7439999999999998,"5min":5.8280000000000003,"15min":9.7639999999999993},"last":0.252},{"interface":"front","average":{"1min":0.497,"5min":0.56200000000000006,"15min":0.42299999999999999},"min":{"1min":0.20399999999999999,"5min":0.187,"15min":0.16800000000000001},"max":{"1min":3.9319999999999999,"5min":8.8040000000000003,"15min":8.8040000000000003},"last":0.29899999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738839,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":650100,"kb_used_data":5356,"kb_used_omap":214,"kb_used_meta":644521,"kb_avail":93721740,"statfs":{"total":96636764160,"available":95971061760,"internally_reserved":0,"allocated":5484544,"data_stored":2698616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":219335,"internal_metadata":659990329},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:56 2023","interfaces":[{"interface":"back","average":{"1min":0.30399999999999999,"5min":0.34699999999999998,"15min":0.36499999999999999},"min":{"1min":0.23999999999999999,"5min":0.14199999999999999,"15min":0.14199999999999999},"max":{"1min":0.42399999999999999,"5min":1.595,"15min":7.4249999999999998},"last":0.17100000000000001},{"interface":"front","average":{"1min":0.35499999999999998,"5min":0.40899999999999997,"15min":0.376},"min":{"1min":0.23999999999999999,"5min":0.19600000000000001,"15min":0.183},"max":{"1min":0.46999999999999997,"5min":2.6859999999999999,"15min":2.6859999999999999},"last":0.23000000000000001}]},{"osd":2,"last update":"Wed Nov 8 05:30:57 2023","interfaces":[{"interface":"back","average":{"1min":0.28599999999999998,"5min":0.34699999999999998,"15min":0.36899999999999999},"min":{"1min":0.19700000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":0.41399999999999998,"5min":2.2650000000000001,"15min":5.8209999999999997},"last":0.161},{"interface":"front","average":{"1min":0.30599999999999999,"5min":0.313,"15min":0.35799999999999998},"min":{"1min":0.17799999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.435,"5min":1.137,"15min":4.2350000000000003},"last":0.17000000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-08T05:31:41.267 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-11-08T05:31:41.550 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T05:31:41.552 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2023-11-08T05:31:41.563 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":1624,"stamp":"2023-11-08T05:31:40.071068+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":10,"acting":13,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":109,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2634368,"kb_used_data":409012,"kb_used_omap":780,"kb_used_meta":2220403,"kb_avail":280481152,"statfs":{"total":289910292480,"available":287212699648,"internally_reserved":0,"allocated":418828288,"data_stored":410500163,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":799447,"internal_metadata":2273692969},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-536782902,"num_objects":-164,"num_object_clones":0,"num_object_copies":-328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-115,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-164,"num_whiteouts":0,"num_read":-122056,"num_read_kb":-39011942,"num_write":-413642,"num_write_kb":-151705339,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-8,"num_bytes_recovered":-16922624,"num_keys_recovered":-2,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.424817"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490644+0000","last_change":"2023-11-08T05:28:13.557071+0000","last_active":"2023-11-08T05:30:44.490644+0000","last_peered":"2023-11-08T05:30:44.490644+0000","last_clean":"2023-11-08T05:28:01.458439+0000","last_became_active":"2023-11-08T05:28:13.557071+0000","last_became_peered":"2023-11-08T05:28:13.557071+0000","last_unstale":"2023-11-08T05:30:44.490644+0000","last_undegraded":"2023-11-08T05:30:44.490644+0000","last_fullsized":"2023-11-08T05:28:13.547025+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":459,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:45.794760+0000","last_change":"2023-11-08T05:30:45.794760+0000","last_active":"2023-11-08T05:30:45.794760+0000","last_peered":"2023-11-08T05:30:45.794760+0000","last_clean":"2023-11-08T05:30:45.794760+0000","last_became_active":"2023-11-08T05:30:45.794398+0000","last_became_peered":"2023-11-08T05:30:45.794398+0000","last_unstale":"2023-11-08T05:30:45.794760+0000","last_undegraded":"2023-11-08T05:30:45.794760+0000","last_fullsized":"2023-11-08T05:30:45.794760+0000","mapping_epoch":458,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":459,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":465,"reported_epoch":459,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:45.795085+0000","last_change":"2023-11-08T05:30:45.795085+0000","last_active":"2023-11-08T05:30:45.795085+0000","last_peered":"2023-11-08T05:30:45.795085+0000","last_clean":"2023-11-08T05:30:45.795085+0000","last_became_active":"2023-11-08T05:30:45.794743+0000","last_became_peered":"2023-11-08T05:30:45.794743+0000","last_unstale":"2023-11-08T05:30:45.795085+0000","last_undegraded":"2023-11-08T05:30:45.795085+0000","last_fullsized":"2023-11-08T05:30:45.795085+0000","mapping_epoch":458,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":459,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":454,"reported_epoch":458,"state":"active+clean","last_fresh":"2023-11-08T05:30:44.490618+0000","last_change":"2023-11-08T04:52:47.548597+0000","last_active":"2023-11-08T05:30:44.490618+0000","last_peered":"2023-11-08T05:30:44.490618+0000","last_clean":"2023-11-08T05:30:44.490618+0000","last_became_active":"2023-11-08T04:52:47.548294+0000","last_became_peered":"2023-11-08T04:52:47.548294+0000","last_unstale":"2023-11-08T05:30:44.490618+0000","last_undegraded":"2023-11-08T05:30:44.490618+0000","last_fullsized":"2023-11-08T05:30:44.490618+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":460,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490523+0000","last_change":"2023-11-08T05:28:13.545468+0000","last_active":"2023-11-08T05:30:44.490523+0000","last_peered":"2023-11-08T05:30:44.490523+0000","last_clean":"2023-11-08T05:28:01.456723+0000","last_became_active":"2023-11-08T05:28:13.545468+0000","last_became_peered":"2023-11-08T05:28:13.545468+0000","last_unstale":"2023-11-08T05:30:44.490523+0000","last_undegraded":"2023-11-08T05:30:44.490523+0000","last_fullsized":"2023-11-08T05:28:13.537207+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_clean_scrub_stamp":"2023-11-08T04:52:45.528796+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":465,"reported_epoch":458,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:44.491907+0000","last_change":"2023-11-08T05:29:44.395350+0000","last_active":"2023-11-08T05:30:44.491907+0000","last_peered":"2023-11-08T05:30:44.491907+0000","last_clean":"2023-11-08T05:30:44.491907+0000","last_became_active":"2023-11-08T05:29:44.395012+0000","last_became_peered":"2023-11-08T05:29:44.395012+0000","last_unstale":"2023-11-08T05:30:44.491907+0000","last_undegraded":"2023-11-08T05:30:44.491907+0000","last_fullsized":"2023-11-08T05:30:44.491907+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490629+0000","last_change":"2023-11-08T05:28:13.563357+0000","last_active":"2023-11-08T05:30:44.490629+0000","last_peered":"2023-11-08T05:30:44.490629+0000","last_clean":"2023-11-08T05:28:01.457691+0000","last_became_active":"2023-11-08T05:28:13.563357+0000","last_became_peered":"2023-11-08T05:28:13.563357+0000","last_unstale":"2023-11-08T05:30:44.490629+0000","last_undegraded":"2023-11-08T05:30:44.490629+0000","last_fullsized":"2023-11-08T05:28:13.541614+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490691+0000","last_change":"2023-11-08T05:28:13.550672+0000","last_active":"2023-11-08T05:30:44.490691+0000","last_peered":"2023-11-08T05:30:44.490691+0000","last_clean":"2023-11-08T05:28:01.457904+0000","last_became_active":"2023-11-08T05:28:13.550672+0000","last_became_peered":"2023-11-08T05:28:13.550672+0000","last_unstale":"2023-11-08T05:30:44.490691+0000","last_undegraded":"2023-11-08T05:30:44.490691+0000","last_fullsized":"2023-11-08T05:28:13.539595+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490805+0000","last_change":"2023-11-08T05:28:13.557968+0000","last_active":"2023-11-08T05:30:44.490805+0000","last_peered":"2023-11-08T05:30:44.490805+0000","last_clean":"2023-11-08T05:28:01.458433+0000","last_became_active":"2023-11-08T05:28:13.557968+0000","last_became_peered":"2023-11-08T05:28:13.557968+0000","last_unstale":"2023-11-08T05:30:44.490805+0000","last_undegraded":"2023-11-08T05:30:44.490805+0000","last_fullsized":"2023-11-08T05:28:13.542832+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":1,"acting":1,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":12,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738838,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":645040,"kb_used_data":5352,"kb_used_omap":275,"kb_used_meta":639404,"kb_avail":93726800,"statfs":{"total":96636764160,"available":95976243200,"internally_reserved":0,"allocated":5480448,"data_stored":2697647,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":281650,"internal_metadata":654750670},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:59 2023","interfaces":[{"interface":"back","average":{"1min":0.30499999999999999,"5min":0.439,"15min":0.36099999999999999},"min":{"1min":0.17399999999999999,"5min":0.14499999999999999,"15min":0.14499999999999999},"max":{"1min":0.60099999999999998,"5min":4.9379999999999997,"15min":4.9379999999999997},"last":0.379},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.49399999999999999,"15min":0.42399999999999999},"min":{"1min":0.221,"5min":0.19600000000000001,"15min":0.17000000000000001},"max":{"1min":0.63200000000000001,"5min":4.9859999999999998,"15min":4.9859999999999998},"last":0.40899999999999997}]},{"osd":1,"last update":"Wed Nov 8 05:31:06 2023","interfaces":[{"interface":"back","average":{"1min":0.27500000000000002,"5min":0.40300000000000002,"15min":0.46999999999999997},"min":{"1min":0.20899999999999999,"5min":0.16,"15min":0.153},"max":{"1min":0.41099999999999998,"5min":4.0830000000000002,"15min":23.242999999999999},"last":0.26300000000000001},{"interface":"front","average":{"1min":0.29899999999999999,"5min":0.40100000000000002,"15min":0.46000000000000002},"min":{"1min":0.23799999999999999,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":0.45300000000000001,"5min":3.5590000000000002,"15min":23.251000000000001},"last":0.246}]}]},{"osd":0,"up_from":8,"seq":34359738798,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1339228,"kb_used_data":398304,"kb_used_omap":291,"kb_used_meta":936476,"kb_avail":93032612,"statfs":{"total":96636764160,"available":95265394688,"internally_reserved":0,"allocated":407863296,"data_stored":405103900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":298462,"internal_metadata":958951970},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.436,"15min":0.36299999999999999},"min":{"1min":0.19700000000000001,"5min":0.14699999999999999,"15min":0.13700000000000001},"max":{"1min":3.222,"5min":6.5499999999999998,"15min":6.9989999999999997},"last":0.245},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.46300000000000002,"15min":0.42299999999999999},"min":{"1min":0.17799999999999999,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":5.0030000000000001,"5min":6.0629999999999997,"15min":9.6869999999999994},"last":0.28599999999999998}]},{"osd":2,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.51600000000000001,"15min":0.47099999999999997},"min":{"1min":0.21099999999999999,"5min":0.153,"15min":0.153},"max":{"1min":5.7439999999999998,"5min":5.8280000000000003,"15min":9.7639999999999993},"last":0.252},{"interface":"front","average":{"1min":0.497,"5min":0.56200000000000006,"15min":0.42299999999999999},"min":{"1min":0.20399999999999999,"5min":0.187,"15min":0.16800000000000001},"max":{"1min":3.9319999999999999,"5min":8.8040000000000003,"15min":8.8040000000000003},"last":0.29899999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738839,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":650100,"kb_used_data":5356,"kb_used_omap":214,"kb_used_meta":644521,"kb_avail":93721740,"statfs":{"total":96636764160,"available":95971061760,"internally_reserved":0,"allocated":5484544,"data_stored":2698616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":219335,"internal_metadata":659990329},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:56 2023","interfaces":[{"interface":"back","average":{"1min":0.30399999999999999,"5min":0.34699999999999998,"15min":0.36499999999999999},"min":{"1min":0.23999999999999999,"5min":0.14199999999999999,"15min":0.14199999999999999},"max":{"1min":0.42399999999999999,"5min":1.595,"15min":7.4249999999999998},"last":0.17100000000000001},{"interface":"front","average":{"1min":0.35499999999999998,"5min":0.40899999999999997,"15min":0.376},"min":{"1min":0.23999999999999999,"5min":0.19600000000000001,"15min":0.183},"max":{"1min":0.46999999999999997,"5min":2.6859999999999999,"15min":2.6859999999999999},"last":0.23000000000000001}]},{"osd":2,"last update":"Wed Nov 8 05:30:57 2023","interfaces":[{"interface":"back","average":{"1min":0.28599999999999998,"5min":0.34699999999999998,"15min":0.36899999999999999},"min":{"1min":0.19700000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":0.41399999999999998,"5min":2.2650000000000001,"15min":5.8209999999999997},"last":0.161},{"interface":"front","average":{"1min":0.30599999999999999,"5min":0.313,"15min":0.35799999999999998},"min":{"1min":0.17799999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.435,"5min":1.137,"15min":4.2350000000000003},"last":0.17000000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-08T05:31:41.565 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-11-08T05:31:41.566 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 459, 'reported_epoch': 458, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:30:44.490644+0000', 'last_change': '2023-11-08T05:28:13.557071+0000', 'last_active': '2023-11-08T05:30:44.490644+0000', 'last_peered': '2023-11-08T05:30:44.490644+0000', 'last_clean': '2023-11-08T05:28:01.458439+0000', 'last_became_active': '2023-11-08T05:28:13.557071+0000', 'last_became_peered': '2023-11-08T05:28:13.557071+0000', 'last_unstale': '2023-11-08T05:30:44.490644+0000', 'last_undegraded': '2023-11-08T05:30:44.490644+0000', 'last_fullsized': '2023-11-08T05:28:13.547025+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:31:41.566 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-11-08T05:31:41.567 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 11, 'reported_epoch': 459, 'state': 'active+clean+remapped', 'last_fresh': '2023-11-08T05:30:45.794760+0000', 'last_change': '2023-11-08T05:30:45.794760+0000', 'last_active': '2023-11-08T05:30:45.794760+0000', 'last_peered': '2023-11-08T05:30:45.794760+0000', 'last_clean': '2023-11-08T05:30:45.794760+0000', 'last_became_active': '2023-11-08T05:30:45.794398+0000', 'last_became_peered': '2023-11-08T05:30:45.794398+0000', 'last_unstale': '2023-11-08T05:30:45.794760+0000', 'last_undegraded': '2023-11-08T05:30:45.794760+0000', 'last_fullsized': '2023-11-08T05:30:45.794760+0000', 'mapping_epoch': 458, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 459, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2], 'acting': [2, 1], 'avail_no_missing': ['2', '1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2023-11-08T05:31:41.567 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.1 is not active+clean 2023-11-08T05:31:41.568 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.1', 'version': "0'0", 'reported_seq': 465, 'reported_epoch': 459, 'state': 'active+clean+remapped', 'last_fresh': '2023-11-08T05:30:45.795085+0000', 'last_change': '2023-11-08T05:30:45.795085+0000', 'last_active': '2023-11-08T05:30:45.795085+0000', 'last_peered': '2023-11-08T05:30:45.795085+0000', 'last_clean': '2023-11-08T05:30:45.795085+0000', 'last_became_active': '2023-11-08T05:30:45.794743+0000', 'last_became_peered': '2023-11-08T05:30:45.794743+0000', 'last_unstale': '2023-11-08T05:30:45.795085+0000', 'last_undegraded': '2023-11-08T05:30:45.795085+0000', 'last_fullsized': '2023-11-08T05:30:45.795085+0000', 'mapping_epoch': 458, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 459, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2], 'acting': [2, 1], 'avail_no_missing': ['2', '1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2023-11-08T05:31:41.568 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-11-08T05:31:41.569 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "0'0", 'reported_seq': 460, 'reported_epoch': 458, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:30:44.490523+0000', 'last_change': '2023-11-08T05:28:13.545468+0000', 'last_active': '2023-11-08T05:30:44.490523+0000', 'last_peered': '2023-11-08T05:30:44.490523+0000', 'last_clean': '2023-11-08T05:28:01.456723+0000', 'last_became_active': '2023-11-08T05:28:13.545468+0000', 'last_became_peered': '2023-11-08T05:28:13.545468+0000', 'last_unstale': '2023-11-08T05:30:44.490523+0000', 'last_undegraded': '2023-11-08T05:30:44.490523+0000', 'last_fullsized': '2023-11-08T05:28:13.537207+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 9, 'last_epoch_clean': 10, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:45.528796+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:45.528796+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:45.528796+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:31:41.569 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.0 is not active+clean 2023-11-08T05:31:41.570 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.0', 'version': "0'0", 'reported_seq': 465, 'reported_epoch': 458, 'state': 'active+clean+remapped', 'last_fresh': '2023-11-08T05:30:44.491907+0000', 'last_change': '2023-11-08T05:29:44.395350+0000', 'last_active': '2023-11-08T05:30:44.491907+0000', 'last_peered': '2023-11-08T05:30:44.491907+0000', 'last_clean': '2023-11-08T05:30:44.491907+0000', 'last_became_active': '2023-11-08T05:29:44.395012+0000', 'last_became_peered': '2023-11-08T05:29:44.395012+0000', 'last_unstale': '2023-11-08T05:30:44.491907+0000', 'last_undegraded': '2023-11-08T05:30:44.491907+0000', 'last_fullsized': '2023-11-08T05:30:44.491907+0000', 'mapping_epoch': 455, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 456, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2], 'acting': [2, 1], 'avail_no_missing': ['2', '1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2023-11-08T05:31:41.570 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-11-08T05:31:41.570 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 459, 'reported_epoch': 458, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:30:44.490629+0000', 'last_change': '2023-11-08T05:28:13.563357+0000', 'last_active': '2023-11-08T05:30:44.490629+0000', 'last_peered': '2023-11-08T05:30:44.490629+0000', 'last_clean': '2023-11-08T05:28:01.457691+0000', 'last_became_active': '2023-11-08T05:28:13.563357+0000', 'last_became_peered': '2023-11-08T05:28:13.563357+0000', 'last_unstale': '2023-11-08T05:30:44.490629+0000', 'last_undegraded': '2023-11-08T05:30:44.490629+0000', 'last_fullsized': '2023-11-08T05:28:13.541614+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:31:41.571 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-11-08T05:31:41.571 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 459, 'reported_epoch': 458, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:30:44.490691+0000', 'last_change': '2023-11-08T05:28:13.550672+0000', 'last_active': '2023-11-08T05:30:44.490691+0000', 'last_peered': '2023-11-08T05:30:44.490691+0000', 'last_clean': '2023-11-08T05:28:01.457904+0000', 'last_became_active': '2023-11-08T05:28:13.550672+0000', 'last_became_peered': '2023-11-08T05:28:13.550672+0000', 'last_unstale': '2023-11-08T05:30:44.490691+0000', 'last_undegraded': '2023-11-08T05:30:44.490691+0000', 'last_fullsized': '2023-11-08T05:28:13.539595+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:31:41.571 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-11-08T05:31:41.571 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 459, 'reported_epoch': 458, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:30:44.490805+0000', 'last_change': '2023-11-08T05:28:13.557968+0000', 'last_active': '2023-11-08T05:30:44.490805+0000', 'last_peered': '2023-11-08T05:30:44.490805+0000', 'last_clean': '2023-11-08T05:28:01.458433+0000', 'last_became_active': '2023-11-08T05:28:13.557968+0000', 'last_became_peered': '2023-11-08T05:28:13.557968+0000', 'last_unstale': '2023-11-08T05:30:44.490805+0000', 'last_undegraded': '2023-11-08T05:30:44.490805+0000', 'last_fullsized': '2023-11-08T05:28:13.542832+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:31:41.572 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-11-08T05:31:41.918 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T05:31:41.918 INFO:teuthology.orchestra.run.smithi087.stdout:{"fsid":"ab4cbc06-449d-497d-b513-220d9db2b590","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 5 pgs undersized","count":5},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2344,"monmap":{"epoch":1,"min_mon_release_name":"pacific","num_mons":1},"osdmap":{"epoch":459,"num_osds":3,"num_up_osds":2,"osd_up_since":1699421292,"num_in_osds":3,"osd_in_since":1699419158,"num_remapped_pgs":3},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean+remapped","count":3},{"state_name":"active+clean","count":1}],"num_pgs":9,"num_pools":2,"num_objects":0,"data_bytes":0,"bytes_used":2697592832,"bytes_avail":287212699648,"bytes_total":289910292480},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":41,"modified":"2023-11-08T05:29:32.044544+0000","services":{}},"progress_events":{}} 2023-11-08T05:31:41.931 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-11-08T05:31:41.931 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-11-08T05:31:42.214 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T05:31:42.215 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2023-11-08T05:31:42.226 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":1625,"stamp":"2023-11-08T05:31:42.071404+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":10,"acting":13,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":109,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2634368,"kb_used_data":409012,"kb_used_omap":780,"kb_used_meta":2220403,"kb_avail":280481152,"statfs":{"total":289910292480,"available":287212699648,"internally_reserved":0,"allocated":418828288,"data_stored":410500163,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":799447,"internal_metadata":2273692969},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-536782902,"num_objects":-164,"num_object_clones":0,"num_object_copies":-328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-115,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-164,"num_whiteouts":0,"num_read":-122056,"num_read_kb":-39011942,"num_write":-413642,"num_write_kb":-151705339,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-8,"num_bytes_recovered":-16922624,"num_keys_recovered":-2,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.424817"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490644+0000","last_change":"2023-11-08T05:28:13.557071+0000","last_active":"2023-11-08T05:30:44.490644+0000","last_peered":"2023-11-08T05:30:44.490644+0000","last_clean":"2023-11-08T05:28:01.458439+0000","last_became_active":"2023-11-08T05:28:13.557071+0000","last_became_peered":"2023-11-08T05:28:13.557071+0000","last_unstale":"2023-11-08T05:30:44.490644+0000","last_undegraded":"2023-11-08T05:30:44.490644+0000","last_fullsized":"2023-11-08T05:28:13.547025+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":459,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:45.794760+0000","last_change":"2023-11-08T05:30:45.794760+0000","last_active":"2023-11-08T05:30:45.794760+0000","last_peered":"2023-11-08T05:30:45.794760+0000","last_clean":"2023-11-08T05:30:45.794760+0000","last_became_active":"2023-11-08T05:30:45.794398+0000","last_became_peered":"2023-11-08T05:30:45.794398+0000","last_unstale":"2023-11-08T05:30:45.794760+0000","last_undegraded":"2023-11-08T05:30:45.794760+0000","last_fullsized":"2023-11-08T05:30:45.794760+0000","mapping_epoch":458,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":459,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":465,"reported_epoch":459,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:45.795085+0000","last_change":"2023-11-08T05:30:45.795085+0000","last_active":"2023-11-08T05:30:45.795085+0000","last_peered":"2023-11-08T05:30:45.795085+0000","last_clean":"2023-11-08T05:30:45.795085+0000","last_became_active":"2023-11-08T05:30:45.794743+0000","last_became_peered":"2023-11-08T05:30:45.794743+0000","last_unstale":"2023-11-08T05:30:45.795085+0000","last_undegraded":"2023-11-08T05:30:45.795085+0000","last_fullsized":"2023-11-08T05:30:45.795085+0000","mapping_epoch":458,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":459,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":454,"reported_epoch":458,"state":"active+clean","last_fresh":"2023-11-08T05:30:44.490618+0000","last_change":"2023-11-08T04:52:47.548597+0000","last_active":"2023-11-08T05:30:44.490618+0000","last_peered":"2023-11-08T05:30:44.490618+0000","last_clean":"2023-11-08T05:30:44.490618+0000","last_became_active":"2023-11-08T04:52:47.548294+0000","last_became_peered":"2023-11-08T04:52:47.548294+0000","last_unstale":"2023-11-08T05:30:44.490618+0000","last_undegraded":"2023-11-08T05:30:44.490618+0000","last_fullsized":"2023-11-08T05:30:44.490618+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":460,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490523+0000","last_change":"2023-11-08T05:28:13.545468+0000","last_active":"2023-11-08T05:30:44.490523+0000","last_peered":"2023-11-08T05:30:44.490523+0000","last_clean":"2023-11-08T05:28:01.456723+0000","last_became_active":"2023-11-08T05:28:13.545468+0000","last_became_peered":"2023-11-08T05:28:13.545468+0000","last_unstale":"2023-11-08T05:30:44.490523+0000","last_undegraded":"2023-11-08T05:30:44.490523+0000","last_fullsized":"2023-11-08T05:28:13.537207+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_clean_scrub_stamp":"2023-11-08T04:52:45.528796+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":465,"reported_epoch":458,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:44.491907+0000","last_change":"2023-11-08T05:29:44.395350+0000","last_active":"2023-11-08T05:30:44.491907+0000","last_peered":"2023-11-08T05:30:44.491907+0000","last_clean":"2023-11-08T05:30:44.491907+0000","last_became_active":"2023-11-08T05:29:44.395012+0000","last_became_peered":"2023-11-08T05:29:44.395012+0000","last_unstale":"2023-11-08T05:30:44.491907+0000","last_undegraded":"2023-11-08T05:30:44.491907+0000","last_fullsized":"2023-11-08T05:30:44.491907+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490629+0000","last_change":"2023-11-08T05:28:13.563357+0000","last_active":"2023-11-08T05:30:44.490629+0000","last_peered":"2023-11-08T05:30:44.490629+0000","last_clean":"2023-11-08T05:28:01.457691+0000","last_became_active":"2023-11-08T05:28:13.563357+0000","last_became_peered":"2023-11-08T05:28:13.563357+0000","last_unstale":"2023-11-08T05:30:44.490629+0000","last_undegraded":"2023-11-08T05:30:44.490629+0000","last_fullsized":"2023-11-08T05:28:13.541614+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490691+0000","last_change":"2023-11-08T05:28:13.550672+0000","last_active":"2023-11-08T05:30:44.490691+0000","last_peered":"2023-11-08T05:30:44.490691+0000","last_clean":"2023-11-08T05:28:01.457904+0000","last_became_active":"2023-11-08T05:28:13.550672+0000","last_became_peered":"2023-11-08T05:28:13.550672+0000","last_unstale":"2023-11-08T05:30:44.490691+0000","last_undegraded":"2023-11-08T05:30:44.490691+0000","last_fullsized":"2023-11-08T05:28:13.539595+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490805+0000","last_change":"2023-11-08T05:28:13.557968+0000","last_active":"2023-11-08T05:30:44.490805+0000","last_peered":"2023-11-08T05:30:44.490805+0000","last_clean":"2023-11-08T05:28:01.458433+0000","last_became_active":"2023-11-08T05:28:13.557968+0000","last_became_peered":"2023-11-08T05:28:13.557968+0000","last_unstale":"2023-11-08T05:30:44.490805+0000","last_undegraded":"2023-11-08T05:30:44.490805+0000","last_fullsized":"2023-11-08T05:28:13.542832+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":1,"acting":1,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":12,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738838,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":645040,"kb_used_data":5352,"kb_used_omap":275,"kb_used_meta":639404,"kb_avail":93726800,"statfs":{"total":96636764160,"available":95976243200,"internally_reserved":0,"allocated":5480448,"data_stored":2697647,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":281650,"internal_metadata":654750670},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:59 2023","interfaces":[{"interface":"back","average":{"1min":0.30499999999999999,"5min":0.439,"15min":0.36099999999999999},"min":{"1min":0.17399999999999999,"5min":0.14499999999999999,"15min":0.14499999999999999},"max":{"1min":0.60099999999999998,"5min":4.9379999999999997,"15min":4.9379999999999997},"last":0.379},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.49399999999999999,"15min":0.42399999999999999},"min":{"1min":0.221,"5min":0.19600000000000001,"15min":0.17000000000000001},"max":{"1min":0.63200000000000001,"5min":4.9859999999999998,"15min":4.9859999999999998},"last":0.40899999999999997}]},{"osd":1,"last update":"Wed Nov 8 05:31:06 2023","interfaces":[{"interface":"back","average":{"1min":0.27500000000000002,"5min":0.40300000000000002,"15min":0.46999999999999997},"min":{"1min":0.20899999999999999,"5min":0.16,"15min":0.153},"max":{"1min":0.41099999999999998,"5min":4.0830000000000002,"15min":23.242999999999999},"last":0.26300000000000001},{"interface":"front","average":{"1min":0.29899999999999999,"5min":0.40100000000000002,"15min":0.46000000000000002},"min":{"1min":0.23799999999999999,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":0.45300000000000001,"5min":3.5590000000000002,"15min":23.251000000000001},"last":0.246}]}]},{"osd":0,"up_from":8,"seq":34359738798,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1339228,"kb_used_data":398304,"kb_used_omap":291,"kb_used_meta":936476,"kb_avail":93032612,"statfs":{"total":96636764160,"available":95265394688,"internally_reserved":0,"allocated":407863296,"data_stored":405103900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":298462,"internal_metadata":958951970},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.436,"15min":0.36299999999999999},"min":{"1min":0.19700000000000001,"5min":0.14699999999999999,"15min":0.13700000000000001},"max":{"1min":3.222,"5min":6.5499999999999998,"15min":6.9989999999999997},"last":0.245},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.46300000000000002,"15min":0.42299999999999999},"min":{"1min":0.17799999999999999,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":5.0030000000000001,"5min":6.0629999999999997,"15min":9.6869999999999994},"last":0.28599999999999998}]},{"osd":2,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.51600000000000001,"15min":0.47099999999999997},"min":{"1min":0.21099999999999999,"5min":0.153,"15min":0.153},"max":{"1min":5.7439999999999998,"5min":5.8280000000000003,"15min":9.7639999999999993},"last":0.252},{"interface":"front","average":{"1min":0.497,"5min":0.56200000000000006,"15min":0.42299999999999999},"min":{"1min":0.20399999999999999,"5min":0.187,"15min":0.16800000000000001},"max":{"1min":3.9319999999999999,"5min":8.8040000000000003,"15min":8.8040000000000003},"last":0.29899999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738840,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":650100,"kb_used_data":5356,"kb_used_omap":214,"kb_used_meta":644521,"kb_avail":93721740,"statfs":{"total":96636764160,"available":95971061760,"internally_reserved":0,"allocated":5484544,"data_stored":2698616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":219335,"internal_metadata":659990329},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:56 2023","interfaces":[{"interface":"back","average":{"1min":0.30399999999999999,"5min":0.34699999999999998,"15min":0.36499999999999999},"min":{"1min":0.23999999999999999,"5min":0.14199999999999999,"15min":0.14199999999999999},"max":{"1min":0.42399999999999999,"5min":1.595,"15min":7.4249999999999998},"last":0.17100000000000001},{"interface":"front","average":{"1min":0.35499999999999998,"5min":0.40899999999999997,"15min":0.376},"min":{"1min":0.23999999999999999,"5min":0.19600000000000001,"15min":0.183},"max":{"1min":0.46999999999999997,"5min":2.6859999999999999,"15min":2.6859999999999999},"last":0.23000000000000001}]},{"osd":2,"last update":"Wed Nov 8 05:30:57 2023","interfaces":[{"interface":"back","average":{"1min":0.28599999999999998,"5min":0.34699999999999998,"15min":0.36899999999999999},"min":{"1min":0.19700000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":0.41399999999999998,"5min":2.2650000000000001,"15min":5.8209999999999997},"last":0.26200000000000001},{"interface":"front","average":{"1min":0.30599999999999999,"5min":0.313,"15min":0.35799999999999998},"min":{"1min":0.17799999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.435,"5min":1.137,"15min":4.2350000000000003},"last":0.254}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-08T05:31:43.400 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~205s 2023-11-08T05:31:43.487 INFO:tasks.ceph.cluster2.mgr.x.smithi077.stderr:2023-11-08T05:31:43.485+0000 7f16c17cf700 -1 calc_pg_upmaps abort due to max <= 0 2023-11-08T05:31:45.228 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-11-08T05:31:45.524 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T05:31:45.526 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2023-11-08T05:31:45.537 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":1626,"stamp":"2023-11-08T05:31:44.071814+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":10,"acting":13,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":109,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2634368,"kb_used_data":409012,"kb_used_omap":780,"kb_used_meta":2220403,"kb_avail":280481152,"statfs":{"total":289910292480,"available":287212699648,"internally_reserved":0,"allocated":418828288,"data_stored":410500163,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":799447,"internal_metadata":2273692969},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-536782902,"num_objects":-164,"num_object_clones":0,"num_object_copies":-328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-115,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-164,"num_whiteouts":0,"num_read":-122056,"num_read_kb":-39011942,"num_write":-413642,"num_write_kb":-151705339,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-8,"num_bytes_recovered":-16922624,"num_keys_recovered":-2,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.424817"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490644+0000","last_change":"2023-11-08T05:28:13.557071+0000","last_active":"2023-11-08T05:30:44.490644+0000","last_peered":"2023-11-08T05:30:44.490644+0000","last_clean":"2023-11-08T05:28:01.458439+0000","last_became_active":"2023-11-08T05:28:13.557071+0000","last_became_peered":"2023-11-08T05:28:13.557071+0000","last_unstale":"2023-11-08T05:30:44.490644+0000","last_undegraded":"2023-11-08T05:30:44.490644+0000","last_fullsized":"2023-11-08T05:28:13.547025+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":459,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:45.794760+0000","last_change":"2023-11-08T05:30:45.794760+0000","last_active":"2023-11-08T05:30:45.794760+0000","last_peered":"2023-11-08T05:30:45.794760+0000","last_clean":"2023-11-08T05:30:45.794760+0000","last_became_active":"2023-11-08T05:30:45.794398+0000","last_became_peered":"2023-11-08T05:30:45.794398+0000","last_unstale":"2023-11-08T05:30:45.794760+0000","last_undegraded":"2023-11-08T05:30:45.794760+0000","last_fullsized":"2023-11-08T05:30:45.794760+0000","mapping_epoch":458,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":459,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":465,"reported_epoch":459,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:45.795085+0000","last_change":"2023-11-08T05:30:45.795085+0000","last_active":"2023-11-08T05:30:45.795085+0000","last_peered":"2023-11-08T05:30:45.795085+0000","last_clean":"2023-11-08T05:30:45.795085+0000","last_became_active":"2023-11-08T05:30:45.794743+0000","last_became_peered":"2023-11-08T05:30:45.794743+0000","last_unstale":"2023-11-08T05:30:45.795085+0000","last_undegraded":"2023-11-08T05:30:45.795085+0000","last_fullsized":"2023-11-08T05:30:45.795085+0000","mapping_epoch":458,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":459,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":454,"reported_epoch":458,"state":"active+clean","last_fresh":"2023-11-08T05:30:44.490618+0000","last_change":"2023-11-08T04:52:47.548597+0000","last_active":"2023-11-08T05:30:44.490618+0000","last_peered":"2023-11-08T05:30:44.490618+0000","last_clean":"2023-11-08T05:30:44.490618+0000","last_became_active":"2023-11-08T04:52:47.548294+0000","last_became_peered":"2023-11-08T04:52:47.548294+0000","last_unstale":"2023-11-08T05:30:44.490618+0000","last_undegraded":"2023-11-08T05:30:44.490618+0000","last_fullsized":"2023-11-08T05:30:44.490618+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":460,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490523+0000","last_change":"2023-11-08T05:28:13.545468+0000","last_active":"2023-11-08T05:30:44.490523+0000","last_peered":"2023-11-08T05:30:44.490523+0000","last_clean":"2023-11-08T05:28:01.456723+0000","last_became_active":"2023-11-08T05:28:13.545468+0000","last_became_peered":"2023-11-08T05:28:13.545468+0000","last_unstale":"2023-11-08T05:30:44.490523+0000","last_undegraded":"2023-11-08T05:30:44.490523+0000","last_fullsized":"2023-11-08T05:28:13.537207+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_clean_scrub_stamp":"2023-11-08T04:52:45.528796+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":465,"reported_epoch":458,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:44.491907+0000","last_change":"2023-11-08T05:29:44.395350+0000","last_active":"2023-11-08T05:30:44.491907+0000","last_peered":"2023-11-08T05:30:44.491907+0000","last_clean":"2023-11-08T05:30:44.491907+0000","last_became_active":"2023-11-08T05:29:44.395012+0000","last_became_peered":"2023-11-08T05:29:44.395012+0000","last_unstale":"2023-11-08T05:30:44.491907+0000","last_undegraded":"2023-11-08T05:30:44.491907+0000","last_fullsized":"2023-11-08T05:30:44.491907+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490629+0000","last_change":"2023-11-08T05:28:13.563357+0000","last_active":"2023-11-08T05:30:44.490629+0000","last_peered":"2023-11-08T05:30:44.490629+0000","last_clean":"2023-11-08T05:28:01.457691+0000","last_became_active":"2023-11-08T05:28:13.563357+0000","last_became_peered":"2023-11-08T05:28:13.563357+0000","last_unstale":"2023-11-08T05:30:44.490629+0000","last_undegraded":"2023-11-08T05:30:44.490629+0000","last_fullsized":"2023-11-08T05:28:13.541614+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490691+0000","last_change":"2023-11-08T05:28:13.550672+0000","last_active":"2023-11-08T05:30:44.490691+0000","last_peered":"2023-11-08T05:30:44.490691+0000","last_clean":"2023-11-08T05:28:01.457904+0000","last_became_active":"2023-11-08T05:28:13.550672+0000","last_became_peered":"2023-11-08T05:28:13.550672+0000","last_unstale":"2023-11-08T05:30:44.490691+0000","last_undegraded":"2023-11-08T05:30:44.490691+0000","last_fullsized":"2023-11-08T05:28:13.539595+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490805+0000","last_change":"2023-11-08T05:28:13.557968+0000","last_active":"2023-11-08T05:30:44.490805+0000","last_peered":"2023-11-08T05:30:44.490805+0000","last_clean":"2023-11-08T05:28:01.458433+0000","last_became_active":"2023-11-08T05:28:13.557968+0000","last_became_peered":"2023-11-08T05:28:13.557968+0000","last_unstale":"2023-11-08T05:30:44.490805+0000","last_undegraded":"2023-11-08T05:30:44.490805+0000","last_fullsized":"2023-11-08T05:28:13.542832+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":1,"acting":1,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":12,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738838,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":645040,"kb_used_data":5352,"kb_used_omap":275,"kb_used_meta":639404,"kb_avail":93726800,"statfs":{"total":96636764160,"available":95976243200,"internally_reserved":0,"allocated":5480448,"data_stored":2697647,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":281650,"internal_metadata":654750670},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:59 2023","interfaces":[{"interface":"back","average":{"1min":0.30499999999999999,"5min":0.439,"15min":0.36099999999999999},"min":{"1min":0.17399999999999999,"5min":0.14499999999999999,"15min":0.14499999999999999},"max":{"1min":0.60099999999999998,"5min":4.9379999999999997,"15min":4.9379999999999997},"last":0.379},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.49399999999999999,"15min":0.42399999999999999},"min":{"1min":0.221,"5min":0.19600000000000001,"15min":0.17000000000000001},"max":{"1min":0.63200000000000001,"5min":4.9859999999999998,"15min":4.9859999999999998},"last":0.40899999999999997}]},{"osd":1,"last update":"Wed Nov 8 05:31:06 2023","interfaces":[{"interface":"back","average":{"1min":0.27500000000000002,"5min":0.40300000000000002,"15min":0.46999999999999997},"min":{"1min":0.20899999999999999,"5min":0.16,"15min":0.153},"max":{"1min":0.41099999999999998,"5min":4.0830000000000002,"15min":23.242999999999999},"last":0.26300000000000001},{"interface":"front","average":{"1min":0.29899999999999999,"5min":0.40100000000000002,"15min":0.46000000000000002},"min":{"1min":0.23799999999999999,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":0.45300000000000001,"5min":3.5590000000000002,"15min":23.251000000000001},"last":0.246}]}]},{"osd":0,"up_from":8,"seq":34359738798,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1339228,"kb_used_data":398304,"kb_used_omap":291,"kb_used_meta":936476,"kb_avail":93032612,"statfs":{"total":96636764160,"available":95265394688,"internally_reserved":0,"allocated":407863296,"data_stored":405103900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":298462,"internal_metadata":958951970},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.436,"15min":0.36299999999999999},"min":{"1min":0.19700000000000001,"5min":0.14699999999999999,"15min":0.13700000000000001},"max":{"1min":3.222,"5min":6.5499999999999998,"15min":6.9989999999999997},"last":0.245},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.46300000000000002,"15min":0.42299999999999999},"min":{"1min":0.17799999999999999,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":5.0030000000000001,"5min":6.0629999999999997,"15min":9.6869999999999994},"last":0.28599999999999998}]},{"osd":2,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.51600000000000001,"15min":0.47099999999999997},"min":{"1min":0.21099999999999999,"5min":0.153,"15min":0.153},"max":{"1min":5.7439999999999998,"5min":5.8280000000000003,"15min":9.7639999999999993},"last":0.252},{"interface":"front","average":{"1min":0.497,"5min":0.56200000000000006,"15min":0.42299999999999999},"min":{"1min":0.20399999999999999,"5min":0.187,"15min":0.16800000000000001},"max":{"1min":3.9319999999999999,"5min":8.8040000000000003,"15min":8.8040000000000003},"last":0.29899999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738840,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":650100,"kb_used_data":5356,"kb_used_omap":214,"kb_used_meta":644521,"kb_avail":93721740,"statfs":{"total":96636764160,"available":95971061760,"internally_reserved":0,"allocated":5484544,"data_stored":2698616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":219335,"internal_metadata":659990329},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:56 2023","interfaces":[{"interface":"back","average":{"1min":0.30399999999999999,"5min":0.34699999999999998,"15min":0.36499999999999999},"min":{"1min":0.23999999999999999,"5min":0.14199999999999999,"15min":0.14199999999999999},"max":{"1min":0.42399999999999999,"5min":1.595,"15min":7.4249999999999998},"last":0.17100000000000001},{"interface":"front","average":{"1min":0.35499999999999998,"5min":0.40899999999999997,"15min":0.376},"min":{"1min":0.23999999999999999,"5min":0.19600000000000001,"15min":0.183},"max":{"1min":0.46999999999999997,"5min":2.6859999999999999,"15min":2.6859999999999999},"last":0.23000000000000001}]},{"osd":2,"last update":"Wed Nov 8 05:30:57 2023","interfaces":[{"interface":"back","average":{"1min":0.28599999999999998,"5min":0.34699999999999998,"15min":0.36899999999999999},"min":{"1min":0.19700000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":0.41399999999999998,"5min":2.2650000000000001,"15min":5.8209999999999997},"last":0.26200000000000001},{"interface":"front","average":{"1min":0.30599999999999999,"5min":0.313,"15min":0.35799999999999998},"min":{"1min":0.17799999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.435,"5min":1.137,"15min":4.2350000000000003},"last":0.254}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-08T05:31:45.539 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-11-08T05:31:45.823 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T05:31:45.824 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2023-11-08T05:31:45.836 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":1626,"stamp":"2023-11-08T05:31:44.071814+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":10,"acting":13,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":109,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2634368,"kb_used_data":409012,"kb_used_omap":780,"kb_used_meta":2220403,"kb_avail":280481152,"statfs":{"total":289910292480,"available":287212699648,"internally_reserved":0,"allocated":418828288,"data_stored":410500163,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":799447,"internal_metadata":2273692969},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-536782902,"num_objects":-164,"num_object_clones":0,"num_object_copies":-328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-115,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-164,"num_whiteouts":0,"num_read":-122056,"num_read_kb":-39011942,"num_write":-413642,"num_write_kb":-151705339,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-8,"num_bytes_recovered":-16922624,"num_keys_recovered":-2,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.424817"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490644+0000","last_change":"2023-11-08T05:28:13.557071+0000","last_active":"2023-11-08T05:30:44.490644+0000","last_peered":"2023-11-08T05:30:44.490644+0000","last_clean":"2023-11-08T05:28:01.458439+0000","last_became_active":"2023-11-08T05:28:13.557071+0000","last_became_peered":"2023-11-08T05:28:13.557071+0000","last_unstale":"2023-11-08T05:30:44.490644+0000","last_undegraded":"2023-11-08T05:30:44.490644+0000","last_fullsized":"2023-11-08T05:28:13.547025+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":459,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:45.794760+0000","last_change":"2023-11-08T05:30:45.794760+0000","last_active":"2023-11-08T05:30:45.794760+0000","last_peered":"2023-11-08T05:30:45.794760+0000","last_clean":"2023-11-08T05:30:45.794760+0000","last_became_active":"2023-11-08T05:30:45.794398+0000","last_became_peered":"2023-11-08T05:30:45.794398+0000","last_unstale":"2023-11-08T05:30:45.794760+0000","last_undegraded":"2023-11-08T05:30:45.794760+0000","last_fullsized":"2023-11-08T05:30:45.794760+0000","mapping_epoch":458,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":459,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":465,"reported_epoch":459,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:45.795085+0000","last_change":"2023-11-08T05:30:45.795085+0000","last_active":"2023-11-08T05:30:45.795085+0000","last_peered":"2023-11-08T05:30:45.795085+0000","last_clean":"2023-11-08T05:30:45.795085+0000","last_became_active":"2023-11-08T05:30:45.794743+0000","last_became_peered":"2023-11-08T05:30:45.794743+0000","last_unstale":"2023-11-08T05:30:45.795085+0000","last_undegraded":"2023-11-08T05:30:45.795085+0000","last_fullsized":"2023-11-08T05:30:45.795085+0000","mapping_epoch":458,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":459,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":454,"reported_epoch":458,"state":"active+clean","last_fresh":"2023-11-08T05:30:44.490618+0000","last_change":"2023-11-08T04:52:47.548597+0000","last_active":"2023-11-08T05:30:44.490618+0000","last_peered":"2023-11-08T05:30:44.490618+0000","last_clean":"2023-11-08T05:30:44.490618+0000","last_became_active":"2023-11-08T04:52:47.548294+0000","last_became_peered":"2023-11-08T04:52:47.548294+0000","last_unstale":"2023-11-08T05:30:44.490618+0000","last_undegraded":"2023-11-08T05:30:44.490618+0000","last_fullsized":"2023-11-08T05:30:44.490618+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":460,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490523+0000","last_change":"2023-11-08T05:28:13.545468+0000","last_active":"2023-11-08T05:30:44.490523+0000","last_peered":"2023-11-08T05:30:44.490523+0000","last_clean":"2023-11-08T05:28:01.456723+0000","last_became_active":"2023-11-08T05:28:13.545468+0000","last_became_peered":"2023-11-08T05:28:13.545468+0000","last_unstale":"2023-11-08T05:30:44.490523+0000","last_undegraded":"2023-11-08T05:30:44.490523+0000","last_fullsized":"2023-11-08T05:28:13.537207+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_clean_scrub_stamp":"2023-11-08T04:52:45.528796+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":465,"reported_epoch":458,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:44.491907+0000","last_change":"2023-11-08T05:29:44.395350+0000","last_active":"2023-11-08T05:30:44.491907+0000","last_peered":"2023-11-08T05:30:44.491907+0000","last_clean":"2023-11-08T05:30:44.491907+0000","last_became_active":"2023-11-08T05:29:44.395012+0000","last_became_peered":"2023-11-08T05:29:44.395012+0000","last_unstale":"2023-11-08T05:30:44.491907+0000","last_undegraded":"2023-11-08T05:30:44.491907+0000","last_fullsized":"2023-11-08T05:30:44.491907+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490629+0000","last_change":"2023-11-08T05:28:13.563357+0000","last_active":"2023-11-08T05:30:44.490629+0000","last_peered":"2023-11-08T05:30:44.490629+0000","last_clean":"2023-11-08T05:28:01.457691+0000","last_became_active":"2023-11-08T05:28:13.563357+0000","last_became_peered":"2023-11-08T05:28:13.563357+0000","last_unstale":"2023-11-08T05:30:44.490629+0000","last_undegraded":"2023-11-08T05:30:44.490629+0000","last_fullsized":"2023-11-08T05:28:13.541614+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490691+0000","last_change":"2023-11-08T05:28:13.550672+0000","last_active":"2023-11-08T05:30:44.490691+0000","last_peered":"2023-11-08T05:30:44.490691+0000","last_clean":"2023-11-08T05:28:01.457904+0000","last_became_active":"2023-11-08T05:28:13.550672+0000","last_became_peered":"2023-11-08T05:28:13.550672+0000","last_unstale":"2023-11-08T05:30:44.490691+0000","last_undegraded":"2023-11-08T05:30:44.490691+0000","last_fullsized":"2023-11-08T05:28:13.539595+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490805+0000","last_change":"2023-11-08T05:28:13.557968+0000","last_active":"2023-11-08T05:30:44.490805+0000","last_peered":"2023-11-08T05:30:44.490805+0000","last_clean":"2023-11-08T05:28:01.458433+0000","last_became_active":"2023-11-08T05:28:13.557968+0000","last_became_peered":"2023-11-08T05:28:13.557968+0000","last_unstale":"2023-11-08T05:30:44.490805+0000","last_undegraded":"2023-11-08T05:30:44.490805+0000","last_fullsized":"2023-11-08T05:28:13.542832+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":1,"acting":1,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":12,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738838,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":645040,"kb_used_data":5352,"kb_used_omap":275,"kb_used_meta":639404,"kb_avail":93726800,"statfs":{"total":96636764160,"available":95976243200,"internally_reserved":0,"allocated":5480448,"data_stored":2697647,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":281650,"internal_metadata":654750670},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:59 2023","interfaces":[{"interface":"back","average":{"1min":0.30499999999999999,"5min":0.439,"15min":0.36099999999999999},"min":{"1min":0.17399999999999999,"5min":0.14499999999999999,"15min":0.14499999999999999},"max":{"1min":0.60099999999999998,"5min":4.9379999999999997,"15min":4.9379999999999997},"last":0.379},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.49399999999999999,"15min":0.42399999999999999},"min":{"1min":0.221,"5min":0.19600000000000001,"15min":0.17000000000000001},"max":{"1min":0.63200000000000001,"5min":4.9859999999999998,"15min":4.9859999999999998},"last":0.40899999999999997}]},{"osd":1,"last update":"Wed Nov 8 05:31:06 2023","interfaces":[{"interface":"back","average":{"1min":0.27500000000000002,"5min":0.40300000000000002,"15min":0.46999999999999997},"min":{"1min":0.20899999999999999,"5min":0.16,"15min":0.153},"max":{"1min":0.41099999999999998,"5min":4.0830000000000002,"15min":23.242999999999999},"last":0.26300000000000001},{"interface":"front","average":{"1min":0.29899999999999999,"5min":0.40100000000000002,"15min":0.46000000000000002},"min":{"1min":0.23799999999999999,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":0.45300000000000001,"5min":3.5590000000000002,"15min":23.251000000000001},"last":0.246}]}]},{"osd":0,"up_from":8,"seq":34359738798,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1339228,"kb_used_data":398304,"kb_used_omap":291,"kb_used_meta":936476,"kb_avail":93032612,"statfs":{"total":96636764160,"available":95265394688,"internally_reserved":0,"allocated":407863296,"data_stored":405103900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":298462,"internal_metadata":958951970},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.436,"15min":0.36299999999999999},"min":{"1min":0.19700000000000001,"5min":0.14699999999999999,"15min":0.13700000000000001},"max":{"1min":3.222,"5min":6.5499999999999998,"15min":6.9989999999999997},"last":0.245},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.46300000000000002,"15min":0.42299999999999999},"min":{"1min":0.17799999999999999,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":5.0030000000000001,"5min":6.0629999999999997,"15min":9.6869999999999994},"last":0.28599999999999998}]},{"osd":2,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.51600000000000001,"15min":0.47099999999999997},"min":{"1min":0.21099999999999999,"5min":0.153,"15min":0.153},"max":{"1min":5.7439999999999998,"5min":5.8280000000000003,"15min":9.7639999999999993},"last":0.252},{"interface":"front","average":{"1min":0.497,"5min":0.56200000000000006,"15min":0.42299999999999999},"min":{"1min":0.20399999999999999,"5min":0.187,"15min":0.16800000000000001},"max":{"1min":3.9319999999999999,"5min":8.8040000000000003,"15min":8.8040000000000003},"last":0.29899999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738840,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":650100,"kb_used_data":5356,"kb_used_omap":214,"kb_used_meta":644521,"kb_avail":93721740,"statfs":{"total":96636764160,"available":95971061760,"internally_reserved":0,"allocated":5484544,"data_stored":2698616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":219335,"internal_metadata":659990329},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:56 2023","interfaces":[{"interface":"back","average":{"1min":0.30399999999999999,"5min":0.34699999999999998,"15min":0.36499999999999999},"min":{"1min":0.23999999999999999,"5min":0.14199999999999999,"15min":0.14199999999999999},"max":{"1min":0.42399999999999999,"5min":1.595,"15min":7.4249999999999998},"last":0.17100000000000001},{"interface":"front","average":{"1min":0.35499999999999998,"5min":0.40899999999999997,"15min":0.376},"min":{"1min":0.23999999999999999,"5min":0.19600000000000001,"15min":0.183},"max":{"1min":0.46999999999999997,"5min":2.6859999999999999,"15min":2.6859999999999999},"last":0.23000000000000001}]},{"osd":2,"last update":"Wed Nov 8 05:30:57 2023","interfaces":[{"interface":"back","average":{"1min":0.28599999999999998,"5min":0.34699999999999998,"15min":0.36899999999999999},"min":{"1min":0.19700000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":0.41399999999999998,"5min":2.2650000000000001,"15min":5.8209999999999997},"last":0.26200000000000001},{"interface":"front","average":{"1min":0.30599999999999999,"5min":0.313,"15min":0.35799999999999998},"min":{"1min":0.17799999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.435,"5min":1.137,"15min":4.2350000000000003},"last":0.254}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-08T05:31:45.838 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-11-08T05:31:45.841 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 459, 'reported_epoch': 458, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:30:44.490644+0000', 'last_change': '2023-11-08T05:28:13.557071+0000', 'last_active': '2023-11-08T05:30:44.490644+0000', 'last_peered': '2023-11-08T05:30:44.490644+0000', 'last_clean': '2023-11-08T05:28:01.458439+0000', 'last_became_active': '2023-11-08T05:28:13.557071+0000', 'last_became_peered': '2023-11-08T05:28:13.557071+0000', 'last_unstale': '2023-11-08T05:30:44.490644+0000', 'last_undegraded': '2023-11-08T05:30:44.490644+0000', 'last_fullsized': '2023-11-08T05:28:13.547025+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:31:45.842 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-11-08T05:31:45.843 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 11, 'reported_epoch': 459, 'state': 'active+clean+remapped', 'last_fresh': '2023-11-08T05:30:45.794760+0000', 'last_change': '2023-11-08T05:30:45.794760+0000', 'last_active': '2023-11-08T05:30:45.794760+0000', 'last_peered': '2023-11-08T05:30:45.794760+0000', 'last_clean': '2023-11-08T05:30:45.794760+0000', 'last_became_active': '2023-11-08T05:30:45.794398+0000', 'last_became_peered': '2023-11-08T05:30:45.794398+0000', 'last_unstale': '2023-11-08T05:30:45.794760+0000', 'last_undegraded': '2023-11-08T05:30:45.794760+0000', 'last_fullsized': '2023-11-08T05:30:45.794760+0000', 'mapping_epoch': 458, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 459, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2], 'acting': [2, 1], 'avail_no_missing': ['2', '1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2023-11-08T05:31:45.843 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.1 is not active+clean 2023-11-08T05:31:45.844 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.1', 'version': "0'0", 'reported_seq': 465, 'reported_epoch': 459, 'state': 'active+clean+remapped', 'last_fresh': '2023-11-08T05:30:45.795085+0000', 'last_change': '2023-11-08T05:30:45.795085+0000', 'last_active': '2023-11-08T05:30:45.795085+0000', 'last_peered': '2023-11-08T05:30:45.795085+0000', 'last_clean': '2023-11-08T05:30:45.795085+0000', 'last_became_active': '2023-11-08T05:30:45.794743+0000', 'last_became_peered': '2023-11-08T05:30:45.794743+0000', 'last_unstale': '2023-11-08T05:30:45.795085+0000', 'last_undegraded': '2023-11-08T05:30:45.795085+0000', 'last_fullsized': '2023-11-08T05:30:45.795085+0000', 'mapping_epoch': 458, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 459, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2], 'acting': [2, 1], 'avail_no_missing': ['2', '1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2023-11-08T05:31:45.844 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-11-08T05:31:45.844 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "0'0", 'reported_seq': 460, 'reported_epoch': 458, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:30:44.490523+0000', 'last_change': '2023-11-08T05:28:13.545468+0000', 'last_active': '2023-11-08T05:30:44.490523+0000', 'last_peered': '2023-11-08T05:30:44.490523+0000', 'last_clean': '2023-11-08T05:28:01.456723+0000', 'last_became_active': '2023-11-08T05:28:13.545468+0000', 'last_became_peered': '2023-11-08T05:28:13.545468+0000', 'last_unstale': '2023-11-08T05:30:44.490523+0000', 'last_undegraded': '2023-11-08T05:30:44.490523+0000', 'last_fullsized': '2023-11-08T05:28:13.537207+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 9, 'last_epoch_clean': 10, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:45.528796+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:45.528796+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:45.528796+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:31:45.845 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.0 is not active+clean 2023-11-08T05:31:45.845 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.0', 'version': "0'0", 'reported_seq': 465, 'reported_epoch': 458, 'state': 'active+clean+remapped', 'last_fresh': '2023-11-08T05:30:44.491907+0000', 'last_change': '2023-11-08T05:29:44.395350+0000', 'last_active': '2023-11-08T05:30:44.491907+0000', 'last_peered': '2023-11-08T05:30:44.491907+0000', 'last_clean': '2023-11-08T05:30:44.491907+0000', 'last_became_active': '2023-11-08T05:29:44.395012+0000', 'last_became_peered': '2023-11-08T05:29:44.395012+0000', 'last_unstale': '2023-11-08T05:30:44.491907+0000', 'last_undegraded': '2023-11-08T05:30:44.491907+0000', 'last_fullsized': '2023-11-08T05:30:44.491907+0000', 'mapping_epoch': 455, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 456, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2], 'acting': [2, 1], 'avail_no_missing': ['2', '1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2023-11-08T05:31:45.845 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-11-08T05:31:45.846 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 459, 'reported_epoch': 458, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:30:44.490629+0000', 'last_change': '2023-11-08T05:28:13.563357+0000', 'last_active': '2023-11-08T05:30:44.490629+0000', 'last_peered': '2023-11-08T05:30:44.490629+0000', 'last_clean': '2023-11-08T05:28:01.457691+0000', 'last_became_active': '2023-11-08T05:28:13.563357+0000', 'last_became_peered': '2023-11-08T05:28:13.563357+0000', 'last_unstale': '2023-11-08T05:30:44.490629+0000', 'last_undegraded': '2023-11-08T05:30:44.490629+0000', 'last_fullsized': '2023-11-08T05:28:13.541614+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:31:45.846 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-11-08T05:31:45.846 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 459, 'reported_epoch': 458, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:30:44.490691+0000', 'last_change': '2023-11-08T05:28:13.550672+0000', 'last_active': '2023-11-08T05:30:44.490691+0000', 'last_peered': '2023-11-08T05:30:44.490691+0000', 'last_clean': '2023-11-08T05:28:01.457904+0000', 'last_became_active': '2023-11-08T05:28:13.550672+0000', 'last_became_peered': '2023-11-08T05:28:13.550672+0000', 'last_unstale': '2023-11-08T05:30:44.490691+0000', 'last_undegraded': '2023-11-08T05:30:44.490691+0000', 'last_fullsized': '2023-11-08T05:28:13.539595+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:31:45.847 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-11-08T05:31:45.847 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 459, 'reported_epoch': 458, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:30:44.490805+0000', 'last_change': '2023-11-08T05:28:13.557968+0000', 'last_active': '2023-11-08T05:30:44.490805+0000', 'last_peered': '2023-11-08T05:30:44.490805+0000', 'last_clean': '2023-11-08T05:28:01.458433+0000', 'last_became_active': '2023-11-08T05:28:13.557968+0000', 'last_became_peered': '2023-11-08T05:28:13.557968+0000', 'last_unstale': '2023-11-08T05:30:44.490805+0000', 'last_undegraded': '2023-11-08T05:30:44.490805+0000', 'last_fullsized': '2023-11-08T05:28:13.542832+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:31:45.847 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-11-08T05:31:46.194 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T05:31:46.194 INFO:teuthology.orchestra.run.smithi087.stdout:{"fsid":"ab4cbc06-449d-497d-b513-220d9db2b590","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 5 pgs undersized","count":5},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2348,"monmap":{"epoch":1,"min_mon_release_name":"pacific","num_mons":1},"osdmap":{"epoch":459,"num_osds":3,"num_up_osds":2,"osd_up_since":1699421292,"num_in_osds":3,"osd_in_since":1699419158,"num_remapped_pgs":3},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean+remapped","count":3},{"state_name":"active+clean","count":1}],"num_pgs":9,"num_pools":2,"num_objects":0,"data_bytes":0,"bytes_used":2697592832,"bytes_avail":287212699648,"bytes_total":289910292480},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":41,"modified":"2023-11-08T05:29:32.044544+0000","services":{}},"progress_events":{}} 2023-11-08T05:31:46.207 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-11-08T05:31:46.208 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-11-08T05:31:46.490 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T05:31:46.492 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2023-11-08T05:31:46.503 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":1627,"stamp":"2023-11-08T05:31:46.072263+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":10,"acting":13,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":109,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2634368,"kb_used_data":409012,"kb_used_omap":780,"kb_used_meta":2220403,"kb_avail":280481152,"statfs":{"total":289910292480,"available":287212699648,"internally_reserved":0,"allocated":418828288,"data_stored":410500163,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":799447,"internal_metadata":2273692969},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-536782902,"num_objects":-164,"num_object_clones":0,"num_object_copies":-328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-115,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-164,"num_whiteouts":0,"num_read":-122056,"num_read_kb":-39011942,"num_write":-413642,"num_write_kb":-151705339,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-8,"num_bytes_recovered":-16922624,"num_keys_recovered":-2,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.424817"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490644+0000","last_change":"2023-11-08T05:28:13.557071+0000","last_active":"2023-11-08T05:30:44.490644+0000","last_peered":"2023-11-08T05:30:44.490644+0000","last_clean":"2023-11-08T05:28:01.458439+0000","last_became_active":"2023-11-08T05:28:13.557071+0000","last_became_peered":"2023-11-08T05:28:13.557071+0000","last_unstale":"2023-11-08T05:30:44.490644+0000","last_undegraded":"2023-11-08T05:30:44.490644+0000","last_fullsized":"2023-11-08T05:28:13.547025+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":459,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:45.794760+0000","last_change":"2023-11-08T05:30:45.794760+0000","last_active":"2023-11-08T05:30:45.794760+0000","last_peered":"2023-11-08T05:30:45.794760+0000","last_clean":"2023-11-08T05:30:45.794760+0000","last_became_active":"2023-11-08T05:30:45.794398+0000","last_became_peered":"2023-11-08T05:30:45.794398+0000","last_unstale":"2023-11-08T05:30:45.794760+0000","last_undegraded":"2023-11-08T05:30:45.794760+0000","last_fullsized":"2023-11-08T05:30:45.794760+0000","mapping_epoch":458,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":459,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":465,"reported_epoch":459,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:45.795085+0000","last_change":"2023-11-08T05:30:45.795085+0000","last_active":"2023-11-08T05:30:45.795085+0000","last_peered":"2023-11-08T05:30:45.795085+0000","last_clean":"2023-11-08T05:30:45.795085+0000","last_became_active":"2023-11-08T05:30:45.794743+0000","last_became_peered":"2023-11-08T05:30:45.794743+0000","last_unstale":"2023-11-08T05:30:45.795085+0000","last_undegraded":"2023-11-08T05:30:45.795085+0000","last_fullsized":"2023-11-08T05:30:45.795085+0000","mapping_epoch":458,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":459,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":454,"reported_epoch":458,"state":"active+clean","last_fresh":"2023-11-08T05:30:44.490618+0000","last_change":"2023-11-08T04:52:47.548597+0000","last_active":"2023-11-08T05:30:44.490618+0000","last_peered":"2023-11-08T05:30:44.490618+0000","last_clean":"2023-11-08T05:30:44.490618+0000","last_became_active":"2023-11-08T04:52:47.548294+0000","last_became_peered":"2023-11-08T04:52:47.548294+0000","last_unstale":"2023-11-08T05:30:44.490618+0000","last_undegraded":"2023-11-08T05:30:44.490618+0000","last_fullsized":"2023-11-08T05:30:44.490618+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":460,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490523+0000","last_change":"2023-11-08T05:28:13.545468+0000","last_active":"2023-11-08T05:30:44.490523+0000","last_peered":"2023-11-08T05:30:44.490523+0000","last_clean":"2023-11-08T05:28:01.456723+0000","last_became_active":"2023-11-08T05:28:13.545468+0000","last_became_peered":"2023-11-08T05:28:13.545468+0000","last_unstale":"2023-11-08T05:30:44.490523+0000","last_undegraded":"2023-11-08T05:30:44.490523+0000","last_fullsized":"2023-11-08T05:28:13.537207+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_clean_scrub_stamp":"2023-11-08T04:52:45.528796+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":465,"reported_epoch":458,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:44.491907+0000","last_change":"2023-11-08T05:29:44.395350+0000","last_active":"2023-11-08T05:30:44.491907+0000","last_peered":"2023-11-08T05:30:44.491907+0000","last_clean":"2023-11-08T05:30:44.491907+0000","last_became_active":"2023-11-08T05:29:44.395012+0000","last_became_peered":"2023-11-08T05:29:44.395012+0000","last_unstale":"2023-11-08T05:30:44.491907+0000","last_undegraded":"2023-11-08T05:30:44.491907+0000","last_fullsized":"2023-11-08T05:30:44.491907+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490629+0000","last_change":"2023-11-08T05:28:13.563357+0000","last_active":"2023-11-08T05:30:44.490629+0000","last_peered":"2023-11-08T05:30:44.490629+0000","last_clean":"2023-11-08T05:28:01.457691+0000","last_became_active":"2023-11-08T05:28:13.563357+0000","last_became_peered":"2023-11-08T05:28:13.563357+0000","last_unstale":"2023-11-08T05:30:44.490629+0000","last_undegraded":"2023-11-08T05:30:44.490629+0000","last_fullsized":"2023-11-08T05:28:13.541614+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490691+0000","last_change":"2023-11-08T05:28:13.550672+0000","last_active":"2023-11-08T05:30:44.490691+0000","last_peered":"2023-11-08T05:30:44.490691+0000","last_clean":"2023-11-08T05:28:01.457904+0000","last_became_active":"2023-11-08T05:28:13.550672+0000","last_became_peered":"2023-11-08T05:28:13.550672+0000","last_unstale":"2023-11-08T05:30:44.490691+0000","last_undegraded":"2023-11-08T05:30:44.490691+0000","last_fullsized":"2023-11-08T05:28:13.539595+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490805+0000","last_change":"2023-11-08T05:28:13.557968+0000","last_active":"2023-11-08T05:30:44.490805+0000","last_peered":"2023-11-08T05:30:44.490805+0000","last_clean":"2023-11-08T05:28:01.458433+0000","last_became_active":"2023-11-08T05:28:13.557968+0000","last_became_peered":"2023-11-08T05:28:13.557968+0000","last_unstale":"2023-11-08T05:30:44.490805+0000","last_undegraded":"2023-11-08T05:30:44.490805+0000","last_fullsized":"2023-11-08T05:28:13.542832+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":1,"acting":1,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":12,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738839,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":645040,"kb_used_data":5352,"kb_used_omap":275,"kb_used_meta":639404,"kb_avail":93726800,"statfs":{"total":96636764160,"available":95976243200,"internally_reserved":0,"allocated":5480448,"data_stored":2697647,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":281650,"internal_metadata":654750670},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:59 2023","interfaces":[{"interface":"back","average":{"1min":0.30499999999999999,"5min":0.439,"15min":0.36099999999999999},"min":{"1min":0.17399999999999999,"5min":0.14499999999999999,"15min":0.14499999999999999},"max":{"1min":0.60099999999999998,"5min":4.9379999999999997,"15min":4.9379999999999997},"last":0.379},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.49399999999999999,"15min":0.42399999999999999},"min":{"1min":0.221,"5min":0.19600000000000001,"15min":0.17000000000000001},"max":{"1min":0.63200000000000001,"5min":4.9859999999999998,"15min":4.9859999999999998},"last":0.40899999999999997}]},{"osd":1,"last update":"Wed Nov 8 05:31:06 2023","interfaces":[{"interface":"back","average":{"1min":0.27500000000000002,"5min":0.40300000000000002,"15min":0.46999999999999997},"min":{"1min":0.20899999999999999,"5min":0.16,"15min":0.153},"max":{"1min":0.41099999999999998,"5min":4.0830000000000002,"15min":23.242999999999999},"last":0.33200000000000002},{"interface":"front","average":{"1min":0.29899999999999999,"5min":0.40100000000000002,"15min":0.46000000000000002},"min":{"1min":0.23799999999999999,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":0.45300000000000001,"5min":3.5590000000000002,"15min":23.251000000000001},"last":0.35499999999999998}]}]},{"osd":0,"up_from":8,"seq":34359738798,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1339228,"kb_used_data":398304,"kb_used_omap":291,"kb_used_meta":936476,"kb_avail":93032612,"statfs":{"total":96636764160,"available":95265394688,"internally_reserved":0,"allocated":407863296,"data_stored":405103900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":298462,"internal_metadata":958951970},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.436,"15min":0.36299999999999999},"min":{"1min":0.19700000000000001,"5min":0.14699999999999999,"15min":0.13700000000000001},"max":{"1min":3.222,"5min":6.5499999999999998,"15min":6.9989999999999997},"last":0.245},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.46300000000000002,"15min":0.42299999999999999},"min":{"1min":0.17799999999999999,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":5.0030000000000001,"5min":6.0629999999999997,"15min":9.6869999999999994},"last":0.28599999999999998}]},{"osd":2,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.51600000000000001,"15min":0.47099999999999997},"min":{"1min":0.21099999999999999,"5min":0.153,"15min":0.153},"max":{"1min":5.7439999999999998,"5min":5.8280000000000003,"15min":9.7639999999999993},"last":0.252},{"interface":"front","average":{"1min":0.497,"5min":0.56200000000000006,"15min":0.42299999999999999},"min":{"1min":0.20399999999999999,"5min":0.187,"15min":0.16800000000000001},"max":{"1min":3.9319999999999999,"5min":8.8040000000000003,"15min":8.8040000000000003},"last":0.29899999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738841,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":650100,"kb_used_data":5356,"kb_used_omap":214,"kb_used_meta":644521,"kb_avail":93721740,"statfs":{"total":96636764160,"available":95971061760,"internally_reserved":0,"allocated":5484544,"data_stored":2698616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":219335,"internal_metadata":659990329},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:56 2023","interfaces":[{"interface":"back","average":{"1min":0.30399999999999999,"5min":0.34699999999999998,"15min":0.36499999999999999},"min":{"1min":0.23999999999999999,"5min":0.14199999999999999,"15min":0.14199999999999999},"max":{"1min":0.42399999999999999,"5min":1.595,"15min":7.4249999999999998},"last":0.17100000000000001},{"interface":"front","average":{"1min":0.35499999999999998,"5min":0.40899999999999997,"15min":0.376},"min":{"1min":0.23999999999999999,"5min":0.19600000000000001,"15min":0.183},"max":{"1min":0.46999999999999997,"5min":2.6859999999999999,"15min":2.6859999999999999},"last":0.23000000000000001}]},{"osd":2,"last update":"Wed Nov 8 05:30:57 2023","interfaces":[{"interface":"back","average":{"1min":0.28599999999999998,"5min":0.34699999999999998,"15min":0.36899999999999999},"min":{"1min":0.19700000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":0.41399999999999998,"5min":2.2650000000000001,"15min":5.8209999999999997},"last":0.30499999999999999},{"interface":"front","average":{"1min":0.30599999999999999,"5min":0.313,"15min":0.35799999999999998},"min":{"1min":0.17799999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.435,"5min":1.137,"15min":4.2350000000000003},"last":0.26000000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-08T05:31:48.804 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~211s 2023-11-08T05:31:49.505 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-11-08T05:31:49.804 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T05:31:49.806 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2023-11-08T05:31:49.816 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":1628,"stamp":"2023-11-08T05:31:48.072623+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":10,"acting":13,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":109,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2634368,"kb_used_data":409012,"kb_used_omap":780,"kb_used_meta":2220403,"kb_avail":280481152,"statfs":{"total":289910292480,"available":287212699648,"internally_reserved":0,"allocated":418828288,"data_stored":410500163,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":799447,"internal_metadata":2273692969},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-536782902,"num_objects":-164,"num_object_clones":0,"num_object_copies":-328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-115,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-164,"num_whiteouts":0,"num_read":-122056,"num_read_kb":-39011942,"num_write":-413642,"num_write_kb":-151705339,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-8,"num_bytes_recovered":-16922624,"num_keys_recovered":-2,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.424817"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490644+0000","last_change":"2023-11-08T05:28:13.557071+0000","last_active":"2023-11-08T05:30:44.490644+0000","last_peered":"2023-11-08T05:30:44.490644+0000","last_clean":"2023-11-08T05:28:01.458439+0000","last_became_active":"2023-11-08T05:28:13.557071+0000","last_became_peered":"2023-11-08T05:28:13.557071+0000","last_unstale":"2023-11-08T05:30:44.490644+0000","last_undegraded":"2023-11-08T05:30:44.490644+0000","last_fullsized":"2023-11-08T05:28:13.547025+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":459,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:45.794760+0000","last_change":"2023-11-08T05:30:45.794760+0000","last_active":"2023-11-08T05:30:45.794760+0000","last_peered":"2023-11-08T05:30:45.794760+0000","last_clean":"2023-11-08T05:30:45.794760+0000","last_became_active":"2023-11-08T05:30:45.794398+0000","last_became_peered":"2023-11-08T05:30:45.794398+0000","last_unstale":"2023-11-08T05:30:45.794760+0000","last_undegraded":"2023-11-08T05:30:45.794760+0000","last_fullsized":"2023-11-08T05:30:45.794760+0000","mapping_epoch":458,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":459,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":465,"reported_epoch":459,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:45.795085+0000","last_change":"2023-11-08T05:30:45.795085+0000","last_active":"2023-11-08T05:30:45.795085+0000","last_peered":"2023-11-08T05:30:45.795085+0000","last_clean":"2023-11-08T05:30:45.795085+0000","last_became_active":"2023-11-08T05:30:45.794743+0000","last_became_peered":"2023-11-08T05:30:45.794743+0000","last_unstale":"2023-11-08T05:30:45.795085+0000","last_undegraded":"2023-11-08T05:30:45.795085+0000","last_fullsized":"2023-11-08T05:30:45.795085+0000","mapping_epoch":458,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":459,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":454,"reported_epoch":458,"state":"active+clean","last_fresh":"2023-11-08T05:30:44.490618+0000","last_change":"2023-11-08T04:52:47.548597+0000","last_active":"2023-11-08T05:30:44.490618+0000","last_peered":"2023-11-08T05:30:44.490618+0000","last_clean":"2023-11-08T05:30:44.490618+0000","last_became_active":"2023-11-08T04:52:47.548294+0000","last_became_peered":"2023-11-08T04:52:47.548294+0000","last_unstale":"2023-11-08T05:30:44.490618+0000","last_undegraded":"2023-11-08T05:30:44.490618+0000","last_fullsized":"2023-11-08T05:30:44.490618+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":460,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490523+0000","last_change":"2023-11-08T05:28:13.545468+0000","last_active":"2023-11-08T05:30:44.490523+0000","last_peered":"2023-11-08T05:30:44.490523+0000","last_clean":"2023-11-08T05:28:01.456723+0000","last_became_active":"2023-11-08T05:28:13.545468+0000","last_became_peered":"2023-11-08T05:28:13.545468+0000","last_unstale":"2023-11-08T05:30:44.490523+0000","last_undegraded":"2023-11-08T05:30:44.490523+0000","last_fullsized":"2023-11-08T05:28:13.537207+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_clean_scrub_stamp":"2023-11-08T04:52:45.528796+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":465,"reported_epoch":458,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:44.491907+0000","last_change":"2023-11-08T05:29:44.395350+0000","last_active":"2023-11-08T05:30:44.491907+0000","last_peered":"2023-11-08T05:30:44.491907+0000","last_clean":"2023-11-08T05:30:44.491907+0000","last_became_active":"2023-11-08T05:29:44.395012+0000","last_became_peered":"2023-11-08T05:29:44.395012+0000","last_unstale":"2023-11-08T05:30:44.491907+0000","last_undegraded":"2023-11-08T05:30:44.491907+0000","last_fullsized":"2023-11-08T05:30:44.491907+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490629+0000","last_change":"2023-11-08T05:28:13.563357+0000","last_active":"2023-11-08T05:30:44.490629+0000","last_peered":"2023-11-08T05:30:44.490629+0000","last_clean":"2023-11-08T05:28:01.457691+0000","last_became_active":"2023-11-08T05:28:13.563357+0000","last_became_peered":"2023-11-08T05:28:13.563357+0000","last_unstale":"2023-11-08T05:30:44.490629+0000","last_undegraded":"2023-11-08T05:30:44.490629+0000","last_fullsized":"2023-11-08T05:28:13.541614+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490691+0000","last_change":"2023-11-08T05:28:13.550672+0000","last_active":"2023-11-08T05:30:44.490691+0000","last_peered":"2023-11-08T05:30:44.490691+0000","last_clean":"2023-11-08T05:28:01.457904+0000","last_became_active":"2023-11-08T05:28:13.550672+0000","last_became_peered":"2023-11-08T05:28:13.550672+0000","last_unstale":"2023-11-08T05:30:44.490691+0000","last_undegraded":"2023-11-08T05:30:44.490691+0000","last_fullsized":"2023-11-08T05:28:13.539595+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490805+0000","last_change":"2023-11-08T05:28:13.557968+0000","last_active":"2023-11-08T05:30:44.490805+0000","last_peered":"2023-11-08T05:30:44.490805+0000","last_clean":"2023-11-08T05:28:01.458433+0000","last_became_active":"2023-11-08T05:28:13.557968+0000","last_became_peered":"2023-11-08T05:28:13.557968+0000","last_unstale":"2023-11-08T05:30:44.490805+0000","last_undegraded":"2023-11-08T05:30:44.490805+0000","last_fullsized":"2023-11-08T05:28:13.542832+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":1,"acting":1,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":12,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738839,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":645040,"kb_used_data":5352,"kb_used_omap":275,"kb_used_meta":639404,"kb_avail":93726800,"statfs":{"total":96636764160,"available":95976243200,"internally_reserved":0,"allocated":5480448,"data_stored":2697647,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":281650,"internal_metadata":654750670},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:59 2023","interfaces":[{"interface":"back","average":{"1min":0.30499999999999999,"5min":0.439,"15min":0.36099999999999999},"min":{"1min":0.17399999999999999,"5min":0.14499999999999999,"15min":0.14499999999999999},"max":{"1min":0.60099999999999998,"5min":4.9379999999999997,"15min":4.9379999999999997},"last":0.379},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.49399999999999999,"15min":0.42399999999999999},"min":{"1min":0.221,"5min":0.19600000000000001,"15min":0.17000000000000001},"max":{"1min":0.63200000000000001,"5min":4.9859999999999998,"15min":4.9859999999999998},"last":0.40899999999999997}]},{"osd":1,"last update":"Wed Nov 8 05:31:06 2023","interfaces":[{"interface":"back","average":{"1min":0.27500000000000002,"5min":0.40300000000000002,"15min":0.46999999999999997},"min":{"1min":0.20899999999999999,"5min":0.16,"15min":0.153},"max":{"1min":0.41099999999999998,"5min":4.0830000000000002,"15min":23.242999999999999},"last":0.33200000000000002},{"interface":"front","average":{"1min":0.29899999999999999,"5min":0.40100000000000002,"15min":0.46000000000000002},"min":{"1min":0.23799999999999999,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":0.45300000000000001,"5min":3.5590000000000002,"15min":23.251000000000001},"last":0.35499999999999998}]}]},{"osd":0,"up_from":8,"seq":34359738798,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1339228,"kb_used_data":398304,"kb_used_omap":291,"kb_used_meta":936476,"kb_avail":93032612,"statfs":{"total":96636764160,"available":95265394688,"internally_reserved":0,"allocated":407863296,"data_stored":405103900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":298462,"internal_metadata":958951970},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.436,"15min":0.36299999999999999},"min":{"1min":0.19700000000000001,"5min":0.14699999999999999,"15min":0.13700000000000001},"max":{"1min":3.222,"5min":6.5499999999999998,"15min":6.9989999999999997},"last":0.245},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.46300000000000002,"15min":0.42299999999999999},"min":{"1min":0.17799999999999999,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":5.0030000000000001,"5min":6.0629999999999997,"15min":9.6869999999999994},"last":0.28599999999999998}]},{"osd":2,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.51600000000000001,"15min":0.47099999999999997},"min":{"1min":0.21099999999999999,"5min":0.153,"15min":0.153},"max":{"1min":5.7439999999999998,"5min":5.8280000000000003,"15min":9.7639999999999993},"last":0.252},{"interface":"front","average":{"1min":0.497,"5min":0.56200000000000006,"15min":0.42299999999999999},"min":{"1min":0.20399999999999999,"5min":0.187,"15min":0.16800000000000001},"max":{"1min":3.9319999999999999,"5min":8.8040000000000003,"15min":8.8040000000000003},"last":0.29899999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738841,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":650100,"kb_used_data":5356,"kb_used_omap":214,"kb_used_meta":644521,"kb_avail":93721740,"statfs":{"total":96636764160,"available":95971061760,"internally_reserved":0,"allocated":5484544,"data_stored":2698616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":219335,"internal_metadata":659990329},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:56 2023","interfaces":[{"interface":"back","average":{"1min":0.30399999999999999,"5min":0.34699999999999998,"15min":0.36499999999999999},"min":{"1min":0.23999999999999999,"5min":0.14199999999999999,"15min":0.14199999999999999},"max":{"1min":0.42399999999999999,"5min":1.595,"15min":7.4249999999999998},"last":0.17100000000000001},{"interface":"front","average":{"1min":0.35499999999999998,"5min":0.40899999999999997,"15min":0.376},"min":{"1min":0.23999999999999999,"5min":0.19600000000000001,"15min":0.183},"max":{"1min":0.46999999999999997,"5min":2.6859999999999999,"15min":2.6859999999999999},"last":0.23000000000000001}]},{"osd":2,"last update":"Wed Nov 8 05:30:57 2023","interfaces":[{"interface":"back","average":{"1min":0.28599999999999998,"5min":0.34699999999999998,"15min":0.36899999999999999},"min":{"1min":0.19700000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":0.41399999999999998,"5min":2.2650000000000001,"15min":5.8209999999999997},"last":0.30499999999999999},{"interface":"front","average":{"1min":0.30599999999999999,"5min":0.313,"15min":0.35799999999999998},"min":{"1min":0.17799999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.435,"5min":1.137,"15min":4.2350000000000003},"last":0.26000000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-08T05:31:49.818 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-11-08T05:31:50.099 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T05:31:50.101 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2023-11-08T05:31:50.112 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":1629,"stamp":"2023-11-08T05:31:50.073050+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":10,"acting":13,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":109,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2634368,"kb_used_data":409012,"kb_used_omap":780,"kb_used_meta":2220403,"kb_avail":280481152,"statfs":{"total":289910292480,"available":287212699648,"internally_reserved":0,"allocated":418828288,"data_stored":410500163,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":799447,"internal_metadata":2273692969},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-536782902,"num_objects":-164,"num_object_clones":0,"num_object_copies":-328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-115,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-164,"num_whiteouts":0,"num_read":-122056,"num_read_kb":-39011942,"num_write":-413642,"num_write_kb":-151705339,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-8,"num_bytes_recovered":-16922624,"num_keys_recovered":-2,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.424817"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490644+0000","last_change":"2023-11-08T05:28:13.557071+0000","last_active":"2023-11-08T05:30:44.490644+0000","last_peered":"2023-11-08T05:30:44.490644+0000","last_clean":"2023-11-08T05:28:01.458439+0000","last_became_active":"2023-11-08T05:28:13.557071+0000","last_became_peered":"2023-11-08T05:28:13.557071+0000","last_unstale":"2023-11-08T05:30:44.490644+0000","last_undegraded":"2023-11-08T05:30:44.490644+0000","last_fullsized":"2023-11-08T05:28:13.547025+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":459,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:45.794760+0000","last_change":"2023-11-08T05:30:45.794760+0000","last_active":"2023-11-08T05:30:45.794760+0000","last_peered":"2023-11-08T05:30:45.794760+0000","last_clean":"2023-11-08T05:30:45.794760+0000","last_became_active":"2023-11-08T05:30:45.794398+0000","last_became_peered":"2023-11-08T05:30:45.794398+0000","last_unstale":"2023-11-08T05:30:45.794760+0000","last_undegraded":"2023-11-08T05:30:45.794760+0000","last_fullsized":"2023-11-08T05:30:45.794760+0000","mapping_epoch":458,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":459,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":465,"reported_epoch":459,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:45.795085+0000","last_change":"2023-11-08T05:30:45.795085+0000","last_active":"2023-11-08T05:30:45.795085+0000","last_peered":"2023-11-08T05:30:45.795085+0000","last_clean":"2023-11-08T05:30:45.795085+0000","last_became_active":"2023-11-08T05:30:45.794743+0000","last_became_peered":"2023-11-08T05:30:45.794743+0000","last_unstale":"2023-11-08T05:30:45.795085+0000","last_undegraded":"2023-11-08T05:30:45.795085+0000","last_fullsized":"2023-11-08T05:30:45.795085+0000","mapping_epoch":458,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":459,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":454,"reported_epoch":458,"state":"active+clean","last_fresh":"2023-11-08T05:30:44.490618+0000","last_change":"2023-11-08T04:52:47.548597+0000","last_active":"2023-11-08T05:30:44.490618+0000","last_peered":"2023-11-08T05:30:44.490618+0000","last_clean":"2023-11-08T05:30:44.490618+0000","last_became_active":"2023-11-08T04:52:47.548294+0000","last_became_peered":"2023-11-08T04:52:47.548294+0000","last_unstale":"2023-11-08T05:30:44.490618+0000","last_undegraded":"2023-11-08T05:30:44.490618+0000","last_fullsized":"2023-11-08T05:30:44.490618+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":460,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490523+0000","last_change":"2023-11-08T05:28:13.545468+0000","last_active":"2023-11-08T05:30:44.490523+0000","last_peered":"2023-11-08T05:30:44.490523+0000","last_clean":"2023-11-08T05:28:01.456723+0000","last_became_active":"2023-11-08T05:28:13.545468+0000","last_became_peered":"2023-11-08T05:28:13.545468+0000","last_unstale":"2023-11-08T05:30:44.490523+0000","last_undegraded":"2023-11-08T05:30:44.490523+0000","last_fullsized":"2023-11-08T05:28:13.537207+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_clean_scrub_stamp":"2023-11-08T04:52:45.528796+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":465,"reported_epoch":458,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:44.491907+0000","last_change":"2023-11-08T05:29:44.395350+0000","last_active":"2023-11-08T05:30:44.491907+0000","last_peered":"2023-11-08T05:30:44.491907+0000","last_clean":"2023-11-08T05:30:44.491907+0000","last_became_active":"2023-11-08T05:29:44.395012+0000","last_became_peered":"2023-11-08T05:29:44.395012+0000","last_unstale":"2023-11-08T05:30:44.491907+0000","last_undegraded":"2023-11-08T05:30:44.491907+0000","last_fullsized":"2023-11-08T05:30:44.491907+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490629+0000","last_change":"2023-11-08T05:28:13.563357+0000","last_active":"2023-11-08T05:30:44.490629+0000","last_peered":"2023-11-08T05:30:44.490629+0000","last_clean":"2023-11-08T05:28:01.457691+0000","last_became_active":"2023-11-08T05:28:13.563357+0000","last_became_peered":"2023-11-08T05:28:13.563357+0000","last_unstale":"2023-11-08T05:30:44.490629+0000","last_undegraded":"2023-11-08T05:30:44.490629+0000","last_fullsized":"2023-11-08T05:28:13.541614+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490691+0000","last_change":"2023-11-08T05:28:13.550672+0000","last_active":"2023-11-08T05:30:44.490691+0000","last_peered":"2023-11-08T05:30:44.490691+0000","last_clean":"2023-11-08T05:28:01.457904+0000","last_became_active":"2023-11-08T05:28:13.550672+0000","last_became_peered":"2023-11-08T05:28:13.550672+0000","last_unstale":"2023-11-08T05:30:44.490691+0000","last_undegraded":"2023-11-08T05:30:44.490691+0000","last_fullsized":"2023-11-08T05:28:13.539595+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490805+0000","last_change":"2023-11-08T05:28:13.557968+0000","last_active":"2023-11-08T05:30:44.490805+0000","last_peered":"2023-11-08T05:30:44.490805+0000","last_clean":"2023-11-08T05:28:01.458433+0000","last_became_active":"2023-11-08T05:28:13.557968+0000","last_became_peered":"2023-11-08T05:28:13.557968+0000","last_unstale":"2023-11-08T05:30:44.490805+0000","last_undegraded":"2023-11-08T05:30:44.490805+0000","last_fullsized":"2023-11-08T05:28:13.542832+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":1,"acting":1,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":12,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738840,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":645040,"kb_used_data":5352,"kb_used_omap":275,"kb_used_meta":639404,"kb_avail":93726800,"statfs":{"total":96636764160,"available":95976243200,"internally_reserved":0,"allocated":5480448,"data_stored":2697647,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":281650,"internal_metadata":654750670},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:59 2023","interfaces":[{"interface":"back","average":{"1min":0.30499999999999999,"5min":0.439,"15min":0.36099999999999999},"min":{"1min":0.17399999999999999,"5min":0.14499999999999999,"15min":0.14499999999999999},"max":{"1min":0.60099999999999998,"5min":4.9379999999999997,"15min":4.9379999999999997},"last":0.379},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.49399999999999999,"15min":0.42399999999999999},"min":{"1min":0.221,"5min":0.19600000000000001,"15min":0.17000000000000001},"max":{"1min":0.63200000000000001,"5min":4.9859999999999998,"15min":4.9859999999999998},"last":0.40899999999999997}]},{"osd":1,"last update":"Wed Nov 8 05:31:06 2023","interfaces":[{"interface":"back","average":{"1min":0.27500000000000002,"5min":0.40300000000000002,"15min":0.46999999999999997},"min":{"1min":0.20899999999999999,"5min":0.16,"15min":0.153},"max":{"1min":0.41099999999999998,"5min":4.0830000000000002,"15min":23.242999999999999},"last":0.34000000000000002},{"interface":"front","average":{"1min":0.29899999999999999,"5min":0.40100000000000002,"15min":0.46000000000000002},"min":{"1min":0.23799999999999999,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":0.45300000000000001,"5min":3.5590000000000002,"15min":23.251000000000001},"last":0.38}]}]},{"osd":0,"up_from":8,"seq":34359738798,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1339228,"kb_used_data":398304,"kb_used_omap":291,"kb_used_meta":936476,"kb_avail":93032612,"statfs":{"total":96636764160,"available":95265394688,"internally_reserved":0,"allocated":407863296,"data_stored":405103900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":298462,"internal_metadata":958951970},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.436,"15min":0.36299999999999999},"min":{"1min":0.19700000000000001,"5min":0.14699999999999999,"15min":0.13700000000000001},"max":{"1min":3.222,"5min":6.5499999999999998,"15min":6.9989999999999997},"last":0.245},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.46300000000000002,"15min":0.42299999999999999},"min":{"1min":0.17799999999999999,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":5.0030000000000001,"5min":6.0629999999999997,"15min":9.6869999999999994},"last":0.28599999999999998}]},{"osd":2,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.51600000000000001,"15min":0.47099999999999997},"min":{"1min":0.21099999999999999,"5min":0.153,"15min":0.153},"max":{"1min":5.7439999999999998,"5min":5.8280000000000003,"15min":9.7639999999999993},"last":0.252},{"interface":"front","average":{"1min":0.497,"5min":0.56200000000000006,"15min":0.42299999999999999},"min":{"1min":0.20399999999999999,"5min":0.187,"15min":0.16800000000000001},"max":{"1min":3.9319999999999999,"5min":8.8040000000000003,"15min":8.8040000000000003},"last":0.29899999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738841,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":650100,"kb_used_data":5356,"kb_used_omap":214,"kb_used_meta":644521,"kb_avail":93721740,"statfs":{"total":96636764160,"available":95971061760,"internally_reserved":0,"allocated":5484544,"data_stored":2698616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":219335,"internal_metadata":659990329},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:56 2023","interfaces":[{"interface":"back","average":{"1min":0.30399999999999999,"5min":0.34699999999999998,"15min":0.36499999999999999},"min":{"1min":0.23999999999999999,"5min":0.14199999999999999,"15min":0.14199999999999999},"max":{"1min":0.42399999999999999,"5min":1.595,"15min":7.4249999999999998},"last":0.17100000000000001},{"interface":"front","average":{"1min":0.35499999999999998,"5min":0.40899999999999997,"15min":0.376},"min":{"1min":0.23999999999999999,"5min":0.19600000000000001,"15min":0.183},"max":{"1min":0.46999999999999997,"5min":2.6859999999999999,"15min":2.6859999999999999},"last":0.23000000000000001}]},{"osd":2,"last update":"Wed Nov 8 05:30:57 2023","interfaces":[{"interface":"back","average":{"1min":0.28599999999999998,"5min":0.34699999999999998,"15min":0.36899999999999999},"min":{"1min":0.19700000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":0.41399999999999998,"5min":2.2650000000000001,"15min":5.8209999999999997},"last":0.30499999999999999},{"interface":"front","average":{"1min":0.30599999999999999,"5min":0.313,"15min":0.35799999999999998},"min":{"1min":0.17799999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.435,"5min":1.137,"15min":4.2350000000000003},"last":0.26000000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-08T05:31:50.113 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-11-08T05:31:50.114 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 459, 'reported_epoch': 458, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:30:44.490644+0000', 'last_change': '2023-11-08T05:28:13.557071+0000', 'last_active': '2023-11-08T05:30:44.490644+0000', 'last_peered': '2023-11-08T05:30:44.490644+0000', 'last_clean': '2023-11-08T05:28:01.458439+0000', 'last_became_active': '2023-11-08T05:28:13.557071+0000', 'last_became_peered': '2023-11-08T05:28:13.557071+0000', 'last_unstale': '2023-11-08T05:30:44.490644+0000', 'last_undegraded': '2023-11-08T05:30:44.490644+0000', 'last_fullsized': '2023-11-08T05:28:13.547025+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:31:50.114 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-11-08T05:31:50.114 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 11, 'reported_epoch': 459, 'state': 'active+clean+remapped', 'last_fresh': '2023-11-08T05:30:45.794760+0000', 'last_change': '2023-11-08T05:30:45.794760+0000', 'last_active': '2023-11-08T05:30:45.794760+0000', 'last_peered': '2023-11-08T05:30:45.794760+0000', 'last_clean': '2023-11-08T05:30:45.794760+0000', 'last_became_active': '2023-11-08T05:30:45.794398+0000', 'last_became_peered': '2023-11-08T05:30:45.794398+0000', 'last_unstale': '2023-11-08T05:30:45.794760+0000', 'last_undegraded': '2023-11-08T05:30:45.794760+0000', 'last_fullsized': '2023-11-08T05:30:45.794760+0000', 'mapping_epoch': 458, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 459, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2], 'acting': [2, 1], 'avail_no_missing': ['2', '1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2023-11-08T05:31:50.115 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.1 is not active+clean 2023-11-08T05:31:50.115 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.1', 'version': "0'0", 'reported_seq': 465, 'reported_epoch': 459, 'state': 'active+clean+remapped', 'last_fresh': '2023-11-08T05:30:45.795085+0000', 'last_change': '2023-11-08T05:30:45.795085+0000', 'last_active': '2023-11-08T05:30:45.795085+0000', 'last_peered': '2023-11-08T05:30:45.795085+0000', 'last_clean': '2023-11-08T05:30:45.795085+0000', 'last_became_active': '2023-11-08T05:30:45.794743+0000', 'last_became_peered': '2023-11-08T05:30:45.794743+0000', 'last_unstale': '2023-11-08T05:30:45.795085+0000', 'last_undegraded': '2023-11-08T05:30:45.795085+0000', 'last_fullsized': '2023-11-08T05:30:45.795085+0000', 'mapping_epoch': 458, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 459, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2], 'acting': [2, 1], 'avail_no_missing': ['2', '1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2023-11-08T05:31:50.115 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-11-08T05:31:50.115 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "0'0", 'reported_seq': 460, 'reported_epoch': 458, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:30:44.490523+0000', 'last_change': '2023-11-08T05:28:13.545468+0000', 'last_active': '2023-11-08T05:30:44.490523+0000', 'last_peered': '2023-11-08T05:30:44.490523+0000', 'last_clean': '2023-11-08T05:28:01.456723+0000', 'last_became_active': '2023-11-08T05:28:13.545468+0000', 'last_became_peered': '2023-11-08T05:28:13.545468+0000', 'last_unstale': '2023-11-08T05:30:44.490523+0000', 'last_undegraded': '2023-11-08T05:30:44.490523+0000', 'last_fullsized': '2023-11-08T05:28:13.537207+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 9, 'last_epoch_clean': 10, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:45.528796+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:45.528796+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:45.528796+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:31:50.116 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.0 is not active+clean 2023-11-08T05:31:50.116 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.0', 'version': "0'0", 'reported_seq': 465, 'reported_epoch': 458, 'state': 'active+clean+remapped', 'last_fresh': '2023-11-08T05:30:44.491907+0000', 'last_change': '2023-11-08T05:29:44.395350+0000', 'last_active': '2023-11-08T05:30:44.491907+0000', 'last_peered': '2023-11-08T05:30:44.491907+0000', 'last_clean': '2023-11-08T05:30:44.491907+0000', 'last_became_active': '2023-11-08T05:29:44.395012+0000', 'last_became_peered': '2023-11-08T05:29:44.395012+0000', 'last_unstale': '2023-11-08T05:30:44.491907+0000', 'last_undegraded': '2023-11-08T05:30:44.491907+0000', 'last_fullsized': '2023-11-08T05:30:44.491907+0000', 'mapping_epoch': 455, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 456, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2], 'acting': [2, 1], 'avail_no_missing': ['2', '1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2023-11-08T05:31:50.116 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-11-08T05:31:50.117 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 459, 'reported_epoch': 458, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:30:44.490629+0000', 'last_change': '2023-11-08T05:28:13.563357+0000', 'last_active': '2023-11-08T05:30:44.490629+0000', 'last_peered': '2023-11-08T05:30:44.490629+0000', 'last_clean': '2023-11-08T05:28:01.457691+0000', 'last_became_active': '2023-11-08T05:28:13.563357+0000', 'last_became_peered': '2023-11-08T05:28:13.563357+0000', 'last_unstale': '2023-11-08T05:30:44.490629+0000', 'last_undegraded': '2023-11-08T05:30:44.490629+0000', 'last_fullsized': '2023-11-08T05:28:13.541614+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:31:50.117 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-11-08T05:31:50.117 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 459, 'reported_epoch': 458, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:30:44.490691+0000', 'last_change': '2023-11-08T05:28:13.550672+0000', 'last_active': '2023-11-08T05:30:44.490691+0000', 'last_peered': '2023-11-08T05:30:44.490691+0000', 'last_clean': '2023-11-08T05:28:01.457904+0000', 'last_became_active': '2023-11-08T05:28:13.550672+0000', 'last_became_peered': '2023-11-08T05:28:13.550672+0000', 'last_unstale': '2023-11-08T05:30:44.490691+0000', 'last_undegraded': '2023-11-08T05:30:44.490691+0000', 'last_fullsized': '2023-11-08T05:28:13.539595+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:31:50.118 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-11-08T05:31:50.118 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 459, 'reported_epoch': 458, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:30:44.490805+0000', 'last_change': '2023-11-08T05:28:13.557968+0000', 'last_active': '2023-11-08T05:30:44.490805+0000', 'last_peered': '2023-11-08T05:30:44.490805+0000', 'last_clean': '2023-11-08T05:28:01.458433+0000', 'last_became_active': '2023-11-08T05:28:13.557968+0000', 'last_became_peered': '2023-11-08T05:28:13.557968+0000', 'last_unstale': '2023-11-08T05:30:44.490805+0000', 'last_undegraded': '2023-11-08T05:30:44.490805+0000', 'last_fullsized': '2023-11-08T05:28:13.542832+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:31:50.118 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-11-08T05:31:50.465 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T05:31:50.465 INFO:teuthology.orchestra.run.smithi087.stdout:{"fsid":"ab4cbc06-449d-497d-b513-220d9db2b590","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 5 pgs undersized","count":5},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2353,"monmap":{"epoch":1,"min_mon_release_name":"pacific","num_mons":1},"osdmap":{"epoch":459,"num_osds":3,"num_up_osds":2,"osd_up_since":1699421292,"num_in_osds":3,"osd_in_since":1699419158,"num_remapped_pgs":3},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean+remapped","count":3},{"state_name":"active+clean","count":1}],"num_pgs":9,"num_pools":2,"num_objects":0,"data_bytes":0,"bytes_used":2697592832,"bytes_avail":287212699648,"bytes_total":289910292480},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":41,"modified":"2023-11-08T05:29:32.044544+0000","services":{}},"progress_events":{}} 2023-11-08T05:31:50.478 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-11-08T05:31:50.478 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-11-08T05:31:50.763 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T05:31:50.764 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2023-11-08T05:31:50.775 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":1629,"stamp":"2023-11-08T05:31:50.073050+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":10,"acting":13,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":109,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2634368,"kb_used_data":409012,"kb_used_omap":780,"kb_used_meta":2220403,"kb_avail":280481152,"statfs":{"total":289910292480,"available":287212699648,"internally_reserved":0,"allocated":418828288,"data_stored":410500163,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":799447,"internal_metadata":2273692969},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-536782902,"num_objects":-164,"num_object_clones":0,"num_object_copies":-328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-115,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-164,"num_whiteouts":0,"num_read":-122056,"num_read_kb":-39011942,"num_write":-413642,"num_write_kb":-151705339,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-8,"num_bytes_recovered":-16922624,"num_keys_recovered":-2,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.424817"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490644+0000","last_change":"2023-11-08T05:28:13.557071+0000","last_active":"2023-11-08T05:30:44.490644+0000","last_peered":"2023-11-08T05:30:44.490644+0000","last_clean":"2023-11-08T05:28:01.458439+0000","last_became_active":"2023-11-08T05:28:13.557071+0000","last_became_peered":"2023-11-08T05:28:13.557071+0000","last_unstale":"2023-11-08T05:30:44.490644+0000","last_undegraded":"2023-11-08T05:30:44.490644+0000","last_fullsized":"2023-11-08T05:28:13.547025+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":459,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:45.794760+0000","last_change":"2023-11-08T05:30:45.794760+0000","last_active":"2023-11-08T05:30:45.794760+0000","last_peered":"2023-11-08T05:30:45.794760+0000","last_clean":"2023-11-08T05:30:45.794760+0000","last_became_active":"2023-11-08T05:30:45.794398+0000","last_became_peered":"2023-11-08T05:30:45.794398+0000","last_unstale":"2023-11-08T05:30:45.794760+0000","last_undegraded":"2023-11-08T05:30:45.794760+0000","last_fullsized":"2023-11-08T05:30:45.794760+0000","mapping_epoch":458,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":459,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":465,"reported_epoch":459,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:45.795085+0000","last_change":"2023-11-08T05:30:45.795085+0000","last_active":"2023-11-08T05:30:45.795085+0000","last_peered":"2023-11-08T05:30:45.795085+0000","last_clean":"2023-11-08T05:30:45.795085+0000","last_became_active":"2023-11-08T05:30:45.794743+0000","last_became_peered":"2023-11-08T05:30:45.794743+0000","last_unstale":"2023-11-08T05:30:45.795085+0000","last_undegraded":"2023-11-08T05:30:45.795085+0000","last_fullsized":"2023-11-08T05:30:45.795085+0000","mapping_epoch":458,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":459,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":454,"reported_epoch":458,"state":"active+clean","last_fresh":"2023-11-08T05:30:44.490618+0000","last_change":"2023-11-08T04:52:47.548597+0000","last_active":"2023-11-08T05:30:44.490618+0000","last_peered":"2023-11-08T05:30:44.490618+0000","last_clean":"2023-11-08T05:30:44.490618+0000","last_became_active":"2023-11-08T04:52:47.548294+0000","last_became_peered":"2023-11-08T04:52:47.548294+0000","last_unstale":"2023-11-08T05:30:44.490618+0000","last_undegraded":"2023-11-08T05:30:44.490618+0000","last_fullsized":"2023-11-08T05:30:44.490618+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":460,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490523+0000","last_change":"2023-11-08T05:28:13.545468+0000","last_active":"2023-11-08T05:30:44.490523+0000","last_peered":"2023-11-08T05:30:44.490523+0000","last_clean":"2023-11-08T05:28:01.456723+0000","last_became_active":"2023-11-08T05:28:13.545468+0000","last_became_peered":"2023-11-08T05:28:13.545468+0000","last_unstale":"2023-11-08T05:30:44.490523+0000","last_undegraded":"2023-11-08T05:30:44.490523+0000","last_fullsized":"2023-11-08T05:28:13.537207+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_clean_scrub_stamp":"2023-11-08T04:52:45.528796+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":465,"reported_epoch":458,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:44.491907+0000","last_change":"2023-11-08T05:29:44.395350+0000","last_active":"2023-11-08T05:30:44.491907+0000","last_peered":"2023-11-08T05:30:44.491907+0000","last_clean":"2023-11-08T05:30:44.491907+0000","last_became_active":"2023-11-08T05:29:44.395012+0000","last_became_peered":"2023-11-08T05:29:44.395012+0000","last_unstale":"2023-11-08T05:30:44.491907+0000","last_undegraded":"2023-11-08T05:30:44.491907+0000","last_fullsized":"2023-11-08T05:30:44.491907+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490629+0000","last_change":"2023-11-08T05:28:13.563357+0000","last_active":"2023-11-08T05:30:44.490629+0000","last_peered":"2023-11-08T05:30:44.490629+0000","last_clean":"2023-11-08T05:28:01.457691+0000","last_became_active":"2023-11-08T05:28:13.563357+0000","last_became_peered":"2023-11-08T05:28:13.563357+0000","last_unstale":"2023-11-08T05:30:44.490629+0000","last_undegraded":"2023-11-08T05:30:44.490629+0000","last_fullsized":"2023-11-08T05:28:13.541614+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490691+0000","last_change":"2023-11-08T05:28:13.550672+0000","last_active":"2023-11-08T05:30:44.490691+0000","last_peered":"2023-11-08T05:30:44.490691+0000","last_clean":"2023-11-08T05:28:01.457904+0000","last_became_active":"2023-11-08T05:28:13.550672+0000","last_became_peered":"2023-11-08T05:28:13.550672+0000","last_unstale":"2023-11-08T05:30:44.490691+0000","last_undegraded":"2023-11-08T05:30:44.490691+0000","last_fullsized":"2023-11-08T05:28:13.539595+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490805+0000","last_change":"2023-11-08T05:28:13.557968+0000","last_active":"2023-11-08T05:30:44.490805+0000","last_peered":"2023-11-08T05:30:44.490805+0000","last_clean":"2023-11-08T05:28:01.458433+0000","last_became_active":"2023-11-08T05:28:13.557968+0000","last_became_peered":"2023-11-08T05:28:13.557968+0000","last_unstale":"2023-11-08T05:30:44.490805+0000","last_undegraded":"2023-11-08T05:30:44.490805+0000","last_fullsized":"2023-11-08T05:28:13.542832+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":1,"acting":1,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":12,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738840,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":645040,"kb_used_data":5352,"kb_used_omap":275,"kb_used_meta":639404,"kb_avail":93726800,"statfs":{"total":96636764160,"available":95976243200,"internally_reserved":0,"allocated":5480448,"data_stored":2697647,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":281650,"internal_metadata":654750670},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:59 2023","interfaces":[{"interface":"back","average":{"1min":0.30499999999999999,"5min":0.439,"15min":0.36099999999999999},"min":{"1min":0.17399999999999999,"5min":0.14499999999999999,"15min":0.14499999999999999},"max":{"1min":0.60099999999999998,"5min":4.9379999999999997,"15min":4.9379999999999997},"last":0.379},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.49399999999999999,"15min":0.42399999999999999},"min":{"1min":0.221,"5min":0.19600000000000001,"15min":0.17000000000000001},"max":{"1min":0.63200000000000001,"5min":4.9859999999999998,"15min":4.9859999999999998},"last":0.40899999999999997}]},{"osd":1,"last update":"Wed Nov 8 05:31:06 2023","interfaces":[{"interface":"back","average":{"1min":0.27500000000000002,"5min":0.40300000000000002,"15min":0.46999999999999997},"min":{"1min":0.20899999999999999,"5min":0.16,"15min":0.153},"max":{"1min":0.41099999999999998,"5min":4.0830000000000002,"15min":23.242999999999999},"last":0.34000000000000002},{"interface":"front","average":{"1min":0.29899999999999999,"5min":0.40100000000000002,"15min":0.46000000000000002},"min":{"1min":0.23799999999999999,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":0.45300000000000001,"5min":3.5590000000000002,"15min":23.251000000000001},"last":0.38}]}]},{"osd":0,"up_from":8,"seq":34359738798,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1339228,"kb_used_data":398304,"kb_used_omap":291,"kb_used_meta":936476,"kb_avail":93032612,"statfs":{"total":96636764160,"available":95265394688,"internally_reserved":0,"allocated":407863296,"data_stored":405103900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":298462,"internal_metadata":958951970},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.436,"15min":0.36299999999999999},"min":{"1min":0.19700000000000001,"5min":0.14699999999999999,"15min":0.13700000000000001},"max":{"1min":3.222,"5min":6.5499999999999998,"15min":6.9989999999999997},"last":0.245},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.46300000000000002,"15min":0.42299999999999999},"min":{"1min":0.17799999999999999,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":5.0030000000000001,"5min":6.0629999999999997,"15min":9.6869999999999994},"last":0.28599999999999998}]},{"osd":2,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.51600000000000001,"15min":0.47099999999999997},"min":{"1min":0.21099999999999999,"5min":0.153,"15min":0.153},"max":{"1min":5.7439999999999998,"5min":5.8280000000000003,"15min":9.7639999999999993},"last":0.252},{"interface":"front","average":{"1min":0.497,"5min":0.56200000000000006,"15min":0.42299999999999999},"min":{"1min":0.20399999999999999,"5min":0.187,"15min":0.16800000000000001},"max":{"1min":3.9319999999999999,"5min":8.8040000000000003,"15min":8.8040000000000003},"last":0.29899999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738841,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":650100,"kb_used_data":5356,"kb_used_omap":214,"kb_used_meta":644521,"kb_avail":93721740,"statfs":{"total":96636764160,"available":95971061760,"internally_reserved":0,"allocated":5484544,"data_stored":2698616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":219335,"internal_metadata":659990329},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:56 2023","interfaces":[{"interface":"back","average":{"1min":0.30399999999999999,"5min":0.34699999999999998,"15min":0.36499999999999999},"min":{"1min":0.23999999999999999,"5min":0.14199999999999999,"15min":0.14199999999999999},"max":{"1min":0.42399999999999999,"5min":1.595,"15min":7.4249999999999998},"last":0.17100000000000001},{"interface":"front","average":{"1min":0.35499999999999998,"5min":0.40899999999999997,"15min":0.376},"min":{"1min":0.23999999999999999,"5min":0.19600000000000001,"15min":0.183},"max":{"1min":0.46999999999999997,"5min":2.6859999999999999,"15min":2.6859999999999999},"last":0.23000000000000001}]},{"osd":2,"last update":"Wed Nov 8 05:30:57 2023","interfaces":[{"interface":"back","average":{"1min":0.28599999999999998,"5min":0.34699999999999998,"15min":0.36899999999999999},"min":{"1min":0.19700000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":0.41399999999999998,"5min":2.2650000000000001,"15min":5.8209999999999997},"last":0.30499999999999999},{"interface":"front","average":{"1min":0.30599999999999999,"5min":0.313,"15min":0.35799999999999998},"min":{"1min":0.17799999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.435,"5min":1.137,"15min":4.2350000000000003},"last":0.26000000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-08T05:31:53.778 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-11-08T05:31:54.073 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T05:31:54.075 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2023-11-08T05:31:54.086 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":1630,"stamp":"2023-11-08T05:31:52.073362+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":10,"acting":13,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":109,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2634368,"kb_used_data":409012,"kb_used_omap":780,"kb_used_meta":2220403,"kb_avail":280481152,"statfs":{"total":289910292480,"available":287212699648,"internally_reserved":0,"allocated":418828288,"data_stored":410500163,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":799447,"internal_metadata":2273692969},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-536782902,"num_objects":-164,"num_object_clones":0,"num_object_copies":-328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-115,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-164,"num_whiteouts":0,"num_read":-122056,"num_read_kb":-39011942,"num_write":-413642,"num_write_kb":-151705339,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-8,"num_bytes_recovered":-16922624,"num_keys_recovered":-2,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.424817"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490644+0000","last_change":"2023-11-08T05:28:13.557071+0000","last_active":"2023-11-08T05:30:44.490644+0000","last_peered":"2023-11-08T05:30:44.490644+0000","last_clean":"2023-11-08T05:28:01.458439+0000","last_became_active":"2023-11-08T05:28:13.557071+0000","last_became_peered":"2023-11-08T05:28:13.557071+0000","last_unstale":"2023-11-08T05:30:44.490644+0000","last_undegraded":"2023-11-08T05:30:44.490644+0000","last_fullsized":"2023-11-08T05:28:13.547025+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":459,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:45.794760+0000","last_change":"2023-11-08T05:30:45.794760+0000","last_active":"2023-11-08T05:30:45.794760+0000","last_peered":"2023-11-08T05:30:45.794760+0000","last_clean":"2023-11-08T05:30:45.794760+0000","last_became_active":"2023-11-08T05:30:45.794398+0000","last_became_peered":"2023-11-08T05:30:45.794398+0000","last_unstale":"2023-11-08T05:30:45.794760+0000","last_undegraded":"2023-11-08T05:30:45.794760+0000","last_fullsized":"2023-11-08T05:30:45.794760+0000","mapping_epoch":458,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":459,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":465,"reported_epoch":459,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:45.795085+0000","last_change":"2023-11-08T05:30:45.795085+0000","last_active":"2023-11-08T05:30:45.795085+0000","last_peered":"2023-11-08T05:30:45.795085+0000","last_clean":"2023-11-08T05:30:45.795085+0000","last_became_active":"2023-11-08T05:30:45.794743+0000","last_became_peered":"2023-11-08T05:30:45.794743+0000","last_unstale":"2023-11-08T05:30:45.795085+0000","last_undegraded":"2023-11-08T05:30:45.795085+0000","last_fullsized":"2023-11-08T05:30:45.795085+0000","mapping_epoch":458,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":459,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":454,"reported_epoch":458,"state":"active+clean","last_fresh":"2023-11-08T05:30:44.490618+0000","last_change":"2023-11-08T04:52:47.548597+0000","last_active":"2023-11-08T05:30:44.490618+0000","last_peered":"2023-11-08T05:30:44.490618+0000","last_clean":"2023-11-08T05:30:44.490618+0000","last_became_active":"2023-11-08T04:52:47.548294+0000","last_became_peered":"2023-11-08T04:52:47.548294+0000","last_unstale":"2023-11-08T05:30:44.490618+0000","last_undegraded":"2023-11-08T05:30:44.490618+0000","last_fullsized":"2023-11-08T05:30:44.490618+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":460,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490523+0000","last_change":"2023-11-08T05:28:13.545468+0000","last_active":"2023-11-08T05:30:44.490523+0000","last_peered":"2023-11-08T05:30:44.490523+0000","last_clean":"2023-11-08T05:28:01.456723+0000","last_became_active":"2023-11-08T05:28:13.545468+0000","last_became_peered":"2023-11-08T05:28:13.545468+0000","last_unstale":"2023-11-08T05:30:44.490523+0000","last_undegraded":"2023-11-08T05:30:44.490523+0000","last_fullsized":"2023-11-08T05:28:13.537207+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_clean_scrub_stamp":"2023-11-08T04:52:45.528796+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":465,"reported_epoch":458,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:44.491907+0000","last_change":"2023-11-08T05:29:44.395350+0000","last_active":"2023-11-08T05:30:44.491907+0000","last_peered":"2023-11-08T05:30:44.491907+0000","last_clean":"2023-11-08T05:30:44.491907+0000","last_became_active":"2023-11-08T05:29:44.395012+0000","last_became_peered":"2023-11-08T05:29:44.395012+0000","last_unstale":"2023-11-08T05:30:44.491907+0000","last_undegraded":"2023-11-08T05:30:44.491907+0000","last_fullsized":"2023-11-08T05:30:44.491907+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490629+0000","last_change":"2023-11-08T05:28:13.563357+0000","last_active":"2023-11-08T05:30:44.490629+0000","last_peered":"2023-11-08T05:30:44.490629+0000","last_clean":"2023-11-08T05:28:01.457691+0000","last_became_active":"2023-11-08T05:28:13.563357+0000","last_became_peered":"2023-11-08T05:28:13.563357+0000","last_unstale":"2023-11-08T05:30:44.490629+0000","last_undegraded":"2023-11-08T05:30:44.490629+0000","last_fullsized":"2023-11-08T05:28:13.541614+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490691+0000","last_change":"2023-11-08T05:28:13.550672+0000","last_active":"2023-11-08T05:30:44.490691+0000","last_peered":"2023-11-08T05:30:44.490691+0000","last_clean":"2023-11-08T05:28:01.457904+0000","last_became_active":"2023-11-08T05:28:13.550672+0000","last_became_peered":"2023-11-08T05:28:13.550672+0000","last_unstale":"2023-11-08T05:30:44.490691+0000","last_undegraded":"2023-11-08T05:30:44.490691+0000","last_fullsized":"2023-11-08T05:28:13.539595+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490805+0000","last_change":"2023-11-08T05:28:13.557968+0000","last_active":"2023-11-08T05:30:44.490805+0000","last_peered":"2023-11-08T05:30:44.490805+0000","last_clean":"2023-11-08T05:28:01.458433+0000","last_became_active":"2023-11-08T05:28:13.557968+0000","last_became_peered":"2023-11-08T05:28:13.557968+0000","last_unstale":"2023-11-08T05:30:44.490805+0000","last_undegraded":"2023-11-08T05:30:44.490805+0000","last_fullsized":"2023-11-08T05:28:13.542832+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":1,"acting":1,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":12,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738840,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":645040,"kb_used_data":5352,"kb_used_omap":275,"kb_used_meta":639404,"kb_avail":93726800,"statfs":{"total":96636764160,"available":95976243200,"internally_reserved":0,"allocated":5480448,"data_stored":2697647,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":281650,"internal_metadata":654750670},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:59 2023","interfaces":[{"interface":"back","average":{"1min":0.30499999999999999,"5min":0.439,"15min":0.36099999999999999},"min":{"1min":0.17399999999999999,"5min":0.14499999999999999,"15min":0.14499999999999999},"max":{"1min":0.60099999999999998,"5min":4.9379999999999997,"15min":4.9379999999999997},"last":0.379},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.49399999999999999,"15min":0.42399999999999999},"min":{"1min":0.221,"5min":0.19600000000000001,"15min":0.17000000000000001},"max":{"1min":0.63200000000000001,"5min":4.9859999999999998,"15min":4.9859999999999998},"last":0.40899999999999997}]},{"osd":1,"last update":"Wed Nov 8 05:31:06 2023","interfaces":[{"interface":"back","average":{"1min":0.27500000000000002,"5min":0.40300000000000002,"15min":0.46999999999999997},"min":{"1min":0.20899999999999999,"5min":0.16,"15min":0.153},"max":{"1min":0.41099999999999998,"5min":4.0830000000000002,"15min":23.242999999999999},"last":0.34000000000000002},{"interface":"front","average":{"1min":0.29899999999999999,"5min":0.40100000000000002,"15min":0.46000000000000002},"min":{"1min":0.23799999999999999,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":0.45300000000000001,"5min":3.5590000000000002,"15min":23.251000000000001},"last":0.38}]}]},{"osd":0,"up_from":8,"seq":34359738798,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1339228,"kb_used_data":398304,"kb_used_omap":291,"kb_used_meta":936476,"kb_avail":93032612,"statfs":{"total":96636764160,"available":95265394688,"internally_reserved":0,"allocated":407863296,"data_stored":405103900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":298462,"internal_metadata":958951970},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.436,"15min":0.36299999999999999},"min":{"1min":0.19700000000000001,"5min":0.14699999999999999,"15min":0.13700000000000001},"max":{"1min":3.222,"5min":6.5499999999999998,"15min":6.9989999999999997},"last":0.245},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.46300000000000002,"15min":0.42299999999999999},"min":{"1min":0.17799999999999999,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":5.0030000000000001,"5min":6.0629999999999997,"15min":9.6869999999999994},"last":0.28599999999999998}]},{"osd":2,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.51600000000000001,"15min":0.47099999999999997},"min":{"1min":0.21099999999999999,"5min":0.153,"15min":0.153},"max":{"1min":5.7439999999999998,"5min":5.8280000000000003,"15min":9.7639999999999993},"last":0.252},{"interface":"front","average":{"1min":0.497,"5min":0.56200000000000006,"15min":0.42299999999999999},"min":{"1min":0.20399999999999999,"5min":0.187,"15min":0.16800000000000001},"max":{"1min":3.9319999999999999,"5min":8.8040000000000003,"15min":8.8040000000000003},"last":0.29899999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738842,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":650100,"kb_used_data":5356,"kb_used_omap":214,"kb_used_meta":644521,"kb_avail":93721740,"statfs":{"total":96636764160,"available":95971061760,"internally_reserved":0,"allocated":5484544,"data_stored":2698616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":219335,"internal_metadata":659990329},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:56 2023","interfaces":[{"interface":"back","average":{"1min":0.30399999999999999,"5min":0.34699999999999998,"15min":0.36499999999999999},"min":{"1min":0.23999999999999999,"5min":0.14199999999999999,"15min":0.14199999999999999},"max":{"1min":0.42399999999999999,"5min":1.595,"15min":7.4249999999999998},"last":0.17100000000000001},{"interface":"front","average":{"1min":0.35499999999999998,"5min":0.40899999999999997,"15min":0.376},"min":{"1min":0.23999999999999999,"5min":0.19600000000000001,"15min":0.183},"max":{"1min":0.46999999999999997,"5min":2.6859999999999999,"15min":2.6859999999999999},"last":0.23000000000000001}]},{"osd":2,"last update":"Wed Nov 8 05:30:57 2023","interfaces":[{"interface":"back","average":{"1min":0.28599999999999998,"5min":0.34699999999999998,"15min":0.36899999999999999},"min":{"1min":0.19700000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":0.41399999999999998,"5min":2.2650000000000001,"15min":5.8209999999999997},"last":0.39600000000000002},{"interface":"front","average":{"1min":0.30599999999999999,"5min":0.313,"15min":0.35799999999999998},"min":{"1min":0.17799999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.435,"5min":1.137,"15min":4.2350000000000003},"last":0.35799999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-08T05:31:54.087 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-11-08T05:31:54.207 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~216s 2023-11-08T05:31:54.367 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T05:31:54.369 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2023-11-08T05:31:54.380 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":1631,"stamp":"2023-11-08T05:31:54.073648+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":10,"acting":13,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":109,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2634368,"kb_used_data":409012,"kb_used_omap":780,"kb_used_meta":2220403,"kb_avail":280481152,"statfs":{"total":289910292480,"available":287212699648,"internally_reserved":0,"allocated":418828288,"data_stored":410500163,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":799447,"internal_metadata":2273692969},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-536782902,"num_objects":-164,"num_object_clones":0,"num_object_copies":-328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-115,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-164,"num_whiteouts":0,"num_read":-122056,"num_read_kb":-39011942,"num_write":-413642,"num_write_kb":-151705339,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-8,"num_bytes_recovered":-16922624,"num_keys_recovered":-2,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.424817"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490644+0000","last_change":"2023-11-08T05:28:13.557071+0000","last_active":"2023-11-08T05:30:44.490644+0000","last_peered":"2023-11-08T05:30:44.490644+0000","last_clean":"2023-11-08T05:28:01.458439+0000","last_became_active":"2023-11-08T05:28:13.557071+0000","last_became_peered":"2023-11-08T05:28:13.557071+0000","last_unstale":"2023-11-08T05:30:44.490644+0000","last_undegraded":"2023-11-08T05:30:44.490644+0000","last_fullsized":"2023-11-08T05:28:13.547025+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":459,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:45.794760+0000","last_change":"2023-11-08T05:30:45.794760+0000","last_active":"2023-11-08T05:30:45.794760+0000","last_peered":"2023-11-08T05:30:45.794760+0000","last_clean":"2023-11-08T05:30:45.794760+0000","last_became_active":"2023-11-08T05:30:45.794398+0000","last_became_peered":"2023-11-08T05:30:45.794398+0000","last_unstale":"2023-11-08T05:30:45.794760+0000","last_undegraded":"2023-11-08T05:30:45.794760+0000","last_fullsized":"2023-11-08T05:30:45.794760+0000","mapping_epoch":458,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":459,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":465,"reported_epoch":459,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:45.795085+0000","last_change":"2023-11-08T05:30:45.795085+0000","last_active":"2023-11-08T05:30:45.795085+0000","last_peered":"2023-11-08T05:30:45.795085+0000","last_clean":"2023-11-08T05:30:45.795085+0000","last_became_active":"2023-11-08T05:30:45.794743+0000","last_became_peered":"2023-11-08T05:30:45.794743+0000","last_unstale":"2023-11-08T05:30:45.795085+0000","last_undegraded":"2023-11-08T05:30:45.795085+0000","last_fullsized":"2023-11-08T05:30:45.795085+0000","mapping_epoch":458,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":459,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":454,"reported_epoch":458,"state":"active+clean","last_fresh":"2023-11-08T05:30:44.490618+0000","last_change":"2023-11-08T04:52:47.548597+0000","last_active":"2023-11-08T05:30:44.490618+0000","last_peered":"2023-11-08T05:30:44.490618+0000","last_clean":"2023-11-08T05:30:44.490618+0000","last_became_active":"2023-11-08T04:52:47.548294+0000","last_became_peered":"2023-11-08T04:52:47.548294+0000","last_unstale":"2023-11-08T05:30:44.490618+0000","last_undegraded":"2023-11-08T05:30:44.490618+0000","last_fullsized":"2023-11-08T05:30:44.490618+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":460,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490523+0000","last_change":"2023-11-08T05:28:13.545468+0000","last_active":"2023-11-08T05:30:44.490523+0000","last_peered":"2023-11-08T05:30:44.490523+0000","last_clean":"2023-11-08T05:28:01.456723+0000","last_became_active":"2023-11-08T05:28:13.545468+0000","last_became_peered":"2023-11-08T05:28:13.545468+0000","last_unstale":"2023-11-08T05:30:44.490523+0000","last_undegraded":"2023-11-08T05:30:44.490523+0000","last_fullsized":"2023-11-08T05:28:13.537207+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_clean_scrub_stamp":"2023-11-08T04:52:45.528796+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":465,"reported_epoch":458,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:44.491907+0000","last_change":"2023-11-08T05:29:44.395350+0000","last_active":"2023-11-08T05:30:44.491907+0000","last_peered":"2023-11-08T05:30:44.491907+0000","last_clean":"2023-11-08T05:30:44.491907+0000","last_became_active":"2023-11-08T05:29:44.395012+0000","last_became_peered":"2023-11-08T05:29:44.395012+0000","last_unstale":"2023-11-08T05:30:44.491907+0000","last_undegraded":"2023-11-08T05:30:44.491907+0000","last_fullsized":"2023-11-08T05:30:44.491907+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490629+0000","last_change":"2023-11-08T05:28:13.563357+0000","last_active":"2023-11-08T05:30:44.490629+0000","last_peered":"2023-11-08T05:30:44.490629+0000","last_clean":"2023-11-08T05:28:01.457691+0000","last_became_active":"2023-11-08T05:28:13.563357+0000","last_became_peered":"2023-11-08T05:28:13.563357+0000","last_unstale":"2023-11-08T05:30:44.490629+0000","last_undegraded":"2023-11-08T05:30:44.490629+0000","last_fullsized":"2023-11-08T05:28:13.541614+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490691+0000","last_change":"2023-11-08T05:28:13.550672+0000","last_active":"2023-11-08T05:30:44.490691+0000","last_peered":"2023-11-08T05:30:44.490691+0000","last_clean":"2023-11-08T05:28:01.457904+0000","last_became_active":"2023-11-08T05:28:13.550672+0000","last_became_peered":"2023-11-08T05:28:13.550672+0000","last_unstale":"2023-11-08T05:30:44.490691+0000","last_undegraded":"2023-11-08T05:30:44.490691+0000","last_fullsized":"2023-11-08T05:28:13.539595+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490805+0000","last_change":"2023-11-08T05:28:13.557968+0000","last_active":"2023-11-08T05:30:44.490805+0000","last_peered":"2023-11-08T05:30:44.490805+0000","last_clean":"2023-11-08T05:28:01.458433+0000","last_became_active":"2023-11-08T05:28:13.557968+0000","last_became_peered":"2023-11-08T05:28:13.557968+0000","last_unstale":"2023-11-08T05:30:44.490805+0000","last_undegraded":"2023-11-08T05:30:44.490805+0000","last_fullsized":"2023-11-08T05:28:13.542832+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":1,"acting":1,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":12,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738840,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":645040,"kb_used_data":5352,"kb_used_omap":275,"kb_used_meta":639404,"kb_avail":93726800,"statfs":{"total":96636764160,"available":95976243200,"internally_reserved":0,"allocated":5480448,"data_stored":2697647,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":281650,"internal_metadata":654750670},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:59 2023","interfaces":[{"interface":"back","average":{"1min":0.30499999999999999,"5min":0.439,"15min":0.36099999999999999},"min":{"1min":0.17399999999999999,"5min":0.14499999999999999,"15min":0.14499999999999999},"max":{"1min":0.60099999999999998,"5min":4.9379999999999997,"15min":4.9379999999999997},"last":0.379},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.49399999999999999,"15min":0.42399999999999999},"min":{"1min":0.221,"5min":0.19600000000000001,"15min":0.17000000000000001},"max":{"1min":0.63200000000000001,"5min":4.9859999999999998,"15min":4.9859999999999998},"last":0.40899999999999997}]},{"osd":1,"last update":"Wed Nov 8 05:31:06 2023","interfaces":[{"interface":"back","average":{"1min":0.27500000000000002,"5min":0.40300000000000002,"15min":0.46999999999999997},"min":{"1min":0.20899999999999999,"5min":0.16,"15min":0.153},"max":{"1min":0.41099999999999998,"5min":4.0830000000000002,"15min":23.242999999999999},"last":0.34000000000000002},{"interface":"front","average":{"1min":0.29899999999999999,"5min":0.40100000000000002,"15min":0.46000000000000002},"min":{"1min":0.23799999999999999,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":0.45300000000000001,"5min":3.5590000000000002,"15min":23.251000000000001},"last":0.38}]}]},{"osd":0,"up_from":8,"seq":34359738798,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1339228,"kb_used_data":398304,"kb_used_omap":291,"kb_used_meta":936476,"kb_avail":93032612,"statfs":{"total":96636764160,"available":95265394688,"internally_reserved":0,"allocated":407863296,"data_stored":405103900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":298462,"internal_metadata":958951970},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.436,"15min":0.36299999999999999},"min":{"1min":0.19700000000000001,"5min":0.14699999999999999,"15min":0.13700000000000001},"max":{"1min":3.222,"5min":6.5499999999999998,"15min":6.9989999999999997},"last":0.245},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.46300000000000002,"15min":0.42299999999999999},"min":{"1min":0.17799999999999999,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":5.0030000000000001,"5min":6.0629999999999997,"15min":9.6869999999999994},"last":0.28599999999999998}]},{"osd":2,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.51600000000000001,"15min":0.47099999999999997},"min":{"1min":0.21099999999999999,"5min":0.153,"15min":0.153},"max":{"1min":5.7439999999999998,"5min":5.8280000000000003,"15min":9.7639999999999993},"last":0.252},{"interface":"front","average":{"1min":0.497,"5min":0.56200000000000006,"15min":0.42299999999999999},"min":{"1min":0.20399999999999999,"5min":0.187,"15min":0.16800000000000001},"max":{"1min":3.9319999999999999,"5min":8.8040000000000003,"15min":8.8040000000000003},"last":0.29899999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738842,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":650100,"kb_used_data":5356,"kb_used_omap":214,"kb_used_meta":644521,"kb_avail":93721740,"statfs":{"total":96636764160,"available":95971061760,"internally_reserved":0,"allocated":5484544,"data_stored":2698616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":219335,"internal_metadata":659990329},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:56 2023","interfaces":[{"interface":"back","average":{"1min":0.30399999999999999,"5min":0.34699999999999998,"15min":0.36499999999999999},"min":{"1min":0.23999999999999999,"5min":0.14199999999999999,"15min":0.14199999999999999},"max":{"1min":0.42399999999999999,"5min":1.595,"15min":7.4249999999999998},"last":0.17100000000000001},{"interface":"front","average":{"1min":0.35499999999999998,"5min":0.40899999999999997,"15min":0.376},"min":{"1min":0.23999999999999999,"5min":0.19600000000000001,"15min":0.183},"max":{"1min":0.46999999999999997,"5min":2.6859999999999999,"15min":2.6859999999999999},"last":0.23000000000000001}]},{"osd":2,"last update":"Wed Nov 8 05:30:57 2023","interfaces":[{"interface":"back","average":{"1min":0.28599999999999998,"5min":0.34699999999999998,"15min":0.36899999999999999},"min":{"1min":0.19700000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":0.41399999999999998,"5min":2.2650000000000001,"15min":5.8209999999999997},"last":0.39600000000000002},{"interface":"front","average":{"1min":0.30599999999999999,"5min":0.313,"15min":0.35799999999999998},"min":{"1min":0.17799999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.435,"5min":1.137,"15min":4.2350000000000003},"last":0.35799999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-08T05:31:54.381 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-11-08T05:31:54.381 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 459, 'reported_epoch': 458, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:30:44.490644+0000', 'last_change': '2023-11-08T05:28:13.557071+0000', 'last_active': '2023-11-08T05:30:44.490644+0000', 'last_peered': '2023-11-08T05:30:44.490644+0000', 'last_clean': '2023-11-08T05:28:01.458439+0000', 'last_became_active': '2023-11-08T05:28:13.557071+0000', 'last_became_peered': '2023-11-08T05:28:13.557071+0000', 'last_unstale': '2023-11-08T05:30:44.490644+0000', 'last_undegraded': '2023-11-08T05:30:44.490644+0000', 'last_fullsized': '2023-11-08T05:28:13.547025+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:31:54.382 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-11-08T05:31:54.382 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 11, 'reported_epoch': 459, 'state': 'active+clean+remapped', 'last_fresh': '2023-11-08T05:30:45.794760+0000', 'last_change': '2023-11-08T05:30:45.794760+0000', 'last_active': '2023-11-08T05:30:45.794760+0000', 'last_peered': '2023-11-08T05:30:45.794760+0000', 'last_clean': '2023-11-08T05:30:45.794760+0000', 'last_became_active': '2023-11-08T05:30:45.794398+0000', 'last_became_peered': '2023-11-08T05:30:45.794398+0000', 'last_unstale': '2023-11-08T05:30:45.794760+0000', 'last_undegraded': '2023-11-08T05:30:45.794760+0000', 'last_fullsized': '2023-11-08T05:30:45.794760+0000', 'mapping_epoch': 458, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 459, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2], 'acting': [2, 1], 'avail_no_missing': ['2', '1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2023-11-08T05:31:54.382 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.1 is not active+clean 2023-11-08T05:31:54.383 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.1', 'version': "0'0", 'reported_seq': 465, 'reported_epoch': 459, 'state': 'active+clean+remapped', 'last_fresh': '2023-11-08T05:30:45.795085+0000', 'last_change': '2023-11-08T05:30:45.795085+0000', 'last_active': '2023-11-08T05:30:45.795085+0000', 'last_peered': '2023-11-08T05:30:45.795085+0000', 'last_clean': '2023-11-08T05:30:45.795085+0000', 'last_became_active': '2023-11-08T05:30:45.794743+0000', 'last_became_peered': '2023-11-08T05:30:45.794743+0000', 'last_unstale': '2023-11-08T05:30:45.795085+0000', 'last_undegraded': '2023-11-08T05:30:45.795085+0000', 'last_fullsized': '2023-11-08T05:30:45.795085+0000', 'mapping_epoch': 458, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 459, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2], 'acting': [2, 1], 'avail_no_missing': ['2', '1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2023-11-08T05:31:54.383 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-11-08T05:31:54.383 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "0'0", 'reported_seq': 460, 'reported_epoch': 458, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:30:44.490523+0000', 'last_change': '2023-11-08T05:28:13.545468+0000', 'last_active': '2023-11-08T05:30:44.490523+0000', 'last_peered': '2023-11-08T05:30:44.490523+0000', 'last_clean': '2023-11-08T05:28:01.456723+0000', 'last_became_active': '2023-11-08T05:28:13.545468+0000', 'last_became_peered': '2023-11-08T05:28:13.545468+0000', 'last_unstale': '2023-11-08T05:30:44.490523+0000', 'last_undegraded': '2023-11-08T05:30:44.490523+0000', 'last_fullsized': '2023-11-08T05:28:13.537207+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 9, 'last_epoch_clean': 10, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:45.528796+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:45.528796+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:45.528796+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:31:54.384 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.0 is not active+clean 2023-11-08T05:31:54.384 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.0', 'version': "0'0", 'reported_seq': 465, 'reported_epoch': 458, 'state': 'active+clean+remapped', 'last_fresh': '2023-11-08T05:30:44.491907+0000', 'last_change': '2023-11-08T05:29:44.395350+0000', 'last_active': '2023-11-08T05:30:44.491907+0000', 'last_peered': '2023-11-08T05:30:44.491907+0000', 'last_clean': '2023-11-08T05:30:44.491907+0000', 'last_became_active': '2023-11-08T05:29:44.395012+0000', 'last_became_peered': '2023-11-08T05:29:44.395012+0000', 'last_unstale': '2023-11-08T05:30:44.491907+0000', 'last_undegraded': '2023-11-08T05:30:44.491907+0000', 'last_fullsized': '2023-11-08T05:30:44.491907+0000', 'mapping_epoch': 455, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 456, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2], 'acting': [2, 1], 'avail_no_missing': ['2', '1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2023-11-08T05:31:54.384 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-11-08T05:31:54.384 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 459, 'reported_epoch': 458, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:30:44.490629+0000', 'last_change': '2023-11-08T05:28:13.563357+0000', 'last_active': '2023-11-08T05:30:44.490629+0000', 'last_peered': '2023-11-08T05:30:44.490629+0000', 'last_clean': '2023-11-08T05:28:01.457691+0000', 'last_became_active': '2023-11-08T05:28:13.563357+0000', 'last_became_peered': '2023-11-08T05:28:13.563357+0000', 'last_unstale': '2023-11-08T05:30:44.490629+0000', 'last_undegraded': '2023-11-08T05:30:44.490629+0000', 'last_fullsized': '2023-11-08T05:28:13.541614+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:31:54.385 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-11-08T05:31:54.385 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 459, 'reported_epoch': 458, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:30:44.490691+0000', 'last_change': '2023-11-08T05:28:13.550672+0000', 'last_active': '2023-11-08T05:30:44.490691+0000', 'last_peered': '2023-11-08T05:30:44.490691+0000', 'last_clean': '2023-11-08T05:28:01.457904+0000', 'last_became_active': '2023-11-08T05:28:13.550672+0000', 'last_became_peered': '2023-11-08T05:28:13.550672+0000', 'last_unstale': '2023-11-08T05:30:44.490691+0000', 'last_undegraded': '2023-11-08T05:30:44.490691+0000', 'last_fullsized': '2023-11-08T05:28:13.539595+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:31:54.385 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-11-08T05:31:54.386 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 459, 'reported_epoch': 458, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:30:44.490805+0000', 'last_change': '2023-11-08T05:28:13.557968+0000', 'last_active': '2023-11-08T05:30:44.490805+0000', 'last_peered': '2023-11-08T05:30:44.490805+0000', 'last_clean': '2023-11-08T05:28:01.458433+0000', 'last_became_active': '2023-11-08T05:28:13.557968+0000', 'last_became_peered': '2023-11-08T05:28:13.557968+0000', 'last_unstale': '2023-11-08T05:30:44.490805+0000', 'last_undegraded': '2023-11-08T05:30:44.490805+0000', 'last_fullsized': '2023-11-08T05:28:13.542832+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:31:54.386 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-11-08T05:31:54.735 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T05:31:54.736 INFO:teuthology.orchestra.run.smithi087.stdout:{"fsid":"ab4cbc06-449d-497d-b513-220d9db2b590","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 5 pgs undersized","count":5},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2357,"monmap":{"epoch":1,"min_mon_release_name":"pacific","num_mons":1},"osdmap":{"epoch":459,"num_osds":3,"num_up_osds":2,"osd_up_since":1699421292,"num_in_osds":3,"osd_in_since":1699419158,"num_remapped_pgs":3},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean+remapped","count":3},{"state_name":"active+clean","count":1}],"num_pgs":9,"num_pools":2,"num_objects":0,"data_bytes":0,"bytes_used":2697592832,"bytes_avail":287212699648,"bytes_total":289910292480},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":41,"modified":"2023-11-08T05:29:32.044544+0000","services":{}},"progress_events":{}} 2023-11-08T05:31:54.749 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-11-08T05:31:54.749 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-11-08T05:31:55.031 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T05:31:55.033 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2023-11-08T05:31:55.044 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":1631,"stamp":"2023-11-08T05:31:54.073648+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":10,"acting":13,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":109,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2634368,"kb_used_data":409012,"kb_used_omap":780,"kb_used_meta":2220403,"kb_avail":280481152,"statfs":{"total":289910292480,"available":287212699648,"internally_reserved":0,"allocated":418828288,"data_stored":410500163,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":799447,"internal_metadata":2273692969},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-536782902,"num_objects":-164,"num_object_clones":0,"num_object_copies":-328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-115,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-164,"num_whiteouts":0,"num_read":-122056,"num_read_kb":-39011942,"num_write":-413642,"num_write_kb":-151705339,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-8,"num_bytes_recovered":-16922624,"num_keys_recovered":-2,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.424817"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490644+0000","last_change":"2023-11-08T05:28:13.557071+0000","last_active":"2023-11-08T05:30:44.490644+0000","last_peered":"2023-11-08T05:30:44.490644+0000","last_clean":"2023-11-08T05:28:01.458439+0000","last_became_active":"2023-11-08T05:28:13.557071+0000","last_became_peered":"2023-11-08T05:28:13.557071+0000","last_unstale":"2023-11-08T05:30:44.490644+0000","last_undegraded":"2023-11-08T05:30:44.490644+0000","last_fullsized":"2023-11-08T05:28:13.547025+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":459,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:45.794760+0000","last_change":"2023-11-08T05:30:45.794760+0000","last_active":"2023-11-08T05:30:45.794760+0000","last_peered":"2023-11-08T05:30:45.794760+0000","last_clean":"2023-11-08T05:30:45.794760+0000","last_became_active":"2023-11-08T05:30:45.794398+0000","last_became_peered":"2023-11-08T05:30:45.794398+0000","last_unstale":"2023-11-08T05:30:45.794760+0000","last_undegraded":"2023-11-08T05:30:45.794760+0000","last_fullsized":"2023-11-08T05:30:45.794760+0000","mapping_epoch":458,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":459,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":465,"reported_epoch":459,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:45.795085+0000","last_change":"2023-11-08T05:30:45.795085+0000","last_active":"2023-11-08T05:30:45.795085+0000","last_peered":"2023-11-08T05:30:45.795085+0000","last_clean":"2023-11-08T05:30:45.795085+0000","last_became_active":"2023-11-08T05:30:45.794743+0000","last_became_peered":"2023-11-08T05:30:45.794743+0000","last_unstale":"2023-11-08T05:30:45.795085+0000","last_undegraded":"2023-11-08T05:30:45.795085+0000","last_fullsized":"2023-11-08T05:30:45.795085+0000","mapping_epoch":458,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":459,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":454,"reported_epoch":458,"state":"active+clean","last_fresh":"2023-11-08T05:30:44.490618+0000","last_change":"2023-11-08T04:52:47.548597+0000","last_active":"2023-11-08T05:30:44.490618+0000","last_peered":"2023-11-08T05:30:44.490618+0000","last_clean":"2023-11-08T05:30:44.490618+0000","last_became_active":"2023-11-08T04:52:47.548294+0000","last_became_peered":"2023-11-08T04:52:47.548294+0000","last_unstale":"2023-11-08T05:30:44.490618+0000","last_undegraded":"2023-11-08T05:30:44.490618+0000","last_fullsized":"2023-11-08T05:30:44.490618+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":460,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490523+0000","last_change":"2023-11-08T05:28:13.545468+0000","last_active":"2023-11-08T05:30:44.490523+0000","last_peered":"2023-11-08T05:30:44.490523+0000","last_clean":"2023-11-08T05:28:01.456723+0000","last_became_active":"2023-11-08T05:28:13.545468+0000","last_became_peered":"2023-11-08T05:28:13.545468+0000","last_unstale":"2023-11-08T05:30:44.490523+0000","last_undegraded":"2023-11-08T05:30:44.490523+0000","last_fullsized":"2023-11-08T05:28:13.537207+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_clean_scrub_stamp":"2023-11-08T04:52:45.528796+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":465,"reported_epoch":458,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:44.491907+0000","last_change":"2023-11-08T05:29:44.395350+0000","last_active":"2023-11-08T05:30:44.491907+0000","last_peered":"2023-11-08T05:30:44.491907+0000","last_clean":"2023-11-08T05:30:44.491907+0000","last_became_active":"2023-11-08T05:29:44.395012+0000","last_became_peered":"2023-11-08T05:29:44.395012+0000","last_unstale":"2023-11-08T05:30:44.491907+0000","last_undegraded":"2023-11-08T05:30:44.491907+0000","last_fullsized":"2023-11-08T05:30:44.491907+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490629+0000","last_change":"2023-11-08T05:28:13.563357+0000","last_active":"2023-11-08T05:30:44.490629+0000","last_peered":"2023-11-08T05:30:44.490629+0000","last_clean":"2023-11-08T05:28:01.457691+0000","last_became_active":"2023-11-08T05:28:13.563357+0000","last_became_peered":"2023-11-08T05:28:13.563357+0000","last_unstale":"2023-11-08T05:30:44.490629+0000","last_undegraded":"2023-11-08T05:30:44.490629+0000","last_fullsized":"2023-11-08T05:28:13.541614+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490691+0000","last_change":"2023-11-08T05:28:13.550672+0000","last_active":"2023-11-08T05:30:44.490691+0000","last_peered":"2023-11-08T05:30:44.490691+0000","last_clean":"2023-11-08T05:28:01.457904+0000","last_became_active":"2023-11-08T05:28:13.550672+0000","last_became_peered":"2023-11-08T05:28:13.550672+0000","last_unstale":"2023-11-08T05:30:44.490691+0000","last_undegraded":"2023-11-08T05:30:44.490691+0000","last_fullsized":"2023-11-08T05:28:13.539595+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490805+0000","last_change":"2023-11-08T05:28:13.557968+0000","last_active":"2023-11-08T05:30:44.490805+0000","last_peered":"2023-11-08T05:30:44.490805+0000","last_clean":"2023-11-08T05:28:01.458433+0000","last_became_active":"2023-11-08T05:28:13.557968+0000","last_became_peered":"2023-11-08T05:28:13.557968+0000","last_unstale":"2023-11-08T05:30:44.490805+0000","last_undegraded":"2023-11-08T05:30:44.490805+0000","last_fullsized":"2023-11-08T05:28:13.542832+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":1,"acting":1,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":12,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738840,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":645040,"kb_used_data":5352,"kb_used_omap":275,"kb_used_meta":639404,"kb_avail":93726800,"statfs":{"total":96636764160,"available":95976243200,"internally_reserved":0,"allocated":5480448,"data_stored":2697647,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":281650,"internal_metadata":654750670},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:59 2023","interfaces":[{"interface":"back","average":{"1min":0.30499999999999999,"5min":0.439,"15min":0.36099999999999999},"min":{"1min":0.17399999999999999,"5min":0.14499999999999999,"15min":0.14499999999999999},"max":{"1min":0.60099999999999998,"5min":4.9379999999999997,"15min":4.9379999999999997},"last":0.379},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.49399999999999999,"15min":0.42399999999999999},"min":{"1min":0.221,"5min":0.19600000000000001,"15min":0.17000000000000001},"max":{"1min":0.63200000000000001,"5min":4.9859999999999998,"15min":4.9859999999999998},"last":0.40899999999999997}]},{"osd":1,"last update":"Wed Nov 8 05:31:06 2023","interfaces":[{"interface":"back","average":{"1min":0.27500000000000002,"5min":0.40300000000000002,"15min":0.46999999999999997},"min":{"1min":0.20899999999999999,"5min":0.16,"15min":0.153},"max":{"1min":0.41099999999999998,"5min":4.0830000000000002,"15min":23.242999999999999},"last":0.34000000000000002},{"interface":"front","average":{"1min":0.29899999999999999,"5min":0.40100000000000002,"15min":0.46000000000000002},"min":{"1min":0.23799999999999999,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":0.45300000000000001,"5min":3.5590000000000002,"15min":23.251000000000001},"last":0.38}]}]},{"osd":0,"up_from":8,"seq":34359738798,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1339228,"kb_used_data":398304,"kb_used_omap":291,"kb_used_meta":936476,"kb_avail":93032612,"statfs":{"total":96636764160,"available":95265394688,"internally_reserved":0,"allocated":407863296,"data_stored":405103900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":298462,"internal_metadata":958951970},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.436,"15min":0.36299999999999999},"min":{"1min":0.19700000000000001,"5min":0.14699999999999999,"15min":0.13700000000000001},"max":{"1min":3.222,"5min":6.5499999999999998,"15min":6.9989999999999997},"last":0.245},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.46300000000000002,"15min":0.42299999999999999},"min":{"1min":0.17799999999999999,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":5.0030000000000001,"5min":6.0629999999999997,"15min":9.6869999999999994},"last":0.28599999999999998}]},{"osd":2,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.51600000000000001,"15min":0.47099999999999997},"min":{"1min":0.21099999999999999,"5min":0.153,"15min":0.153},"max":{"1min":5.7439999999999998,"5min":5.8280000000000003,"15min":9.7639999999999993},"last":0.252},{"interface":"front","average":{"1min":0.497,"5min":0.56200000000000006,"15min":0.42299999999999999},"min":{"1min":0.20399999999999999,"5min":0.187,"15min":0.16800000000000001},"max":{"1min":3.9319999999999999,"5min":8.8040000000000003,"15min":8.8040000000000003},"last":0.29899999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738842,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":650100,"kb_used_data":5356,"kb_used_omap":214,"kb_used_meta":644521,"kb_avail":93721740,"statfs":{"total":96636764160,"available":95971061760,"internally_reserved":0,"allocated":5484544,"data_stored":2698616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":219335,"internal_metadata":659990329},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:56 2023","interfaces":[{"interface":"back","average":{"1min":0.30399999999999999,"5min":0.34699999999999998,"15min":0.36499999999999999},"min":{"1min":0.23999999999999999,"5min":0.14199999999999999,"15min":0.14199999999999999},"max":{"1min":0.42399999999999999,"5min":1.595,"15min":7.4249999999999998},"last":0.17100000000000001},{"interface":"front","average":{"1min":0.35499999999999998,"5min":0.40899999999999997,"15min":0.376},"min":{"1min":0.23999999999999999,"5min":0.19600000000000001,"15min":0.183},"max":{"1min":0.46999999999999997,"5min":2.6859999999999999,"15min":2.6859999999999999},"last":0.23000000000000001}]},{"osd":2,"last update":"Wed Nov 8 05:30:57 2023","interfaces":[{"interface":"back","average":{"1min":0.28599999999999998,"5min":0.34699999999999998,"15min":0.36899999999999999},"min":{"1min":0.19700000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":0.41399999999999998,"5min":2.2650000000000001,"15min":5.8209999999999997},"last":0.39600000000000002},{"interface":"front","average":{"1min":0.30599999999999999,"5min":0.313,"15min":0.35799999999999998},"min":{"1min":0.17799999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.435,"5min":1.137,"15min":4.2350000000000003},"last":0.35799999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-08T05:31:58.046 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-11-08T05:31:58.341 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T05:31:58.343 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2023-11-08T05:31:58.354 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":1633,"stamp":"2023-11-08T05:31:58.074338+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":10,"acting":13,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":109,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2634368,"kb_used_data":409012,"kb_used_omap":780,"kb_used_meta":2220403,"kb_avail":280481152,"statfs":{"total":289910292480,"available":287212699648,"internally_reserved":0,"allocated":418828288,"data_stored":410500163,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":799447,"internal_metadata":2273692969},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-536782902,"num_objects":-164,"num_object_clones":0,"num_object_copies":-328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-115,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-164,"num_whiteouts":0,"num_read":-122056,"num_read_kb":-39011942,"num_write":-413642,"num_write_kb":-151705339,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-8,"num_bytes_recovered":-16922624,"num_keys_recovered":-2,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.424817"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490644+0000","last_change":"2023-11-08T05:28:13.557071+0000","last_active":"2023-11-08T05:30:44.490644+0000","last_peered":"2023-11-08T05:30:44.490644+0000","last_clean":"2023-11-08T05:28:01.458439+0000","last_became_active":"2023-11-08T05:28:13.557071+0000","last_became_peered":"2023-11-08T05:28:13.557071+0000","last_unstale":"2023-11-08T05:30:44.490644+0000","last_undegraded":"2023-11-08T05:30:44.490644+0000","last_fullsized":"2023-11-08T05:28:13.547025+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":459,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:45.794760+0000","last_change":"2023-11-08T05:30:45.794760+0000","last_active":"2023-11-08T05:30:45.794760+0000","last_peered":"2023-11-08T05:30:45.794760+0000","last_clean":"2023-11-08T05:30:45.794760+0000","last_became_active":"2023-11-08T05:30:45.794398+0000","last_became_peered":"2023-11-08T05:30:45.794398+0000","last_unstale":"2023-11-08T05:30:45.794760+0000","last_undegraded":"2023-11-08T05:30:45.794760+0000","last_fullsized":"2023-11-08T05:30:45.794760+0000","mapping_epoch":458,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":459,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":465,"reported_epoch":459,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:45.795085+0000","last_change":"2023-11-08T05:30:45.795085+0000","last_active":"2023-11-08T05:30:45.795085+0000","last_peered":"2023-11-08T05:30:45.795085+0000","last_clean":"2023-11-08T05:30:45.795085+0000","last_became_active":"2023-11-08T05:30:45.794743+0000","last_became_peered":"2023-11-08T05:30:45.794743+0000","last_unstale":"2023-11-08T05:30:45.795085+0000","last_undegraded":"2023-11-08T05:30:45.795085+0000","last_fullsized":"2023-11-08T05:30:45.795085+0000","mapping_epoch":458,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":459,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":454,"reported_epoch":458,"state":"active+clean","last_fresh":"2023-11-08T05:30:44.490618+0000","last_change":"2023-11-08T04:52:47.548597+0000","last_active":"2023-11-08T05:30:44.490618+0000","last_peered":"2023-11-08T05:30:44.490618+0000","last_clean":"2023-11-08T05:30:44.490618+0000","last_became_active":"2023-11-08T04:52:47.548294+0000","last_became_peered":"2023-11-08T04:52:47.548294+0000","last_unstale":"2023-11-08T05:30:44.490618+0000","last_undegraded":"2023-11-08T05:30:44.490618+0000","last_fullsized":"2023-11-08T05:30:44.490618+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":460,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490523+0000","last_change":"2023-11-08T05:28:13.545468+0000","last_active":"2023-11-08T05:30:44.490523+0000","last_peered":"2023-11-08T05:30:44.490523+0000","last_clean":"2023-11-08T05:28:01.456723+0000","last_became_active":"2023-11-08T05:28:13.545468+0000","last_became_peered":"2023-11-08T05:28:13.545468+0000","last_unstale":"2023-11-08T05:30:44.490523+0000","last_undegraded":"2023-11-08T05:30:44.490523+0000","last_fullsized":"2023-11-08T05:28:13.537207+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_clean_scrub_stamp":"2023-11-08T04:52:45.528796+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":465,"reported_epoch":458,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:44.491907+0000","last_change":"2023-11-08T05:29:44.395350+0000","last_active":"2023-11-08T05:30:44.491907+0000","last_peered":"2023-11-08T05:30:44.491907+0000","last_clean":"2023-11-08T05:30:44.491907+0000","last_became_active":"2023-11-08T05:29:44.395012+0000","last_became_peered":"2023-11-08T05:29:44.395012+0000","last_unstale":"2023-11-08T05:30:44.491907+0000","last_undegraded":"2023-11-08T05:30:44.491907+0000","last_fullsized":"2023-11-08T05:30:44.491907+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490629+0000","last_change":"2023-11-08T05:28:13.563357+0000","last_active":"2023-11-08T05:30:44.490629+0000","last_peered":"2023-11-08T05:30:44.490629+0000","last_clean":"2023-11-08T05:28:01.457691+0000","last_became_active":"2023-11-08T05:28:13.563357+0000","last_became_peered":"2023-11-08T05:28:13.563357+0000","last_unstale":"2023-11-08T05:30:44.490629+0000","last_undegraded":"2023-11-08T05:30:44.490629+0000","last_fullsized":"2023-11-08T05:28:13.541614+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490691+0000","last_change":"2023-11-08T05:28:13.550672+0000","last_active":"2023-11-08T05:30:44.490691+0000","last_peered":"2023-11-08T05:30:44.490691+0000","last_clean":"2023-11-08T05:28:01.457904+0000","last_became_active":"2023-11-08T05:28:13.550672+0000","last_became_peered":"2023-11-08T05:28:13.550672+0000","last_unstale":"2023-11-08T05:30:44.490691+0000","last_undegraded":"2023-11-08T05:30:44.490691+0000","last_fullsized":"2023-11-08T05:28:13.539595+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490805+0000","last_change":"2023-11-08T05:28:13.557968+0000","last_active":"2023-11-08T05:30:44.490805+0000","last_peered":"2023-11-08T05:30:44.490805+0000","last_clean":"2023-11-08T05:28:01.458433+0000","last_became_active":"2023-11-08T05:28:13.557968+0000","last_became_peered":"2023-11-08T05:28:13.557968+0000","last_unstale":"2023-11-08T05:30:44.490805+0000","last_undegraded":"2023-11-08T05:30:44.490805+0000","last_fullsized":"2023-11-08T05:28:13.542832+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":1,"acting":1,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":12,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738841,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":645040,"kb_used_data":5352,"kb_used_omap":275,"kb_used_meta":639404,"kb_avail":93726800,"statfs":{"total":96636764160,"available":95976243200,"internally_reserved":0,"allocated":5480448,"data_stored":2697647,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":281650,"internal_metadata":654750670},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:59 2023","interfaces":[{"interface":"back","average":{"1min":0.30499999999999999,"5min":0.439,"15min":0.36099999999999999},"min":{"1min":0.17399999999999999,"5min":0.14499999999999999,"15min":0.14499999999999999},"max":{"1min":0.60099999999999998,"5min":4.9379999999999997,"15min":4.9379999999999997},"last":0.379},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.49399999999999999,"15min":0.42399999999999999},"min":{"1min":0.221,"5min":0.19600000000000001,"15min":0.17000000000000001},"max":{"1min":0.63200000000000001,"5min":4.9859999999999998,"15min":4.9859999999999998},"last":0.40899999999999997}]},{"osd":1,"last update":"Wed Nov 8 05:31:06 2023","interfaces":[{"interface":"back","average":{"1min":0.27500000000000002,"5min":0.40300000000000002,"15min":0.46999999999999997},"min":{"1min":0.20899999999999999,"5min":0.16,"15min":0.153},"max":{"1min":0.41099999999999998,"5min":4.0830000000000002,"15min":23.242999999999999},"last":0.25800000000000001},{"interface":"front","average":{"1min":0.29899999999999999,"5min":0.40100000000000002,"15min":0.46000000000000002},"min":{"1min":0.23799999999999999,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":0.45300000000000001,"5min":3.5590000000000002,"15min":23.251000000000001},"last":0.23499999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738798,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1339228,"kb_used_data":398304,"kb_used_omap":291,"kb_used_meta":936476,"kb_avail":93032612,"statfs":{"total":96636764160,"available":95265394688,"internally_reserved":0,"allocated":407863296,"data_stored":405103900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":298462,"internal_metadata":958951970},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.436,"15min":0.36299999999999999},"min":{"1min":0.19700000000000001,"5min":0.14699999999999999,"15min":0.13700000000000001},"max":{"1min":3.222,"5min":6.5499999999999998,"15min":6.9989999999999997},"last":0.245},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.46300000000000002,"15min":0.42299999999999999},"min":{"1min":0.17799999999999999,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":5.0030000000000001,"5min":6.0629999999999997,"15min":9.6869999999999994},"last":0.28599999999999998}]},{"osd":2,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.51600000000000001,"15min":0.47099999999999997},"min":{"1min":0.21099999999999999,"5min":0.153,"15min":0.153},"max":{"1min":5.7439999999999998,"5min":5.8280000000000003,"15min":9.7639999999999993},"last":0.252},{"interface":"front","average":{"1min":0.497,"5min":0.56200000000000006,"15min":0.42299999999999999},"min":{"1min":0.20399999999999999,"5min":0.187,"15min":0.16800000000000001},"max":{"1min":3.9319999999999999,"5min":8.8040000000000003,"15min":8.8040000000000003},"last":0.29899999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738843,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":650100,"kb_used_data":5356,"kb_used_omap":214,"kb_used_meta":644521,"kb_avail":93721740,"statfs":{"total":96636764160,"available":95971061760,"internally_reserved":0,"allocated":5484544,"data_stored":2698616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":219335,"internal_metadata":659990329},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:56 2023","interfaces":[{"interface":"back","average":{"1min":0.30399999999999999,"5min":0.34699999999999998,"15min":0.36499999999999999},"min":{"1min":0.23999999999999999,"5min":0.14199999999999999,"15min":0.14199999999999999},"max":{"1min":0.42399999999999999,"5min":1.595,"15min":7.4249999999999998},"last":0.17100000000000001},{"interface":"front","average":{"1min":0.35499999999999998,"5min":0.40899999999999997,"15min":0.376},"min":{"1min":0.23999999999999999,"5min":0.19600000000000001,"15min":0.183},"max":{"1min":0.46999999999999997,"5min":2.6859999999999999,"15min":2.6859999999999999},"last":0.23000000000000001}]},{"osd":2,"last update":"Wed Nov 8 05:30:57 2023","interfaces":[{"interface":"back","average":{"1min":0.28599999999999998,"5min":0.34699999999999998,"15min":0.36899999999999999},"min":{"1min":0.19700000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":0.41399999999999998,"5min":2.2650000000000001,"15min":5.8209999999999997},"last":0.25600000000000001},{"interface":"front","average":{"1min":0.30599999999999999,"5min":0.313,"15min":0.35799999999999998},"min":{"1min":0.17799999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.435,"5min":1.137,"15min":4.2350000000000003},"last":0.23799999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-08T05:31:58.355 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-11-08T05:31:58.640 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T05:31:58.641 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2023-11-08T05:31:58.652 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":1633,"stamp":"2023-11-08T05:31:58.074338+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":10,"acting":13,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":109,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2634368,"kb_used_data":409012,"kb_used_omap":780,"kb_used_meta":2220403,"kb_avail":280481152,"statfs":{"total":289910292480,"available":287212699648,"internally_reserved":0,"allocated":418828288,"data_stored":410500163,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":799447,"internal_metadata":2273692969},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-536782902,"num_objects":-164,"num_object_clones":0,"num_object_copies":-328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-115,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-164,"num_whiteouts":0,"num_read":-122056,"num_read_kb":-39011942,"num_write":-413642,"num_write_kb":-151705339,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-8,"num_bytes_recovered":-16922624,"num_keys_recovered":-2,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.424817"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490644+0000","last_change":"2023-11-08T05:28:13.557071+0000","last_active":"2023-11-08T05:30:44.490644+0000","last_peered":"2023-11-08T05:30:44.490644+0000","last_clean":"2023-11-08T05:28:01.458439+0000","last_became_active":"2023-11-08T05:28:13.557071+0000","last_became_peered":"2023-11-08T05:28:13.557071+0000","last_unstale":"2023-11-08T05:30:44.490644+0000","last_undegraded":"2023-11-08T05:30:44.490644+0000","last_fullsized":"2023-11-08T05:28:13.547025+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":459,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:45.794760+0000","last_change":"2023-11-08T05:30:45.794760+0000","last_active":"2023-11-08T05:30:45.794760+0000","last_peered":"2023-11-08T05:30:45.794760+0000","last_clean":"2023-11-08T05:30:45.794760+0000","last_became_active":"2023-11-08T05:30:45.794398+0000","last_became_peered":"2023-11-08T05:30:45.794398+0000","last_unstale":"2023-11-08T05:30:45.794760+0000","last_undegraded":"2023-11-08T05:30:45.794760+0000","last_fullsized":"2023-11-08T05:30:45.794760+0000","mapping_epoch":458,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":459,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":465,"reported_epoch":459,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:45.795085+0000","last_change":"2023-11-08T05:30:45.795085+0000","last_active":"2023-11-08T05:30:45.795085+0000","last_peered":"2023-11-08T05:30:45.795085+0000","last_clean":"2023-11-08T05:30:45.795085+0000","last_became_active":"2023-11-08T05:30:45.794743+0000","last_became_peered":"2023-11-08T05:30:45.794743+0000","last_unstale":"2023-11-08T05:30:45.795085+0000","last_undegraded":"2023-11-08T05:30:45.795085+0000","last_fullsized":"2023-11-08T05:30:45.795085+0000","mapping_epoch":458,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":459,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":454,"reported_epoch":458,"state":"active+clean","last_fresh":"2023-11-08T05:30:44.490618+0000","last_change":"2023-11-08T04:52:47.548597+0000","last_active":"2023-11-08T05:30:44.490618+0000","last_peered":"2023-11-08T05:30:44.490618+0000","last_clean":"2023-11-08T05:30:44.490618+0000","last_became_active":"2023-11-08T04:52:47.548294+0000","last_became_peered":"2023-11-08T04:52:47.548294+0000","last_unstale":"2023-11-08T05:30:44.490618+0000","last_undegraded":"2023-11-08T05:30:44.490618+0000","last_fullsized":"2023-11-08T05:30:44.490618+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":460,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490523+0000","last_change":"2023-11-08T05:28:13.545468+0000","last_active":"2023-11-08T05:30:44.490523+0000","last_peered":"2023-11-08T05:30:44.490523+0000","last_clean":"2023-11-08T05:28:01.456723+0000","last_became_active":"2023-11-08T05:28:13.545468+0000","last_became_peered":"2023-11-08T05:28:13.545468+0000","last_unstale":"2023-11-08T05:30:44.490523+0000","last_undegraded":"2023-11-08T05:30:44.490523+0000","last_fullsized":"2023-11-08T05:28:13.537207+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_clean_scrub_stamp":"2023-11-08T04:52:45.528796+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":465,"reported_epoch":458,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:44.491907+0000","last_change":"2023-11-08T05:29:44.395350+0000","last_active":"2023-11-08T05:30:44.491907+0000","last_peered":"2023-11-08T05:30:44.491907+0000","last_clean":"2023-11-08T05:30:44.491907+0000","last_became_active":"2023-11-08T05:29:44.395012+0000","last_became_peered":"2023-11-08T05:29:44.395012+0000","last_unstale":"2023-11-08T05:30:44.491907+0000","last_undegraded":"2023-11-08T05:30:44.491907+0000","last_fullsized":"2023-11-08T05:30:44.491907+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490629+0000","last_change":"2023-11-08T05:28:13.563357+0000","last_active":"2023-11-08T05:30:44.490629+0000","last_peered":"2023-11-08T05:30:44.490629+0000","last_clean":"2023-11-08T05:28:01.457691+0000","last_became_active":"2023-11-08T05:28:13.563357+0000","last_became_peered":"2023-11-08T05:28:13.563357+0000","last_unstale":"2023-11-08T05:30:44.490629+0000","last_undegraded":"2023-11-08T05:30:44.490629+0000","last_fullsized":"2023-11-08T05:28:13.541614+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490691+0000","last_change":"2023-11-08T05:28:13.550672+0000","last_active":"2023-11-08T05:30:44.490691+0000","last_peered":"2023-11-08T05:30:44.490691+0000","last_clean":"2023-11-08T05:28:01.457904+0000","last_became_active":"2023-11-08T05:28:13.550672+0000","last_became_peered":"2023-11-08T05:28:13.550672+0000","last_unstale":"2023-11-08T05:30:44.490691+0000","last_undegraded":"2023-11-08T05:30:44.490691+0000","last_fullsized":"2023-11-08T05:28:13.539595+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490805+0000","last_change":"2023-11-08T05:28:13.557968+0000","last_active":"2023-11-08T05:30:44.490805+0000","last_peered":"2023-11-08T05:30:44.490805+0000","last_clean":"2023-11-08T05:28:01.458433+0000","last_became_active":"2023-11-08T05:28:13.557968+0000","last_became_peered":"2023-11-08T05:28:13.557968+0000","last_unstale":"2023-11-08T05:30:44.490805+0000","last_undegraded":"2023-11-08T05:30:44.490805+0000","last_fullsized":"2023-11-08T05:28:13.542832+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":1,"acting":1,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":12,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738841,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":645040,"kb_used_data":5352,"kb_used_omap":275,"kb_used_meta":639404,"kb_avail":93726800,"statfs":{"total":96636764160,"available":95976243200,"internally_reserved":0,"allocated":5480448,"data_stored":2697647,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":281650,"internal_metadata":654750670},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:59 2023","interfaces":[{"interface":"back","average":{"1min":0.30499999999999999,"5min":0.439,"15min":0.36099999999999999},"min":{"1min":0.17399999999999999,"5min":0.14499999999999999,"15min":0.14499999999999999},"max":{"1min":0.60099999999999998,"5min":4.9379999999999997,"15min":4.9379999999999997},"last":0.379},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.49399999999999999,"15min":0.42399999999999999},"min":{"1min":0.221,"5min":0.19600000000000001,"15min":0.17000000000000001},"max":{"1min":0.63200000000000001,"5min":4.9859999999999998,"15min":4.9859999999999998},"last":0.40899999999999997}]},{"osd":1,"last update":"Wed Nov 8 05:31:06 2023","interfaces":[{"interface":"back","average":{"1min":0.27500000000000002,"5min":0.40300000000000002,"15min":0.46999999999999997},"min":{"1min":0.20899999999999999,"5min":0.16,"15min":0.153},"max":{"1min":0.41099999999999998,"5min":4.0830000000000002,"15min":23.242999999999999},"last":0.25800000000000001},{"interface":"front","average":{"1min":0.29899999999999999,"5min":0.40100000000000002,"15min":0.46000000000000002},"min":{"1min":0.23799999999999999,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":0.45300000000000001,"5min":3.5590000000000002,"15min":23.251000000000001},"last":0.23499999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738798,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1339228,"kb_used_data":398304,"kb_used_omap":291,"kb_used_meta":936476,"kb_avail":93032612,"statfs":{"total":96636764160,"available":95265394688,"internally_reserved":0,"allocated":407863296,"data_stored":405103900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":298462,"internal_metadata":958951970},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.436,"15min":0.36299999999999999},"min":{"1min":0.19700000000000001,"5min":0.14699999999999999,"15min":0.13700000000000001},"max":{"1min":3.222,"5min":6.5499999999999998,"15min":6.9989999999999997},"last":0.245},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.46300000000000002,"15min":0.42299999999999999},"min":{"1min":0.17799999999999999,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":5.0030000000000001,"5min":6.0629999999999997,"15min":9.6869999999999994},"last":0.28599999999999998}]},{"osd":2,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.51600000000000001,"15min":0.47099999999999997},"min":{"1min":0.21099999999999999,"5min":0.153,"15min":0.153},"max":{"1min":5.7439999999999998,"5min":5.8280000000000003,"15min":9.7639999999999993},"last":0.252},{"interface":"front","average":{"1min":0.497,"5min":0.56200000000000006,"15min":0.42299999999999999},"min":{"1min":0.20399999999999999,"5min":0.187,"15min":0.16800000000000001},"max":{"1min":3.9319999999999999,"5min":8.8040000000000003,"15min":8.8040000000000003},"last":0.29899999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738843,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":650100,"kb_used_data":5356,"kb_used_omap":214,"kb_used_meta":644521,"kb_avail":93721740,"statfs":{"total":96636764160,"available":95971061760,"internally_reserved":0,"allocated":5484544,"data_stored":2698616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":219335,"internal_metadata":659990329},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:56 2023","interfaces":[{"interface":"back","average":{"1min":0.30399999999999999,"5min":0.34699999999999998,"15min":0.36499999999999999},"min":{"1min":0.23999999999999999,"5min":0.14199999999999999,"15min":0.14199999999999999},"max":{"1min":0.42399999999999999,"5min":1.595,"15min":7.4249999999999998},"last":0.17100000000000001},{"interface":"front","average":{"1min":0.35499999999999998,"5min":0.40899999999999997,"15min":0.376},"min":{"1min":0.23999999999999999,"5min":0.19600000000000001,"15min":0.183},"max":{"1min":0.46999999999999997,"5min":2.6859999999999999,"15min":2.6859999999999999},"last":0.23000000000000001}]},{"osd":2,"last update":"Wed Nov 8 05:30:57 2023","interfaces":[{"interface":"back","average":{"1min":0.28599999999999998,"5min":0.34699999999999998,"15min":0.36899999999999999},"min":{"1min":0.19700000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":0.41399999999999998,"5min":2.2650000000000001,"15min":5.8209999999999997},"last":0.25600000000000001},{"interface":"front","average":{"1min":0.30599999999999999,"5min":0.313,"15min":0.35799999999999998},"min":{"1min":0.17799999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.435,"5min":1.137,"15min":4.2350000000000003},"last":0.23799999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-08T05:31:58.654 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-11-08T05:31:58.654 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 459, 'reported_epoch': 458, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:30:44.490644+0000', 'last_change': '2023-11-08T05:28:13.557071+0000', 'last_active': '2023-11-08T05:30:44.490644+0000', 'last_peered': '2023-11-08T05:30:44.490644+0000', 'last_clean': '2023-11-08T05:28:01.458439+0000', 'last_became_active': '2023-11-08T05:28:13.557071+0000', 'last_became_peered': '2023-11-08T05:28:13.557071+0000', 'last_unstale': '2023-11-08T05:30:44.490644+0000', 'last_undegraded': '2023-11-08T05:30:44.490644+0000', 'last_fullsized': '2023-11-08T05:28:13.547025+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:31:58.654 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-11-08T05:31:58.655 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 11, 'reported_epoch': 459, 'state': 'active+clean+remapped', 'last_fresh': '2023-11-08T05:30:45.794760+0000', 'last_change': '2023-11-08T05:30:45.794760+0000', 'last_active': '2023-11-08T05:30:45.794760+0000', 'last_peered': '2023-11-08T05:30:45.794760+0000', 'last_clean': '2023-11-08T05:30:45.794760+0000', 'last_became_active': '2023-11-08T05:30:45.794398+0000', 'last_became_peered': '2023-11-08T05:30:45.794398+0000', 'last_unstale': '2023-11-08T05:30:45.794760+0000', 'last_undegraded': '2023-11-08T05:30:45.794760+0000', 'last_fullsized': '2023-11-08T05:30:45.794760+0000', 'mapping_epoch': 458, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 459, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2], 'acting': [2, 1], 'avail_no_missing': ['2', '1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2023-11-08T05:31:58.655 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.1 is not active+clean 2023-11-08T05:31:58.655 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.1', 'version': "0'0", 'reported_seq': 465, 'reported_epoch': 459, 'state': 'active+clean+remapped', 'last_fresh': '2023-11-08T05:30:45.795085+0000', 'last_change': '2023-11-08T05:30:45.795085+0000', 'last_active': '2023-11-08T05:30:45.795085+0000', 'last_peered': '2023-11-08T05:30:45.795085+0000', 'last_clean': '2023-11-08T05:30:45.795085+0000', 'last_became_active': '2023-11-08T05:30:45.794743+0000', 'last_became_peered': '2023-11-08T05:30:45.794743+0000', 'last_unstale': '2023-11-08T05:30:45.795085+0000', 'last_undegraded': '2023-11-08T05:30:45.795085+0000', 'last_fullsized': '2023-11-08T05:30:45.795085+0000', 'mapping_epoch': 458, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 459, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2], 'acting': [2, 1], 'avail_no_missing': ['2', '1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2023-11-08T05:31:58.656 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-11-08T05:31:58.656 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "0'0", 'reported_seq': 460, 'reported_epoch': 458, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:30:44.490523+0000', 'last_change': '2023-11-08T05:28:13.545468+0000', 'last_active': '2023-11-08T05:30:44.490523+0000', 'last_peered': '2023-11-08T05:30:44.490523+0000', 'last_clean': '2023-11-08T05:28:01.456723+0000', 'last_became_active': '2023-11-08T05:28:13.545468+0000', 'last_became_peered': '2023-11-08T05:28:13.545468+0000', 'last_unstale': '2023-11-08T05:30:44.490523+0000', 'last_undegraded': '2023-11-08T05:30:44.490523+0000', 'last_fullsized': '2023-11-08T05:28:13.537207+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 9, 'last_epoch_clean': 10, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:45.528796+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:45.528796+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:45.528796+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:31:58.656 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.0 is not active+clean 2023-11-08T05:31:58.656 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.0', 'version': "0'0", 'reported_seq': 465, 'reported_epoch': 458, 'state': 'active+clean+remapped', 'last_fresh': '2023-11-08T05:30:44.491907+0000', 'last_change': '2023-11-08T05:29:44.395350+0000', 'last_active': '2023-11-08T05:30:44.491907+0000', 'last_peered': '2023-11-08T05:30:44.491907+0000', 'last_clean': '2023-11-08T05:30:44.491907+0000', 'last_became_active': '2023-11-08T05:29:44.395012+0000', 'last_became_peered': '2023-11-08T05:29:44.395012+0000', 'last_unstale': '2023-11-08T05:30:44.491907+0000', 'last_undegraded': '2023-11-08T05:30:44.491907+0000', 'last_fullsized': '2023-11-08T05:30:44.491907+0000', 'mapping_epoch': 455, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 456, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2], 'acting': [2, 1], 'avail_no_missing': ['2', '1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2023-11-08T05:31:58.657 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-11-08T05:31:58.657 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 459, 'reported_epoch': 458, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:30:44.490629+0000', 'last_change': '2023-11-08T05:28:13.563357+0000', 'last_active': '2023-11-08T05:30:44.490629+0000', 'last_peered': '2023-11-08T05:30:44.490629+0000', 'last_clean': '2023-11-08T05:28:01.457691+0000', 'last_became_active': '2023-11-08T05:28:13.563357+0000', 'last_became_peered': '2023-11-08T05:28:13.563357+0000', 'last_unstale': '2023-11-08T05:30:44.490629+0000', 'last_undegraded': '2023-11-08T05:30:44.490629+0000', 'last_fullsized': '2023-11-08T05:28:13.541614+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:31:58.657 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-11-08T05:31:58.658 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 459, 'reported_epoch': 458, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:30:44.490691+0000', 'last_change': '2023-11-08T05:28:13.550672+0000', 'last_active': '2023-11-08T05:30:44.490691+0000', 'last_peered': '2023-11-08T05:30:44.490691+0000', 'last_clean': '2023-11-08T05:28:01.457904+0000', 'last_became_active': '2023-11-08T05:28:13.550672+0000', 'last_became_peered': '2023-11-08T05:28:13.550672+0000', 'last_unstale': '2023-11-08T05:30:44.490691+0000', 'last_undegraded': '2023-11-08T05:30:44.490691+0000', 'last_fullsized': '2023-11-08T05:28:13.539595+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:31:58.658 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-11-08T05:31:58.658 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 459, 'reported_epoch': 458, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:30:44.490805+0000', 'last_change': '2023-11-08T05:28:13.557968+0000', 'last_active': '2023-11-08T05:30:44.490805+0000', 'last_peered': '2023-11-08T05:30:44.490805+0000', 'last_clean': '2023-11-08T05:28:01.458433+0000', 'last_became_active': '2023-11-08T05:28:13.557968+0000', 'last_became_peered': '2023-11-08T05:28:13.557968+0000', 'last_unstale': '2023-11-08T05:30:44.490805+0000', 'last_undegraded': '2023-11-08T05:30:44.490805+0000', 'last_fullsized': '2023-11-08T05:28:13.542832+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:31:58.659 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-11-08T05:31:59.006 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T05:31:59.006 INFO:teuthology.orchestra.run.smithi087.stdout:{"fsid":"ab4cbc06-449d-497d-b513-220d9db2b590","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 5 pgs undersized","count":5},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2361,"monmap":{"epoch":1,"min_mon_release_name":"pacific","num_mons":1},"osdmap":{"epoch":459,"num_osds":3,"num_up_osds":2,"osd_up_since":1699421292,"num_in_osds":3,"osd_in_since":1699419158,"num_remapped_pgs":3},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean+remapped","count":3},{"state_name":"active+clean","count":1}],"num_pgs":9,"num_pools":2,"num_objects":0,"data_bytes":0,"bytes_used":2697592832,"bytes_avail":287212699648,"bytes_total":289910292480},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":41,"modified":"2023-11-08T05:29:32.044544+0000","services":{}},"progress_events":{}} 2023-11-08T05:31:59.019 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-11-08T05:31:59.020 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-11-08T05:31:59.302 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T05:31:59.304 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2023-11-08T05:31:59.315 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":1633,"stamp":"2023-11-08T05:31:58.074338+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":10,"acting":13,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":109,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2634368,"kb_used_data":409012,"kb_used_omap":780,"kb_used_meta":2220403,"kb_avail":280481152,"statfs":{"total":289910292480,"available":287212699648,"internally_reserved":0,"allocated":418828288,"data_stored":410500163,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":799447,"internal_metadata":2273692969},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-536782902,"num_objects":-164,"num_object_clones":0,"num_object_copies":-328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-115,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-164,"num_whiteouts":0,"num_read":-122056,"num_read_kb":-39011942,"num_write":-413642,"num_write_kb":-151705339,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-8,"num_bytes_recovered":-16922624,"num_keys_recovered":-2,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.424817"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490644+0000","last_change":"2023-11-08T05:28:13.557071+0000","last_active":"2023-11-08T05:30:44.490644+0000","last_peered":"2023-11-08T05:30:44.490644+0000","last_clean":"2023-11-08T05:28:01.458439+0000","last_became_active":"2023-11-08T05:28:13.557071+0000","last_became_peered":"2023-11-08T05:28:13.557071+0000","last_unstale":"2023-11-08T05:30:44.490644+0000","last_undegraded":"2023-11-08T05:30:44.490644+0000","last_fullsized":"2023-11-08T05:28:13.547025+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":459,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:45.794760+0000","last_change":"2023-11-08T05:30:45.794760+0000","last_active":"2023-11-08T05:30:45.794760+0000","last_peered":"2023-11-08T05:30:45.794760+0000","last_clean":"2023-11-08T05:30:45.794760+0000","last_became_active":"2023-11-08T05:30:45.794398+0000","last_became_peered":"2023-11-08T05:30:45.794398+0000","last_unstale":"2023-11-08T05:30:45.794760+0000","last_undegraded":"2023-11-08T05:30:45.794760+0000","last_fullsized":"2023-11-08T05:30:45.794760+0000","mapping_epoch":458,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":459,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":465,"reported_epoch":459,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:45.795085+0000","last_change":"2023-11-08T05:30:45.795085+0000","last_active":"2023-11-08T05:30:45.795085+0000","last_peered":"2023-11-08T05:30:45.795085+0000","last_clean":"2023-11-08T05:30:45.795085+0000","last_became_active":"2023-11-08T05:30:45.794743+0000","last_became_peered":"2023-11-08T05:30:45.794743+0000","last_unstale":"2023-11-08T05:30:45.795085+0000","last_undegraded":"2023-11-08T05:30:45.795085+0000","last_fullsized":"2023-11-08T05:30:45.795085+0000","mapping_epoch":458,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":459,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":454,"reported_epoch":458,"state":"active+clean","last_fresh":"2023-11-08T05:30:44.490618+0000","last_change":"2023-11-08T04:52:47.548597+0000","last_active":"2023-11-08T05:30:44.490618+0000","last_peered":"2023-11-08T05:30:44.490618+0000","last_clean":"2023-11-08T05:30:44.490618+0000","last_became_active":"2023-11-08T04:52:47.548294+0000","last_became_peered":"2023-11-08T04:52:47.548294+0000","last_unstale":"2023-11-08T05:30:44.490618+0000","last_undegraded":"2023-11-08T05:30:44.490618+0000","last_fullsized":"2023-11-08T05:30:44.490618+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":460,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490523+0000","last_change":"2023-11-08T05:28:13.545468+0000","last_active":"2023-11-08T05:30:44.490523+0000","last_peered":"2023-11-08T05:30:44.490523+0000","last_clean":"2023-11-08T05:28:01.456723+0000","last_became_active":"2023-11-08T05:28:13.545468+0000","last_became_peered":"2023-11-08T05:28:13.545468+0000","last_unstale":"2023-11-08T05:30:44.490523+0000","last_undegraded":"2023-11-08T05:30:44.490523+0000","last_fullsized":"2023-11-08T05:28:13.537207+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_clean_scrub_stamp":"2023-11-08T04:52:45.528796+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":465,"reported_epoch":458,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:44.491907+0000","last_change":"2023-11-08T05:29:44.395350+0000","last_active":"2023-11-08T05:30:44.491907+0000","last_peered":"2023-11-08T05:30:44.491907+0000","last_clean":"2023-11-08T05:30:44.491907+0000","last_became_active":"2023-11-08T05:29:44.395012+0000","last_became_peered":"2023-11-08T05:29:44.395012+0000","last_unstale":"2023-11-08T05:30:44.491907+0000","last_undegraded":"2023-11-08T05:30:44.491907+0000","last_fullsized":"2023-11-08T05:30:44.491907+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490629+0000","last_change":"2023-11-08T05:28:13.563357+0000","last_active":"2023-11-08T05:30:44.490629+0000","last_peered":"2023-11-08T05:30:44.490629+0000","last_clean":"2023-11-08T05:28:01.457691+0000","last_became_active":"2023-11-08T05:28:13.563357+0000","last_became_peered":"2023-11-08T05:28:13.563357+0000","last_unstale":"2023-11-08T05:30:44.490629+0000","last_undegraded":"2023-11-08T05:30:44.490629+0000","last_fullsized":"2023-11-08T05:28:13.541614+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490691+0000","last_change":"2023-11-08T05:28:13.550672+0000","last_active":"2023-11-08T05:30:44.490691+0000","last_peered":"2023-11-08T05:30:44.490691+0000","last_clean":"2023-11-08T05:28:01.457904+0000","last_became_active":"2023-11-08T05:28:13.550672+0000","last_became_peered":"2023-11-08T05:28:13.550672+0000","last_unstale":"2023-11-08T05:30:44.490691+0000","last_undegraded":"2023-11-08T05:30:44.490691+0000","last_fullsized":"2023-11-08T05:28:13.539595+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490805+0000","last_change":"2023-11-08T05:28:13.557968+0000","last_active":"2023-11-08T05:30:44.490805+0000","last_peered":"2023-11-08T05:30:44.490805+0000","last_clean":"2023-11-08T05:28:01.458433+0000","last_became_active":"2023-11-08T05:28:13.557968+0000","last_became_peered":"2023-11-08T05:28:13.557968+0000","last_unstale":"2023-11-08T05:30:44.490805+0000","last_undegraded":"2023-11-08T05:30:44.490805+0000","last_fullsized":"2023-11-08T05:28:13.542832+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":1,"acting":1,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":12,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738841,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":645040,"kb_used_data":5352,"kb_used_omap":275,"kb_used_meta":639404,"kb_avail":93726800,"statfs":{"total":96636764160,"available":95976243200,"internally_reserved":0,"allocated":5480448,"data_stored":2697647,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":281650,"internal_metadata":654750670},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:59 2023","interfaces":[{"interface":"back","average":{"1min":0.30499999999999999,"5min":0.439,"15min":0.36099999999999999},"min":{"1min":0.17399999999999999,"5min":0.14499999999999999,"15min":0.14499999999999999},"max":{"1min":0.60099999999999998,"5min":4.9379999999999997,"15min":4.9379999999999997},"last":0.379},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.49399999999999999,"15min":0.42399999999999999},"min":{"1min":0.221,"5min":0.19600000000000001,"15min":0.17000000000000001},"max":{"1min":0.63200000000000001,"5min":4.9859999999999998,"15min":4.9859999999999998},"last":0.40899999999999997}]},{"osd":1,"last update":"Wed Nov 8 05:31:06 2023","interfaces":[{"interface":"back","average":{"1min":0.27500000000000002,"5min":0.40300000000000002,"15min":0.46999999999999997},"min":{"1min":0.20899999999999999,"5min":0.16,"15min":0.153},"max":{"1min":0.41099999999999998,"5min":4.0830000000000002,"15min":23.242999999999999},"last":0.25800000000000001},{"interface":"front","average":{"1min":0.29899999999999999,"5min":0.40100000000000002,"15min":0.46000000000000002},"min":{"1min":0.23799999999999999,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":0.45300000000000001,"5min":3.5590000000000002,"15min":23.251000000000001},"last":0.23499999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738798,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1339228,"kb_used_data":398304,"kb_used_omap":291,"kb_used_meta":936476,"kb_avail":93032612,"statfs":{"total":96636764160,"available":95265394688,"internally_reserved":0,"allocated":407863296,"data_stored":405103900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":298462,"internal_metadata":958951970},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.436,"15min":0.36299999999999999},"min":{"1min":0.19700000000000001,"5min":0.14699999999999999,"15min":0.13700000000000001},"max":{"1min":3.222,"5min":6.5499999999999998,"15min":6.9989999999999997},"last":0.245},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.46300000000000002,"15min":0.42299999999999999},"min":{"1min":0.17799999999999999,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":5.0030000000000001,"5min":6.0629999999999997,"15min":9.6869999999999994},"last":0.28599999999999998}]},{"osd":2,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.51600000000000001,"15min":0.47099999999999997},"min":{"1min":0.21099999999999999,"5min":0.153,"15min":0.153},"max":{"1min":5.7439999999999998,"5min":5.8280000000000003,"15min":9.7639999999999993},"last":0.252},{"interface":"front","average":{"1min":0.497,"5min":0.56200000000000006,"15min":0.42299999999999999},"min":{"1min":0.20399999999999999,"5min":0.187,"15min":0.16800000000000001},"max":{"1min":3.9319999999999999,"5min":8.8040000000000003,"15min":8.8040000000000003},"last":0.29899999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738843,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":650100,"kb_used_data":5356,"kb_used_omap":214,"kb_used_meta":644521,"kb_avail":93721740,"statfs":{"total":96636764160,"available":95971061760,"internally_reserved":0,"allocated":5484544,"data_stored":2698616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":219335,"internal_metadata":659990329},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:56 2023","interfaces":[{"interface":"back","average":{"1min":0.30399999999999999,"5min":0.34699999999999998,"15min":0.36499999999999999},"min":{"1min":0.23999999999999999,"5min":0.14199999999999999,"15min":0.14199999999999999},"max":{"1min":0.42399999999999999,"5min":1.595,"15min":7.4249999999999998},"last":0.17100000000000001},{"interface":"front","average":{"1min":0.35499999999999998,"5min":0.40899999999999997,"15min":0.376},"min":{"1min":0.23999999999999999,"5min":0.19600000000000001,"15min":0.183},"max":{"1min":0.46999999999999997,"5min":2.6859999999999999,"15min":2.6859999999999999},"last":0.23000000000000001}]},{"osd":2,"last update":"Wed Nov 8 05:30:57 2023","interfaces":[{"interface":"back","average":{"1min":0.28599999999999998,"5min":0.34699999999999998,"15min":0.36899999999999999},"min":{"1min":0.19700000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":0.41399999999999998,"5min":2.2650000000000001,"15min":5.8209999999999997},"last":0.25600000000000001},{"interface":"front","average":{"1min":0.30599999999999999,"5min":0.313,"15min":0.35799999999999998},"min":{"1min":0.17799999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.435,"5min":1.137,"15min":4.2350000000000003},"last":0.23799999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-08T05:31:59.612 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~222s 2023-11-08T05:32:02.317 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-11-08T05:32:02.614 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T05:32:02.616 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2023-11-08T05:32:02.627 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":1635,"stamp":"2023-11-08T05:32:02.075192+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":10,"acting":13,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":109,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2634368,"kb_used_data":409012,"kb_used_omap":780,"kb_used_meta":2220403,"kb_avail":280481152,"statfs":{"total":289910292480,"available":287212699648,"internally_reserved":0,"allocated":418828288,"data_stored":410500163,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":799447,"internal_metadata":2273692969},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-536782902,"num_objects":-164,"num_object_clones":0,"num_object_copies":-328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-115,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-164,"num_whiteouts":0,"num_read":-122056,"num_read_kb":-39011942,"num_write":-413642,"num_write_kb":-151705339,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-8,"num_bytes_recovered":-16922624,"num_keys_recovered":-2,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.424817"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490644+0000","last_change":"2023-11-08T05:28:13.557071+0000","last_active":"2023-11-08T05:30:44.490644+0000","last_peered":"2023-11-08T05:30:44.490644+0000","last_clean":"2023-11-08T05:28:01.458439+0000","last_became_active":"2023-11-08T05:28:13.557071+0000","last_became_peered":"2023-11-08T05:28:13.557071+0000","last_unstale":"2023-11-08T05:30:44.490644+0000","last_undegraded":"2023-11-08T05:30:44.490644+0000","last_fullsized":"2023-11-08T05:28:13.547025+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":459,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:45.794760+0000","last_change":"2023-11-08T05:30:45.794760+0000","last_active":"2023-11-08T05:30:45.794760+0000","last_peered":"2023-11-08T05:30:45.794760+0000","last_clean":"2023-11-08T05:30:45.794760+0000","last_became_active":"2023-11-08T05:30:45.794398+0000","last_became_peered":"2023-11-08T05:30:45.794398+0000","last_unstale":"2023-11-08T05:30:45.794760+0000","last_undegraded":"2023-11-08T05:30:45.794760+0000","last_fullsized":"2023-11-08T05:30:45.794760+0000","mapping_epoch":458,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":459,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":465,"reported_epoch":459,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:45.795085+0000","last_change":"2023-11-08T05:30:45.795085+0000","last_active":"2023-11-08T05:30:45.795085+0000","last_peered":"2023-11-08T05:30:45.795085+0000","last_clean":"2023-11-08T05:30:45.795085+0000","last_became_active":"2023-11-08T05:30:45.794743+0000","last_became_peered":"2023-11-08T05:30:45.794743+0000","last_unstale":"2023-11-08T05:30:45.795085+0000","last_undegraded":"2023-11-08T05:30:45.795085+0000","last_fullsized":"2023-11-08T05:30:45.795085+0000","mapping_epoch":458,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":459,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":454,"reported_epoch":458,"state":"active+clean","last_fresh":"2023-11-08T05:30:44.490618+0000","last_change":"2023-11-08T04:52:47.548597+0000","last_active":"2023-11-08T05:30:44.490618+0000","last_peered":"2023-11-08T05:30:44.490618+0000","last_clean":"2023-11-08T05:30:44.490618+0000","last_became_active":"2023-11-08T04:52:47.548294+0000","last_became_peered":"2023-11-08T04:52:47.548294+0000","last_unstale":"2023-11-08T05:30:44.490618+0000","last_undegraded":"2023-11-08T05:30:44.490618+0000","last_fullsized":"2023-11-08T05:30:44.490618+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":460,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490523+0000","last_change":"2023-11-08T05:28:13.545468+0000","last_active":"2023-11-08T05:30:44.490523+0000","last_peered":"2023-11-08T05:30:44.490523+0000","last_clean":"2023-11-08T05:28:01.456723+0000","last_became_active":"2023-11-08T05:28:13.545468+0000","last_became_peered":"2023-11-08T05:28:13.545468+0000","last_unstale":"2023-11-08T05:30:44.490523+0000","last_undegraded":"2023-11-08T05:30:44.490523+0000","last_fullsized":"2023-11-08T05:28:13.537207+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_clean_scrub_stamp":"2023-11-08T04:52:45.528796+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":465,"reported_epoch":458,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:44.491907+0000","last_change":"2023-11-08T05:29:44.395350+0000","last_active":"2023-11-08T05:30:44.491907+0000","last_peered":"2023-11-08T05:30:44.491907+0000","last_clean":"2023-11-08T05:30:44.491907+0000","last_became_active":"2023-11-08T05:29:44.395012+0000","last_became_peered":"2023-11-08T05:29:44.395012+0000","last_unstale":"2023-11-08T05:30:44.491907+0000","last_undegraded":"2023-11-08T05:30:44.491907+0000","last_fullsized":"2023-11-08T05:30:44.491907+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490629+0000","last_change":"2023-11-08T05:28:13.563357+0000","last_active":"2023-11-08T05:30:44.490629+0000","last_peered":"2023-11-08T05:30:44.490629+0000","last_clean":"2023-11-08T05:28:01.457691+0000","last_became_active":"2023-11-08T05:28:13.563357+0000","last_became_peered":"2023-11-08T05:28:13.563357+0000","last_unstale":"2023-11-08T05:30:44.490629+0000","last_undegraded":"2023-11-08T05:30:44.490629+0000","last_fullsized":"2023-11-08T05:28:13.541614+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490691+0000","last_change":"2023-11-08T05:28:13.550672+0000","last_active":"2023-11-08T05:30:44.490691+0000","last_peered":"2023-11-08T05:30:44.490691+0000","last_clean":"2023-11-08T05:28:01.457904+0000","last_became_active":"2023-11-08T05:28:13.550672+0000","last_became_peered":"2023-11-08T05:28:13.550672+0000","last_unstale":"2023-11-08T05:30:44.490691+0000","last_undegraded":"2023-11-08T05:30:44.490691+0000","last_fullsized":"2023-11-08T05:28:13.539595+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490805+0000","last_change":"2023-11-08T05:28:13.557968+0000","last_active":"2023-11-08T05:30:44.490805+0000","last_peered":"2023-11-08T05:30:44.490805+0000","last_clean":"2023-11-08T05:28:01.458433+0000","last_became_active":"2023-11-08T05:28:13.557968+0000","last_became_peered":"2023-11-08T05:28:13.557968+0000","last_unstale":"2023-11-08T05:30:44.490805+0000","last_undegraded":"2023-11-08T05:30:44.490805+0000","last_fullsized":"2023-11-08T05:28:13.542832+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":1,"acting":1,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":12,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738842,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":645040,"kb_used_data":5352,"kb_used_omap":275,"kb_used_meta":639404,"kb_avail":93726800,"statfs":{"total":96636764160,"available":95976243200,"internally_reserved":0,"allocated":5480448,"data_stored":2697647,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":281650,"internal_metadata":654750670},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:59 2023","interfaces":[{"interface":"back","average":{"1min":0.30499999999999999,"5min":0.439,"15min":0.36099999999999999},"min":{"1min":0.17399999999999999,"5min":0.14499999999999999,"15min":0.14499999999999999},"max":{"1min":0.60099999999999998,"5min":4.9379999999999997,"15min":4.9379999999999997},"last":0.379},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.49399999999999999,"15min":0.42399999999999999},"min":{"1min":0.221,"5min":0.19600000000000001,"15min":0.17000000000000001},"max":{"1min":0.63200000000000001,"5min":4.9859999999999998,"15min":4.9859999999999998},"last":0.40899999999999997}]},{"osd":1,"last update":"Wed Nov 8 05:31:06 2023","interfaces":[{"interface":"back","average":{"1min":0.27500000000000002,"5min":0.40300000000000002,"15min":0.46999999999999997},"min":{"1min":0.20899999999999999,"5min":0.16,"15min":0.153},"max":{"1min":0.41099999999999998,"5min":4.0830000000000002,"15min":23.242999999999999},"last":0.39000000000000001},{"interface":"front","average":{"1min":0.29899999999999999,"5min":0.40100000000000002,"15min":0.46000000000000002},"min":{"1min":0.23799999999999999,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":0.45300000000000001,"5min":3.5590000000000002,"15min":23.251000000000001},"last":0.41699999999999998}]}]},{"osd":0,"up_from":8,"seq":34359738798,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1339228,"kb_used_data":398304,"kb_used_omap":291,"kb_used_meta":936476,"kb_avail":93032612,"statfs":{"total":96636764160,"available":95265394688,"internally_reserved":0,"allocated":407863296,"data_stored":405103900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":298462,"internal_metadata":958951970},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.436,"15min":0.36299999999999999},"min":{"1min":0.19700000000000001,"5min":0.14699999999999999,"15min":0.13700000000000001},"max":{"1min":3.222,"5min":6.5499999999999998,"15min":6.9989999999999997},"last":0.245},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.46300000000000002,"15min":0.42299999999999999},"min":{"1min":0.17799999999999999,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":5.0030000000000001,"5min":6.0629999999999997,"15min":9.6869999999999994},"last":0.28599999999999998}]},{"osd":2,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.51600000000000001,"15min":0.47099999999999997},"min":{"1min":0.21099999999999999,"5min":0.153,"15min":0.153},"max":{"1min":5.7439999999999998,"5min":5.8280000000000003,"15min":9.7639999999999993},"last":0.252},{"interface":"front","average":{"1min":0.497,"5min":0.56200000000000006,"15min":0.42299999999999999},"min":{"1min":0.20399999999999999,"5min":0.187,"15min":0.16800000000000001},"max":{"1min":3.9319999999999999,"5min":8.8040000000000003,"15min":8.8040000000000003},"last":0.29899999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738844,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":650100,"kb_used_data":5356,"kb_used_omap":214,"kb_used_meta":644521,"kb_avail":93721740,"statfs":{"total":96636764160,"available":95971061760,"internally_reserved":0,"allocated":5484544,"data_stored":2698616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":219335,"internal_metadata":659990329},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:56 2023","interfaces":[{"interface":"back","average":{"1min":0.30399999999999999,"5min":0.34699999999999998,"15min":0.36499999999999999},"min":{"1min":0.23999999999999999,"5min":0.14199999999999999,"15min":0.14199999999999999},"max":{"1min":0.42399999999999999,"5min":1.595,"15min":7.4249999999999998},"last":0.17100000000000001},{"interface":"front","average":{"1min":0.35499999999999998,"5min":0.40899999999999997,"15min":0.376},"min":{"1min":0.23999999999999999,"5min":0.19600000000000001,"15min":0.183},"max":{"1min":0.46999999999999997,"5min":2.6859999999999999,"15min":2.6859999999999999},"last":0.23000000000000001}]},{"osd":2,"last update":"Wed Nov 8 05:30:57 2023","interfaces":[{"interface":"back","average":{"1min":0.28599999999999998,"5min":0.34699999999999998,"15min":0.36899999999999999},"min":{"1min":0.19700000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":0.41399999999999998,"5min":2.2650000000000001,"15min":5.8209999999999997},"last":0.27700000000000002},{"interface":"front","average":{"1min":0.30599999999999999,"5min":0.313,"15min":0.35799999999999998},"min":{"1min":0.17799999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.435,"5min":1.137,"15min":4.2350000000000003},"last":0.25600000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-08T05:32:02.629 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-11-08T05:32:02.912 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T05:32:02.914 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2023-11-08T05:32:02.926 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":1635,"stamp":"2023-11-08T05:32:02.075192+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":10,"acting":13,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":109,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2634368,"kb_used_data":409012,"kb_used_omap":780,"kb_used_meta":2220403,"kb_avail":280481152,"statfs":{"total":289910292480,"available":287212699648,"internally_reserved":0,"allocated":418828288,"data_stored":410500163,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":799447,"internal_metadata":2273692969},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-536782902,"num_objects":-164,"num_object_clones":0,"num_object_copies":-328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-115,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-164,"num_whiteouts":0,"num_read":-122056,"num_read_kb":-39011942,"num_write":-413642,"num_write_kb":-151705339,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-8,"num_bytes_recovered":-16922624,"num_keys_recovered":-2,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.424817"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490644+0000","last_change":"2023-11-08T05:28:13.557071+0000","last_active":"2023-11-08T05:30:44.490644+0000","last_peered":"2023-11-08T05:30:44.490644+0000","last_clean":"2023-11-08T05:28:01.458439+0000","last_became_active":"2023-11-08T05:28:13.557071+0000","last_became_peered":"2023-11-08T05:28:13.557071+0000","last_unstale":"2023-11-08T05:30:44.490644+0000","last_undegraded":"2023-11-08T05:30:44.490644+0000","last_fullsized":"2023-11-08T05:28:13.547025+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":459,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:45.794760+0000","last_change":"2023-11-08T05:30:45.794760+0000","last_active":"2023-11-08T05:30:45.794760+0000","last_peered":"2023-11-08T05:30:45.794760+0000","last_clean":"2023-11-08T05:30:45.794760+0000","last_became_active":"2023-11-08T05:30:45.794398+0000","last_became_peered":"2023-11-08T05:30:45.794398+0000","last_unstale":"2023-11-08T05:30:45.794760+0000","last_undegraded":"2023-11-08T05:30:45.794760+0000","last_fullsized":"2023-11-08T05:30:45.794760+0000","mapping_epoch":458,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":459,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":465,"reported_epoch":459,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:45.795085+0000","last_change":"2023-11-08T05:30:45.795085+0000","last_active":"2023-11-08T05:30:45.795085+0000","last_peered":"2023-11-08T05:30:45.795085+0000","last_clean":"2023-11-08T05:30:45.795085+0000","last_became_active":"2023-11-08T05:30:45.794743+0000","last_became_peered":"2023-11-08T05:30:45.794743+0000","last_unstale":"2023-11-08T05:30:45.795085+0000","last_undegraded":"2023-11-08T05:30:45.795085+0000","last_fullsized":"2023-11-08T05:30:45.795085+0000","mapping_epoch":458,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":459,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":454,"reported_epoch":458,"state":"active+clean","last_fresh":"2023-11-08T05:30:44.490618+0000","last_change":"2023-11-08T04:52:47.548597+0000","last_active":"2023-11-08T05:30:44.490618+0000","last_peered":"2023-11-08T05:30:44.490618+0000","last_clean":"2023-11-08T05:30:44.490618+0000","last_became_active":"2023-11-08T04:52:47.548294+0000","last_became_peered":"2023-11-08T04:52:47.548294+0000","last_unstale":"2023-11-08T05:30:44.490618+0000","last_undegraded":"2023-11-08T05:30:44.490618+0000","last_fullsized":"2023-11-08T05:30:44.490618+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":460,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490523+0000","last_change":"2023-11-08T05:28:13.545468+0000","last_active":"2023-11-08T05:30:44.490523+0000","last_peered":"2023-11-08T05:30:44.490523+0000","last_clean":"2023-11-08T05:28:01.456723+0000","last_became_active":"2023-11-08T05:28:13.545468+0000","last_became_peered":"2023-11-08T05:28:13.545468+0000","last_unstale":"2023-11-08T05:30:44.490523+0000","last_undegraded":"2023-11-08T05:30:44.490523+0000","last_fullsized":"2023-11-08T05:28:13.537207+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_clean_scrub_stamp":"2023-11-08T04:52:45.528796+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":465,"reported_epoch":458,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:44.491907+0000","last_change":"2023-11-08T05:29:44.395350+0000","last_active":"2023-11-08T05:30:44.491907+0000","last_peered":"2023-11-08T05:30:44.491907+0000","last_clean":"2023-11-08T05:30:44.491907+0000","last_became_active":"2023-11-08T05:29:44.395012+0000","last_became_peered":"2023-11-08T05:29:44.395012+0000","last_unstale":"2023-11-08T05:30:44.491907+0000","last_undegraded":"2023-11-08T05:30:44.491907+0000","last_fullsized":"2023-11-08T05:30:44.491907+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490629+0000","last_change":"2023-11-08T05:28:13.563357+0000","last_active":"2023-11-08T05:30:44.490629+0000","last_peered":"2023-11-08T05:30:44.490629+0000","last_clean":"2023-11-08T05:28:01.457691+0000","last_became_active":"2023-11-08T05:28:13.563357+0000","last_became_peered":"2023-11-08T05:28:13.563357+0000","last_unstale":"2023-11-08T05:30:44.490629+0000","last_undegraded":"2023-11-08T05:30:44.490629+0000","last_fullsized":"2023-11-08T05:28:13.541614+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490691+0000","last_change":"2023-11-08T05:28:13.550672+0000","last_active":"2023-11-08T05:30:44.490691+0000","last_peered":"2023-11-08T05:30:44.490691+0000","last_clean":"2023-11-08T05:28:01.457904+0000","last_became_active":"2023-11-08T05:28:13.550672+0000","last_became_peered":"2023-11-08T05:28:13.550672+0000","last_unstale":"2023-11-08T05:30:44.490691+0000","last_undegraded":"2023-11-08T05:30:44.490691+0000","last_fullsized":"2023-11-08T05:28:13.539595+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490805+0000","last_change":"2023-11-08T05:28:13.557968+0000","last_active":"2023-11-08T05:30:44.490805+0000","last_peered":"2023-11-08T05:30:44.490805+0000","last_clean":"2023-11-08T05:28:01.458433+0000","last_became_active":"2023-11-08T05:28:13.557968+0000","last_became_peered":"2023-11-08T05:28:13.557968+0000","last_unstale":"2023-11-08T05:30:44.490805+0000","last_undegraded":"2023-11-08T05:30:44.490805+0000","last_fullsized":"2023-11-08T05:28:13.542832+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":1,"acting":1,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":12,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738842,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":645040,"kb_used_data":5352,"kb_used_omap":275,"kb_used_meta":639404,"kb_avail":93726800,"statfs":{"total":96636764160,"available":95976243200,"internally_reserved":0,"allocated":5480448,"data_stored":2697647,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":281650,"internal_metadata":654750670},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:59 2023","interfaces":[{"interface":"back","average":{"1min":0.30499999999999999,"5min":0.439,"15min":0.36099999999999999},"min":{"1min":0.17399999999999999,"5min":0.14499999999999999,"15min":0.14499999999999999},"max":{"1min":0.60099999999999998,"5min":4.9379999999999997,"15min":4.9379999999999997},"last":0.379},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.49399999999999999,"15min":0.42399999999999999},"min":{"1min":0.221,"5min":0.19600000000000001,"15min":0.17000000000000001},"max":{"1min":0.63200000000000001,"5min":4.9859999999999998,"15min":4.9859999999999998},"last":0.40899999999999997}]},{"osd":1,"last update":"Wed Nov 8 05:31:06 2023","interfaces":[{"interface":"back","average":{"1min":0.27500000000000002,"5min":0.40300000000000002,"15min":0.46999999999999997},"min":{"1min":0.20899999999999999,"5min":0.16,"15min":0.153},"max":{"1min":0.41099999999999998,"5min":4.0830000000000002,"15min":23.242999999999999},"last":0.39000000000000001},{"interface":"front","average":{"1min":0.29899999999999999,"5min":0.40100000000000002,"15min":0.46000000000000002},"min":{"1min":0.23799999999999999,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":0.45300000000000001,"5min":3.5590000000000002,"15min":23.251000000000001},"last":0.41699999999999998}]}]},{"osd":0,"up_from":8,"seq":34359738798,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1339228,"kb_used_data":398304,"kb_used_omap":291,"kb_used_meta":936476,"kb_avail":93032612,"statfs":{"total":96636764160,"available":95265394688,"internally_reserved":0,"allocated":407863296,"data_stored":405103900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":298462,"internal_metadata":958951970},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.436,"15min":0.36299999999999999},"min":{"1min":0.19700000000000001,"5min":0.14699999999999999,"15min":0.13700000000000001},"max":{"1min":3.222,"5min":6.5499999999999998,"15min":6.9989999999999997},"last":0.245},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.46300000000000002,"15min":0.42299999999999999},"min":{"1min":0.17799999999999999,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":5.0030000000000001,"5min":6.0629999999999997,"15min":9.6869999999999994},"last":0.28599999999999998}]},{"osd":2,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.51600000000000001,"15min":0.47099999999999997},"min":{"1min":0.21099999999999999,"5min":0.153,"15min":0.153},"max":{"1min":5.7439999999999998,"5min":5.8280000000000003,"15min":9.7639999999999993},"last":0.252},{"interface":"front","average":{"1min":0.497,"5min":0.56200000000000006,"15min":0.42299999999999999},"min":{"1min":0.20399999999999999,"5min":0.187,"15min":0.16800000000000001},"max":{"1min":3.9319999999999999,"5min":8.8040000000000003,"15min":8.8040000000000003},"last":0.29899999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738844,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":650100,"kb_used_data":5356,"kb_used_omap":214,"kb_used_meta":644521,"kb_avail":93721740,"statfs":{"total":96636764160,"available":95971061760,"internally_reserved":0,"allocated":5484544,"data_stored":2698616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":219335,"internal_metadata":659990329},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:56 2023","interfaces":[{"interface":"back","average":{"1min":0.30399999999999999,"5min":0.34699999999999998,"15min":0.36499999999999999},"min":{"1min":0.23999999999999999,"5min":0.14199999999999999,"15min":0.14199999999999999},"max":{"1min":0.42399999999999999,"5min":1.595,"15min":7.4249999999999998},"last":0.17100000000000001},{"interface":"front","average":{"1min":0.35499999999999998,"5min":0.40899999999999997,"15min":0.376},"min":{"1min":0.23999999999999999,"5min":0.19600000000000001,"15min":0.183},"max":{"1min":0.46999999999999997,"5min":2.6859999999999999,"15min":2.6859999999999999},"last":0.23000000000000001}]},{"osd":2,"last update":"Wed Nov 8 05:30:57 2023","interfaces":[{"interface":"back","average":{"1min":0.28599999999999998,"5min":0.34699999999999998,"15min":0.36899999999999999},"min":{"1min":0.19700000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":0.41399999999999998,"5min":2.2650000000000001,"15min":5.8209999999999997},"last":0.27700000000000002},{"interface":"front","average":{"1min":0.30599999999999999,"5min":0.313,"15min":0.35799999999999998},"min":{"1min":0.17799999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.435,"5min":1.137,"15min":4.2350000000000003},"last":0.25600000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-08T05:32:02.928 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-11-08T05:32:02.928 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 459, 'reported_epoch': 458, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:30:44.490644+0000', 'last_change': '2023-11-08T05:28:13.557071+0000', 'last_active': '2023-11-08T05:30:44.490644+0000', 'last_peered': '2023-11-08T05:30:44.490644+0000', 'last_clean': '2023-11-08T05:28:01.458439+0000', 'last_became_active': '2023-11-08T05:28:13.557071+0000', 'last_became_peered': '2023-11-08T05:28:13.557071+0000', 'last_unstale': '2023-11-08T05:30:44.490644+0000', 'last_undegraded': '2023-11-08T05:30:44.490644+0000', 'last_fullsized': '2023-11-08T05:28:13.547025+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:32:02.929 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-11-08T05:32:02.929 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 11, 'reported_epoch': 459, 'state': 'active+clean+remapped', 'last_fresh': '2023-11-08T05:30:45.794760+0000', 'last_change': '2023-11-08T05:30:45.794760+0000', 'last_active': '2023-11-08T05:30:45.794760+0000', 'last_peered': '2023-11-08T05:30:45.794760+0000', 'last_clean': '2023-11-08T05:30:45.794760+0000', 'last_became_active': '2023-11-08T05:30:45.794398+0000', 'last_became_peered': '2023-11-08T05:30:45.794398+0000', 'last_unstale': '2023-11-08T05:30:45.794760+0000', 'last_undegraded': '2023-11-08T05:30:45.794760+0000', 'last_fullsized': '2023-11-08T05:30:45.794760+0000', 'mapping_epoch': 458, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 459, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2], 'acting': [2, 1], 'avail_no_missing': ['2', '1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2023-11-08T05:32:02.930 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.1 is not active+clean 2023-11-08T05:32:02.930 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.1', 'version': "0'0", 'reported_seq': 465, 'reported_epoch': 459, 'state': 'active+clean+remapped', 'last_fresh': '2023-11-08T05:30:45.795085+0000', 'last_change': '2023-11-08T05:30:45.795085+0000', 'last_active': '2023-11-08T05:30:45.795085+0000', 'last_peered': '2023-11-08T05:30:45.795085+0000', 'last_clean': '2023-11-08T05:30:45.795085+0000', 'last_became_active': '2023-11-08T05:30:45.794743+0000', 'last_became_peered': '2023-11-08T05:30:45.794743+0000', 'last_unstale': '2023-11-08T05:30:45.795085+0000', 'last_undegraded': '2023-11-08T05:30:45.795085+0000', 'last_fullsized': '2023-11-08T05:30:45.795085+0000', 'mapping_epoch': 458, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 459, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2], 'acting': [2, 1], 'avail_no_missing': ['2', '1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2023-11-08T05:32:02.931 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-11-08T05:32:02.932 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "0'0", 'reported_seq': 460, 'reported_epoch': 458, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:30:44.490523+0000', 'last_change': '2023-11-08T05:28:13.545468+0000', 'last_active': '2023-11-08T05:30:44.490523+0000', 'last_peered': '2023-11-08T05:30:44.490523+0000', 'last_clean': '2023-11-08T05:28:01.456723+0000', 'last_became_active': '2023-11-08T05:28:13.545468+0000', 'last_became_peered': '2023-11-08T05:28:13.545468+0000', 'last_unstale': '2023-11-08T05:30:44.490523+0000', 'last_undegraded': '2023-11-08T05:30:44.490523+0000', 'last_fullsized': '2023-11-08T05:28:13.537207+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 9, 'last_epoch_clean': 10, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:45.528796+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:45.528796+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:45.528796+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:32:02.932 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.0 is not active+clean 2023-11-08T05:32:02.933 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.0', 'version': "0'0", 'reported_seq': 465, 'reported_epoch': 458, 'state': 'active+clean+remapped', 'last_fresh': '2023-11-08T05:30:44.491907+0000', 'last_change': '2023-11-08T05:29:44.395350+0000', 'last_active': '2023-11-08T05:30:44.491907+0000', 'last_peered': '2023-11-08T05:30:44.491907+0000', 'last_clean': '2023-11-08T05:30:44.491907+0000', 'last_became_active': '2023-11-08T05:29:44.395012+0000', 'last_became_peered': '2023-11-08T05:29:44.395012+0000', 'last_unstale': '2023-11-08T05:30:44.491907+0000', 'last_undegraded': '2023-11-08T05:30:44.491907+0000', 'last_fullsized': '2023-11-08T05:30:44.491907+0000', 'mapping_epoch': 455, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 456, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2], 'acting': [2, 1], 'avail_no_missing': ['2', '1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2023-11-08T05:32:02.933 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-11-08T05:32:02.934 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 459, 'reported_epoch': 458, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:30:44.490629+0000', 'last_change': '2023-11-08T05:28:13.563357+0000', 'last_active': '2023-11-08T05:30:44.490629+0000', 'last_peered': '2023-11-08T05:30:44.490629+0000', 'last_clean': '2023-11-08T05:28:01.457691+0000', 'last_became_active': '2023-11-08T05:28:13.563357+0000', 'last_became_peered': '2023-11-08T05:28:13.563357+0000', 'last_unstale': '2023-11-08T05:30:44.490629+0000', 'last_undegraded': '2023-11-08T05:30:44.490629+0000', 'last_fullsized': '2023-11-08T05:28:13.541614+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:32:02.935 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-11-08T05:32:02.935 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 459, 'reported_epoch': 458, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:30:44.490691+0000', 'last_change': '2023-11-08T05:28:13.550672+0000', 'last_active': '2023-11-08T05:30:44.490691+0000', 'last_peered': '2023-11-08T05:30:44.490691+0000', 'last_clean': '2023-11-08T05:28:01.457904+0000', 'last_became_active': '2023-11-08T05:28:13.550672+0000', 'last_became_peered': '2023-11-08T05:28:13.550672+0000', 'last_unstale': '2023-11-08T05:30:44.490691+0000', 'last_undegraded': '2023-11-08T05:30:44.490691+0000', 'last_fullsized': '2023-11-08T05:28:13.539595+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:32:02.936 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-11-08T05:32:02.936 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 459, 'reported_epoch': 458, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:30:44.490805+0000', 'last_change': '2023-11-08T05:28:13.557968+0000', 'last_active': '2023-11-08T05:30:44.490805+0000', 'last_peered': '2023-11-08T05:30:44.490805+0000', 'last_clean': '2023-11-08T05:28:01.458433+0000', 'last_became_active': '2023-11-08T05:28:13.557968+0000', 'last_became_peered': '2023-11-08T05:28:13.557968+0000', 'last_unstale': '2023-11-08T05:30:44.490805+0000', 'last_undegraded': '2023-11-08T05:30:44.490805+0000', 'last_fullsized': '2023-11-08T05:28:13.542832+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:32:02.937 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-11-08T05:32:03.283 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T05:32:03.283 INFO:teuthology.orchestra.run.smithi087.stdout:{"fsid":"ab4cbc06-449d-497d-b513-220d9db2b590","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 5 pgs undersized","count":5},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2365,"monmap":{"epoch":1,"min_mon_release_name":"pacific","num_mons":1},"osdmap":{"epoch":459,"num_osds":3,"num_up_osds":2,"osd_up_since":1699421292,"num_in_osds":3,"osd_in_since":1699419158,"num_remapped_pgs":3},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean+remapped","count":3},{"state_name":"active+clean","count":1}],"num_pgs":9,"num_pools":2,"num_objects":0,"data_bytes":0,"bytes_used":2697592832,"bytes_avail":287212699648,"bytes_total":289910292480},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":41,"modified":"2023-11-08T05:29:32.044544+0000","services":{}},"progress_events":{}} 2023-11-08T05:32:03.297 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-11-08T05:32:03.297 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-11-08T05:32:03.579 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T05:32:03.581 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2023-11-08T05:32:03.592 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":1635,"stamp":"2023-11-08T05:32:02.075192+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":10,"acting":13,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":109,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2634368,"kb_used_data":409012,"kb_used_omap":780,"kb_used_meta":2220403,"kb_avail":280481152,"statfs":{"total":289910292480,"available":287212699648,"internally_reserved":0,"allocated":418828288,"data_stored":410500163,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":799447,"internal_metadata":2273692969},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-536782902,"num_objects":-164,"num_object_clones":0,"num_object_copies":-328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-115,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-164,"num_whiteouts":0,"num_read":-122056,"num_read_kb":-39011942,"num_write":-413642,"num_write_kb":-151705339,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-8,"num_bytes_recovered":-16922624,"num_keys_recovered":-2,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.424817"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490644+0000","last_change":"2023-11-08T05:28:13.557071+0000","last_active":"2023-11-08T05:30:44.490644+0000","last_peered":"2023-11-08T05:30:44.490644+0000","last_clean":"2023-11-08T05:28:01.458439+0000","last_became_active":"2023-11-08T05:28:13.557071+0000","last_became_peered":"2023-11-08T05:28:13.557071+0000","last_unstale":"2023-11-08T05:30:44.490644+0000","last_undegraded":"2023-11-08T05:30:44.490644+0000","last_fullsized":"2023-11-08T05:28:13.547025+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":459,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:45.794760+0000","last_change":"2023-11-08T05:30:45.794760+0000","last_active":"2023-11-08T05:30:45.794760+0000","last_peered":"2023-11-08T05:30:45.794760+0000","last_clean":"2023-11-08T05:30:45.794760+0000","last_became_active":"2023-11-08T05:30:45.794398+0000","last_became_peered":"2023-11-08T05:30:45.794398+0000","last_unstale":"2023-11-08T05:30:45.794760+0000","last_undegraded":"2023-11-08T05:30:45.794760+0000","last_fullsized":"2023-11-08T05:30:45.794760+0000","mapping_epoch":458,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":459,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":465,"reported_epoch":459,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:45.795085+0000","last_change":"2023-11-08T05:30:45.795085+0000","last_active":"2023-11-08T05:30:45.795085+0000","last_peered":"2023-11-08T05:30:45.795085+0000","last_clean":"2023-11-08T05:30:45.795085+0000","last_became_active":"2023-11-08T05:30:45.794743+0000","last_became_peered":"2023-11-08T05:30:45.794743+0000","last_unstale":"2023-11-08T05:30:45.795085+0000","last_undegraded":"2023-11-08T05:30:45.795085+0000","last_fullsized":"2023-11-08T05:30:45.795085+0000","mapping_epoch":458,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":459,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":454,"reported_epoch":458,"state":"active+clean","last_fresh":"2023-11-08T05:30:44.490618+0000","last_change":"2023-11-08T04:52:47.548597+0000","last_active":"2023-11-08T05:30:44.490618+0000","last_peered":"2023-11-08T05:30:44.490618+0000","last_clean":"2023-11-08T05:30:44.490618+0000","last_became_active":"2023-11-08T04:52:47.548294+0000","last_became_peered":"2023-11-08T04:52:47.548294+0000","last_unstale":"2023-11-08T05:30:44.490618+0000","last_undegraded":"2023-11-08T05:30:44.490618+0000","last_fullsized":"2023-11-08T05:30:44.490618+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":460,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490523+0000","last_change":"2023-11-08T05:28:13.545468+0000","last_active":"2023-11-08T05:30:44.490523+0000","last_peered":"2023-11-08T05:30:44.490523+0000","last_clean":"2023-11-08T05:28:01.456723+0000","last_became_active":"2023-11-08T05:28:13.545468+0000","last_became_peered":"2023-11-08T05:28:13.545468+0000","last_unstale":"2023-11-08T05:30:44.490523+0000","last_undegraded":"2023-11-08T05:30:44.490523+0000","last_fullsized":"2023-11-08T05:28:13.537207+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_clean_scrub_stamp":"2023-11-08T04:52:45.528796+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":465,"reported_epoch":458,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:44.491907+0000","last_change":"2023-11-08T05:29:44.395350+0000","last_active":"2023-11-08T05:30:44.491907+0000","last_peered":"2023-11-08T05:30:44.491907+0000","last_clean":"2023-11-08T05:30:44.491907+0000","last_became_active":"2023-11-08T05:29:44.395012+0000","last_became_peered":"2023-11-08T05:29:44.395012+0000","last_unstale":"2023-11-08T05:30:44.491907+0000","last_undegraded":"2023-11-08T05:30:44.491907+0000","last_fullsized":"2023-11-08T05:30:44.491907+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490629+0000","last_change":"2023-11-08T05:28:13.563357+0000","last_active":"2023-11-08T05:30:44.490629+0000","last_peered":"2023-11-08T05:30:44.490629+0000","last_clean":"2023-11-08T05:28:01.457691+0000","last_became_active":"2023-11-08T05:28:13.563357+0000","last_became_peered":"2023-11-08T05:28:13.563357+0000","last_unstale":"2023-11-08T05:30:44.490629+0000","last_undegraded":"2023-11-08T05:30:44.490629+0000","last_fullsized":"2023-11-08T05:28:13.541614+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490691+0000","last_change":"2023-11-08T05:28:13.550672+0000","last_active":"2023-11-08T05:30:44.490691+0000","last_peered":"2023-11-08T05:30:44.490691+0000","last_clean":"2023-11-08T05:28:01.457904+0000","last_became_active":"2023-11-08T05:28:13.550672+0000","last_became_peered":"2023-11-08T05:28:13.550672+0000","last_unstale":"2023-11-08T05:30:44.490691+0000","last_undegraded":"2023-11-08T05:30:44.490691+0000","last_fullsized":"2023-11-08T05:28:13.539595+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490805+0000","last_change":"2023-11-08T05:28:13.557968+0000","last_active":"2023-11-08T05:30:44.490805+0000","last_peered":"2023-11-08T05:30:44.490805+0000","last_clean":"2023-11-08T05:28:01.458433+0000","last_became_active":"2023-11-08T05:28:13.557968+0000","last_became_peered":"2023-11-08T05:28:13.557968+0000","last_unstale":"2023-11-08T05:30:44.490805+0000","last_undegraded":"2023-11-08T05:30:44.490805+0000","last_fullsized":"2023-11-08T05:28:13.542832+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":1,"acting":1,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":12,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738842,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":645040,"kb_used_data":5352,"kb_used_omap":275,"kb_used_meta":639404,"kb_avail":93726800,"statfs":{"total":96636764160,"available":95976243200,"internally_reserved":0,"allocated":5480448,"data_stored":2697647,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":281650,"internal_metadata":654750670},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:59 2023","interfaces":[{"interface":"back","average":{"1min":0.30499999999999999,"5min":0.439,"15min":0.36099999999999999},"min":{"1min":0.17399999999999999,"5min":0.14499999999999999,"15min":0.14499999999999999},"max":{"1min":0.60099999999999998,"5min":4.9379999999999997,"15min":4.9379999999999997},"last":0.379},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.49399999999999999,"15min":0.42399999999999999},"min":{"1min":0.221,"5min":0.19600000000000001,"15min":0.17000000000000001},"max":{"1min":0.63200000000000001,"5min":4.9859999999999998,"15min":4.9859999999999998},"last":0.40899999999999997}]},{"osd":1,"last update":"Wed Nov 8 05:31:06 2023","interfaces":[{"interface":"back","average":{"1min":0.27500000000000002,"5min":0.40300000000000002,"15min":0.46999999999999997},"min":{"1min":0.20899999999999999,"5min":0.16,"15min":0.153},"max":{"1min":0.41099999999999998,"5min":4.0830000000000002,"15min":23.242999999999999},"last":0.39000000000000001},{"interface":"front","average":{"1min":0.29899999999999999,"5min":0.40100000000000002,"15min":0.46000000000000002},"min":{"1min":0.23799999999999999,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":0.45300000000000001,"5min":3.5590000000000002,"15min":23.251000000000001},"last":0.41699999999999998}]}]},{"osd":0,"up_from":8,"seq":34359738798,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1339228,"kb_used_data":398304,"kb_used_omap":291,"kb_used_meta":936476,"kb_avail":93032612,"statfs":{"total":96636764160,"available":95265394688,"internally_reserved":0,"allocated":407863296,"data_stored":405103900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":298462,"internal_metadata":958951970},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.436,"15min":0.36299999999999999},"min":{"1min":0.19700000000000001,"5min":0.14699999999999999,"15min":0.13700000000000001},"max":{"1min":3.222,"5min":6.5499999999999998,"15min":6.9989999999999997},"last":0.245},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.46300000000000002,"15min":0.42299999999999999},"min":{"1min":0.17799999999999999,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":5.0030000000000001,"5min":6.0629999999999997,"15min":9.6869999999999994},"last":0.28599999999999998}]},{"osd":2,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.51600000000000001,"15min":0.47099999999999997},"min":{"1min":0.21099999999999999,"5min":0.153,"15min":0.153},"max":{"1min":5.7439999999999998,"5min":5.8280000000000003,"15min":9.7639999999999993},"last":0.252},{"interface":"front","average":{"1min":0.497,"5min":0.56200000000000006,"15min":0.42299999999999999},"min":{"1min":0.20399999999999999,"5min":0.187,"15min":0.16800000000000001},"max":{"1min":3.9319999999999999,"5min":8.8040000000000003,"15min":8.8040000000000003},"last":0.29899999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738844,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":650100,"kb_used_data":5356,"kb_used_omap":214,"kb_used_meta":644521,"kb_avail":93721740,"statfs":{"total":96636764160,"available":95971061760,"internally_reserved":0,"allocated":5484544,"data_stored":2698616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":219335,"internal_metadata":659990329},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:56 2023","interfaces":[{"interface":"back","average":{"1min":0.30399999999999999,"5min":0.34699999999999998,"15min":0.36499999999999999},"min":{"1min":0.23999999999999999,"5min":0.14199999999999999,"15min":0.14199999999999999},"max":{"1min":0.42399999999999999,"5min":1.595,"15min":7.4249999999999998},"last":0.17100000000000001},{"interface":"front","average":{"1min":0.35499999999999998,"5min":0.40899999999999997,"15min":0.376},"min":{"1min":0.23999999999999999,"5min":0.19600000000000001,"15min":0.183},"max":{"1min":0.46999999999999997,"5min":2.6859999999999999,"15min":2.6859999999999999},"last":0.23000000000000001}]},{"osd":2,"last update":"Wed Nov 8 05:30:57 2023","interfaces":[{"interface":"back","average":{"1min":0.28599999999999998,"5min":0.34699999999999998,"15min":0.36899999999999999},"min":{"1min":0.19700000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":0.41399999999999998,"5min":2.2650000000000001,"15min":5.8209999999999997},"last":0.27700000000000002},{"interface":"front","average":{"1min":0.30599999999999999,"5min":0.313,"15min":0.35799999999999998},"min":{"1min":0.17799999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.435,"5min":1.137,"15min":4.2350000000000003},"last":0.25600000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-08T05:32:05.016 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~227s 2023-11-08T05:32:06.594 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-11-08T05:32:06.889 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T05:32:06.891 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2023-11-08T05:32:06.902 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":1637,"stamp":"2023-11-08T05:32:06.076014+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":10,"acting":13,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":109,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2634368,"kb_used_data":409012,"kb_used_omap":780,"kb_used_meta":2220403,"kb_avail":280481152,"statfs":{"total":289910292480,"available":287212699648,"internally_reserved":0,"allocated":418828288,"data_stored":410500163,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":799447,"internal_metadata":2273692969},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-536782902,"num_objects":-164,"num_object_clones":0,"num_object_copies":-328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-115,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-164,"num_whiteouts":0,"num_read":-122056,"num_read_kb":-39011942,"num_write":-413642,"num_write_kb":-151705339,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-8,"num_bytes_recovered":-16922624,"num_keys_recovered":-2,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.424817"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490644+0000","last_change":"2023-11-08T05:28:13.557071+0000","last_active":"2023-11-08T05:30:44.490644+0000","last_peered":"2023-11-08T05:30:44.490644+0000","last_clean":"2023-11-08T05:28:01.458439+0000","last_became_active":"2023-11-08T05:28:13.557071+0000","last_became_peered":"2023-11-08T05:28:13.557071+0000","last_unstale":"2023-11-08T05:30:44.490644+0000","last_undegraded":"2023-11-08T05:30:44.490644+0000","last_fullsized":"2023-11-08T05:28:13.547025+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":459,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:45.794760+0000","last_change":"2023-11-08T05:30:45.794760+0000","last_active":"2023-11-08T05:30:45.794760+0000","last_peered":"2023-11-08T05:30:45.794760+0000","last_clean":"2023-11-08T05:30:45.794760+0000","last_became_active":"2023-11-08T05:30:45.794398+0000","last_became_peered":"2023-11-08T05:30:45.794398+0000","last_unstale":"2023-11-08T05:30:45.794760+0000","last_undegraded":"2023-11-08T05:30:45.794760+0000","last_fullsized":"2023-11-08T05:30:45.794760+0000","mapping_epoch":458,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":459,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":465,"reported_epoch":459,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:45.795085+0000","last_change":"2023-11-08T05:30:45.795085+0000","last_active":"2023-11-08T05:30:45.795085+0000","last_peered":"2023-11-08T05:30:45.795085+0000","last_clean":"2023-11-08T05:30:45.795085+0000","last_became_active":"2023-11-08T05:30:45.794743+0000","last_became_peered":"2023-11-08T05:30:45.794743+0000","last_unstale":"2023-11-08T05:30:45.795085+0000","last_undegraded":"2023-11-08T05:30:45.795085+0000","last_fullsized":"2023-11-08T05:30:45.795085+0000","mapping_epoch":458,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":459,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":454,"reported_epoch":458,"state":"active+clean","last_fresh":"2023-11-08T05:30:44.490618+0000","last_change":"2023-11-08T04:52:47.548597+0000","last_active":"2023-11-08T05:30:44.490618+0000","last_peered":"2023-11-08T05:30:44.490618+0000","last_clean":"2023-11-08T05:30:44.490618+0000","last_became_active":"2023-11-08T04:52:47.548294+0000","last_became_peered":"2023-11-08T04:52:47.548294+0000","last_unstale":"2023-11-08T05:30:44.490618+0000","last_undegraded":"2023-11-08T05:30:44.490618+0000","last_fullsized":"2023-11-08T05:30:44.490618+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":460,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490523+0000","last_change":"2023-11-08T05:28:13.545468+0000","last_active":"2023-11-08T05:30:44.490523+0000","last_peered":"2023-11-08T05:30:44.490523+0000","last_clean":"2023-11-08T05:28:01.456723+0000","last_became_active":"2023-11-08T05:28:13.545468+0000","last_became_peered":"2023-11-08T05:28:13.545468+0000","last_unstale":"2023-11-08T05:30:44.490523+0000","last_undegraded":"2023-11-08T05:30:44.490523+0000","last_fullsized":"2023-11-08T05:28:13.537207+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_clean_scrub_stamp":"2023-11-08T04:52:45.528796+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":465,"reported_epoch":458,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:44.491907+0000","last_change":"2023-11-08T05:29:44.395350+0000","last_active":"2023-11-08T05:30:44.491907+0000","last_peered":"2023-11-08T05:30:44.491907+0000","last_clean":"2023-11-08T05:30:44.491907+0000","last_became_active":"2023-11-08T05:29:44.395012+0000","last_became_peered":"2023-11-08T05:29:44.395012+0000","last_unstale":"2023-11-08T05:30:44.491907+0000","last_undegraded":"2023-11-08T05:30:44.491907+0000","last_fullsized":"2023-11-08T05:30:44.491907+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490629+0000","last_change":"2023-11-08T05:28:13.563357+0000","last_active":"2023-11-08T05:30:44.490629+0000","last_peered":"2023-11-08T05:30:44.490629+0000","last_clean":"2023-11-08T05:28:01.457691+0000","last_became_active":"2023-11-08T05:28:13.563357+0000","last_became_peered":"2023-11-08T05:28:13.563357+0000","last_unstale":"2023-11-08T05:30:44.490629+0000","last_undegraded":"2023-11-08T05:30:44.490629+0000","last_fullsized":"2023-11-08T05:28:13.541614+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490691+0000","last_change":"2023-11-08T05:28:13.550672+0000","last_active":"2023-11-08T05:30:44.490691+0000","last_peered":"2023-11-08T05:30:44.490691+0000","last_clean":"2023-11-08T05:28:01.457904+0000","last_became_active":"2023-11-08T05:28:13.550672+0000","last_became_peered":"2023-11-08T05:28:13.550672+0000","last_unstale":"2023-11-08T05:30:44.490691+0000","last_undegraded":"2023-11-08T05:30:44.490691+0000","last_fullsized":"2023-11-08T05:28:13.539595+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490805+0000","last_change":"2023-11-08T05:28:13.557968+0000","last_active":"2023-11-08T05:30:44.490805+0000","last_peered":"2023-11-08T05:30:44.490805+0000","last_clean":"2023-11-08T05:28:01.458433+0000","last_became_active":"2023-11-08T05:28:13.557968+0000","last_became_peered":"2023-11-08T05:28:13.557968+0000","last_unstale":"2023-11-08T05:30:44.490805+0000","last_undegraded":"2023-11-08T05:30:44.490805+0000","last_fullsized":"2023-11-08T05:28:13.542832+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":1,"acting":1,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":12,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738843,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":645040,"kb_used_data":5352,"kb_used_omap":275,"kb_used_meta":639404,"kb_avail":93726800,"statfs":{"total":96636764160,"available":95976243200,"internally_reserved":0,"allocated":5480448,"data_stored":2697647,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":281650,"internal_metadata":654750670},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:59 2023","interfaces":[{"interface":"back","average":{"1min":0.30499999999999999,"5min":0.439,"15min":0.36099999999999999},"min":{"1min":0.17399999999999999,"5min":0.14499999999999999,"15min":0.14499999999999999},"max":{"1min":0.60099999999999998,"5min":4.9379999999999997,"15min":4.9379999999999997},"last":0.379},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.49399999999999999,"15min":0.42399999999999999},"min":{"1min":0.221,"5min":0.19600000000000001,"15min":0.17000000000000001},"max":{"1min":0.63200000000000001,"5min":4.9859999999999998,"15min":4.9859999999999998},"last":0.40899999999999997}]},{"osd":1,"last update":"Wed Nov 8 05:31:06 2023","interfaces":[{"interface":"back","average":{"1min":0.27500000000000002,"5min":0.40300000000000002,"15min":0.46999999999999997},"min":{"1min":0.20899999999999999,"5min":0.16,"15min":0.153},"max":{"1min":0.41099999999999998,"5min":4.0830000000000002,"15min":23.242999999999999},"last":0.36599999999999999},{"interface":"front","average":{"1min":0.29899999999999999,"5min":0.40100000000000002,"15min":0.46000000000000002},"min":{"1min":0.23799999999999999,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":0.45300000000000001,"5min":3.5590000000000002,"15min":23.251000000000001},"last":0.39400000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738798,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1339228,"kb_used_data":398304,"kb_used_omap":291,"kb_used_meta":936476,"kb_avail":93032612,"statfs":{"total":96636764160,"available":95265394688,"internally_reserved":0,"allocated":407863296,"data_stored":405103900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":298462,"internal_metadata":958951970},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.436,"15min":0.36299999999999999},"min":{"1min":0.19700000000000001,"5min":0.14699999999999999,"15min":0.13700000000000001},"max":{"1min":3.222,"5min":6.5499999999999998,"15min":6.9989999999999997},"last":0.245},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.46300000000000002,"15min":0.42299999999999999},"min":{"1min":0.17799999999999999,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":5.0030000000000001,"5min":6.0629999999999997,"15min":9.6869999999999994},"last":0.28599999999999998}]},{"osd":2,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.51600000000000001,"15min":0.47099999999999997},"min":{"1min":0.21099999999999999,"5min":0.153,"15min":0.153},"max":{"1min":5.7439999999999998,"5min":5.8280000000000003,"15min":9.7639999999999993},"last":0.252},{"interface":"front","average":{"1min":0.497,"5min":0.56200000000000006,"15min":0.42299999999999999},"min":{"1min":0.20399999999999999,"5min":0.187,"15min":0.16800000000000001},"max":{"1min":3.9319999999999999,"5min":8.8040000000000003,"15min":8.8040000000000003},"last":0.29899999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738845,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":650100,"kb_used_data":5356,"kb_used_omap":214,"kb_used_meta":644521,"kb_avail":93721740,"statfs":{"total":96636764160,"available":95971061760,"internally_reserved":0,"allocated":5484544,"data_stored":2698616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":219335,"internal_metadata":659990329},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:56 2023","interfaces":[{"interface":"back","average":{"1min":0.30399999999999999,"5min":0.34699999999999998,"15min":0.36499999999999999},"min":{"1min":0.23999999999999999,"5min":0.14199999999999999,"15min":0.14199999999999999},"max":{"1min":0.42399999999999999,"5min":1.595,"15min":7.4249999999999998},"last":0.17100000000000001},{"interface":"front","average":{"1min":0.35499999999999998,"5min":0.40899999999999997,"15min":0.376},"min":{"1min":0.23999999999999999,"5min":0.19600000000000001,"15min":0.183},"max":{"1min":0.46999999999999997,"5min":2.6859999999999999,"15min":2.6859999999999999},"last":0.23000000000000001}]},{"osd":2,"last update":"Wed Nov 8 05:32:02 2023","interfaces":[{"interface":"back","average":{"1min":0.252,"5min":0.33000000000000002,"15min":0.36499999999999999},"min":{"1min":0.161,"5min":0.161,"15min":0.161},"max":{"1min":0.39600000000000002,"5min":2.2650000000000001,"15min":5.8209999999999997},"last":0.249},{"interface":"front","average":{"1min":0.25800000000000001,"5min":0.29899999999999999,"15min":0.35599999999999998},"min":{"1min":0.17000000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.35799999999999998,"5min":1.137,"15min":4.2350000000000003},"last":0.222}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-08T05:32:06.903 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-11-08T05:32:07.185 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T05:32:07.187 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2023-11-08T05:32:07.198 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":1637,"stamp":"2023-11-08T05:32:06.076014+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":10,"acting":13,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":109,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2634368,"kb_used_data":409012,"kb_used_omap":780,"kb_used_meta":2220403,"kb_avail":280481152,"statfs":{"total":289910292480,"available":287212699648,"internally_reserved":0,"allocated":418828288,"data_stored":410500163,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":799447,"internal_metadata":2273692969},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-536782902,"num_objects":-164,"num_object_clones":0,"num_object_copies":-328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-115,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-164,"num_whiteouts":0,"num_read":-122056,"num_read_kb":-39011942,"num_write":-413642,"num_write_kb":-151705339,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-8,"num_bytes_recovered":-16922624,"num_keys_recovered":-2,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.424817"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490644+0000","last_change":"2023-11-08T05:28:13.557071+0000","last_active":"2023-11-08T05:30:44.490644+0000","last_peered":"2023-11-08T05:30:44.490644+0000","last_clean":"2023-11-08T05:28:01.458439+0000","last_became_active":"2023-11-08T05:28:13.557071+0000","last_became_peered":"2023-11-08T05:28:13.557071+0000","last_unstale":"2023-11-08T05:30:44.490644+0000","last_undegraded":"2023-11-08T05:30:44.490644+0000","last_fullsized":"2023-11-08T05:28:13.547025+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":459,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:45.794760+0000","last_change":"2023-11-08T05:30:45.794760+0000","last_active":"2023-11-08T05:30:45.794760+0000","last_peered":"2023-11-08T05:30:45.794760+0000","last_clean":"2023-11-08T05:30:45.794760+0000","last_became_active":"2023-11-08T05:30:45.794398+0000","last_became_peered":"2023-11-08T05:30:45.794398+0000","last_unstale":"2023-11-08T05:30:45.794760+0000","last_undegraded":"2023-11-08T05:30:45.794760+0000","last_fullsized":"2023-11-08T05:30:45.794760+0000","mapping_epoch":458,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":459,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":465,"reported_epoch":459,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:45.795085+0000","last_change":"2023-11-08T05:30:45.795085+0000","last_active":"2023-11-08T05:30:45.795085+0000","last_peered":"2023-11-08T05:30:45.795085+0000","last_clean":"2023-11-08T05:30:45.795085+0000","last_became_active":"2023-11-08T05:30:45.794743+0000","last_became_peered":"2023-11-08T05:30:45.794743+0000","last_unstale":"2023-11-08T05:30:45.795085+0000","last_undegraded":"2023-11-08T05:30:45.795085+0000","last_fullsized":"2023-11-08T05:30:45.795085+0000","mapping_epoch":458,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":459,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":454,"reported_epoch":458,"state":"active+clean","last_fresh":"2023-11-08T05:30:44.490618+0000","last_change":"2023-11-08T04:52:47.548597+0000","last_active":"2023-11-08T05:30:44.490618+0000","last_peered":"2023-11-08T05:30:44.490618+0000","last_clean":"2023-11-08T05:30:44.490618+0000","last_became_active":"2023-11-08T04:52:47.548294+0000","last_became_peered":"2023-11-08T04:52:47.548294+0000","last_unstale":"2023-11-08T05:30:44.490618+0000","last_undegraded":"2023-11-08T05:30:44.490618+0000","last_fullsized":"2023-11-08T05:30:44.490618+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":460,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490523+0000","last_change":"2023-11-08T05:28:13.545468+0000","last_active":"2023-11-08T05:30:44.490523+0000","last_peered":"2023-11-08T05:30:44.490523+0000","last_clean":"2023-11-08T05:28:01.456723+0000","last_became_active":"2023-11-08T05:28:13.545468+0000","last_became_peered":"2023-11-08T05:28:13.545468+0000","last_unstale":"2023-11-08T05:30:44.490523+0000","last_undegraded":"2023-11-08T05:30:44.490523+0000","last_fullsized":"2023-11-08T05:28:13.537207+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_clean_scrub_stamp":"2023-11-08T04:52:45.528796+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":465,"reported_epoch":458,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:44.491907+0000","last_change":"2023-11-08T05:29:44.395350+0000","last_active":"2023-11-08T05:30:44.491907+0000","last_peered":"2023-11-08T05:30:44.491907+0000","last_clean":"2023-11-08T05:30:44.491907+0000","last_became_active":"2023-11-08T05:29:44.395012+0000","last_became_peered":"2023-11-08T05:29:44.395012+0000","last_unstale":"2023-11-08T05:30:44.491907+0000","last_undegraded":"2023-11-08T05:30:44.491907+0000","last_fullsized":"2023-11-08T05:30:44.491907+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490629+0000","last_change":"2023-11-08T05:28:13.563357+0000","last_active":"2023-11-08T05:30:44.490629+0000","last_peered":"2023-11-08T05:30:44.490629+0000","last_clean":"2023-11-08T05:28:01.457691+0000","last_became_active":"2023-11-08T05:28:13.563357+0000","last_became_peered":"2023-11-08T05:28:13.563357+0000","last_unstale":"2023-11-08T05:30:44.490629+0000","last_undegraded":"2023-11-08T05:30:44.490629+0000","last_fullsized":"2023-11-08T05:28:13.541614+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490691+0000","last_change":"2023-11-08T05:28:13.550672+0000","last_active":"2023-11-08T05:30:44.490691+0000","last_peered":"2023-11-08T05:30:44.490691+0000","last_clean":"2023-11-08T05:28:01.457904+0000","last_became_active":"2023-11-08T05:28:13.550672+0000","last_became_peered":"2023-11-08T05:28:13.550672+0000","last_unstale":"2023-11-08T05:30:44.490691+0000","last_undegraded":"2023-11-08T05:30:44.490691+0000","last_fullsized":"2023-11-08T05:28:13.539595+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490805+0000","last_change":"2023-11-08T05:28:13.557968+0000","last_active":"2023-11-08T05:30:44.490805+0000","last_peered":"2023-11-08T05:30:44.490805+0000","last_clean":"2023-11-08T05:28:01.458433+0000","last_became_active":"2023-11-08T05:28:13.557968+0000","last_became_peered":"2023-11-08T05:28:13.557968+0000","last_unstale":"2023-11-08T05:30:44.490805+0000","last_undegraded":"2023-11-08T05:30:44.490805+0000","last_fullsized":"2023-11-08T05:28:13.542832+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":1,"acting":1,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":12,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738843,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":645040,"kb_used_data":5352,"kb_used_omap":275,"kb_used_meta":639404,"kb_avail":93726800,"statfs":{"total":96636764160,"available":95976243200,"internally_reserved":0,"allocated":5480448,"data_stored":2697647,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":281650,"internal_metadata":654750670},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:59 2023","interfaces":[{"interface":"back","average":{"1min":0.30499999999999999,"5min":0.439,"15min":0.36099999999999999},"min":{"1min":0.17399999999999999,"5min":0.14499999999999999,"15min":0.14499999999999999},"max":{"1min":0.60099999999999998,"5min":4.9379999999999997,"15min":4.9379999999999997},"last":0.379},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.49399999999999999,"15min":0.42399999999999999},"min":{"1min":0.221,"5min":0.19600000000000001,"15min":0.17000000000000001},"max":{"1min":0.63200000000000001,"5min":4.9859999999999998,"15min":4.9859999999999998},"last":0.40899999999999997}]},{"osd":1,"last update":"Wed Nov 8 05:31:06 2023","interfaces":[{"interface":"back","average":{"1min":0.27500000000000002,"5min":0.40300000000000002,"15min":0.46999999999999997},"min":{"1min":0.20899999999999999,"5min":0.16,"15min":0.153},"max":{"1min":0.41099999999999998,"5min":4.0830000000000002,"15min":23.242999999999999},"last":0.36599999999999999},{"interface":"front","average":{"1min":0.29899999999999999,"5min":0.40100000000000002,"15min":0.46000000000000002},"min":{"1min":0.23799999999999999,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":0.45300000000000001,"5min":3.5590000000000002,"15min":23.251000000000001},"last":0.39400000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738798,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1339228,"kb_used_data":398304,"kb_used_omap":291,"kb_used_meta":936476,"kb_avail":93032612,"statfs":{"total":96636764160,"available":95265394688,"internally_reserved":0,"allocated":407863296,"data_stored":405103900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":298462,"internal_metadata":958951970},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.436,"15min":0.36299999999999999},"min":{"1min":0.19700000000000001,"5min":0.14699999999999999,"15min":0.13700000000000001},"max":{"1min":3.222,"5min":6.5499999999999998,"15min":6.9989999999999997},"last":0.245},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.46300000000000002,"15min":0.42299999999999999},"min":{"1min":0.17799999999999999,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":5.0030000000000001,"5min":6.0629999999999997,"15min":9.6869999999999994},"last":0.28599999999999998}]},{"osd":2,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.51600000000000001,"15min":0.47099999999999997},"min":{"1min":0.21099999999999999,"5min":0.153,"15min":0.153},"max":{"1min":5.7439999999999998,"5min":5.8280000000000003,"15min":9.7639999999999993},"last":0.252},{"interface":"front","average":{"1min":0.497,"5min":0.56200000000000006,"15min":0.42299999999999999},"min":{"1min":0.20399999999999999,"5min":0.187,"15min":0.16800000000000001},"max":{"1min":3.9319999999999999,"5min":8.8040000000000003,"15min":8.8040000000000003},"last":0.29899999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738845,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":650100,"kb_used_data":5356,"kb_used_omap":214,"kb_used_meta":644521,"kb_avail":93721740,"statfs":{"total":96636764160,"available":95971061760,"internally_reserved":0,"allocated":5484544,"data_stored":2698616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":219335,"internal_metadata":659990329},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:56 2023","interfaces":[{"interface":"back","average":{"1min":0.30399999999999999,"5min":0.34699999999999998,"15min":0.36499999999999999},"min":{"1min":0.23999999999999999,"5min":0.14199999999999999,"15min":0.14199999999999999},"max":{"1min":0.42399999999999999,"5min":1.595,"15min":7.4249999999999998},"last":0.17100000000000001},{"interface":"front","average":{"1min":0.35499999999999998,"5min":0.40899999999999997,"15min":0.376},"min":{"1min":0.23999999999999999,"5min":0.19600000000000001,"15min":0.183},"max":{"1min":0.46999999999999997,"5min":2.6859999999999999,"15min":2.6859999999999999},"last":0.23000000000000001}]},{"osd":2,"last update":"Wed Nov 8 05:32:02 2023","interfaces":[{"interface":"back","average":{"1min":0.252,"5min":0.33000000000000002,"15min":0.36499999999999999},"min":{"1min":0.161,"5min":0.161,"15min":0.161},"max":{"1min":0.39600000000000002,"5min":2.2650000000000001,"15min":5.8209999999999997},"last":0.249},{"interface":"front","average":{"1min":0.25800000000000001,"5min":0.29899999999999999,"15min":0.35599999999999998},"min":{"1min":0.17000000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.35799999999999998,"5min":1.137,"15min":4.2350000000000003},"last":0.222}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-08T05:32:07.199 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-11-08T05:32:07.199 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 459, 'reported_epoch': 458, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:30:44.490644+0000', 'last_change': '2023-11-08T05:28:13.557071+0000', 'last_active': '2023-11-08T05:30:44.490644+0000', 'last_peered': '2023-11-08T05:30:44.490644+0000', 'last_clean': '2023-11-08T05:28:01.458439+0000', 'last_became_active': '2023-11-08T05:28:13.557071+0000', 'last_became_peered': '2023-11-08T05:28:13.557071+0000', 'last_unstale': '2023-11-08T05:30:44.490644+0000', 'last_undegraded': '2023-11-08T05:30:44.490644+0000', 'last_fullsized': '2023-11-08T05:28:13.547025+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:32:07.200 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-11-08T05:32:07.200 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 11, 'reported_epoch': 459, 'state': 'active+clean+remapped', 'last_fresh': '2023-11-08T05:30:45.794760+0000', 'last_change': '2023-11-08T05:30:45.794760+0000', 'last_active': '2023-11-08T05:30:45.794760+0000', 'last_peered': '2023-11-08T05:30:45.794760+0000', 'last_clean': '2023-11-08T05:30:45.794760+0000', 'last_became_active': '2023-11-08T05:30:45.794398+0000', 'last_became_peered': '2023-11-08T05:30:45.794398+0000', 'last_unstale': '2023-11-08T05:30:45.794760+0000', 'last_undegraded': '2023-11-08T05:30:45.794760+0000', 'last_fullsized': '2023-11-08T05:30:45.794760+0000', 'mapping_epoch': 458, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 459, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2], 'acting': [2, 1], 'avail_no_missing': ['2', '1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2023-11-08T05:32:07.200 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.1 is not active+clean 2023-11-08T05:32:07.200 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.1', 'version': "0'0", 'reported_seq': 465, 'reported_epoch': 459, 'state': 'active+clean+remapped', 'last_fresh': '2023-11-08T05:30:45.795085+0000', 'last_change': '2023-11-08T05:30:45.795085+0000', 'last_active': '2023-11-08T05:30:45.795085+0000', 'last_peered': '2023-11-08T05:30:45.795085+0000', 'last_clean': '2023-11-08T05:30:45.795085+0000', 'last_became_active': '2023-11-08T05:30:45.794743+0000', 'last_became_peered': '2023-11-08T05:30:45.794743+0000', 'last_unstale': '2023-11-08T05:30:45.795085+0000', 'last_undegraded': '2023-11-08T05:30:45.795085+0000', 'last_fullsized': '2023-11-08T05:30:45.795085+0000', 'mapping_epoch': 458, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 459, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2], 'acting': [2, 1], 'avail_no_missing': ['2', '1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2023-11-08T05:32:07.201 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-11-08T05:32:07.201 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "0'0", 'reported_seq': 460, 'reported_epoch': 458, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:30:44.490523+0000', 'last_change': '2023-11-08T05:28:13.545468+0000', 'last_active': '2023-11-08T05:30:44.490523+0000', 'last_peered': '2023-11-08T05:30:44.490523+0000', 'last_clean': '2023-11-08T05:28:01.456723+0000', 'last_became_active': '2023-11-08T05:28:13.545468+0000', 'last_became_peered': '2023-11-08T05:28:13.545468+0000', 'last_unstale': '2023-11-08T05:30:44.490523+0000', 'last_undegraded': '2023-11-08T05:30:44.490523+0000', 'last_fullsized': '2023-11-08T05:28:13.537207+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 9, 'last_epoch_clean': 10, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:45.528796+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:45.528796+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:45.528796+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:32:07.201 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.0 is not active+clean 2023-11-08T05:32:07.202 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.0', 'version': "0'0", 'reported_seq': 465, 'reported_epoch': 458, 'state': 'active+clean+remapped', 'last_fresh': '2023-11-08T05:30:44.491907+0000', 'last_change': '2023-11-08T05:29:44.395350+0000', 'last_active': '2023-11-08T05:30:44.491907+0000', 'last_peered': '2023-11-08T05:30:44.491907+0000', 'last_clean': '2023-11-08T05:30:44.491907+0000', 'last_became_active': '2023-11-08T05:29:44.395012+0000', 'last_became_peered': '2023-11-08T05:29:44.395012+0000', 'last_unstale': '2023-11-08T05:30:44.491907+0000', 'last_undegraded': '2023-11-08T05:30:44.491907+0000', 'last_fullsized': '2023-11-08T05:30:44.491907+0000', 'mapping_epoch': 455, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 456, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2], 'acting': [2, 1], 'avail_no_missing': ['2', '1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2023-11-08T05:32:07.202 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-11-08T05:32:07.202 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 459, 'reported_epoch': 458, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:30:44.490629+0000', 'last_change': '2023-11-08T05:28:13.563357+0000', 'last_active': '2023-11-08T05:30:44.490629+0000', 'last_peered': '2023-11-08T05:30:44.490629+0000', 'last_clean': '2023-11-08T05:28:01.457691+0000', 'last_became_active': '2023-11-08T05:28:13.563357+0000', 'last_became_peered': '2023-11-08T05:28:13.563357+0000', 'last_unstale': '2023-11-08T05:30:44.490629+0000', 'last_undegraded': '2023-11-08T05:30:44.490629+0000', 'last_fullsized': '2023-11-08T05:28:13.541614+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:32:07.203 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-11-08T05:32:07.203 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 459, 'reported_epoch': 458, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:30:44.490691+0000', 'last_change': '2023-11-08T05:28:13.550672+0000', 'last_active': '2023-11-08T05:30:44.490691+0000', 'last_peered': '2023-11-08T05:30:44.490691+0000', 'last_clean': '2023-11-08T05:28:01.457904+0000', 'last_became_active': '2023-11-08T05:28:13.550672+0000', 'last_became_peered': '2023-11-08T05:28:13.550672+0000', 'last_unstale': '2023-11-08T05:30:44.490691+0000', 'last_undegraded': '2023-11-08T05:30:44.490691+0000', 'last_fullsized': '2023-11-08T05:28:13.539595+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:32:07.203 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-11-08T05:32:07.203 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 459, 'reported_epoch': 458, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:30:44.490805+0000', 'last_change': '2023-11-08T05:28:13.557968+0000', 'last_active': '2023-11-08T05:30:44.490805+0000', 'last_peered': '2023-11-08T05:30:44.490805+0000', 'last_clean': '2023-11-08T05:28:01.458433+0000', 'last_became_active': '2023-11-08T05:28:13.557968+0000', 'last_became_peered': '2023-11-08T05:28:13.557968+0000', 'last_unstale': '2023-11-08T05:30:44.490805+0000', 'last_undegraded': '2023-11-08T05:30:44.490805+0000', 'last_fullsized': '2023-11-08T05:28:13.542832+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:32:07.204 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-11-08T05:32:07.552 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T05:32:07.553 INFO:teuthology.orchestra.run.smithi087.stdout:{"fsid":"ab4cbc06-449d-497d-b513-220d9db2b590","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 5 pgs undersized","count":5},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2370,"monmap":{"epoch":1,"min_mon_release_name":"pacific","num_mons":1},"osdmap":{"epoch":459,"num_osds":3,"num_up_osds":2,"osd_up_since":1699421292,"num_in_osds":3,"osd_in_since":1699419158,"num_remapped_pgs":3},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean+remapped","count":3},{"state_name":"active+clean","count":1}],"num_pgs":9,"num_pools":2,"num_objects":0,"data_bytes":0,"bytes_used":2697592832,"bytes_avail":287212699648,"bytes_total":289910292480},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":41,"modified":"2023-11-08T05:29:32.044544+0000","services":{}},"progress_events":{}} 2023-11-08T05:32:07.566 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-11-08T05:32:07.567 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-11-08T05:32:07.849 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T05:32:07.851 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2023-11-08T05:32:07.862 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":1637,"stamp":"2023-11-08T05:32:06.076014+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":10,"acting":13,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":109,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2634368,"kb_used_data":409012,"kb_used_omap":780,"kb_used_meta":2220403,"kb_avail":280481152,"statfs":{"total":289910292480,"available":287212699648,"internally_reserved":0,"allocated":418828288,"data_stored":410500163,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":799447,"internal_metadata":2273692969},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-536782902,"num_objects":-164,"num_object_clones":0,"num_object_copies":-328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-115,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-164,"num_whiteouts":0,"num_read":-122056,"num_read_kb":-39011942,"num_write":-413642,"num_write_kb":-151705339,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-8,"num_bytes_recovered":-16922624,"num_keys_recovered":-2,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.424817"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490644+0000","last_change":"2023-11-08T05:28:13.557071+0000","last_active":"2023-11-08T05:30:44.490644+0000","last_peered":"2023-11-08T05:30:44.490644+0000","last_clean":"2023-11-08T05:28:01.458439+0000","last_became_active":"2023-11-08T05:28:13.557071+0000","last_became_peered":"2023-11-08T05:28:13.557071+0000","last_unstale":"2023-11-08T05:30:44.490644+0000","last_undegraded":"2023-11-08T05:30:44.490644+0000","last_fullsized":"2023-11-08T05:28:13.547025+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":459,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:45.794760+0000","last_change":"2023-11-08T05:30:45.794760+0000","last_active":"2023-11-08T05:30:45.794760+0000","last_peered":"2023-11-08T05:30:45.794760+0000","last_clean":"2023-11-08T05:30:45.794760+0000","last_became_active":"2023-11-08T05:30:45.794398+0000","last_became_peered":"2023-11-08T05:30:45.794398+0000","last_unstale":"2023-11-08T05:30:45.794760+0000","last_undegraded":"2023-11-08T05:30:45.794760+0000","last_fullsized":"2023-11-08T05:30:45.794760+0000","mapping_epoch":458,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":459,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":465,"reported_epoch":459,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:45.795085+0000","last_change":"2023-11-08T05:30:45.795085+0000","last_active":"2023-11-08T05:30:45.795085+0000","last_peered":"2023-11-08T05:30:45.795085+0000","last_clean":"2023-11-08T05:30:45.795085+0000","last_became_active":"2023-11-08T05:30:45.794743+0000","last_became_peered":"2023-11-08T05:30:45.794743+0000","last_unstale":"2023-11-08T05:30:45.795085+0000","last_undegraded":"2023-11-08T05:30:45.795085+0000","last_fullsized":"2023-11-08T05:30:45.795085+0000","mapping_epoch":458,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":459,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":454,"reported_epoch":458,"state":"active+clean","last_fresh":"2023-11-08T05:30:44.490618+0000","last_change":"2023-11-08T04:52:47.548597+0000","last_active":"2023-11-08T05:30:44.490618+0000","last_peered":"2023-11-08T05:30:44.490618+0000","last_clean":"2023-11-08T05:30:44.490618+0000","last_became_active":"2023-11-08T04:52:47.548294+0000","last_became_peered":"2023-11-08T04:52:47.548294+0000","last_unstale":"2023-11-08T05:30:44.490618+0000","last_undegraded":"2023-11-08T05:30:44.490618+0000","last_fullsized":"2023-11-08T05:30:44.490618+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":460,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490523+0000","last_change":"2023-11-08T05:28:13.545468+0000","last_active":"2023-11-08T05:30:44.490523+0000","last_peered":"2023-11-08T05:30:44.490523+0000","last_clean":"2023-11-08T05:28:01.456723+0000","last_became_active":"2023-11-08T05:28:13.545468+0000","last_became_peered":"2023-11-08T05:28:13.545468+0000","last_unstale":"2023-11-08T05:30:44.490523+0000","last_undegraded":"2023-11-08T05:30:44.490523+0000","last_fullsized":"2023-11-08T05:28:13.537207+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_clean_scrub_stamp":"2023-11-08T04:52:45.528796+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":465,"reported_epoch":458,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:44.491907+0000","last_change":"2023-11-08T05:29:44.395350+0000","last_active":"2023-11-08T05:30:44.491907+0000","last_peered":"2023-11-08T05:30:44.491907+0000","last_clean":"2023-11-08T05:30:44.491907+0000","last_became_active":"2023-11-08T05:29:44.395012+0000","last_became_peered":"2023-11-08T05:29:44.395012+0000","last_unstale":"2023-11-08T05:30:44.491907+0000","last_undegraded":"2023-11-08T05:30:44.491907+0000","last_fullsized":"2023-11-08T05:30:44.491907+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490629+0000","last_change":"2023-11-08T05:28:13.563357+0000","last_active":"2023-11-08T05:30:44.490629+0000","last_peered":"2023-11-08T05:30:44.490629+0000","last_clean":"2023-11-08T05:28:01.457691+0000","last_became_active":"2023-11-08T05:28:13.563357+0000","last_became_peered":"2023-11-08T05:28:13.563357+0000","last_unstale":"2023-11-08T05:30:44.490629+0000","last_undegraded":"2023-11-08T05:30:44.490629+0000","last_fullsized":"2023-11-08T05:28:13.541614+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490691+0000","last_change":"2023-11-08T05:28:13.550672+0000","last_active":"2023-11-08T05:30:44.490691+0000","last_peered":"2023-11-08T05:30:44.490691+0000","last_clean":"2023-11-08T05:28:01.457904+0000","last_became_active":"2023-11-08T05:28:13.550672+0000","last_became_peered":"2023-11-08T05:28:13.550672+0000","last_unstale":"2023-11-08T05:30:44.490691+0000","last_undegraded":"2023-11-08T05:30:44.490691+0000","last_fullsized":"2023-11-08T05:28:13.539595+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490805+0000","last_change":"2023-11-08T05:28:13.557968+0000","last_active":"2023-11-08T05:30:44.490805+0000","last_peered":"2023-11-08T05:30:44.490805+0000","last_clean":"2023-11-08T05:28:01.458433+0000","last_became_active":"2023-11-08T05:28:13.557968+0000","last_became_peered":"2023-11-08T05:28:13.557968+0000","last_unstale":"2023-11-08T05:30:44.490805+0000","last_undegraded":"2023-11-08T05:30:44.490805+0000","last_fullsized":"2023-11-08T05:28:13.542832+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":1,"acting":1,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":12,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738843,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":645040,"kb_used_data":5352,"kb_used_omap":275,"kb_used_meta":639404,"kb_avail":93726800,"statfs":{"total":96636764160,"available":95976243200,"internally_reserved":0,"allocated":5480448,"data_stored":2697647,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":281650,"internal_metadata":654750670},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:59 2023","interfaces":[{"interface":"back","average":{"1min":0.30499999999999999,"5min":0.439,"15min":0.36099999999999999},"min":{"1min":0.17399999999999999,"5min":0.14499999999999999,"15min":0.14499999999999999},"max":{"1min":0.60099999999999998,"5min":4.9379999999999997,"15min":4.9379999999999997},"last":0.379},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.49399999999999999,"15min":0.42399999999999999},"min":{"1min":0.221,"5min":0.19600000000000001,"15min":0.17000000000000001},"max":{"1min":0.63200000000000001,"5min":4.9859999999999998,"15min":4.9859999999999998},"last":0.40899999999999997}]},{"osd":1,"last update":"Wed Nov 8 05:31:06 2023","interfaces":[{"interface":"back","average":{"1min":0.27500000000000002,"5min":0.40300000000000002,"15min":0.46999999999999997},"min":{"1min":0.20899999999999999,"5min":0.16,"15min":0.153},"max":{"1min":0.41099999999999998,"5min":4.0830000000000002,"15min":23.242999999999999},"last":0.36599999999999999},{"interface":"front","average":{"1min":0.29899999999999999,"5min":0.40100000000000002,"15min":0.46000000000000002},"min":{"1min":0.23799999999999999,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":0.45300000000000001,"5min":3.5590000000000002,"15min":23.251000000000001},"last":0.39400000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738798,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1339228,"kb_used_data":398304,"kb_used_omap":291,"kb_used_meta":936476,"kb_avail":93032612,"statfs":{"total":96636764160,"available":95265394688,"internally_reserved":0,"allocated":407863296,"data_stored":405103900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":298462,"internal_metadata":958951970},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.436,"15min":0.36299999999999999},"min":{"1min":0.19700000000000001,"5min":0.14699999999999999,"15min":0.13700000000000001},"max":{"1min":3.222,"5min":6.5499999999999998,"15min":6.9989999999999997},"last":0.245},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.46300000000000002,"15min":0.42299999999999999},"min":{"1min":0.17799999999999999,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":5.0030000000000001,"5min":6.0629999999999997,"15min":9.6869999999999994},"last":0.28599999999999998}]},{"osd":2,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.51600000000000001,"15min":0.47099999999999997},"min":{"1min":0.21099999999999999,"5min":0.153,"15min":0.153},"max":{"1min":5.7439999999999998,"5min":5.8280000000000003,"15min":9.7639999999999993},"last":0.252},{"interface":"front","average":{"1min":0.497,"5min":0.56200000000000006,"15min":0.42299999999999999},"min":{"1min":0.20399999999999999,"5min":0.187,"15min":0.16800000000000001},"max":{"1min":3.9319999999999999,"5min":8.8040000000000003,"15min":8.8040000000000003},"last":0.29899999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738845,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":650100,"kb_used_data":5356,"kb_used_omap":214,"kb_used_meta":644521,"kb_avail":93721740,"statfs":{"total":96636764160,"available":95971061760,"internally_reserved":0,"allocated":5484544,"data_stored":2698616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":219335,"internal_metadata":659990329},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:56 2023","interfaces":[{"interface":"back","average":{"1min":0.30399999999999999,"5min":0.34699999999999998,"15min":0.36499999999999999},"min":{"1min":0.23999999999999999,"5min":0.14199999999999999,"15min":0.14199999999999999},"max":{"1min":0.42399999999999999,"5min":1.595,"15min":7.4249999999999998},"last":0.17100000000000001},{"interface":"front","average":{"1min":0.35499999999999998,"5min":0.40899999999999997,"15min":0.376},"min":{"1min":0.23999999999999999,"5min":0.19600000000000001,"15min":0.183},"max":{"1min":0.46999999999999997,"5min":2.6859999999999999,"15min":2.6859999999999999},"last":0.23000000000000001}]},{"osd":2,"last update":"Wed Nov 8 05:32:02 2023","interfaces":[{"interface":"back","average":{"1min":0.252,"5min":0.33000000000000002,"15min":0.36499999999999999},"min":{"1min":0.161,"5min":0.161,"15min":0.161},"max":{"1min":0.39600000000000002,"5min":2.2650000000000001,"15min":5.8209999999999997},"last":0.249},{"interface":"front","average":{"1min":0.25800000000000001,"5min":0.29899999999999999,"15min":0.35599999999999998},"min":{"1min":0.17000000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.35799999999999998,"5min":1.137,"15min":4.2350000000000003},"last":0.222}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-08T05:32:10.420 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~232s 2023-11-08T05:32:10.864 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-11-08T05:32:11.160 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T05:32:11.162 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2023-11-08T05:32:11.173 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":1639,"stamp":"2023-11-08T05:32:10.077094+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":10,"acting":13,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":109,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2634368,"kb_used_data":409012,"kb_used_omap":780,"kb_used_meta":2220403,"kb_avail":280481152,"statfs":{"total":289910292480,"available":287212699648,"internally_reserved":0,"allocated":418828288,"data_stored":410500163,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":799447,"internal_metadata":2273692969},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-536782902,"num_objects":-164,"num_object_clones":0,"num_object_copies":-328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-115,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-164,"num_whiteouts":0,"num_read":-122056,"num_read_kb":-39011942,"num_write":-413642,"num_write_kb":-151705339,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-8,"num_bytes_recovered":-16922624,"num_keys_recovered":-2,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.424817"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490644+0000","last_change":"2023-11-08T05:28:13.557071+0000","last_active":"2023-11-08T05:30:44.490644+0000","last_peered":"2023-11-08T05:30:44.490644+0000","last_clean":"2023-11-08T05:28:01.458439+0000","last_became_active":"2023-11-08T05:28:13.557071+0000","last_became_peered":"2023-11-08T05:28:13.557071+0000","last_unstale":"2023-11-08T05:30:44.490644+0000","last_undegraded":"2023-11-08T05:30:44.490644+0000","last_fullsized":"2023-11-08T05:28:13.547025+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":459,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:45.794760+0000","last_change":"2023-11-08T05:30:45.794760+0000","last_active":"2023-11-08T05:30:45.794760+0000","last_peered":"2023-11-08T05:30:45.794760+0000","last_clean":"2023-11-08T05:30:45.794760+0000","last_became_active":"2023-11-08T05:30:45.794398+0000","last_became_peered":"2023-11-08T05:30:45.794398+0000","last_unstale":"2023-11-08T05:30:45.794760+0000","last_undegraded":"2023-11-08T05:30:45.794760+0000","last_fullsized":"2023-11-08T05:30:45.794760+0000","mapping_epoch":458,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":459,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":465,"reported_epoch":459,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:45.795085+0000","last_change":"2023-11-08T05:30:45.795085+0000","last_active":"2023-11-08T05:30:45.795085+0000","last_peered":"2023-11-08T05:30:45.795085+0000","last_clean":"2023-11-08T05:30:45.795085+0000","last_became_active":"2023-11-08T05:30:45.794743+0000","last_became_peered":"2023-11-08T05:30:45.794743+0000","last_unstale":"2023-11-08T05:30:45.795085+0000","last_undegraded":"2023-11-08T05:30:45.795085+0000","last_fullsized":"2023-11-08T05:30:45.795085+0000","mapping_epoch":458,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":459,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":454,"reported_epoch":458,"state":"active+clean","last_fresh":"2023-11-08T05:30:44.490618+0000","last_change":"2023-11-08T04:52:47.548597+0000","last_active":"2023-11-08T05:30:44.490618+0000","last_peered":"2023-11-08T05:30:44.490618+0000","last_clean":"2023-11-08T05:30:44.490618+0000","last_became_active":"2023-11-08T04:52:47.548294+0000","last_became_peered":"2023-11-08T04:52:47.548294+0000","last_unstale":"2023-11-08T05:30:44.490618+0000","last_undegraded":"2023-11-08T05:30:44.490618+0000","last_fullsized":"2023-11-08T05:30:44.490618+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":460,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490523+0000","last_change":"2023-11-08T05:28:13.545468+0000","last_active":"2023-11-08T05:30:44.490523+0000","last_peered":"2023-11-08T05:30:44.490523+0000","last_clean":"2023-11-08T05:28:01.456723+0000","last_became_active":"2023-11-08T05:28:13.545468+0000","last_became_peered":"2023-11-08T05:28:13.545468+0000","last_unstale":"2023-11-08T05:30:44.490523+0000","last_undegraded":"2023-11-08T05:30:44.490523+0000","last_fullsized":"2023-11-08T05:28:13.537207+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_clean_scrub_stamp":"2023-11-08T04:52:45.528796+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":465,"reported_epoch":458,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:44.491907+0000","last_change":"2023-11-08T05:29:44.395350+0000","last_active":"2023-11-08T05:30:44.491907+0000","last_peered":"2023-11-08T05:30:44.491907+0000","last_clean":"2023-11-08T05:30:44.491907+0000","last_became_active":"2023-11-08T05:29:44.395012+0000","last_became_peered":"2023-11-08T05:29:44.395012+0000","last_unstale":"2023-11-08T05:30:44.491907+0000","last_undegraded":"2023-11-08T05:30:44.491907+0000","last_fullsized":"2023-11-08T05:30:44.491907+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490629+0000","last_change":"2023-11-08T05:28:13.563357+0000","last_active":"2023-11-08T05:30:44.490629+0000","last_peered":"2023-11-08T05:30:44.490629+0000","last_clean":"2023-11-08T05:28:01.457691+0000","last_became_active":"2023-11-08T05:28:13.563357+0000","last_became_peered":"2023-11-08T05:28:13.563357+0000","last_unstale":"2023-11-08T05:30:44.490629+0000","last_undegraded":"2023-11-08T05:30:44.490629+0000","last_fullsized":"2023-11-08T05:28:13.541614+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490691+0000","last_change":"2023-11-08T05:28:13.550672+0000","last_active":"2023-11-08T05:30:44.490691+0000","last_peered":"2023-11-08T05:30:44.490691+0000","last_clean":"2023-11-08T05:28:01.457904+0000","last_became_active":"2023-11-08T05:28:13.550672+0000","last_became_peered":"2023-11-08T05:28:13.550672+0000","last_unstale":"2023-11-08T05:30:44.490691+0000","last_undegraded":"2023-11-08T05:30:44.490691+0000","last_fullsized":"2023-11-08T05:28:13.539595+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490805+0000","last_change":"2023-11-08T05:28:13.557968+0000","last_active":"2023-11-08T05:30:44.490805+0000","last_peered":"2023-11-08T05:30:44.490805+0000","last_clean":"2023-11-08T05:28:01.458433+0000","last_became_active":"2023-11-08T05:28:13.557968+0000","last_became_peered":"2023-11-08T05:28:13.557968+0000","last_unstale":"2023-11-08T05:30:44.490805+0000","last_undegraded":"2023-11-08T05:30:44.490805+0000","last_fullsized":"2023-11-08T05:28:13.542832+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":1,"acting":1,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":12,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738844,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":645040,"kb_used_data":5352,"kb_used_omap":275,"kb_used_meta":639404,"kb_avail":93726800,"statfs":{"total":96636764160,"available":95976243200,"internally_reserved":0,"allocated":5480448,"data_stored":2697647,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":281650,"internal_metadata":654750670},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:59 2023","interfaces":[{"interface":"back","average":{"1min":0.30499999999999999,"5min":0.439,"15min":0.36099999999999999},"min":{"1min":0.17399999999999999,"5min":0.14499999999999999,"15min":0.14499999999999999},"max":{"1min":0.60099999999999998,"5min":4.9379999999999997,"15min":4.9379999999999997},"last":0.379},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.49399999999999999,"15min":0.42399999999999999},"min":{"1min":0.221,"5min":0.19600000000000001,"15min":0.17000000000000001},"max":{"1min":0.63200000000000001,"5min":4.9859999999999998,"15min":4.9859999999999998},"last":0.40899999999999997}]},{"osd":1,"last update":"Wed Nov 8 05:32:06 2023","interfaces":[{"interface":"back","average":{"1min":0.27400000000000002,"5min":0.307,"15min":0.46800000000000003},"min":{"1min":0.17599999999999999,"5min":0.16,"15min":0.153},"max":{"1min":0.39000000000000001,"5min":1.825,"15min":23.242999999999999},"last":0.22700000000000001},{"interface":"front","average":{"1min":0.28000000000000003,"5min":0.34799999999999998,"15min":0.45700000000000002},"min":{"1min":0.20200000000000001,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":0.41699999999999998,"5min":2.0419999999999998,"15min":23.251000000000001},"last":0.217}]}]},{"osd":0,"up_from":8,"seq":34359738798,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1339228,"kb_used_data":398304,"kb_used_omap":291,"kb_used_meta":936476,"kb_avail":93032612,"statfs":{"total":96636764160,"available":95265394688,"internally_reserved":0,"allocated":407863296,"data_stored":405103900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":298462,"internal_metadata":958951970},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.436,"15min":0.36299999999999999},"min":{"1min":0.19700000000000001,"5min":0.14699999999999999,"15min":0.13700000000000001},"max":{"1min":3.222,"5min":6.5499999999999998,"15min":6.9989999999999997},"last":0.245},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.46300000000000002,"15min":0.42299999999999999},"min":{"1min":0.17799999999999999,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":5.0030000000000001,"5min":6.0629999999999997,"15min":9.6869999999999994},"last":0.28599999999999998}]},{"osd":2,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.51600000000000001,"15min":0.47099999999999997},"min":{"1min":0.21099999999999999,"5min":0.153,"15min":0.153},"max":{"1min":5.7439999999999998,"5min":5.8280000000000003,"15min":9.7639999999999993},"last":0.252},{"interface":"front","average":{"1min":0.497,"5min":0.56200000000000006,"15min":0.42299999999999999},"min":{"1min":0.20399999999999999,"5min":0.187,"15min":0.16800000000000001},"max":{"1min":3.9319999999999999,"5min":8.8040000000000003,"15min":8.8040000000000003},"last":0.29899999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738845,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":650100,"kb_used_data":5356,"kb_used_omap":214,"kb_used_meta":644521,"kb_avail":93721740,"statfs":{"total":96636764160,"available":95971061760,"internally_reserved":0,"allocated":5484544,"data_stored":2698616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":219335,"internal_metadata":659990329},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:56 2023","interfaces":[{"interface":"back","average":{"1min":0.30399999999999999,"5min":0.34699999999999998,"15min":0.36499999999999999},"min":{"1min":0.23999999999999999,"5min":0.14199999999999999,"15min":0.14199999999999999},"max":{"1min":0.42399999999999999,"5min":1.595,"15min":7.4249999999999998},"last":0.17100000000000001},{"interface":"front","average":{"1min":0.35499999999999998,"5min":0.40899999999999997,"15min":0.376},"min":{"1min":0.23999999999999999,"5min":0.19600000000000001,"15min":0.183},"max":{"1min":0.46999999999999997,"5min":2.6859999999999999,"15min":2.6859999999999999},"last":0.23000000000000001}]},{"osd":2,"last update":"Wed Nov 8 05:32:02 2023","interfaces":[{"interface":"back","average":{"1min":0.252,"5min":0.33000000000000002,"15min":0.36499999999999999},"min":{"1min":0.161,"5min":0.161,"15min":0.161},"max":{"1min":0.39600000000000002,"5min":2.2650000000000001,"15min":5.8209999999999997},"last":0.249},{"interface":"front","average":{"1min":0.25800000000000001,"5min":0.29899999999999999,"15min":0.35599999999999998},"min":{"1min":0.17000000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.35799999999999998,"5min":1.137,"15min":4.2350000000000003},"last":0.222}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-08T05:32:11.174 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-11-08T05:32:11.456 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T05:32:11.457 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2023-11-08T05:32:11.468 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":1639,"stamp":"2023-11-08T05:32:10.077094+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":10,"acting":13,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":109,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2634368,"kb_used_data":409012,"kb_used_omap":780,"kb_used_meta":2220403,"kb_avail":280481152,"statfs":{"total":289910292480,"available":287212699648,"internally_reserved":0,"allocated":418828288,"data_stored":410500163,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":799447,"internal_metadata":2273692969},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-536782902,"num_objects":-164,"num_object_clones":0,"num_object_copies":-328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-115,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-164,"num_whiteouts":0,"num_read":-122056,"num_read_kb":-39011942,"num_write":-413642,"num_write_kb":-151705339,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-8,"num_bytes_recovered":-16922624,"num_keys_recovered":-2,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.424817"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490644+0000","last_change":"2023-11-08T05:28:13.557071+0000","last_active":"2023-11-08T05:30:44.490644+0000","last_peered":"2023-11-08T05:30:44.490644+0000","last_clean":"2023-11-08T05:28:01.458439+0000","last_became_active":"2023-11-08T05:28:13.557071+0000","last_became_peered":"2023-11-08T05:28:13.557071+0000","last_unstale":"2023-11-08T05:30:44.490644+0000","last_undegraded":"2023-11-08T05:30:44.490644+0000","last_fullsized":"2023-11-08T05:28:13.547025+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":459,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:45.794760+0000","last_change":"2023-11-08T05:30:45.794760+0000","last_active":"2023-11-08T05:30:45.794760+0000","last_peered":"2023-11-08T05:30:45.794760+0000","last_clean":"2023-11-08T05:30:45.794760+0000","last_became_active":"2023-11-08T05:30:45.794398+0000","last_became_peered":"2023-11-08T05:30:45.794398+0000","last_unstale":"2023-11-08T05:30:45.794760+0000","last_undegraded":"2023-11-08T05:30:45.794760+0000","last_fullsized":"2023-11-08T05:30:45.794760+0000","mapping_epoch":458,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":459,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":465,"reported_epoch":459,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:45.795085+0000","last_change":"2023-11-08T05:30:45.795085+0000","last_active":"2023-11-08T05:30:45.795085+0000","last_peered":"2023-11-08T05:30:45.795085+0000","last_clean":"2023-11-08T05:30:45.795085+0000","last_became_active":"2023-11-08T05:30:45.794743+0000","last_became_peered":"2023-11-08T05:30:45.794743+0000","last_unstale":"2023-11-08T05:30:45.795085+0000","last_undegraded":"2023-11-08T05:30:45.795085+0000","last_fullsized":"2023-11-08T05:30:45.795085+0000","mapping_epoch":458,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":459,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":454,"reported_epoch":458,"state":"active+clean","last_fresh":"2023-11-08T05:30:44.490618+0000","last_change":"2023-11-08T04:52:47.548597+0000","last_active":"2023-11-08T05:30:44.490618+0000","last_peered":"2023-11-08T05:30:44.490618+0000","last_clean":"2023-11-08T05:30:44.490618+0000","last_became_active":"2023-11-08T04:52:47.548294+0000","last_became_peered":"2023-11-08T04:52:47.548294+0000","last_unstale":"2023-11-08T05:30:44.490618+0000","last_undegraded":"2023-11-08T05:30:44.490618+0000","last_fullsized":"2023-11-08T05:30:44.490618+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":460,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490523+0000","last_change":"2023-11-08T05:28:13.545468+0000","last_active":"2023-11-08T05:30:44.490523+0000","last_peered":"2023-11-08T05:30:44.490523+0000","last_clean":"2023-11-08T05:28:01.456723+0000","last_became_active":"2023-11-08T05:28:13.545468+0000","last_became_peered":"2023-11-08T05:28:13.545468+0000","last_unstale":"2023-11-08T05:30:44.490523+0000","last_undegraded":"2023-11-08T05:30:44.490523+0000","last_fullsized":"2023-11-08T05:28:13.537207+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_clean_scrub_stamp":"2023-11-08T04:52:45.528796+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":465,"reported_epoch":458,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:44.491907+0000","last_change":"2023-11-08T05:29:44.395350+0000","last_active":"2023-11-08T05:30:44.491907+0000","last_peered":"2023-11-08T05:30:44.491907+0000","last_clean":"2023-11-08T05:30:44.491907+0000","last_became_active":"2023-11-08T05:29:44.395012+0000","last_became_peered":"2023-11-08T05:29:44.395012+0000","last_unstale":"2023-11-08T05:30:44.491907+0000","last_undegraded":"2023-11-08T05:30:44.491907+0000","last_fullsized":"2023-11-08T05:30:44.491907+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490629+0000","last_change":"2023-11-08T05:28:13.563357+0000","last_active":"2023-11-08T05:30:44.490629+0000","last_peered":"2023-11-08T05:30:44.490629+0000","last_clean":"2023-11-08T05:28:01.457691+0000","last_became_active":"2023-11-08T05:28:13.563357+0000","last_became_peered":"2023-11-08T05:28:13.563357+0000","last_unstale":"2023-11-08T05:30:44.490629+0000","last_undegraded":"2023-11-08T05:30:44.490629+0000","last_fullsized":"2023-11-08T05:28:13.541614+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490691+0000","last_change":"2023-11-08T05:28:13.550672+0000","last_active":"2023-11-08T05:30:44.490691+0000","last_peered":"2023-11-08T05:30:44.490691+0000","last_clean":"2023-11-08T05:28:01.457904+0000","last_became_active":"2023-11-08T05:28:13.550672+0000","last_became_peered":"2023-11-08T05:28:13.550672+0000","last_unstale":"2023-11-08T05:30:44.490691+0000","last_undegraded":"2023-11-08T05:30:44.490691+0000","last_fullsized":"2023-11-08T05:28:13.539595+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490805+0000","last_change":"2023-11-08T05:28:13.557968+0000","last_active":"2023-11-08T05:30:44.490805+0000","last_peered":"2023-11-08T05:30:44.490805+0000","last_clean":"2023-11-08T05:28:01.458433+0000","last_became_active":"2023-11-08T05:28:13.557968+0000","last_became_peered":"2023-11-08T05:28:13.557968+0000","last_unstale":"2023-11-08T05:30:44.490805+0000","last_undegraded":"2023-11-08T05:30:44.490805+0000","last_fullsized":"2023-11-08T05:28:13.542832+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":1,"acting":1,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":12,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738844,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":645040,"kb_used_data":5352,"kb_used_omap":275,"kb_used_meta":639404,"kb_avail":93726800,"statfs":{"total":96636764160,"available":95976243200,"internally_reserved":0,"allocated":5480448,"data_stored":2697647,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":281650,"internal_metadata":654750670},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:59 2023","interfaces":[{"interface":"back","average":{"1min":0.30499999999999999,"5min":0.439,"15min":0.36099999999999999},"min":{"1min":0.17399999999999999,"5min":0.14499999999999999,"15min":0.14499999999999999},"max":{"1min":0.60099999999999998,"5min":4.9379999999999997,"15min":4.9379999999999997},"last":0.379},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.49399999999999999,"15min":0.42399999999999999},"min":{"1min":0.221,"5min":0.19600000000000001,"15min":0.17000000000000001},"max":{"1min":0.63200000000000001,"5min":4.9859999999999998,"15min":4.9859999999999998},"last":0.40899999999999997}]},{"osd":1,"last update":"Wed Nov 8 05:32:06 2023","interfaces":[{"interface":"back","average":{"1min":0.27400000000000002,"5min":0.307,"15min":0.46800000000000003},"min":{"1min":0.17599999999999999,"5min":0.16,"15min":0.153},"max":{"1min":0.39000000000000001,"5min":1.825,"15min":23.242999999999999},"last":0.22700000000000001},{"interface":"front","average":{"1min":0.28000000000000003,"5min":0.34799999999999998,"15min":0.45700000000000002},"min":{"1min":0.20200000000000001,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":0.41699999999999998,"5min":2.0419999999999998,"15min":23.251000000000001},"last":0.217}]}]},{"osd":0,"up_from":8,"seq":34359738798,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1339228,"kb_used_data":398304,"kb_used_omap":291,"kb_used_meta":936476,"kb_avail":93032612,"statfs":{"total":96636764160,"available":95265394688,"internally_reserved":0,"allocated":407863296,"data_stored":405103900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":298462,"internal_metadata":958951970},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.436,"15min":0.36299999999999999},"min":{"1min":0.19700000000000001,"5min":0.14699999999999999,"15min":0.13700000000000001},"max":{"1min":3.222,"5min":6.5499999999999998,"15min":6.9989999999999997},"last":0.245},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.46300000000000002,"15min":0.42299999999999999},"min":{"1min":0.17799999999999999,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":5.0030000000000001,"5min":6.0629999999999997,"15min":9.6869999999999994},"last":0.28599999999999998}]},{"osd":2,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.51600000000000001,"15min":0.47099999999999997},"min":{"1min":0.21099999999999999,"5min":0.153,"15min":0.153},"max":{"1min":5.7439999999999998,"5min":5.8280000000000003,"15min":9.7639999999999993},"last":0.252},{"interface":"front","average":{"1min":0.497,"5min":0.56200000000000006,"15min":0.42299999999999999},"min":{"1min":0.20399999999999999,"5min":0.187,"15min":0.16800000000000001},"max":{"1min":3.9319999999999999,"5min":8.8040000000000003,"15min":8.8040000000000003},"last":0.29899999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738845,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":650100,"kb_used_data":5356,"kb_used_omap":214,"kb_used_meta":644521,"kb_avail":93721740,"statfs":{"total":96636764160,"available":95971061760,"internally_reserved":0,"allocated":5484544,"data_stored":2698616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":219335,"internal_metadata":659990329},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:56 2023","interfaces":[{"interface":"back","average":{"1min":0.30399999999999999,"5min":0.34699999999999998,"15min":0.36499999999999999},"min":{"1min":0.23999999999999999,"5min":0.14199999999999999,"15min":0.14199999999999999},"max":{"1min":0.42399999999999999,"5min":1.595,"15min":7.4249999999999998},"last":0.17100000000000001},{"interface":"front","average":{"1min":0.35499999999999998,"5min":0.40899999999999997,"15min":0.376},"min":{"1min":0.23999999999999999,"5min":0.19600000000000001,"15min":0.183},"max":{"1min":0.46999999999999997,"5min":2.6859999999999999,"15min":2.6859999999999999},"last":0.23000000000000001}]},{"osd":2,"last update":"Wed Nov 8 05:32:02 2023","interfaces":[{"interface":"back","average":{"1min":0.252,"5min":0.33000000000000002,"15min":0.36499999999999999},"min":{"1min":0.161,"5min":0.161,"15min":0.161},"max":{"1min":0.39600000000000002,"5min":2.2650000000000001,"15min":5.8209999999999997},"last":0.249},{"interface":"front","average":{"1min":0.25800000000000001,"5min":0.29899999999999999,"15min":0.35599999999999998},"min":{"1min":0.17000000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.35799999999999998,"5min":1.137,"15min":4.2350000000000003},"last":0.222}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-08T05:32:11.470 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-11-08T05:32:11.471 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 459, 'reported_epoch': 458, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:30:44.490644+0000', 'last_change': '2023-11-08T05:28:13.557071+0000', 'last_active': '2023-11-08T05:30:44.490644+0000', 'last_peered': '2023-11-08T05:30:44.490644+0000', 'last_clean': '2023-11-08T05:28:01.458439+0000', 'last_became_active': '2023-11-08T05:28:13.557071+0000', 'last_became_peered': '2023-11-08T05:28:13.557071+0000', 'last_unstale': '2023-11-08T05:30:44.490644+0000', 'last_undegraded': '2023-11-08T05:30:44.490644+0000', 'last_fullsized': '2023-11-08T05:28:13.547025+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:32:11.471 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-11-08T05:32:11.472 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 11, 'reported_epoch': 459, 'state': 'active+clean+remapped', 'last_fresh': '2023-11-08T05:30:45.794760+0000', 'last_change': '2023-11-08T05:30:45.794760+0000', 'last_active': '2023-11-08T05:30:45.794760+0000', 'last_peered': '2023-11-08T05:30:45.794760+0000', 'last_clean': '2023-11-08T05:30:45.794760+0000', 'last_became_active': '2023-11-08T05:30:45.794398+0000', 'last_became_peered': '2023-11-08T05:30:45.794398+0000', 'last_unstale': '2023-11-08T05:30:45.794760+0000', 'last_undegraded': '2023-11-08T05:30:45.794760+0000', 'last_fullsized': '2023-11-08T05:30:45.794760+0000', 'mapping_epoch': 458, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 459, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2], 'acting': [2, 1], 'avail_no_missing': ['2', '1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2023-11-08T05:32:11.472 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.1 is not active+clean 2023-11-08T05:32:11.472 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.1', 'version': "0'0", 'reported_seq': 465, 'reported_epoch': 459, 'state': 'active+clean+remapped', 'last_fresh': '2023-11-08T05:30:45.795085+0000', 'last_change': '2023-11-08T05:30:45.795085+0000', 'last_active': '2023-11-08T05:30:45.795085+0000', 'last_peered': '2023-11-08T05:30:45.795085+0000', 'last_clean': '2023-11-08T05:30:45.795085+0000', 'last_became_active': '2023-11-08T05:30:45.794743+0000', 'last_became_peered': '2023-11-08T05:30:45.794743+0000', 'last_unstale': '2023-11-08T05:30:45.795085+0000', 'last_undegraded': '2023-11-08T05:30:45.795085+0000', 'last_fullsized': '2023-11-08T05:30:45.795085+0000', 'mapping_epoch': 458, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 459, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2], 'acting': [2, 1], 'avail_no_missing': ['2', '1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2023-11-08T05:32:11.473 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-11-08T05:32:11.473 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "0'0", 'reported_seq': 460, 'reported_epoch': 458, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:30:44.490523+0000', 'last_change': '2023-11-08T05:28:13.545468+0000', 'last_active': '2023-11-08T05:30:44.490523+0000', 'last_peered': '2023-11-08T05:30:44.490523+0000', 'last_clean': '2023-11-08T05:28:01.456723+0000', 'last_became_active': '2023-11-08T05:28:13.545468+0000', 'last_became_peered': '2023-11-08T05:28:13.545468+0000', 'last_unstale': '2023-11-08T05:30:44.490523+0000', 'last_undegraded': '2023-11-08T05:30:44.490523+0000', 'last_fullsized': '2023-11-08T05:28:13.537207+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 9, 'last_epoch_clean': 10, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:45.528796+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:45.528796+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:45.528796+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:32:11.474 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.0 is not active+clean 2023-11-08T05:32:11.474 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.0', 'version': "0'0", 'reported_seq': 465, 'reported_epoch': 458, 'state': 'active+clean+remapped', 'last_fresh': '2023-11-08T05:30:44.491907+0000', 'last_change': '2023-11-08T05:29:44.395350+0000', 'last_active': '2023-11-08T05:30:44.491907+0000', 'last_peered': '2023-11-08T05:30:44.491907+0000', 'last_clean': '2023-11-08T05:30:44.491907+0000', 'last_became_active': '2023-11-08T05:29:44.395012+0000', 'last_became_peered': '2023-11-08T05:29:44.395012+0000', 'last_unstale': '2023-11-08T05:30:44.491907+0000', 'last_undegraded': '2023-11-08T05:30:44.491907+0000', 'last_fullsized': '2023-11-08T05:30:44.491907+0000', 'mapping_epoch': 455, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 456, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2], 'acting': [2, 1], 'avail_no_missing': ['2', '1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2023-11-08T05:32:11.475 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-11-08T05:32:11.475 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 459, 'reported_epoch': 458, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:30:44.490629+0000', 'last_change': '2023-11-08T05:28:13.563357+0000', 'last_active': '2023-11-08T05:30:44.490629+0000', 'last_peered': '2023-11-08T05:30:44.490629+0000', 'last_clean': '2023-11-08T05:28:01.457691+0000', 'last_became_active': '2023-11-08T05:28:13.563357+0000', 'last_became_peered': '2023-11-08T05:28:13.563357+0000', 'last_unstale': '2023-11-08T05:30:44.490629+0000', 'last_undegraded': '2023-11-08T05:30:44.490629+0000', 'last_fullsized': '2023-11-08T05:28:13.541614+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:32:11.476 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-11-08T05:32:11.476 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 459, 'reported_epoch': 458, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:30:44.490691+0000', 'last_change': '2023-11-08T05:28:13.550672+0000', 'last_active': '2023-11-08T05:30:44.490691+0000', 'last_peered': '2023-11-08T05:30:44.490691+0000', 'last_clean': '2023-11-08T05:28:01.457904+0000', 'last_became_active': '2023-11-08T05:28:13.550672+0000', 'last_became_peered': '2023-11-08T05:28:13.550672+0000', 'last_unstale': '2023-11-08T05:30:44.490691+0000', 'last_undegraded': '2023-11-08T05:30:44.490691+0000', 'last_fullsized': '2023-11-08T05:28:13.539595+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:32:11.476 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-11-08T05:32:11.477 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 459, 'reported_epoch': 458, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:30:44.490805+0000', 'last_change': '2023-11-08T05:28:13.557968+0000', 'last_active': '2023-11-08T05:30:44.490805+0000', 'last_peered': '2023-11-08T05:30:44.490805+0000', 'last_clean': '2023-11-08T05:28:01.458433+0000', 'last_became_active': '2023-11-08T05:28:13.557968+0000', 'last_became_peered': '2023-11-08T05:28:13.557968+0000', 'last_unstale': '2023-11-08T05:30:44.490805+0000', 'last_undegraded': '2023-11-08T05:30:44.490805+0000', 'last_fullsized': '2023-11-08T05:28:13.542832+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:32:11.477 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-11-08T05:32:11.823 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T05:32:11.823 INFO:teuthology.orchestra.run.smithi087.stdout:{"fsid":"ab4cbc06-449d-497d-b513-220d9db2b590","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 5 pgs undersized","count":5},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2374,"monmap":{"epoch":1,"min_mon_release_name":"pacific","num_mons":1},"osdmap":{"epoch":459,"num_osds":3,"num_up_osds":2,"osd_up_since":1699421292,"num_in_osds":3,"osd_in_since":1699419158,"num_remapped_pgs":3},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean+remapped","count":3},{"state_name":"active+clean","count":1}],"num_pgs":9,"num_pools":2,"num_objects":0,"data_bytes":0,"bytes_used":2697592832,"bytes_avail":287212699648,"bytes_total":289910292480},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":41,"modified":"2023-11-08T05:29:32.044544+0000","services":{}},"progress_events":{}} 2023-11-08T05:32:11.836 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-11-08T05:32:11.837 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-11-08T05:32:12.118 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T05:32:12.119 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2023-11-08T05:32:12.130 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":1640,"stamp":"2023-11-08T05:32:12.077594+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":10,"acting":13,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":109,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2634368,"kb_used_data":409012,"kb_used_omap":780,"kb_used_meta":2220403,"kb_avail":280481152,"statfs":{"total":289910292480,"available":287212699648,"internally_reserved":0,"allocated":418828288,"data_stored":410500163,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":799447,"internal_metadata":2273692969},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-536782902,"num_objects":-164,"num_object_clones":0,"num_object_copies":-328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-115,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-164,"num_whiteouts":0,"num_read":-122056,"num_read_kb":-39011942,"num_write":-413642,"num_write_kb":-151705339,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-8,"num_bytes_recovered":-16922624,"num_keys_recovered":-2,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.424817"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490644+0000","last_change":"2023-11-08T05:28:13.557071+0000","last_active":"2023-11-08T05:30:44.490644+0000","last_peered":"2023-11-08T05:30:44.490644+0000","last_clean":"2023-11-08T05:28:01.458439+0000","last_became_active":"2023-11-08T05:28:13.557071+0000","last_became_peered":"2023-11-08T05:28:13.557071+0000","last_unstale":"2023-11-08T05:30:44.490644+0000","last_undegraded":"2023-11-08T05:30:44.490644+0000","last_fullsized":"2023-11-08T05:28:13.547025+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":459,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:45.794760+0000","last_change":"2023-11-08T05:30:45.794760+0000","last_active":"2023-11-08T05:30:45.794760+0000","last_peered":"2023-11-08T05:30:45.794760+0000","last_clean":"2023-11-08T05:30:45.794760+0000","last_became_active":"2023-11-08T05:30:45.794398+0000","last_became_peered":"2023-11-08T05:30:45.794398+0000","last_unstale":"2023-11-08T05:30:45.794760+0000","last_undegraded":"2023-11-08T05:30:45.794760+0000","last_fullsized":"2023-11-08T05:30:45.794760+0000","mapping_epoch":458,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":459,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":465,"reported_epoch":459,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:45.795085+0000","last_change":"2023-11-08T05:30:45.795085+0000","last_active":"2023-11-08T05:30:45.795085+0000","last_peered":"2023-11-08T05:30:45.795085+0000","last_clean":"2023-11-08T05:30:45.795085+0000","last_became_active":"2023-11-08T05:30:45.794743+0000","last_became_peered":"2023-11-08T05:30:45.794743+0000","last_unstale":"2023-11-08T05:30:45.795085+0000","last_undegraded":"2023-11-08T05:30:45.795085+0000","last_fullsized":"2023-11-08T05:30:45.795085+0000","mapping_epoch":458,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":459,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":454,"reported_epoch":458,"state":"active+clean","last_fresh":"2023-11-08T05:30:44.490618+0000","last_change":"2023-11-08T04:52:47.548597+0000","last_active":"2023-11-08T05:30:44.490618+0000","last_peered":"2023-11-08T05:30:44.490618+0000","last_clean":"2023-11-08T05:30:44.490618+0000","last_became_active":"2023-11-08T04:52:47.548294+0000","last_became_peered":"2023-11-08T04:52:47.548294+0000","last_unstale":"2023-11-08T05:30:44.490618+0000","last_undegraded":"2023-11-08T05:30:44.490618+0000","last_fullsized":"2023-11-08T05:30:44.490618+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":460,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490523+0000","last_change":"2023-11-08T05:28:13.545468+0000","last_active":"2023-11-08T05:30:44.490523+0000","last_peered":"2023-11-08T05:30:44.490523+0000","last_clean":"2023-11-08T05:28:01.456723+0000","last_became_active":"2023-11-08T05:28:13.545468+0000","last_became_peered":"2023-11-08T05:28:13.545468+0000","last_unstale":"2023-11-08T05:30:44.490523+0000","last_undegraded":"2023-11-08T05:30:44.490523+0000","last_fullsized":"2023-11-08T05:28:13.537207+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_clean_scrub_stamp":"2023-11-08T04:52:45.528796+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":465,"reported_epoch":458,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:44.491907+0000","last_change":"2023-11-08T05:29:44.395350+0000","last_active":"2023-11-08T05:30:44.491907+0000","last_peered":"2023-11-08T05:30:44.491907+0000","last_clean":"2023-11-08T05:30:44.491907+0000","last_became_active":"2023-11-08T05:29:44.395012+0000","last_became_peered":"2023-11-08T05:29:44.395012+0000","last_unstale":"2023-11-08T05:30:44.491907+0000","last_undegraded":"2023-11-08T05:30:44.491907+0000","last_fullsized":"2023-11-08T05:30:44.491907+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490629+0000","last_change":"2023-11-08T05:28:13.563357+0000","last_active":"2023-11-08T05:30:44.490629+0000","last_peered":"2023-11-08T05:30:44.490629+0000","last_clean":"2023-11-08T05:28:01.457691+0000","last_became_active":"2023-11-08T05:28:13.563357+0000","last_became_peered":"2023-11-08T05:28:13.563357+0000","last_unstale":"2023-11-08T05:30:44.490629+0000","last_undegraded":"2023-11-08T05:30:44.490629+0000","last_fullsized":"2023-11-08T05:28:13.541614+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490691+0000","last_change":"2023-11-08T05:28:13.550672+0000","last_active":"2023-11-08T05:30:44.490691+0000","last_peered":"2023-11-08T05:30:44.490691+0000","last_clean":"2023-11-08T05:28:01.457904+0000","last_became_active":"2023-11-08T05:28:13.550672+0000","last_became_peered":"2023-11-08T05:28:13.550672+0000","last_unstale":"2023-11-08T05:30:44.490691+0000","last_undegraded":"2023-11-08T05:30:44.490691+0000","last_fullsized":"2023-11-08T05:28:13.539595+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490805+0000","last_change":"2023-11-08T05:28:13.557968+0000","last_active":"2023-11-08T05:30:44.490805+0000","last_peered":"2023-11-08T05:30:44.490805+0000","last_clean":"2023-11-08T05:28:01.458433+0000","last_became_active":"2023-11-08T05:28:13.557968+0000","last_became_peered":"2023-11-08T05:28:13.557968+0000","last_unstale":"2023-11-08T05:30:44.490805+0000","last_undegraded":"2023-11-08T05:30:44.490805+0000","last_fullsized":"2023-11-08T05:28:13.542832+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":1,"acting":1,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":12,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738844,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":645040,"kb_used_data":5352,"kb_used_omap":275,"kb_used_meta":639404,"kb_avail":93726800,"statfs":{"total":96636764160,"available":95976243200,"internally_reserved":0,"allocated":5480448,"data_stored":2697647,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":281650,"internal_metadata":654750670},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:59 2023","interfaces":[{"interface":"back","average":{"1min":0.30499999999999999,"5min":0.439,"15min":0.36099999999999999},"min":{"1min":0.17399999999999999,"5min":0.14499999999999999,"15min":0.14499999999999999},"max":{"1min":0.60099999999999998,"5min":4.9379999999999997,"15min":4.9379999999999997},"last":0.379},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.49399999999999999,"15min":0.42399999999999999},"min":{"1min":0.221,"5min":0.19600000000000001,"15min":0.17000000000000001},"max":{"1min":0.63200000000000001,"5min":4.9859999999999998,"15min":4.9859999999999998},"last":0.40899999999999997}]},{"osd":1,"last update":"Wed Nov 8 05:32:06 2023","interfaces":[{"interface":"back","average":{"1min":0.27400000000000002,"5min":0.307,"15min":0.46800000000000003},"min":{"1min":0.17599999999999999,"5min":0.16,"15min":0.153},"max":{"1min":0.39000000000000001,"5min":1.825,"15min":23.242999999999999},"last":0.22700000000000001},{"interface":"front","average":{"1min":0.28000000000000003,"5min":0.34799999999999998,"15min":0.45700000000000002},"min":{"1min":0.20200000000000001,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":0.41699999999999998,"5min":2.0419999999999998,"15min":23.251000000000001},"last":0.217}]}]},{"osd":0,"up_from":8,"seq":34359738798,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1339228,"kb_used_data":398304,"kb_used_omap":291,"kb_used_meta":936476,"kb_avail":93032612,"statfs":{"total":96636764160,"available":95265394688,"internally_reserved":0,"allocated":407863296,"data_stored":405103900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":298462,"internal_metadata":958951970},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.436,"15min":0.36299999999999999},"min":{"1min":0.19700000000000001,"5min":0.14699999999999999,"15min":0.13700000000000001},"max":{"1min":3.222,"5min":6.5499999999999998,"15min":6.9989999999999997},"last":0.245},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.46300000000000002,"15min":0.42299999999999999},"min":{"1min":0.17799999999999999,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":5.0030000000000001,"5min":6.0629999999999997,"15min":9.6869999999999994},"last":0.28599999999999998}]},{"osd":2,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.51600000000000001,"15min":0.47099999999999997},"min":{"1min":0.21099999999999999,"5min":0.153,"15min":0.153},"max":{"1min":5.7439999999999998,"5min":5.8280000000000003,"15min":9.7639999999999993},"last":0.252},{"interface":"front","average":{"1min":0.497,"5min":0.56200000000000006,"15min":0.42299999999999999},"min":{"1min":0.20399999999999999,"5min":0.187,"15min":0.16800000000000001},"max":{"1min":3.9319999999999999,"5min":8.8040000000000003,"15min":8.8040000000000003},"last":0.29899999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738846,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":650100,"kb_used_data":5356,"kb_used_omap":214,"kb_used_meta":644521,"kb_avail":93721740,"statfs":{"total":96636764160,"available":95971061760,"internally_reserved":0,"allocated":5484544,"data_stored":2698616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":219335,"internal_metadata":659990329},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:56 2023","interfaces":[{"interface":"back","average":{"1min":0.30399999999999999,"5min":0.34699999999999998,"15min":0.36499999999999999},"min":{"1min":0.23999999999999999,"5min":0.14199999999999999,"15min":0.14199999999999999},"max":{"1min":0.42399999999999999,"5min":1.595,"15min":7.4249999999999998},"last":0.17100000000000001},{"interface":"front","average":{"1min":0.35499999999999998,"5min":0.40899999999999997,"15min":0.376},"min":{"1min":0.23999999999999999,"5min":0.19600000000000001,"15min":0.183},"max":{"1min":0.46999999999999997,"5min":2.6859999999999999,"15min":2.6859999999999999},"last":0.23000000000000001}]},{"osd":2,"last update":"Wed Nov 8 05:32:02 2023","interfaces":[{"interface":"back","average":{"1min":0.252,"5min":0.33000000000000002,"15min":0.36499999999999999},"min":{"1min":0.161,"5min":0.161,"15min":0.161},"max":{"1min":0.39600000000000002,"5min":2.2650000000000001,"15min":5.8209999999999997},"last":0.29999999999999999},{"interface":"front","average":{"1min":0.25800000000000001,"5min":0.29899999999999999,"15min":0.35599999999999998},"min":{"1min":0.17000000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.35799999999999998,"5min":1.137,"15min":4.2350000000000003},"last":0.28100000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-08T05:32:15.132 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-11-08T05:32:15.428 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T05:32:15.429 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2023-11-08T05:32:15.440 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":1641,"stamp":"2023-11-08T05:32:14.078126+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":10,"acting":13,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":109,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2634368,"kb_used_data":409012,"kb_used_omap":780,"kb_used_meta":2220403,"kb_avail":280481152,"statfs":{"total":289910292480,"available":287212699648,"internally_reserved":0,"allocated":418828288,"data_stored":410500163,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":799447,"internal_metadata":2273692969},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-536782902,"num_objects":-164,"num_object_clones":0,"num_object_copies":-328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-115,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-164,"num_whiteouts":0,"num_read":-122056,"num_read_kb":-39011942,"num_write":-413642,"num_write_kb":-151705339,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-8,"num_bytes_recovered":-16922624,"num_keys_recovered":-2,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.424817"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490644+0000","last_change":"2023-11-08T05:28:13.557071+0000","last_active":"2023-11-08T05:30:44.490644+0000","last_peered":"2023-11-08T05:30:44.490644+0000","last_clean":"2023-11-08T05:28:01.458439+0000","last_became_active":"2023-11-08T05:28:13.557071+0000","last_became_peered":"2023-11-08T05:28:13.557071+0000","last_unstale":"2023-11-08T05:30:44.490644+0000","last_undegraded":"2023-11-08T05:30:44.490644+0000","last_fullsized":"2023-11-08T05:28:13.547025+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":459,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:45.794760+0000","last_change":"2023-11-08T05:30:45.794760+0000","last_active":"2023-11-08T05:30:45.794760+0000","last_peered":"2023-11-08T05:30:45.794760+0000","last_clean":"2023-11-08T05:30:45.794760+0000","last_became_active":"2023-11-08T05:30:45.794398+0000","last_became_peered":"2023-11-08T05:30:45.794398+0000","last_unstale":"2023-11-08T05:30:45.794760+0000","last_undegraded":"2023-11-08T05:30:45.794760+0000","last_fullsized":"2023-11-08T05:30:45.794760+0000","mapping_epoch":458,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":459,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":465,"reported_epoch":459,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:45.795085+0000","last_change":"2023-11-08T05:30:45.795085+0000","last_active":"2023-11-08T05:30:45.795085+0000","last_peered":"2023-11-08T05:30:45.795085+0000","last_clean":"2023-11-08T05:30:45.795085+0000","last_became_active":"2023-11-08T05:30:45.794743+0000","last_became_peered":"2023-11-08T05:30:45.794743+0000","last_unstale":"2023-11-08T05:30:45.795085+0000","last_undegraded":"2023-11-08T05:30:45.795085+0000","last_fullsized":"2023-11-08T05:30:45.795085+0000","mapping_epoch":458,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":459,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":454,"reported_epoch":458,"state":"active+clean","last_fresh":"2023-11-08T05:30:44.490618+0000","last_change":"2023-11-08T04:52:47.548597+0000","last_active":"2023-11-08T05:30:44.490618+0000","last_peered":"2023-11-08T05:30:44.490618+0000","last_clean":"2023-11-08T05:30:44.490618+0000","last_became_active":"2023-11-08T04:52:47.548294+0000","last_became_peered":"2023-11-08T04:52:47.548294+0000","last_unstale":"2023-11-08T05:30:44.490618+0000","last_undegraded":"2023-11-08T05:30:44.490618+0000","last_fullsized":"2023-11-08T05:30:44.490618+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":460,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490523+0000","last_change":"2023-11-08T05:28:13.545468+0000","last_active":"2023-11-08T05:30:44.490523+0000","last_peered":"2023-11-08T05:30:44.490523+0000","last_clean":"2023-11-08T05:28:01.456723+0000","last_became_active":"2023-11-08T05:28:13.545468+0000","last_became_peered":"2023-11-08T05:28:13.545468+0000","last_unstale":"2023-11-08T05:30:44.490523+0000","last_undegraded":"2023-11-08T05:30:44.490523+0000","last_fullsized":"2023-11-08T05:28:13.537207+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_clean_scrub_stamp":"2023-11-08T04:52:45.528796+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":465,"reported_epoch":458,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:44.491907+0000","last_change":"2023-11-08T05:29:44.395350+0000","last_active":"2023-11-08T05:30:44.491907+0000","last_peered":"2023-11-08T05:30:44.491907+0000","last_clean":"2023-11-08T05:30:44.491907+0000","last_became_active":"2023-11-08T05:29:44.395012+0000","last_became_peered":"2023-11-08T05:29:44.395012+0000","last_unstale":"2023-11-08T05:30:44.491907+0000","last_undegraded":"2023-11-08T05:30:44.491907+0000","last_fullsized":"2023-11-08T05:30:44.491907+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490629+0000","last_change":"2023-11-08T05:28:13.563357+0000","last_active":"2023-11-08T05:30:44.490629+0000","last_peered":"2023-11-08T05:30:44.490629+0000","last_clean":"2023-11-08T05:28:01.457691+0000","last_became_active":"2023-11-08T05:28:13.563357+0000","last_became_peered":"2023-11-08T05:28:13.563357+0000","last_unstale":"2023-11-08T05:30:44.490629+0000","last_undegraded":"2023-11-08T05:30:44.490629+0000","last_fullsized":"2023-11-08T05:28:13.541614+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490691+0000","last_change":"2023-11-08T05:28:13.550672+0000","last_active":"2023-11-08T05:30:44.490691+0000","last_peered":"2023-11-08T05:30:44.490691+0000","last_clean":"2023-11-08T05:28:01.457904+0000","last_became_active":"2023-11-08T05:28:13.550672+0000","last_became_peered":"2023-11-08T05:28:13.550672+0000","last_unstale":"2023-11-08T05:30:44.490691+0000","last_undegraded":"2023-11-08T05:30:44.490691+0000","last_fullsized":"2023-11-08T05:28:13.539595+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490805+0000","last_change":"2023-11-08T05:28:13.557968+0000","last_active":"2023-11-08T05:30:44.490805+0000","last_peered":"2023-11-08T05:30:44.490805+0000","last_clean":"2023-11-08T05:28:01.458433+0000","last_became_active":"2023-11-08T05:28:13.557968+0000","last_became_peered":"2023-11-08T05:28:13.557968+0000","last_unstale":"2023-11-08T05:30:44.490805+0000","last_undegraded":"2023-11-08T05:30:44.490805+0000","last_fullsized":"2023-11-08T05:28:13.542832+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":1,"acting":1,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":12,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738844,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":645040,"kb_used_data":5352,"kb_used_omap":275,"kb_used_meta":639404,"kb_avail":93726800,"statfs":{"total":96636764160,"available":95976243200,"internally_reserved":0,"allocated":5480448,"data_stored":2697647,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":281650,"internal_metadata":654750670},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:59 2023","interfaces":[{"interface":"back","average":{"1min":0.30499999999999999,"5min":0.439,"15min":0.36099999999999999},"min":{"1min":0.17399999999999999,"5min":0.14499999999999999,"15min":0.14499999999999999},"max":{"1min":0.60099999999999998,"5min":4.9379999999999997,"15min":4.9379999999999997},"last":0.379},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.49399999999999999,"15min":0.42399999999999999},"min":{"1min":0.221,"5min":0.19600000000000001,"15min":0.17000000000000001},"max":{"1min":0.63200000000000001,"5min":4.9859999999999998,"15min":4.9859999999999998},"last":0.40899999999999997}]},{"osd":1,"last update":"Wed Nov 8 05:32:06 2023","interfaces":[{"interface":"back","average":{"1min":0.27400000000000002,"5min":0.307,"15min":0.46800000000000003},"min":{"1min":0.17599999999999999,"5min":0.16,"15min":0.153},"max":{"1min":0.39000000000000001,"5min":1.825,"15min":23.242999999999999},"last":0.22700000000000001},{"interface":"front","average":{"1min":0.28000000000000003,"5min":0.34799999999999998,"15min":0.45700000000000002},"min":{"1min":0.20200000000000001,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":0.41699999999999998,"5min":2.0419999999999998,"15min":23.251000000000001},"last":0.217}]}]},{"osd":0,"up_from":8,"seq":34359738798,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1339228,"kb_used_data":398304,"kb_used_omap":291,"kb_used_meta":936476,"kb_avail":93032612,"statfs":{"total":96636764160,"available":95265394688,"internally_reserved":0,"allocated":407863296,"data_stored":405103900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":298462,"internal_metadata":958951970},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.436,"15min":0.36299999999999999},"min":{"1min":0.19700000000000001,"5min":0.14699999999999999,"15min":0.13700000000000001},"max":{"1min":3.222,"5min":6.5499999999999998,"15min":6.9989999999999997},"last":0.245},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.46300000000000002,"15min":0.42299999999999999},"min":{"1min":0.17799999999999999,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":5.0030000000000001,"5min":6.0629999999999997,"15min":9.6869999999999994},"last":0.28599999999999998}]},{"osd":2,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.51600000000000001,"15min":0.47099999999999997},"min":{"1min":0.21099999999999999,"5min":0.153,"15min":0.153},"max":{"1min":5.7439999999999998,"5min":5.8280000000000003,"15min":9.7639999999999993},"last":0.252},{"interface":"front","average":{"1min":0.497,"5min":0.56200000000000006,"15min":0.42299999999999999},"min":{"1min":0.20399999999999999,"5min":0.187,"15min":0.16800000000000001},"max":{"1min":3.9319999999999999,"5min":8.8040000000000003,"15min":8.8040000000000003},"last":0.29899999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738846,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":650100,"kb_used_data":5356,"kb_used_omap":214,"kb_used_meta":644521,"kb_avail":93721740,"statfs":{"total":96636764160,"available":95971061760,"internally_reserved":0,"allocated":5484544,"data_stored":2698616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":219335,"internal_metadata":659990329},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:56 2023","interfaces":[{"interface":"back","average":{"1min":0.30399999999999999,"5min":0.34699999999999998,"15min":0.36499999999999999},"min":{"1min":0.23999999999999999,"5min":0.14199999999999999,"15min":0.14199999999999999},"max":{"1min":0.42399999999999999,"5min":1.595,"15min":7.4249999999999998},"last":0.17100000000000001},{"interface":"front","average":{"1min":0.35499999999999998,"5min":0.40899999999999997,"15min":0.376},"min":{"1min":0.23999999999999999,"5min":0.19600000000000001,"15min":0.183},"max":{"1min":0.46999999999999997,"5min":2.6859999999999999,"15min":2.6859999999999999},"last":0.23000000000000001}]},{"osd":2,"last update":"Wed Nov 8 05:32:02 2023","interfaces":[{"interface":"back","average":{"1min":0.252,"5min":0.33000000000000002,"15min":0.36499999999999999},"min":{"1min":0.161,"5min":0.161,"15min":0.161},"max":{"1min":0.39600000000000002,"5min":2.2650000000000001,"15min":5.8209999999999997},"last":0.29999999999999999},{"interface":"front","average":{"1min":0.25800000000000001,"5min":0.29899999999999999,"15min":0.35599999999999998},"min":{"1min":0.17000000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.35799999999999998,"5min":1.137,"15min":4.2350000000000003},"last":0.28100000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-08T05:32:15.442 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-11-08T05:32:15.724 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T05:32:15.726 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2023-11-08T05:32:15.737 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":1641,"stamp":"2023-11-08T05:32:14.078126+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":10,"acting":13,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":109,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2634368,"kb_used_data":409012,"kb_used_omap":780,"kb_used_meta":2220403,"kb_avail":280481152,"statfs":{"total":289910292480,"available":287212699648,"internally_reserved":0,"allocated":418828288,"data_stored":410500163,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":799447,"internal_metadata":2273692969},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-536782902,"num_objects":-164,"num_object_clones":0,"num_object_copies":-328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-115,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-164,"num_whiteouts":0,"num_read":-122056,"num_read_kb":-39011942,"num_write":-413642,"num_write_kb":-151705339,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-8,"num_bytes_recovered":-16922624,"num_keys_recovered":-2,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.424817"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490644+0000","last_change":"2023-11-08T05:28:13.557071+0000","last_active":"2023-11-08T05:30:44.490644+0000","last_peered":"2023-11-08T05:30:44.490644+0000","last_clean":"2023-11-08T05:28:01.458439+0000","last_became_active":"2023-11-08T05:28:13.557071+0000","last_became_peered":"2023-11-08T05:28:13.557071+0000","last_unstale":"2023-11-08T05:30:44.490644+0000","last_undegraded":"2023-11-08T05:30:44.490644+0000","last_fullsized":"2023-11-08T05:28:13.547025+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":459,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:45.794760+0000","last_change":"2023-11-08T05:30:45.794760+0000","last_active":"2023-11-08T05:30:45.794760+0000","last_peered":"2023-11-08T05:30:45.794760+0000","last_clean":"2023-11-08T05:30:45.794760+0000","last_became_active":"2023-11-08T05:30:45.794398+0000","last_became_peered":"2023-11-08T05:30:45.794398+0000","last_unstale":"2023-11-08T05:30:45.794760+0000","last_undegraded":"2023-11-08T05:30:45.794760+0000","last_fullsized":"2023-11-08T05:30:45.794760+0000","mapping_epoch":458,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":459,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":465,"reported_epoch":459,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:45.795085+0000","last_change":"2023-11-08T05:30:45.795085+0000","last_active":"2023-11-08T05:30:45.795085+0000","last_peered":"2023-11-08T05:30:45.795085+0000","last_clean":"2023-11-08T05:30:45.795085+0000","last_became_active":"2023-11-08T05:30:45.794743+0000","last_became_peered":"2023-11-08T05:30:45.794743+0000","last_unstale":"2023-11-08T05:30:45.795085+0000","last_undegraded":"2023-11-08T05:30:45.795085+0000","last_fullsized":"2023-11-08T05:30:45.795085+0000","mapping_epoch":458,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":459,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":454,"reported_epoch":458,"state":"active+clean","last_fresh":"2023-11-08T05:30:44.490618+0000","last_change":"2023-11-08T04:52:47.548597+0000","last_active":"2023-11-08T05:30:44.490618+0000","last_peered":"2023-11-08T05:30:44.490618+0000","last_clean":"2023-11-08T05:30:44.490618+0000","last_became_active":"2023-11-08T04:52:47.548294+0000","last_became_peered":"2023-11-08T04:52:47.548294+0000","last_unstale":"2023-11-08T05:30:44.490618+0000","last_undegraded":"2023-11-08T05:30:44.490618+0000","last_fullsized":"2023-11-08T05:30:44.490618+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":460,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490523+0000","last_change":"2023-11-08T05:28:13.545468+0000","last_active":"2023-11-08T05:30:44.490523+0000","last_peered":"2023-11-08T05:30:44.490523+0000","last_clean":"2023-11-08T05:28:01.456723+0000","last_became_active":"2023-11-08T05:28:13.545468+0000","last_became_peered":"2023-11-08T05:28:13.545468+0000","last_unstale":"2023-11-08T05:30:44.490523+0000","last_undegraded":"2023-11-08T05:30:44.490523+0000","last_fullsized":"2023-11-08T05:28:13.537207+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_clean_scrub_stamp":"2023-11-08T04:52:45.528796+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":465,"reported_epoch":458,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:44.491907+0000","last_change":"2023-11-08T05:29:44.395350+0000","last_active":"2023-11-08T05:30:44.491907+0000","last_peered":"2023-11-08T05:30:44.491907+0000","last_clean":"2023-11-08T05:30:44.491907+0000","last_became_active":"2023-11-08T05:29:44.395012+0000","last_became_peered":"2023-11-08T05:29:44.395012+0000","last_unstale":"2023-11-08T05:30:44.491907+0000","last_undegraded":"2023-11-08T05:30:44.491907+0000","last_fullsized":"2023-11-08T05:30:44.491907+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490629+0000","last_change":"2023-11-08T05:28:13.563357+0000","last_active":"2023-11-08T05:30:44.490629+0000","last_peered":"2023-11-08T05:30:44.490629+0000","last_clean":"2023-11-08T05:28:01.457691+0000","last_became_active":"2023-11-08T05:28:13.563357+0000","last_became_peered":"2023-11-08T05:28:13.563357+0000","last_unstale":"2023-11-08T05:30:44.490629+0000","last_undegraded":"2023-11-08T05:30:44.490629+0000","last_fullsized":"2023-11-08T05:28:13.541614+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490691+0000","last_change":"2023-11-08T05:28:13.550672+0000","last_active":"2023-11-08T05:30:44.490691+0000","last_peered":"2023-11-08T05:30:44.490691+0000","last_clean":"2023-11-08T05:28:01.457904+0000","last_became_active":"2023-11-08T05:28:13.550672+0000","last_became_peered":"2023-11-08T05:28:13.550672+0000","last_unstale":"2023-11-08T05:30:44.490691+0000","last_undegraded":"2023-11-08T05:30:44.490691+0000","last_fullsized":"2023-11-08T05:28:13.539595+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490805+0000","last_change":"2023-11-08T05:28:13.557968+0000","last_active":"2023-11-08T05:30:44.490805+0000","last_peered":"2023-11-08T05:30:44.490805+0000","last_clean":"2023-11-08T05:28:01.458433+0000","last_became_active":"2023-11-08T05:28:13.557968+0000","last_became_peered":"2023-11-08T05:28:13.557968+0000","last_unstale":"2023-11-08T05:30:44.490805+0000","last_undegraded":"2023-11-08T05:30:44.490805+0000","last_fullsized":"2023-11-08T05:28:13.542832+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":1,"acting":1,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":12,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738844,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":645040,"kb_used_data":5352,"kb_used_omap":275,"kb_used_meta":639404,"kb_avail":93726800,"statfs":{"total":96636764160,"available":95976243200,"internally_reserved":0,"allocated":5480448,"data_stored":2697647,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":281650,"internal_metadata":654750670},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:59 2023","interfaces":[{"interface":"back","average":{"1min":0.30499999999999999,"5min":0.439,"15min":0.36099999999999999},"min":{"1min":0.17399999999999999,"5min":0.14499999999999999,"15min":0.14499999999999999},"max":{"1min":0.60099999999999998,"5min":4.9379999999999997,"15min":4.9379999999999997},"last":0.379},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.49399999999999999,"15min":0.42399999999999999},"min":{"1min":0.221,"5min":0.19600000000000001,"15min":0.17000000000000001},"max":{"1min":0.63200000000000001,"5min":4.9859999999999998,"15min":4.9859999999999998},"last":0.40899999999999997}]},{"osd":1,"last update":"Wed Nov 8 05:32:06 2023","interfaces":[{"interface":"back","average":{"1min":0.27400000000000002,"5min":0.307,"15min":0.46800000000000003},"min":{"1min":0.17599999999999999,"5min":0.16,"15min":0.153},"max":{"1min":0.39000000000000001,"5min":1.825,"15min":23.242999999999999},"last":0.22700000000000001},{"interface":"front","average":{"1min":0.28000000000000003,"5min":0.34799999999999998,"15min":0.45700000000000002},"min":{"1min":0.20200000000000001,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":0.41699999999999998,"5min":2.0419999999999998,"15min":23.251000000000001},"last":0.217}]}]},{"osd":0,"up_from":8,"seq":34359738798,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1339228,"kb_used_data":398304,"kb_used_omap":291,"kb_used_meta":936476,"kb_avail":93032612,"statfs":{"total":96636764160,"available":95265394688,"internally_reserved":0,"allocated":407863296,"data_stored":405103900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":298462,"internal_metadata":958951970},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.436,"15min":0.36299999999999999},"min":{"1min":0.19700000000000001,"5min":0.14699999999999999,"15min":0.13700000000000001},"max":{"1min":3.222,"5min":6.5499999999999998,"15min":6.9989999999999997},"last":0.245},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.46300000000000002,"15min":0.42299999999999999},"min":{"1min":0.17799999999999999,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":5.0030000000000001,"5min":6.0629999999999997,"15min":9.6869999999999994},"last":0.28599999999999998}]},{"osd":2,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.51600000000000001,"15min":0.47099999999999997},"min":{"1min":0.21099999999999999,"5min":0.153,"15min":0.153},"max":{"1min":5.7439999999999998,"5min":5.8280000000000003,"15min":9.7639999999999993},"last":0.252},{"interface":"front","average":{"1min":0.497,"5min":0.56200000000000006,"15min":0.42299999999999999},"min":{"1min":0.20399999999999999,"5min":0.187,"15min":0.16800000000000001},"max":{"1min":3.9319999999999999,"5min":8.8040000000000003,"15min":8.8040000000000003},"last":0.29899999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738846,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":650100,"kb_used_data":5356,"kb_used_omap":214,"kb_used_meta":644521,"kb_avail":93721740,"statfs":{"total":96636764160,"available":95971061760,"internally_reserved":0,"allocated":5484544,"data_stored":2698616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":219335,"internal_metadata":659990329},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:56 2023","interfaces":[{"interface":"back","average":{"1min":0.30399999999999999,"5min":0.34699999999999998,"15min":0.36499999999999999},"min":{"1min":0.23999999999999999,"5min":0.14199999999999999,"15min":0.14199999999999999},"max":{"1min":0.42399999999999999,"5min":1.595,"15min":7.4249999999999998},"last":0.17100000000000001},{"interface":"front","average":{"1min":0.35499999999999998,"5min":0.40899999999999997,"15min":0.376},"min":{"1min":0.23999999999999999,"5min":0.19600000000000001,"15min":0.183},"max":{"1min":0.46999999999999997,"5min":2.6859999999999999,"15min":2.6859999999999999},"last":0.23000000000000001}]},{"osd":2,"last update":"Wed Nov 8 05:32:02 2023","interfaces":[{"interface":"back","average":{"1min":0.252,"5min":0.33000000000000002,"15min":0.36499999999999999},"min":{"1min":0.161,"5min":0.161,"15min":0.161},"max":{"1min":0.39600000000000002,"5min":2.2650000000000001,"15min":5.8209999999999997},"last":0.29999999999999999},{"interface":"front","average":{"1min":0.25800000000000001,"5min":0.29899999999999999,"15min":0.35599999999999998},"min":{"1min":0.17000000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.35799999999999998,"5min":1.137,"15min":4.2350000000000003},"last":0.28100000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-08T05:32:15.739 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-11-08T05:32:15.739 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 459, 'reported_epoch': 458, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:30:44.490644+0000', 'last_change': '2023-11-08T05:28:13.557071+0000', 'last_active': '2023-11-08T05:30:44.490644+0000', 'last_peered': '2023-11-08T05:30:44.490644+0000', 'last_clean': '2023-11-08T05:28:01.458439+0000', 'last_became_active': '2023-11-08T05:28:13.557071+0000', 'last_became_peered': '2023-11-08T05:28:13.557071+0000', 'last_unstale': '2023-11-08T05:30:44.490644+0000', 'last_undegraded': '2023-11-08T05:30:44.490644+0000', 'last_fullsized': '2023-11-08T05:28:13.547025+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:32:15.739 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-11-08T05:32:15.740 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 11, 'reported_epoch': 459, 'state': 'active+clean+remapped', 'last_fresh': '2023-11-08T05:30:45.794760+0000', 'last_change': '2023-11-08T05:30:45.794760+0000', 'last_active': '2023-11-08T05:30:45.794760+0000', 'last_peered': '2023-11-08T05:30:45.794760+0000', 'last_clean': '2023-11-08T05:30:45.794760+0000', 'last_became_active': '2023-11-08T05:30:45.794398+0000', 'last_became_peered': '2023-11-08T05:30:45.794398+0000', 'last_unstale': '2023-11-08T05:30:45.794760+0000', 'last_undegraded': '2023-11-08T05:30:45.794760+0000', 'last_fullsized': '2023-11-08T05:30:45.794760+0000', 'mapping_epoch': 458, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 459, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2], 'acting': [2, 1], 'avail_no_missing': ['2', '1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2023-11-08T05:32:15.740 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.1 is not active+clean 2023-11-08T05:32:15.740 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.1', 'version': "0'0", 'reported_seq': 465, 'reported_epoch': 459, 'state': 'active+clean+remapped', 'last_fresh': '2023-11-08T05:30:45.795085+0000', 'last_change': '2023-11-08T05:30:45.795085+0000', 'last_active': '2023-11-08T05:30:45.795085+0000', 'last_peered': '2023-11-08T05:30:45.795085+0000', 'last_clean': '2023-11-08T05:30:45.795085+0000', 'last_became_active': '2023-11-08T05:30:45.794743+0000', 'last_became_peered': '2023-11-08T05:30:45.794743+0000', 'last_unstale': '2023-11-08T05:30:45.795085+0000', 'last_undegraded': '2023-11-08T05:30:45.795085+0000', 'last_fullsized': '2023-11-08T05:30:45.795085+0000', 'mapping_epoch': 458, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 459, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2], 'acting': [2, 1], 'avail_no_missing': ['2', '1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2023-11-08T05:32:15.741 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-11-08T05:32:15.741 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "0'0", 'reported_seq': 460, 'reported_epoch': 458, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:30:44.490523+0000', 'last_change': '2023-11-08T05:28:13.545468+0000', 'last_active': '2023-11-08T05:30:44.490523+0000', 'last_peered': '2023-11-08T05:30:44.490523+0000', 'last_clean': '2023-11-08T05:28:01.456723+0000', 'last_became_active': '2023-11-08T05:28:13.545468+0000', 'last_became_peered': '2023-11-08T05:28:13.545468+0000', 'last_unstale': '2023-11-08T05:30:44.490523+0000', 'last_undegraded': '2023-11-08T05:30:44.490523+0000', 'last_fullsized': '2023-11-08T05:28:13.537207+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 9, 'last_epoch_clean': 10, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:45.528796+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:45.528796+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:45.528796+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:32:15.741 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.0 is not active+clean 2023-11-08T05:32:15.742 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.0', 'version': "0'0", 'reported_seq': 465, 'reported_epoch': 458, 'state': 'active+clean+remapped', 'last_fresh': '2023-11-08T05:30:44.491907+0000', 'last_change': '2023-11-08T05:29:44.395350+0000', 'last_active': '2023-11-08T05:30:44.491907+0000', 'last_peered': '2023-11-08T05:30:44.491907+0000', 'last_clean': '2023-11-08T05:30:44.491907+0000', 'last_became_active': '2023-11-08T05:29:44.395012+0000', 'last_became_peered': '2023-11-08T05:29:44.395012+0000', 'last_unstale': '2023-11-08T05:30:44.491907+0000', 'last_undegraded': '2023-11-08T05:30:44.491907+0000', 'last_fullsized': '2023-11-08T05:30:44.491907+0000', 'mapping_epoch': 455, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 456, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2], 'acting': [2, 1], 'avail_no_missing': ['2', '1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2023-11-08T05:32:15.742 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-11-08T05:32:15.742 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 459, 'reported_epoch': 458, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:30:44.490629+0000', 'last_change': '2023-11-08T05:28:13.563357+0000', 'last_active': '2023-11-08T05:30:44.490629+0000', 'last_peered': '2023-11-08T05:30:44.490629+0000', 'last_clean': '2023-11-08T05:28:01.457691+0000', 'last_became_active': '2023-11-08T05:28:13.563357+0000', 'last_became_peered': '2023-11-08T05:28:13.563357+0000', 'last_unstale': '2023-11-08T05:30:44.490629+0000', 'last_undegraded': '2023-11-08T05:30:44.490629+0000', 'last_fullsized': '2023-11-08T05:28:13.541614+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:32:15.743 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-11-08T05:32:15.743 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 459, 'reported_epoch': 458, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:30:44.490691+0000', 'last_change': '2023-11-08T05:28:13.550672+0000', 'last_active': '2023-11-08T05:30:44.490691+0000', 'last_peered': '2023-11-08T05:30:44.490691+0000', 'last_clean': '2023-11-08T05:28:01.457904+0000', 'last_became_active': '2023-11-08T05:28:13.550672+0000', 'last_became_peered': '2023-11-08T05:28:13.550672+0000', 'last_unstale': '2023-11-08T05:30:44.490691+0000', 'last_undegraded': '2023-11-08T05:30:44.490691+0000', 'last_fullsized': '2023-11-08T05:28:13.539595+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:32:15.743 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-11-08T05:32:15.743 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 459, 'reported_epoch': 458, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:30:44.490805+0000', 'last_change': '2023-11-08T05:28:13.557968+0000', 'last_active': '2023-11-08T05:30:44.490805+0000', 'last_peered': '2023-11-08T05:30:44.490805+0000', 'last_clean': '2023-11-08T05:28:01.458433+0000', 'last_became_active': '2023-11-08T05:28:13.557968+0000', 'last_became_peered': '2023-11-08T05:28:13.557968+0000', 'last_unstale': '2023-11-08T05:30:44.490805+0000', 'last_undegraded': '2023-11-08T05:30:44.490805+0000', 'last_fullsized': '2023-11-08T05:28:13.542832+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:32:15.744 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-11-08T05:32:15.824 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~238s 2023-11-08T05:32:16.089 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T05:32:16.089 INFO:teuthology.orchestra.run.smithi087.stdout:{"fsid":"ab4cbc06-449d-497d-b513-220d9db2b590","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 5 pgs undersized","count":5},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2378,"monmap":{"epoch":1,"min_mon_release_name":"pacific","num_mons":1},"osdmap":{"epoch":459,"num_osds":3,"num_up_osds":2,"osd_up_since":1699421292,"num_in_osds":3,"osd_in_since":1699419158,"num_remapped_pgs":3},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean+remapped","count":3},{"state_name":"active+clean","count":1}],"num_pgs":9,"num_pools":2,"num_objects":0,"data_bytes":0,"bytes_used":2697592832,"bytes_avail":287212699648,"bytes_total":289910292480},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":41,"modified":"2023-11-08T05:29:32.044544+0000","services":{}},"progress_events":{}} 2023-11-08T05:32:16.102 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-11-08T05:32:16.102 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-11-08T05:32:16.384 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T05:32:16.386 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2023-11-08T05:32:16.397 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":1642,"stamp":"2023-11-08T05:32:16.078506+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":10,"acting":13,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":109,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2634368,"kb_used_data":409012,"kb_used_omap":780,"kb_used_meta":2220403,"kb_avail":280481152,"statfs":{"total":289910292480,"available":287212699648,"internally_reserved":0,"allocated":418828288,"data_stored":410500163,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":799447,"internal_metadata":2273692969},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-536782902,"num_objects":-164,"num_object_clones":0,"num_object_copies":-328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-115,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-164,"num_whiteouts":0,"num_read":-122056,"num_read_kb":-39011942,"num_write":-413642,"num_write_kb":-151705339,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-8,"num_bytes_recovered":-16922624,"num_keys_recovered":-2,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.424817"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490644+0000","last_change":"2023-11-08T05:28:13.557071+0000","last_active":"2023-11-08T05:30:44.490644+0000","last_peered":"2023-11-08T05:30:44.490644+0000","last_clean":"2023-11-08T05:28:01.458439+0000","last_became_active":"2023-11-08T05:28:13.557071+0000","last_became_peered":"2023-11-08T05:28:13.557071+0000","last_unstale":"2023-11-08T05:30:44.490644+0000","last_undegraded":"2023-11-08T05:30:44.490644+0000","last_fullsized":"2023-11-08T05:28:13.547025+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":459,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:45.794760+0000","last_change":"2023-11-08T05:30:45.794760+0000","last_active":"2023-11-08T05:30:45.794760+0000","last_peered":"2023-11-08T05:30:45.794760+0000","last_clean":"2023-11-08T05:30:45.794760+0000","last_became_active":"2023-11-08T05:30:45.794398+0000","last_became_peered":"2023-11-08T05:30:45.794398+0000","last_unstale":"2023-11-08T05:30:45.794760+0000","last_undegraded":"2023-11-08T05:30:45.794760+0000","last_fullsized":"2023-11-08T05:30:45.794760+0000","mapping_epoch":458,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":459,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":465,"reported_epoch":459,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:45.795085+0000","last_change":"2023-11-08T05:30:45.795085+0000","last_active":"2023-11-08T05:30:45.795085+0000","last_peered":"2023-11-08T05:30:45.795085+0000","last_clean":"2023-11-08T05:30:45.795085+0000","last_became_active":"2023-11-08T05:30:45.794743+0000","last_became_peered":"2023-11-08T05:30:45.794743+0000","last_unstale":"2023-11-08T05:30:45.795085+0000","last_undegraded":"2023-11-08T05:30:45.795085+0000","last_fullsized":"2023-11-08T05:30:45.795085+0000","mapping_epoch":458,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":459,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":454,"reported_epoch":458,"state":"active+clean","last_fresh":"2023-11-08T05:30:44.490618+0000","last_change":"2023-11-08T04:52:47.548597+0000","last_active":"2023-11-08T05:30:44.490618+0000","last_peered":"2023-11-08T05:30:44.490618+0000","last_clean":"2023-11-08T05:30:44.490618+0000","last_became_active":"2023-11-08T04:52:47.548294+0000","last_became_peered":"2023-11-08T04:52:47.548294+0000","last_unstale":"2023-11-08T05:30:44.490618+0000","last_undegraded":"2023-11-08T05:30:44.490618+0000","last_fullsized":"2023-11-08T05:30:44.490618+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":460,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490523+0000","last_change":"2023-11-08T05:28:13.545468+0000","last_active":"2023-11-08T05:30:44.490523+0000","last_peered":"2023-11-08T05:30:44.490523+0000","last_clean":"2023-11-08T05:28:01.456723+0000","last_became_active":"2023-11-08T05:28:13.545468+0000","last_became_peered":"2023-11-08T05:28:13.545468+0000","last_unstale":"2023-11-08T05:30:44.490523+0000","last_undegraded":"2023-11-08T05:30:44.490523+0000","last_fullsized":"2023-11-08T05:28:13.537207+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_clean_scrub_stamp":"2023-11-08T04:52:45.528796+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":465,"reported_epoch":458,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:44.491907+0000","last_change":"2023-11-08T05:29:44.395350+0000","last_active":"2023-11-08T05:30:44.491907+0000","last_peered":"2023-11-08T05:30:44.491907+0000","last_clean":"2023-11-08T05:30:44.491907+0000","last_became_active":"2023-11-08T05:29:44.395012+0000","last_became_peered":"2023-11-08T05:29:44.395012+0000","last_unstale":"2023-11-08T05:30:44.491907+0000","last_undegraded":"2023-11-08T05:30:44.491907+0000","last_fullsized":"2023-11-08T05:30:44.491907+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490629+0000","last_change":"2023-11-08T05:28:13.563357+0000","last_active":"2023-11-08T05:30:44.490629+0000","last_peered":"2023-11-08T05:30:44.490629+0000","last_clean":"2023-11-08T05:28:01.457691+0000","last_became_active":"2023-11-08T05:28:13.563357+0000","last_became_peered":"2023-11-08T05:28:13.563357+0000","last_unstale":"2023-11-08T05:30:44.490629+0000","last_undegraded":"2023-11-08T05:30:44.490629+0000","last_fullsized":"2023-11-08T05:28:13.541614+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490691+0000","last_change":"2023-11-08T05:28:13.550672+0000","last_active":"2023-11-08T05:30:44.490691+0000","last_peered":"2023-11-08T05:30:44.490691+0000","last_clean":"2023-11-08T05:28:01.457904+0000","last_became_active":"2023-11-08T05:28:13.550672+0000","last_became_peered":"2023-11-08T05:28:13.550672+0000","last_unstale":"2023-11-08T05:30:44.490691+0000","last_undegraded":"2023-11-08T05:30:44.490691+0000","last_fullsized":"2023-11-08T05:28:13.539595+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490805+0000","last_change":"2023-11-08T05:28:13.557968+0000","last_active":"2023-11-08T05:30:44.490805+0000","last_peered":"2023-11-08T05:30:44.490805+0000","last_clean":"2023-11-08T05:28:01.458433+0000","last_became_active":"2023-11-08T05:28:13.557968+0000","last_became_peered":"2023-11-08T05:28:13.557968+0000","last_unstale":"2023-11-08T05:30:44.490805+0000","last_undegraded":"2023-11-08T05:30:44.490805+0000","last_fullsized":"2023-11-08T05:28:13.542832+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":1,"acting":1,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":12,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738845,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":645040,"kb_used_data":5352,"kb_used_omap":275,"kb_used_meta":639404,"kb_avail":93726800,"statfs":{"total":96636764160,"available":95976243200,"internally_reserved":0,"allocated":5480448,"data_stored":2697647,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":281650,"internal_metadata":654750670},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:59 2023","interfaces":[{"interface":"back","average":{"1min":0.30499999999999999,"5min":0.439,"15min":0.36099999999999999},"min":{"1min":0.17399999999999999,"5min":0.14499999999999999,"15min":0.14499999999999999},"max":{"1min":0.60099999999999998,"5min":4.9379999999999997,"15min":4.9379999999999997},"last":0.379},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.49399999999999999,"15min":0.42399999999999999},"min":{"1min":0.221,"5min":0.19600000000000001,"15min":0.17000000000000001},"max":{"1min":0.63200000000000001,"5min":4.9859999999999998,"15min":4.9859999999999998},"last":0.40899999999999997}]},{"osd":1,"last update":"Wed Nov 8 05:32:06 2023","interfaces":[{"interface":"back","average":{"1min":0.27400000000000002,"5min":0.307,"15min":0.46800000000000003},"min":{"1min":0.17599999999999999,"5min":0.16,"15min":0.153},"max":{"1min":0.39000000000000001,"5min":1.825,"15min":23.242999999999999},"last":0.28599999999999998},{"interface":"front","average":{"1min":0.28000000000000003,"5min":0.34799999999999998,"15min":0.45700000000000002},"min":{"1min":0.20200000000000001,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":0.41699999999999998,"5min":2.0419999999999998,"15min":23.251000000000001},"last":0.309}]}]},{"osd":0,"up_from":8,"seq":34359738798,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1339228,"kb_used_data":398304,"kb_used_omap":291,"kb_used_meta":936476,"kb_avail":93032612,"statfs":{"total":96636764160,"available":95265394688,"internally_reserved":0,"allocated":407863296,"data_stored":405103900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":298462,"internal_metadata":958951970},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.436,"15min":0.36299999999999999},"min":{"1min":0.19700000000000001,"5min":0.14699999999999999,"15min":0.13700000000000001},"max":{"1min":3.222,"5min":6.5499999999999998,"15min":6.9989999999999997},"last":0.245},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.46300000000000002,"15min":0.42299999999999999},"min":{"1min":0.17799999999999999,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":5.0030000000000001,"5min":6.0629999999999997,"15min":9.6869999999999994},"last":0.28599999999999998}]},{"osd":2,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.51600000000000001,"15min":0.47099999999999997},"min":{"1min":0.21099999999999999,"5min":0.153,"15min":0.153},"max":{"1min":5.7439999999999998,"5min":5.8280000000000003,"15min":9.7639999999999993},"last":0.252},{"interface":"front","average":{"1min":0.497,"5min":0.56200000000000006,"15min":0.42299999999999999},"min":{"1min":0.20399999999999999,"5min":0.187,"15min":0.16800000000000001},"max":{"1min":3.9319999999999999,"5min":8.8040000000000003,"15min":8.8040000000000003},"last":0.29899999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738847,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":650100,"kb_used_data":5356,"kb_used_omap":214,"kb_used_meta":644521,"kb_avail":93721740,"statfs":{"total":96636764160,"available":95971061760,"internally_reserved":0,"allocated":5484544,"data_stored":2698616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":219335,"internal_metadata":659990329},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:56 2023","interfaces":[{"interface":"back","average":{"1min":0.30399999999999999,"5min":0.34699999999999998,"15min":0.36499999999999999},"min":{"1min":0.23999999999999999,"5min":0.14199999999999999,"15min":0.14199999999999999},"max":{"1min":0.42399999999999999,"5min":1.595,"15min":7.4249999999999998},"last":0.17100000000000001},{"interface":"front","average":{"1min":0.35499999999999998,"5min":0.40899999999999997,"15min":0.376},"min":{"1min":0.23999999999999999,"5min":0.19600000000000001,"15min":0.183},"max":{"1min":0.46999999999999997,"5min":2.6859999999999999,"15min":2.6859999999999999},"last":0.23000000000000001}]},{"osd":2,"last update":"Wed Nov 8 05:32:02 2023","interfaces":[{"interface":"back","average":{"1min":0.252,"5min":0.33000000000000002,"15min":0.36499999999999999},"min":{"1min":0.161,"5min":0.161,"15min":0.161},"max":{"1min":0.39600000000000002,"5min":2.2650000000000001,"15min":5.8209999999999997},"last":0.40799999999999997},{"interface":"front","average":{"1min":0.25800000000000001,"5min":0.29899999999999999,"15min":0.35599999999999998},"min":{"1min":0.17000000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.35799999999999998,"5min":1.137,"15min":4.2350000000000003},"last":0.41999999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-08T05:32:19.399 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-11-08T05:32:19.696 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T05:32:19.698 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2023-11-08T05:32:19.708 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":1643,"stamp":"2023-11-08T05:32:18.078864+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":10,"acting":13,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":109,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2634368,"kb_used_data":409012,"kb_used_omap":780,"kb_used_meta":2220403,"kb_avail":280481152,"statfs":{"total":289910292480,"available":287212699648,"internally_reserved":0,"allocated":418828288,"data_stored":410500163,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":799447,"internal_metadata":2273692969},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-536782902,"num_objects":-164,"num_object_clones":0,"num_object_copies":-328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-115,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-164,"num_whiteouts":0,"num_read":-122056,"num_read_kb":-39011942,"num_write":-413642,"num_write_kb":-151705339,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-8,"num_bytes_recovered":-16922624,"num_keys_recovered":-2,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.424817"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490644+0000","last_change":"2023-11-08T05:28:13.557071+0000","last_active":"2023-11-08T05:30:44.490644+0000","last_peered":"2023-11-08T05:30:44.490644+0000","last_clean":"2023-11-08T05:28:01.458439+0000","last_became_active":"2023-11-08T05:28:13.557071+0000","last_became_peered":"2023-11-08T05:28:13.557071+0000","last_unstale":"2023-11-08T05:30:44.490644+0000","last_undegraded":"2023-11-08T05:30:44.490644+0000","last_fullsized":"2023-11-08T05:28:13.547025+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":459,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:45.794760+0000","last_change":"2023-11-08T05:30:45.794760+0000","last_active":"2023-11-08T05:30:45.794760+0000","last_peered":"2023-11-08T05:30:45.794760+0000","last_clean":"2023-11-08T05:30:45.794760+0000","last_became_active":"2023-11-08T05:30:45.794398+0000","last_became_peered":"2023-11-08T05:30:45.794398+0000","last_unstale":"2023-11-08T05:30:45.794760+0000","last_undegraded":"2023-11-08T05:30:45.794760+0000","last_fullsized":"2023-11-08T05:30:45.794760+0000","mapping_epoch":458,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":459,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":465,"reported_epoch":459,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:45.795085+0000","last_change":"2023-11-08T05:30:45.795085+0000","last_active":"2023-11-08T05:30:45.795085+0000","last_peered":"2023-11-08T05:30:45.795085+0000","last_clean":"2023-11-08T05:30:45.795085+0000","last_became_active":"2023-11-08T05:30:45.794743+0000","last_became_peered":"2023-11-08T05:30:45.794743+0000","last_unstale":"2023-11-08T05:30:45.795085+0000","last_undegraded":"2023-11-08T05:30:45.795085+0000","last_fullsized":"2023-11-08T05:30:45.795085+0000","mapping_epoch":458,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":459,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":454,"reported_epoch":458,"state":"active+clean","last_fresh":"2023-11-08T05:30:44.490618+0000","last_change":"2023-11-08T04:52:47.548597+0000","last_active":"2023-11-08T05:30:44.490618+0000","last_peered":"2023-11-08T05:30:44.490618+0000","last_clean":"2023-11-08T05:30:44.490618+0000","last_became_active":"2023-11-08T04:52:47.548294+0000","last_became_peered":"2023-11-08T04:52:47.548294+0000","last_unstale":"2023-11-08T05:30:44.490618+0000","last_undegraded":"2023-11-08T05:30:44.490618+0000","last_fullsized":"2023-11-08T05:30:44.490618+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":460,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490523+0000","last_change":"2023-11-08T05:28:13.545468+0000","last_active":"2023-11-08T05:30:44.490523+0000","last_peered":"2023-11-08T05:30:44.490523+0000","last_clean":"2023-11-08T05:28:01.456723+0000","last_became_active":"2023-11-08T05:28:13.545468+0000","last_became_peered":"2023-11-08T05:28:13.545468+0000","last_unstale":"2023-11-08T05:30:44.490523+0000","last_undegraded":"2023-11-08T05:30:44.490523+0000","last_fullsized":"2023-11-08T05:28:13.537207+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_clean_scrub_stamp":"2023-11-08T04:52:45.528796+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":465,"reported_epoch":458,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:44.491907+0000","last_change":"2023-11-08T05:29:44.395350+0000","last_active":"2023-11-08T05:30:44.491907+0000","last_peered":"2023-11-08T05:30:44.491907+0000","last_clean":"2023-11-08T05:30:44.491907+0000","last_became_active":"2023-11-08T05:29:44.395012+0000","last_became_peered":"2023-11-08T05:29:44.395012+0000","last_unstale":"2023-11-08T05:30:44.491907+0000","last_undegraded":"2023-11-08T05:30:44.491907+0000","last_fullsized":"2023-11-08T05:30:44.491907+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490629+0000","last_change":"2023-11-08T05:28:13.563357+0000","last_active":"2023-11-08T05:30:44.490629+0000","last_peered":"2023-11-08T05:30:44.490629+0000","last_clean":"2023-11-08T05:28:01.457691+0000","last_became_active":"2023-11-08T05:28:13.563357+0000","last_became_peered":"2023-11-08T05:28:13.563357+0000","last_unstale":"2023-11-08T05:30:44.490629+0000","last_undegraded":"2023-11-08T05:30:44.490629+0000","last_fullsized":"2023-11-08T05:28:13.541614+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490691+0000","last_change":"2023-11-08T05:28:13.550672+0000","last_active":"2023-11-08T05:30:44.490691+0000","last_peered":"2023-11-08T05:30:44.490691+0000","last_clean":"2023-11-08T05:28:01.457904+0000","last_became_active":"2023-11-08T05:28:13.550672+0000","last_became_peered":"2023-11-08T05:28:13.550672+0000","last_unstale":"2023-11-08T05:30:44.490691+0000","last_undegraded":"2023-11-08T05:30:44.490691+0000","last_fullsized":"2023-11-08T05:28:13.539595+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490805+0000","last_change":"2023-11-08T05:28:13.557968+0000","last_active":"2023-11-08T05:30:44.490805+0000","last_peered":"2023-11-08T05:30:44.490805+0000","last_clean":"2023-11-08T05:28:01.458433+0000","last_became_active":"2023-11-08T05:28:13.557968+0000","last_became_peered":"2023-11-08T05:28:13.557968+0000","last_unstale":"2023-11-08T05:30:44.490805+0000","last_undegraded":"2023-11-08T05:30:44.490805+0000","last_fullsized":"2023-11-08T05:28:13.542832+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":1,"acting":1,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":12,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738845,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":645040,"kb_used_data":5352,"kb_used_omap":275,"kb_used_meta":639404,"kb_avail":93726800,"statfs":{"total":96636764160,"available":95976243200,"internally_reserved":0,"allocated":5480448,"data_stored":2697647,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":281650,"internal_metadata":654750670},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:59 2023","interfaces":[{"interface":"back","average":{"1min":0.30499999999999999,"5min":0.439,"15min":0.36099999999999999},"min":{"1min":0.17399999999999999,"5min":0.14499999999999999,"15min":0.14499999999999999},"max":{"1min":0.60099999999999998,"5min":4.9379999999999997,"15min":4.9379999999999997},"last":0.379},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.49399999999999999,"15min":0.42399999999999999},"min":{"1min":0.221,"5min":0.19600000000000001,"15min":0.17000000000000001},"max":{"1min":0.63200000000000001,"5min":4.9859999999999998,"15min":4.9859999999999998},"last":0.40899999999999997}]},{"osd":1,"last update":"Wed Nov 8 05:32:06 2023","interfaces":[{"interface":"back","average":{"1min":0.27400000000000002,"5min":0.307,"15min":0.46800000000000003},"min":{"1min":0.17599999999999999,"5min":0.16,"15min":0.153},"max":{"1min":0.39000000000000001,"5min":1.825,"15min":23.242999999999999},"last":0.28599999999999998},{"interface":"front","average":{"1min":0.28000000000000003,"5min":0.34799999999999998,"15min":0.45700000000000002},"min":{"1min":0.20200000000000001,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":0.41699999999999998,"5min":2.0419999999999998,"15min":23.251000000000001},"last":0.309}]}]},{"osd":0,"up_from":8,"seq":34359738798,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1339228,"kb_used_data":398304,"kb_used_omap":291,"kb_used_meta":936476,"kb_avail":93032612,"statfs":{"total":96636764160,"available":95265394688,"internally_reserved":0,"allocated":407863296,"data_stored":405103900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":298462,"internal_metadata":958951970},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.436,"15min":0.36299999999999999},"min":{"1min":0.19700000000000001,"5min":0.14699999999999999,"15min":0.13700000000000001},"max":{"1min":3.222,"5min":6.5499999999999998,"15min":6.9989999999999997},"last":0.245},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.46300000000000002,"15min":0.42299999999999999},"min":{"1min":0.17799999999999999,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":5.0030000000000001,"5min":6.0629999999999997,"15min":9.6869999999999994},"last":0.28599999999999998}]},{"osd":2,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.51600000000000001,"15min":0.47099999999999997},"min":{"1min":0.21099999999999999,"5min":0.153,"15min":0.153},"max":{"1min":5.7439999999999998,"5min":5.8280000000000003,"15min":9.7639999999999993},"last":0.252},{"interface":"front","average":{"1min":0.497,"5min":0.56200000000000006,"15min":0.42299999999999999},"min":{"1min":0.20399999999999999,"5min":0.187,"15min":0.16800000000000001},"max":{"1min":3.9319999999999999,"5min":8.8040000000000003,"15min":8.8040000000000003},"last":0.29899999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738847,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":650100,"kb_used_data":5356,"kb_used_omap":214,"kb_used_meta":644521,"kb_avail":93721740,"statfs":{"total":96636764160,"available":95971061760,"internally_reserved":0,"allocated":5484544,"data_stored":2698616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":219335,"internal_metadata":659990329},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:56 2023","interfaces":[{"interface":"back","average":{"1min":0.30399999999999999,"5min":0.34699999999999998,"15min":0.36499999999999999},"min":{"1min":0.23999999999999999,"5min":0.14199999999999999,"15min":0.14199999999999999},"max":{"1min":0.42399999999999999,"5min":1.595,"15min":7.4249999999999998},"last":0.17100000000000001},{"interface":"front","average":{"1min":0.35499999999999998,"5min":0.40899999999999997,"15min":0.376},"min":{"1min":0.23999999999999999,"5min":0.19600000000000001,"15min":0.183},"max":{"1min":0.46999999999999997,"5min":2.6859999999999999,"15min":2.6859999999999999},"last":0.23000000000000001}]},{"osd":2,"last update":"Wed Nov 8 05:32:02 2023","interfaces":[{"interface":"back","average":{"1min":0.252,"5min":0.33000000000000002,"15min":0.36499999999999999},"min":{"1min":0.161,"5min":0.161,"15min":0.161},"max":{"1min":0.39600000000000002,"5min":2.2650000000000001,"15min":5.8209999999999997},"last":0.40799999999999997},{"interface":"front","average":{"1min":0.25800000000000001,"5min":0.29899999999999999,"15min":0.35599999999999998},"min":{"1min":0.17000000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.35799999999999998,"5min":1.137,"15min":4.2350000000000003},"last":0.41999999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-08T05:32:19.710 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-11-08T05:32:19.992 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T05:32:19.994 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2023-11-08T05:32:20.005 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":1643,"stamp":"2023-11-08T05:32:18.078864+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":10,"acting":13,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":109,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2634368,"kb_used_data":409012,"kb_used_omap":780,"kb_used_meta":2220403,"kb_avail":280481152,"statfs":{"total":289910292480,"available":287212699648,"internally_reserved":0,"allocated":418828288,"data_stored":410500163,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":799447,"internal_metadata":2273692969},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-536782902,"num_objects":-164,"num_object_clones":0,"num_object_copies":-328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-115,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-164,"num_whiteouts":0,"num_read":-122056,"num_read_kb":-39011942,"num_write":-413642,"num_write_kb":-151705339,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-8,"num_bytes_recovered":-16922624,"num_keys_recovered":-2,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.424817"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490644+0000","last_change":"2023-11-08T05:28:13.557071+0000","last_active":"2023-11-08T05:30:44.490644+0000","last_peered":"2023-11-08T05:30:44.490644+0000","last_clean":"2023-11-08T05:28:01.458439+0000","last_became_active":"2023-11-08T05:28:13.557071+0000","last_became_peered":"2023-11-08T05:28:13.557071+0000","last_unstale":"2023-11-08T05:30:44.490644+0000","last_undegraded":"2023-11-08T05:30:44.490644+0000","last_fullsized":"2023-11-08T05:28:13.547025+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":459,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:45.794760+0000","last_change":"2023-11-08T05:30:45.794760+0000","last_active":"2023-11-08T05:30:45.794760+0000","last_peered":"2023-11-08T05:30:45.794760+0000","last_clean":"2023-11-08T05:30:45.794760+0000","last_became_active":"2023-11-08T05:30:45.794398+0000","last_became_peered":"2023-11-08T05:30:45.794398+0000","last_unstale":"2023-11-08T05:30:45.794760+0000","last_undegraded":"2023-11-08T05:30:45.794760+0000","last_fullsized":"2023-11-08T05:30:45.794760+0000","mapping_epoch":458,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":459,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":465,"reported_epoch":459,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:45.795085+0000","last_change":"2023-11-08T05:30:45.795085+0000","last_active":"2023-11-08T05:30:45.795085+0000","last_peered":"2023-11-08T05:30:45.795085+0000","last_clean":"2023-11-08T05:30:45.795085+0000","last_became_active":"2023-11-08T05:30:45.794743+0000","last_became_peered":"2023-11-08T05:30:45.794743+0000","last_unstale":"2023-11-08T05:30:45.795085+0000","last_undegraded":"2023-11-08T05:30:45.795085+0000","last_fullsized":"2023-11-08T05:30:45.795085+0000","mapping_epoch":458,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":459,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":454,"reported_epoch":458,"state":"active+clean","last_fresh":"2023-11-08T05:30:44.490618+0000","last_change":"2023-11-08T04:52:47.548597+0000","last_active":"2023-11-08T05:30:44.490618+0000","last_peered":"2023-11-08T05:30:44.490618+0000","last_clean":"2023-11-08T05:30:44.490618+0000","last_became_active":"2023-11-08T04:52:47.548294+0000","last_became_peered":"2023-11-08T04:52:47.548294+0000","last_unstale":"2023-11-08T05:30:44.490618+0000","last_undegraded":"2023-11-08T05:30:44.490618+0000","last_fullsized":"2023-11-08T05:30:44.490618+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":460,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490523+0000","last_change":"2023-11-08T05:28:13.545468+0000","last_active":"2023-11-08T05:30:44.490523+0000","last_peered":"2023-11-08T05:30:44.490523+0000","last_clean":"2023-11-08T05:28:01.456723+0000","last_became_active":"2023-11-08T05:28:13.545468+0000","last_became_peered":"2023-11-08T05:28:13.545468+0000","last_unstale":"2023-11-08T05:30:44.490523+0000","last_undegraded":"2023-11-08T05:30:44.490523+0000","last_fullsized":"2023-11-08T05:28:13.537207+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_clean_scrub_stamp":"2023-11-08T04:52:45.528796+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":465,"reported_epoch":458,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:44.491907+0000","last_change":"2023-11-08T05:29:44.395350+0000","last_active":"2023-11-08T05:30:44.491907+0000","last_peered":"2023-11-08T05:30:44.491907+0000","last_clean":"2023-11-08T05:30:44.491907+0000","last_became_active":"2023-11-08T05:29:44.395012+0000","last_became_peered":"2023-11-08T05:29:44.395012+0000","last_unstale":"2023-11-08T05:30:44.491907+0000","last_undegraded":"2023-11-08T05:30:44.491907+0000","last_fullsized":"2023-11-08T05:30:44.491907+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490629+0000","last_change":"2023-11-08T05:28:13.563357+0000","last_active":"2023-11-08T05:30:44.490629+0000","last_peered":"2023-11-08T05:30:44.490629+0000","last_clean":"2023-11-08T05:28:01.457691+0000","last_became_active":"2023-11-08T05:28:13.563357+0000","last_became_peered":"2023-11-08T05:28:13.563357+0000","last_unstale":"2023-11-08T05:30:44.490629+0000","last_undegraded":"2023-11-08T05:30:44.490629+0000","last_fullsized":"2023-11-08T05:28:13.541614+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490691+0000","last_change":"2023-11-08T05:28:13.550672+0000","last_active":"2023-11-08T05:30:44.490691+0000","last_peered":"2023-11-08T05:30:44.490691+0000","last_clean":"2023-11-08T05:28:01.457904+0000","last_became_active":"2023-11-08T05:28:13.550672+0000","last_became_peered":"2023-11-08T05:28:13.550672+0000","last_unstale":"2023-11-08T05:30:44.490691+0000","last_undegraded":"2023-11-08T05:30:44.490691+0000","last_fullsized":"2023-11-08T05:28:13.539595+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490805+0000","last_change":"2023-11-08T05:28:13.557968+0000","last_active":"2023-11-08T05:30:44.490805+0000","last_peered":"2023-11-08T05:30:44.490805+0000","last_clean":"2023-11-08T05:28:01.458433+0000","last_became_active":"2023-11-08T05:28:13.557968+0000","last_became_peered":"2023-11-08T05:28:13.557968+0000","last_unstale":"2023-11-08T05:30:44.490805+0000","last_undegraded":"2023-11-08T05:30:44.490805+0000","last_fullsized":"2023-11-08T05:28:13.542832+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":1,"acting":1,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":12,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738845,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":645040,"kb_used_data":5352,"kb_used_omap":275,"kb_used_meta":639404,"kb_avail":93726800,"statfs":{"total":96636764160,"available":95976243200,"internally_reserved":0,"allocated":5480448,"data_stored":2697647,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":281650,"internal_metadata":654750670},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:59 2023","interfaces":[{"interface":"back","average":{"1min":0.30499999999999999,"5min":0.439,"15min":0.36099999999999999},"min":{"1min":0.17399999999999999,"5min":0.14499999999999999,"15min":0.14499999999999999},"max":{"1min":0.60099999999999998,"5min":4.9379999999999997,"15min":4.9379999999999997},"last":0.379},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.49399999999999999,"15min":0.42399999999999999},"min":{"1min":0.221,"5min":0.19600000000000001,"15min":0.17000000000000001},"max":{"1min":0.63200000000000001,"5min":4.9859999999999998,"15min":4.9859999999999998},"last":0.40899999999999997}]},{"osd":1,"last update":"Wed Nov 8 05:32:06 2023","interfaces":[{"interface":"back","average":{"1min":0.27400000000000002,"5min":0.307,"15min":0.46800000000000003},"min":{"1min":0.17599999999999999,"5min":0.16,"15min":0.153},"max":{"1min":0.39000000000000001,"5min":1.825,"15min":23.242999999999999},"last":0.28599999999999998},{"interface":"front","average":{"1min":0.28000000000000003,"5min":0.34799999999999998,"15min":0.45700000000000002},"min":{"1min":0.20200000000000001,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":0.41699999999999998,"5min":2.0419999999999998,"15min":23.251000000000001},"last":0.309}]}]},{"osd":0,"up_from":8,"seq":34359738798,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1339228,"kb_used_data":398304,"kb_used_omap":291,"kb_used_meta":936476,"kb_avail":93032612,"statfs":{"total":96636764160,"available":95265394688,"internally_reserved":0,"allocated":407863296,"data_stored":405103900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":298462,"internal_metadata":958951970},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.436,"15min":0.36299999999999999},"min":{"1min":0.19700000000000001,"5min":0.14699999999999999,"15min":0.13700000000000001},"max":{"1min":3.222,"5min":6.5499999999999998,"15min":6.9989999999999997},"last":0.245},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.46300000000000002,"15min":0.42299999999999999},"min":{"1min":0.17799999999999999,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":5.0030000000000001,"5min":6.0629999999999997,"15min":9.6869999999999994},"last":0.28599999999999998}]},{"osd":2,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.51600000000000001,"15min":0.47099999999999997},"min":{"1min":0.21099999999999999,"5min":0.153,"15min":0.153},"max":{"1min":5.7439999999999998,"5min":5.8280000000000003,"15min":9.7639999999999993},"last":0.252},{"interface":"front","average":{"1min":0.497,"5min":0.56200000000000006,"15min":0.42299999999999999},"min":{"1min":0.20399999999999999,"5min":0.187,"15min":0.16800000000000001},"max":{"1min":3.9319999999999999,"5min":8.8040000000000003,"15min":8.8040000000000003},"last":0.29899999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738847,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":650100,"kb_used_data":5356,"kb_used_omap":214,"kb_used_meta":644521,"kb_avail":93721740,"statfs":{"total":96636764160,"available":95971061760,"internally_reserved":0,"allocated":5484544,"data_stored":2698616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":219335,"internal_metadata":659990329},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:56 2023","interfaces":[{"interface":"back","average":{"1min":0.30399999999999999,"5min":0.34699999999999998,"15min":0.36499999999999999},"min":{"1min":0.23999999999999999,"5min":0.14199999999999999,"15min":0.14199999999999999},"max":{"1min":0.42399999999999999,"5min":1.595,"15min":7.4249999999999998},"last":0.17100000000000001},{"interface":"front","average":{"1min":0.35499999999999998,"5min":0.40899999999999997,"15min":0.376},"min":{"1min":0.23999999999999999,"5min":0.19600000000000001,"15min":0.183},"max":{"1min":0.46999999999999997,"5min":2.6859999999999999,"15min":2.6859999999999999},"last":0.23000000000000001}]},{"osd":2,"last update":"Wed Nov 8 05:32:02 2023","interfaces":[{"interface":"back","average":{"1min":0.252,"5min":0.33000000000000002,"15min":0.36499999999999999},"min":{"1min":0.161,"5min":0.161,"15min":0.161},"max":{"1min":0.39600000000000002,"5min":2.2650000000000001,"15min":5.8209999999999997},"last":0.40799999999999997},{"interface":"front","average":{"1min":0.25800000000000001,"5min":0.29899999999999999,"15min":0.35599999999999998},"min":{"1min":0.17000000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.35799999999999998,"5min":1.137,"15min":4.2350000000000003},"last":0.41999999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-08T05:32:20.007 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-11-08T05:32:20.007 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 459, 'reported_epoch': 458, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:30:44.490644+0000', 'last_change': '2023-11-08T05:28:13.557071+0000', 'last_active': '2023-11-08T05:30:44.490644+0000', 'last_peered': '2023-11-08T05:30:44.490644+0000', 'last_clean': '2023-11-08T05:28:01.458439+0000', 'last_became_active': '2023-11-08T05:28:13.557071+0000', 'last_became_peered': '2023-11-08T05:28:13.557071+0000', 'last_unstale': '2023-11-08T05:30:44.490644+0000', 'last_undegraded': '2023-11-08T05:30:44.490644+0000', 'last_fullsized': '2023-11-08T05:28:13.547025+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:32:20.008 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-11-08T05:32:20.008 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 11, 'reported_epoch': 459, 'state': 'active+clean+remapped', 'last_fresh': '2023-11-08T05:30:45.794760+0000', 'last_change': '2023-11-08T05:30:45.794760+0000', 'last_active': '2023-11-08T05:30:45.794760+0000', 'last_peered': '2023-11-08T05:30:45.794760+0000', 'last_clean': '2023-11-08T05:30:45.794760+0000', 'last_became_active': '2023-11-08T05:30:45.794398+0000', 'last_became_peered': '2023-11-08T05:30:45.794398+0000', 'last_unstale': '2023-11-08T05:30:45.794760+0000', 'last_undegraded': '2023-11-08T05:30:45.794760+0000', 'last_fullsized': '2023-11-08T05:30:45.794760+0000', 'mapping_epoch': 458, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 459, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2], 'acting': [2, 1], 'avail_no_missing': ['2', '1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2023-11-08T05:32:20.008 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.1 is not active+clean 2023-11-08T05:32:20.009 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.1', 'version': "0'0", 'reported_seq': 465, 'reported_epoch': 459, 'state': 'active+clean+remapped', 'last_fresh': '2023-11-08T05:30:45.795085+0000', 'last_change': '2023-11-08T05:30:45.795085+0000', 'last_active': '2023-11-08T05:30:45.795085+0000', 'last_peered': '2023-11-08T05:30:45.795085+0000', 'last_clean': '2023-11-08T05:30:45.795085+0000', 'last_became_active': '2023-11-08T05:30:45.794743+0000', 'last_became_peered': '2023-11-08T05:30:45.794743+0000', 'last_unstale': '2023-11-08T05:30:45.795085+0000', 'last_undegraded': '2023-11-08T05:30:45.795085+0000', 'last_fullsized': '2023-11-08T05:30:45.795085+0000', 'mapping_epoch': 458, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 459, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2], 'acting': [2, 1], 'avail_no_missing': ['2', '1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2023-11-08T05:32:20.009 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-11-08T05:32:20.009 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "0'0", 'reported_seq': 460, 'reported_epoch': 458, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:30:44.490523+0000', 'last_change': '2023-11-08T05:28:13.545468+0000', 'last_active': '2023-11-08T05:30:44.490523+0000', 'last_peered': '2023-11-08T05:30:44.490523+0000', 'last_clean': '2023-11-08T05:28:01.456723+0000', 'last_became_active': '2023-11-08T05:28:13.545468+0000', 'last_became_peered': '2023-11-08T05:28:13.545468+0000', 'last_unstale': '2023-11-08T05:30:44.490523+0000', 'last_undegraded': '2023-11-08T05:30:44.490523+0000', 'last_fullsized': '2023-11-08T05:28:13.537207+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 9, 'last_epoch_clean': 10, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:45.528796+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:45.528796+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:45.528796+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:32:20.009 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.0 is not active+clean 2023-11-08T05:32:20.010 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.0', 'version': "0'0", 'reported_seq': 465, 'reported_epoch': 458, 'state': 'active+clean+remapped', 'last_fresh': '2023-11-08T05:30:44.491907+0000', 'last_change': '2023-11-08T05:29:44.395350+0000', 'last_active': '2023-11-08T05:30:44.491907+0000', 'last_peered': '2023-11-08T05:30:44.491907+0000', 'last_clean': '2023-11-08T05:30:44.491907+0000', 'last_became_active': '2023-11-08T05:29:44.395012+0000', 'last_became_peered': '2023-11-08T05:29:44.395012+0000', 'last_unstale': '2023-11-08T05:30:44.491907+0000', 'last_undegraded': '2023-11-08T05:30:44.491907+0000', 'last_fullsized': '2023-11-08T05:30:44.491907+0000', 'mapping_epoch': 455, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 456, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2], 'acting': [2, 1], 'avail_no_missing': ['2', '1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2023-11-08T05:32:20.010 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-11-08T05:32:20.011 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 459, 'reported_epoch': 458, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:30:44.490629+0000', 'last_change': '2023-11-08T05:28:13.563357+0000', 'last_active': '2023-11-08T05:30:44.490629+0000', 'last_peered': '2023-11-08T05:30:44.490629+0000', 'last_clean': '2023-11-08T05:28:01.457691+0000', 'last_became_active': '2023-11-08T05:28:13.563357+0000', 'last_became_peered': '2023-11-08T05:28:13.563357+0000', 'last_unstale': '2023-11-08T05:30:44.490629+0000', 'last_undegraded': '2023-11-08T05:30:44.490629+0000', 'last_fullsized': '2023-11-08T05:28:13.541614+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:32:20.011 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-11-08T05:32:20.012 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 459, 'reported_epoch': 458, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:30:44.490691+0000', 'last_change': '2023-11-08T05:28:13.550672+0000', 'last_active': '2023-11-08T05:30:44.490691+0000', 'last_peered': '2023-11-08T05:30:44.490691+0000', 'last_clean': '2023-11-08T05:28:01.457904+0000', 'last_became_active': '2023-11-08T05:28:13.550672+0000', 'last_became_peered': '2023-11-08T05:28:13.550672+0000', 'last_unstale': '2023-11-08T05:30:44.490691+0000', 'last_undegraded': '2023-11-08T05:30:44.490691+0000', 'last_fullsized': '2023-11-08T05:28:13.539595+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:32:20.012 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-11-08T05:32:20.012 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 459, 'reported_epoch': 458, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:30:44.490805+0000', 'last_change': '2023-11-08T05:28:13.557968+0000', 'last_active': '2023-11-08T05:30:44.490805+0000', 'last_peered': '2023-11-08T05:30:44.490805+0000', 'last_clean': '2023-11-08T05:28:01.458433+0000', 'last_became_active': '2023-11-08T05:28:13.557968+0000', 'last_became_peered': '2023-11-08T05:28:13.557968+0000', 'last_unstale': '2023-11-08T05:30:44.490805+0000', 'last_undegraded': '2023-11-08T05:30:44.490805+0000', 'last_fullsized': '2023-11-08T05:28:13.542832+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:32:20.013 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-11-08T05:32:20.358 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T05:32:20.358 INFO:teuthology.orchestra.run.smithi087.stdout:{"fsid":"ab4cbc06-449d-497d-b513-220d9db2b590","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 5 pgs undersized","count":5},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2382,"monmap":{"epoch":1,"min_mon_release_name":"pacific","num_mons":1},"osdmap":{"epoch":459,"num_osds":3,"num_up_osds":2,"osd_up_since":1699421292,"num_in_osds":3,"osd_in_since":1699419158,"num_remapped_pgs":3},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean+remapped","count":3},{"state_name":"active+clean","count":1}],"num_pgs":9,"num_pools":2,"num_objects":0,"data_bytes":0,"bytes_used":2697592832,"bytes_avail":287212699648,"bytes_total":289910292480},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":41,"modified":"2023-11-08T05:29:32.044544+0000","services":{}},"progress_events":{}} 2023-11-08T05:32:20.372 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-11-08T05:32:20.372 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-11-08T05:32:20.658 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T05:32:20.660 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2023-11-08T05:32:20.671 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":1644,"stamp":"2023-11-08T05:32:20.079200+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":10,"acting":13,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":109,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2634368,"kb_used_data":409012,"kb_used_omap":780,"kb_used_meta":2220403,"kb_avail":280481152,"statfs":{"total":289910292480,"available":287212699648,"internally_reserved":0,"allocated":418828288,"data_stored":410500163,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":799447,"internal_metadata":2273692969},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-536782902,"num_objects":-164,"num_object_clones":0,"num_object_copies":-328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-115,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-164,"num_whiteouts":0,"num_read":-122056,"num_read_kb":-39011942,"num_write":-413642,"num_write_kb":-151705339,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-8,"num_bytes_recovered":-16922624,"num_keys_recovered":-2,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.424817"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490644+0000","last_change":"2023-11-08T05:28:13.557071+0000","last_active":"2023-11-08T05:30:44.490644+0000","last_peered":"2023-11-08T05:30:44.490644+0000","last_clean":"2023-11-08T05:28:01.458439+0000","last_became_active":"2023-11-08T05:28:13.557071+0000","last_became_peered":"2023-11-08T05:28:13.557071+0000","last_unstale":"2023-11-08T05:30:44.490644+0000","last_undegraded":"2023-11-08T05:30:44.490644+0000","last_fullsized":"2023-11-08T05:28:13.547025+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":459,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:45.794760+0000","last_change":"2023-11-08T05:30:45.794760+0000","last_active":"2023-11-08T05:30:45.794760+0000","last_peered":"2023-11-08T05:30:45.794760+0000","last_clean":"2023-11-08T05:30:45.794760+0000","last_became_active":"2023-11-08T05:30:45.794398+0000","last_became_peered":"2023-11-08T05:30:45.794398+0000","last_unstale":"2023-11-08T05:30:45.794760+0000","last_undegraded":"2023-11-08T05:30:45.794760+0000","last_fullsized":"2023-11-08T05:30:45.794760+0000","mapping_epoch":458,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":459,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":465,"reported_epoch":459,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:45.795085+0000","last_change":"2023-11-08T05:30:45.795085+0000","last_active":"2023-11-08T05:30:45.795085+0000","last_peered":"2023-11-08T05:30:45.795085+0000","last_clean":"2023-11-08T05:30:45.795085+0000","last_became_active":"2023-11-08T05:30:45.794743+0000","last_became_peered":"2023-11-08T05:30:45.794743+0000","last_unstale":"2023-11-08T05:30:45.795085+0000","last_undegraded":"2023-11-08T05:30:45.795085+0000","last_fullsized":"2023-11-08T05:30:45.795085+0000","mapping_epoch":458,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":459,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":454,"reported_epoch":458,"state":"active+clean","last_fresh":"2023-11-08T05:30:44.490618+0000","last_change":"2023-11-08T04:52:47.548597+0000","last_active":"2023-11-08T05:30:44.490618+0000","last_peered":"2023-11-08T05:30:44.490618+0000","last_clean":"2023-11-08T05:30:44.490618+0000","last_became_active":"2023-11-08T04:52:47.548294+0000","last_became_peered":"2023-11-08T04:52:47.548294+0000","last_unstale":"2023-11-08T05:30:44.490618+0000","last_undegraded":"2023-11-08T05:30:44.490618+0000","last_fullsized":"2023-11-08T05:30:44.490618+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":460,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490523+0000","last_change":"2023-11-08T05:28:13.545468+0000","last_active":"2023-11-08T05:30:44.490523+0000","last_peered":"2023-11-08T05:30:44.490523+0000","last_clean":"2023-11-08T05:28:01.456723+0000","last_became_active":"2023-11-08T05:28:13.545468+0000","last_became_peered":"2023-11-08T05:28:13.545468+0000","last_unstale":"2023-11-08T05:30:44.490523+0000","last_undegraded":"2023-11-08T05:30:44.490523+0000","last_fullsized":"2023-11-08T05:28:13.537207+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_clean_scrub_stamp":"2023-11-08T04:52:45.528796+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":465,"reported_epoch":458,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:44.491907+0000","last_change":"2023-11-08T05:29:44.395350+0000","last_active":"2023-11-08T05:30:44.491907+0000","last_peered":"2023-11-08T05:30:44.491907+0000","last_clean":"2023-11-08T05:30:44.491907+0000","last_became_active":"2023-11-08T05:29:44.395012+0000","last_became_peered":"2023-11-08T05:29:44.395012+0000","last_unstale":"2023-11-08T05:30:44.491907+0000","last_undegraded":"2023-11-08T05:30:44.491907+0000","last_fullsized":"2023-11-08T05:30:44.491907+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490629+0000","last_change":"2023-11-08T05:28:13.563357+0000","last_active":"2023-11-08T05:30:44.490629+0000","last_peered":"2023-11-08T05:30:44.490629+0000","last_clean":"2023-11-08T05:28:01.457691+0000","last_became_active":"2023-11-08T05:28:13.563357+0000","last_became_peered":"2023-11-08T05:28:13.563357+0000","last_unstale":"2023-11-08T05:30:44.490629+0000","last_undegraded":"2023-11-08T05:30:44.490629+0000","last_fullsized":"2023-11-08T05:28:13.541614+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490691+0000","last_change":"2023-11-08T05:28:13.550672+0000","last_active":"2023-11-08T05:30:44.490691+0000","last_peered":"2023-11-08T05:30:44.490691+0000","last_clean":"2023-11-08T05:28:01.457904+0000","last_became_active":"2023-11-08T05:28:13.550672+0000","last_became_peered":"2023-11-08T05:28:13.550672+0000","last_unstale":"2023-11-08T05:30:44.490691+0000","last_undegraded":"2023-11-08T05:30:44.490691+0000","last_fullsized":"2023-11-08T05:28:13.539595+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490805+0000","last_change":"2023-11-08T05:28:13.557968+0000","last_active":"2023-11-08T05:30:44.490805+0000","last_peered":"2023-11-08T05:30:44.490805+0000","last_clean":"2023-11-08T05:28:01.458433+0000","last_became_active":"2023-11-08T05:28:13.557968+0000","last_became_peered":"2023-11-08T05:28:13.557968+0000","last_unstale":"2023-11-08T05:30:44.490805+0000","last_undegraded":"2023-11-08T05:30:44.490805+0000","last_fullsized":"2023-11-08T05:28:13.542832+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":1,"acting":1,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":12,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738846,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":645040,"kb_used_data":5352,"kb_used_omap":275,"kb_used_meta":639404,"kb_avail":93726800,"statfs":{"total":96636764160,"available":95976243200,"internally_reserved":0,"allocated":5480448,"data_stored":2697647,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":281650,"internal_metadata":654750670},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:59 2023","interfaces":[{"interface":"back","average":{"1min":0.30499999999999999,"5min":0.439,"15min":0.36099999999999999},"min":{"1min":0.17399999999999999,"5min":0.14499999999999999,"15min":0.14499999999999999},"max":{"1min":0.60099999999999998,"5min":4.9379999999999997,"15min":4.9379999999999997},"last":0.379},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.49399999999999999,"15min":0.42399999999999999},"min":{"1min":0.221,"5min":0.19600000000000001,"15min":0.17000000000000001},"max":{"1min":0.63200000000000001,"5min":4.9859999999999998,"15min":4.9859999999999998},"last":0.40899999999999997}]},{"osd":1,"last update":"Wed Nov 8 05:32:06 2023","interfaces":[{"interface":"back","average":{"1min":0.27400000000000002,"5min":0.307,"15min":0.46800000000000003},"min":{"1min":0.17599999999999999,"5min":0.16,"15min":0.153},"max":{"1min":0.39000000000000001,"5min":1.825,"15min":23.242999999999999},"last":0.29199999999999998},{"interface":"front","average":{"1min":0.28000000000000003,"5min":0.34799999999999998,"15min":0.45700000000000002},"min":{"1min":0.20200000000000001,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":0.41699999999999998,"5min":2.0419999999999998,"15min":23.251000000000001},"last":0.311}]}]},{"osd":0,"up_from":8,"seq":34359738798,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1339228,"kb_used_data":398304,"kb_used_omap":291,"kb_used_meta":936476,"kb_avail":93032612,"statfs":{"total":96636764160,"available":95265394688,"internally_reserved":0,"allocated":407863296,"data_stored":405103900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":298462,"internal_metadata":958951970},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.436,"15min":0.36299999999999999},"min":{"1min":0.19700000000000001,"5min":0.14699999999999999,"15min":0.13700000000000001},"max":{"1min":3.222,"5min":6.5499999999999998,"15min":6.9989999999999997},"last":0.245},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.46300000000000002,"15min":0.42299999999999999},"min":{"1min":0.17799999999999999,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":5.0030000000000001,"5min":6.0629999999999997,"15min":9.6869999999999994},"last":0.28599999999999998}]},{"osd":2,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.51600000000000001,"15min":0.47099999999999997},"min":{"1min":0.21099999999999999,"5min":0.153,"15min":0.153},"max":{"1min":5.7439999999999998,"5min":5.8280000000000003,"15min":9.7639999999999993},"last":0.252},{"interface":"front","average":{"1min":0.497,"5min":0.56200000000000006,"15min":0.42299999999999999},"min":{"1min":0.20399999999999999,"5min":0.187,"15min":0.16800000000000001},"max":{"1min":3.9319999999999999,"5min":8.8040000000000003,"15min":8.8040000000000003},"last":0.29899999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738847,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":650100,"kb_used_data":5356,"kb_used_omap":214,"kb_used_meta":644521,"kb_avail":93721740,"statfs":{"total":96636764160,"available":95971061760,"internally_reserved":0,"allocated":5484544,"data_stored":2698616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":219335,"internal_metadata":659990329},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:56 2023","interfaces":[{"interface":"back","average":{"1min":0.30399999999999999,"5min":0.34699999999999998,"15min":0.36499999999999999},"min":{"1min":0.23999999999999999,"5min":0.14199999999999999,"15min":0.14199999999999999},"max":{"1min":0.42399999999999999,"5min":1.595,"15min":7.4249999999999998},"last":0.17100000000000001},{"interface":"front","average":{"1min":0.35499999999999998,"5min":0.40899999999999997,"15min":0.376},"min":{"1min":0.23999999999999999,"5min":0.19600000000000001,"15min":0.183},"max":{"1min":0.46999999999999997,"5min":2.6859999999999999,"15min":2.6859999999999999},"last":0.23000000000000001}]},{"osd":2,"last update":"Wed Nov 8 05:32:02 2023","interfaces":[{"interface":"back","average":{"1min":0.252,"5min":0.33000000000000002,"15min":0.36499999999999999},"min":{"1min":0.161,"5min":0.161,"15min":0.161},"max":{"1min":0.39600000000000002,"5min":2.2650000000000001,"15min":5.8209999999999997},"last":0.40799999999999997},{"interface":"front","average":{"1min":0.25800000000000001,"5min":0.29899999999999999,"15min":0.35599999999999998},"min":{"1min":0.17000000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.35799999999999998,"5min":1.137,"15min":4.2350000000000003},"last":0.41999999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-08T05:32:21.228 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~243s 2023-11-08T05:32:23.673 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-11-08T05:32:23.968 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T05:32:23.970 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2023-11-08T05:32:23.981 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":1645,"stamp":"2023-11-08T05:32:22.079564+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":10,"acting":13,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":109,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2634368,"kb_used_data":409012,"kb_used_omap":780,"kb_used_meta":2220403,"kb_avail":280481152,"statfs":{"total":289910292480,"available":287212699648,"internally_reserved":0,"allocated":418828288,"data_stored":410500163,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":799447,"internal_metadata":2273692969},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-536782902,"num_objects":-164,"num_object_clones":0,"num_object_copies":-328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-115,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-164,"num_whiteouts":0,"num_read":-122056,"num_read_kb":-39011942,"num_write":-413642,"num_write_kb":-151705339,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-8,"num_bytes_recovered":-16922624,"num_keys_recovered":-2,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.424817"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490644+0000","last_change":"2023-11-08T05:28:13.557071+0000","last_active":"2023-11-08T05:30:44.490644+0000","last_peered":"2023-11-08T05:30:44.490644+0000","last_clean":"2023-11-08T05:28:01.458439+0000","last_became_active":"2023-11-08T05:28:13.557071+0000","last_became_peered":"2023-11-08T05:28:13.557071+0000","last_unstale":"2023-11-08T05:30:44.490644+0000","last_undegraded":"2023-11-08T05:30:44.490644+0000","last_fullsized":"2023-11-08T05:28:13.547025+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":459,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:45.794760+0000","last_change":"2023-11-08T05:30:45.794760+0000","last_active":"2023-11-08T05:30:45.794760+0000","last_peered":"2023-11-08T05:30:45.794760+0000","last_clean":"2023-11-08T05:30:45.794760+0000","last_became_active":"2023-11-08T05:30:45.794398+0000","last_became_peered":"2023-11-08T05:30:45.794398+0000","last_unstale":"2023-11-08T05:30:45.794760+0000","last_undegraded":"2023-11-08T05:30:45.794760+0000","last_fullsized":"2023-11-08T05:30:45.794760+0000","mapping_epoch":458,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":459,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":465,"reported_epoch":459,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:45.795085+0000","last_change":"2023-11-08T05:30:45.795085+0000","last_active":"2023-11-08T05:30:45.795085+0000","last_peered":"2023-11-08T05:30:45.795085+0000","last_clean":"2023-11-08T05:30:45.795085+0000","last_became_active":"2023-11-08T05:30:45.794743+0000","last_became_peered":"2023-11-08T05:30:45.794743+0000","last_unstale":"2023-11-08T05:30:45.795085+0000","last_undegraded":"2023-11-08T05:30:45.795085+0000","last_fullsized":"2023-11-08T05:30:45.795085+0000","mapping_epoch":458,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":459,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":454,"reported_epoch":458,"state":"active+clean","last_fresh":"2023-11-08T05:30:44.490618+0000","last_change":"2023-11-08T04:52:47.548597+0000","last_active":"2023-11-08T05:30:44.490618+0000","last_peered":"2023-11-08T05:30:44.490618+0000","last_clean":"2023-11-08T05:30:44.490618+0000","last_became_active":"2023-11-08T04:52:47.548294+0000","last_became_peered":"2023-11-08T04:52:47.548294+0000","last_unstale":"2023-11-08T05:30:44.490618+0000","last_undegraded":"2023-11-08T05:30:44.490618+0000","last_fullsized":"2023-11-08T05:30:44.490618+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":460,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490523+0000","last_change":"2023-11-08T05:28:13.545468+0000","last_active":"2023-11-08T05:30:44.490523+0000","last_peered":"2023-11-08T05:30:44.490523+0000","last_clean":"2023-11-08T05:28:01.456723+0000","last_became_active":"2023-11-08T05:28:13.545468+0000","last_became_peered":"2023-11-08T05:28:13.545468+0000","last_unstale":"2023-11-08T05:30:44.490523+0000","last_undegraded":"2023-11-08T05:30:44.490523+0000","last_fullsized":"2023-11-08T05:28:13.537207+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_clean_scrub_stamp":"2023-11-08T04:52:45.528796+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":465,"reported_epoch":458,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:44.491907+0000","last_change":"2023-11-08T05:29:44.395350+0000","last_active":"2023-11-08T05:30:44.491907+0000","last_peered":"2023-11-08T05:30:44.491907+0000","last_clean":"2023-11-08T05:30:44.491907+0000","last_became_active":"2023-11-08T05:29:44.395012+0000","last_became_peered":"2023-11-08T05:29:44.395012+0000","last_unstale":"2023-11-08T05:30:44.491907+0000","last_undegraded":"2023-11-08T05:30:44.491907+0000","last_fullsized":"2023-11-08T05:30:44.491907+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490629+0000","last_change":"2023-11-08T05:28:13.563357+0000","last_active":"2023-11-08T05:30:44.490629+0000","last_peered":"2023-11-08T05:30:44.490629+0000","last_clean":"2023-11-08T05:28:01.457691+0000","last_became_active":"2023-11-08T05:28:13.563357+0000","last_became_peered":"2023-11-08T05:28:13.563357+0000","last_unstale":"2023-11-08T05:30:44.490629+0000","last_undegraded":"2023-11-08T05:30:44.490629+0000","last_fullsized":"2023-11-08T05:28:13.541614+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490691+0000","last_change":"2023-11-08T05:28:13.550672+0000","last_active":"2023-11-08T05:30:44.490691+0000","last_peered":"2023-11-08T05:30:44.490691+0000","last_clean":"2023-11-08T05:28:01.457904+0000","last_became_active":"2023-11-08T05:28:13.550672+0000","last_became_peered":"2023-11-08T05:28:13.550672+0000","last_unstale":"2023-11-08T05:30:44.490691+0000","last_undegraded":"2023-11-08T05:30:44.490691+0000","last_fullsized":"2023-11-08T05:28:13.539595+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490805+0000","last_change":"2023-11-08T05:28:13.557968+0000","last_active":"2023-11-08T05:30:44.490805+0000","last_peered":"2023-11-08T05:30:44.490805+0000","last_clean":"2023-11-08T05:28:01.458433+0000","last_became_active":"2023-11-08T05:28:13.557968+0000","last_became_peered":"2023-11-08T05:28:13.557968+0000","last_unstale":"2023-11-08T05:30:44.490805+0000","last_undegraded":"2023-11-08T05:30:44.490805+0000","last_fullsized":"2023-11-08T05:28:13.542832+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":1,"acting":1,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":12,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738846,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":645040,"kb_used_data":5352,"kb_used_omap":275,"kb_used_meta":639404,"kb_avail":93726800,"statfs":{"total":96636764160,"available":95976243200,"internally_reserved":0,"allocated":5480448,"data_stored":2697647,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":281650,"internal_metadata":654750670},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:59 2023","interfaces":[{"interface":"back","average":{"1min":0.30499999999999999,"5min":0.439,"15min":0.36099999999999999},"min":{"1min":0.17399999999999999,"5min":0.14499999999999999,"15min":0.14499999999999999},"max":{"1min":0.60099999999999998,"5min":4.9379999999999997,"15min":4.9379999999999997},"last":0.379},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.49399999999999999,"15min":0.42399999999999999},"min":{"1min":0.221,"5min":0.19600000000000001,"15min":0.17000000000000001},"max":{"1min":0.63200000000000001,"5min":4.9859999999999998,"15min":4.9859999999999998},"last":0.40899999999999997}]},{"osd":1,"last update":"Wed Nov 8 05:32:06 2023","interfaces":[{"interface":"back","average":{"1min":0.27400000000000002,"5min":0.307,"15min":0.46800000000000003},"min":{"1min":0.17599999999999999,"5min":0.16,"15min":0.153},"max":{"1min":0.39000000000000001,"5min":1.825,"15min":23.242999999999999},"last":0.29199999999999998},{"interface":"front","average":{"1min":0.28000000000000003,"5min":0.34799999999999998,"15min":0.45700000000000002},"min":{"1min":0.20200000000000001,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":0.41699999999999998,"5min":2.0419999999999998,"15min":23.251000000000001},"last":0.311}]}]},{"osd":0,"up_from":8,"seq":34359738798,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1339228,"kb_used_data":398304,"kb_used_omap":291,"kb_used_meta":936476,"kb_avail":93032612,"statfs":{"total":96636764160,"available":95265394688,"internally_reserved":0,"allocated":407863296,"data_stored":405103900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":298462,"internal_metadata":958951970},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.436,"15min":0.36299999999999999},"min":{"1min":0.19700000000000001,"5min":0.14699999999999999,"15min":0.13700000000000001},"max":{"1min":3.222,"5min":6.5499999999999998,"15min":6.9989999999999997},"last":0.245},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.46300000000000002,"15min":0.42299999999999999},"min":{"1min":0.17799999999999999,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":5.0030000000000001,"5min":6.0629999999999997,"15min":9.6869999999999994},"last":0.28599999999999998}]},{"osd":2,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.51600000000000001,"15min":0.47099999999999997},"min":{"1min":0.21099999999999999,"5min":0.153,"15min":0.153},"max":{"1min":5.7439999999999998,"5min":5.8280000000000003,"15min":9.7639999999999993},"last":0.252},{"interface":"front","average":{"1min":0.497,"5min":0.56200000000000006,"15min":0.42299999999999999},"min":{"1min":0.20399999999999999,"5min":0.187,"15min":0.16800000000000001},"max":{"1min":3.9319999999999999,"5min":8.8040000000000003,"15min":8.8040000000000003},"last":0.29899999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738848,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":650100,"kb_used_data":5356,"kb_used_omap":214,"kb_used_meta":644521,"kb_avail":93721740,"statfs":{"total":96636764160,"available":95971061760,"internally_reserved":0,"allocated":5484544,"data_stored":2698616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":219335,"internal_metadata":659990329},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:56 2023","interfaces":[{"interface":"back","average":{"1min":0.30399999999999999,"5min":0.34699999999999998,"15min":0.36499999999999999},"min":{"1min":0.23999999999999999,"5min":0.14199999999999999,"15min":0.14199999999999999},"max":{"1min":0.42399999999999999,"5min":1.595,"15min":7.4249999999999998},"last":0.17100000000000001},{"interface":"front","average":{"1min":0.35499999999999998,"5min":0.40899999999999997,"15min":0.376},"min":{"1min":0.23999999999999999,"5min":0.19600000000000001,"15min":0.183},"max":{"1min":0.46999999999999997,"5min":2.6859999999999999,"15min":2.6859999999999999},"last":0.23000000000000001}]},{"osd":2,"last update":"Wed Nov 8 05:32:02 2023","interfaces":[{"interface":"back","average":{"1min":0.252,"5min":0.33000000000000002,"15min":0.36499999999999999},"min":{"1min":0.161,"5min":0.161,"15min":0.161},"max":{"1min":0.39600000000000002,"5min":2.2650000000000001,"15min":5.8209999999999997},"last":0.28499999999999998},{"interface":"front","average":{"1min":0.25800000000000001,"5min":0.29899999999999999,"15min":0.35599999999999998},"min":{"1min":0.17000000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.35799999999999998,"5min":1.137,"15min":4.2350000000000003},"last":0.30599999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-08T05:32:23.982 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-11-08T05:32:24.267 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T05:32:24.268 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2023-11-08T05:32:24.279 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":1646,"stamp":"2023-11-08T05:32:24.079956+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":10,"acting":13,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":109,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2634368,"kb_used_data":409012,"kb_used_omap":780,"kb_used_meta":2220403,"kb_avail":280481152,"statfs":{"total":289910292480,"available":287212699648,"internally_reserved":0,"allocated":418828288,"data_stored":410500163,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":799447,"internal_metadata":2273692969},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-536782902,"num_objects":-164,"num_object_clones":0,"num_object_copies":-328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-115,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-164,"num_whiteouts":0,"num_read":-122056,"num_read_kb":-39011942,"num_write":-413642,"num_write_kb":-151705339,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-8,"num_bytes_recovered":-16922624,"num_keys_recovered":-2,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.424817"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490644+0000","last_change":"2023-11-08T05:28:13.557071+0000","last_active":"2023-11-08T05:30:44.490644+0000","last_peered":"2023-11-08T05:30:44.490644+0000","last_clean":"2023-11-08T05:28:01.458439+0000","last_became_active":"2023-11-08T05:28:13.557071+0000","last_became_peered":"2023-11-08T05:28:13.557071+0000","last_unstale":"2023-11-08T05:30:44.490644+0000","last_undegraded":"2023-11-08T05:30:44.490644+0000","last_fullsized":"2023-11-08T05:28:13.547025+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":459,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:45.794760+0000","last_change":"2023-11-08T05:30:45.794760+0000","last_active":"2023-11-08T05:30:45.794760+0000","last_peered":"2023-11-08T05:30:45.794760+0000","last_clean":"2023-11-08T05:30:45.794760+0000","last_became_active":"2023-11-08T05:30:45.794398+0000","last_became_peered":"2023-11-08T05:30:45.794398+0000","last_unstale":"2023-11-08T05:30:45.794760+0000","last_undegraded":"2023-11-08T05:30:45.794760+0000","last_fullsized":"2023-11-08T05:30:45.794760+0000","mapping_epoch":458,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":459,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":465,"reported_epoch":459,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:45.795085+0000","last_change":"2023-11-08T05:30:45.795085+0000","last_active":"2023-11-08T05:30:45.795085+0000","last_peered":"2023-11-08T05:30:45.795085+0000","last_clean":"2023-11-08T05:30:45.795085+0000","last_became_active":"2023-11-08T05:30:45.794743+0000","last_became_peered":"2023-11-08T05:30:45.794743+0000","last_unstale":"2023-11-08T05:30:45.795085+0000","last_undegraded":"2023-11-08T05:30:45.795085+0000","last_fullsized":"2023-11-08T05:30:45.795085+0000","mapping_epoch":458,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":459,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":454,"reported_epoch":458,"state":"active+clean","last_fresh":"2023-11-08T05:30:44.490618+0000","last_change":"2023-11-08T04:52:47.548597+0000","last_active":"2023-11-08T05:30:44.490618+0000","last_peered":"2023-11-08T05:30:44.490618+0000","last_clean":"2023-11-08T05:30:44.490618+0000","last_became_active":"2023-11-08T04:52:47.548294+0000","last_became_peered":"2023-11-08T04:52:47.548294+0000","last_unstale":"2023-11-08T05:30:44.490618+0000","last_undegraded":"2023-11-08T05:30:44.490618+0000","last_fullsized":"2023-11-08T05:30:44.490618+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":460,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490523+0000","last_change":"2023-11-08T05:28:13.545468+0000","last_active":"2023-11-08T05:30:44.490523+0000","last_peered":"2023-11-08T05:30:44.490523+0000","last_clean":"2023-11-08T05:28:01.456723+0000","last_became_active":"2023-11-08T05:28:13.545468+0000","last_became_peered":"2023-11-08T05:28:13.545468+0000","last_unstale":"2023-11-08T05:30:44.490523+0000","last_undegraded":"2023-11-08T05:30:44.490523+0000","last_fullsized":"2023-11-08T05:28:13.537207+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_clean_scrub_stamp":"2023-11-08T04:52:45.528796+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":465,"reported_epoch":458,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:44.491907+0000","last_change":"2023-11-08T05:29:44.395350+0000","last_active":"2023-11-08T05:30:44.491907+0000","last_peered":"2023-11-08T05:30:44.491907+0000","last_clean":"2023-11-08T05:30:44.491907+0000","last_became_active":"2023-11-08T05:29:44.395012+0000","last_became_peered":"2023-11-08T05:29:44.395012+0000","last_unstale":"2023-11-08T05:30:44.491907+0000","last_undegraded":"2023-11-08T05:30:44.491907+0000","last_fullsized":"2023-11-08T05:30:44.491907+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490629+0000","last_change":"2023-11-08T05:28:13.563357+0000","last_active":"2023-11-08T05:30:44.490629+0000","last_peered":"2023-11-08T05:30:44.490629+0000","last_clean":"2023-11-08T05:28:01.457691+0000","last_became_active":"2023-11-08T05:28:13.563357+0000","last_became_peered":"2023-11-08T05:28:13.563357+0000","last_unstale":"2023-11-08T05:30:44.490629+0000","last_undegraded":"2023-11-08T05:30:44.490629+0000","last_fullsized":"2023-11-08T05:28:13.541614+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490691+0000","last_change":"2023-11-08T05:28:13.550672+0000","last_active":"2023-11-08T05:30:44.490691+0000","last_peered":"2023-11-08T05:30:44.490691+0000","last_clean":"2023-11-08T05:28:01.457904+0000","last_became_active":"2023-11-08T05:28:13.550672+0000","last_became_peered":"2023-11-08T05:28:13.550672+0000","last_unstale":"2023-11-08T05:30:44.490691+0000","last_undegraded":"2023-11-08T05:30:44.490691+0000","last_fullsized":"2023-11-08T05:28:13.539595+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490805+0000","last_change":"2023-11-08T05:28:13.557968+0000","last_active":"2023-11-08T05:30:44.490805+0000","last_peered":"2023-11-08T05:30:44.490805+0000","last_clean":"2023-11-08T05:28:01.458433+0000","last_became_active":"2023-11-08T05:28:13.557968+0000","last_became_peered":"2023-11-08T05:28:13.557968+0000","last_unstale":"2023-11-08T05:30:44.490805+0000","last_undegraded":"2023-11-08T05:30:44.490805+0000","last_fullsized":"2023-11-08T05:28:13.542832+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":1,"acting":1,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":12,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738846,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":645040,"kb_used_data":5352,"kb_used_omap":275,"kb_used_meta":639404,"kb_avail":93726800,"statfs":{"total":96636764160,"available":95976243200,"internally_reserved":0,"allocated":5480448,"data_stored":2697647,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":281650,"internal_metadata":654750670},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:59 2023","interfaces":[{"interface":"back","average":{"1min":0.30499999999999999,"5min":0.439,"15min":0.36099999999999999},"min":{"1min":0.17399999999999999,"5min":0.14499999999999999,"15min":0.14499999999999999},"max":{"1min":0.60099999999999998,"5min":4.9379999999999997,"15min":4.9379999999999997},"last":0.379},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.49399999999999999,"15min":0.42399999999999999},"min":{"1min":0.221,"5min":0.19600000000000001,"15min":0.17000000000000001},"max":{"1min":0.63200000000000001,"5min":4.9859999999999998,"15min":4.9859999999999998},"last":0.40899999999999997}]},{"osd":1,"last update":"Wed Nov 8 05:32:06 2023","interfaces":[{"interface":"back","average":{"1min":0.27400000000000002,"5min":0.307,"15min":0.46800000000000003},"min":{"1min":0.17599999999999999,"5min":0.16,"15min":0.153},"max":{"1min":0.39000000000000001,"5min":1.825,"15min":23.242999999999999},"last":0.29199999999999998},{"interface":"front","average":{"1min":0.28000000000000003,"5min":0.34799999999999998,"15min":0.45700000000000002},"min":{"1min":0.20200000000000001,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":0.41699999999999998,"5min":2.0419999999999998,"15min":23.251000000000001},"last":0.311}]}]},{"osd":0,"up_from":8,"seq":34359738798,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1339228,"kb_used_data":398304,"kb_used_omap":291,"kb_used_meta":936476,"kb_avail":93032612,"statfs":{"total":96636764160,"available":95265394688,"internally_reserved":0,"allocated":407863296,"data_stored":405103900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":298462,"internal_metadata":958951970},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.436,"15min":0.36299999999999999},"min":{"1min":0.19700000000000001,"5min":0.14699999999999999,"15min":0.13700000000000001},"max":{"1min":3.222,"5min":6.5499999999999998,"15min":6.9989999999999997},"last":0.245},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.46300000000000002,"15min":0.42299999999999999},"min":{"1min":0.17799999999999999,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":5.0030000000000001,"5min":6.0629999999999997,"15min":9.6869999999999994},"last":0.28599999999999998}]},{"osd":2,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.51600000000000001,"15min":0.47099999999999997},"min":{"1min":0.21099999999999999,"5min":0.153,"15min":0.153},"max":{"1min":5.7439999999999998,"5min":5.8280000000000003,"15min":9.7639999999999993},"last":0.252},{"interface":"front","average":{"1min":0.497,"5min":0.56200000000000006,"15min":0.42299999999999999},"min":{"1min":0.20399999999999999,"5min":0.187,"15min":0.16800000000000001},"max":{"1min":3.9319999999999999,"5min":8.8040000000000003,"15min":8.8040000000000003},"last":0.29899999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738848,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":650100,"kb_used_data":5356,"kb_used_omap":214,"kb_used_meta":644521,"kb_avail":93721740,"statfs":{"total":96636764160,"available":95971061760,"internally_reserved":0,"allocated":5484544,"data_stored":2698616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":219335,"internal_metadata":659990329},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:56 2023","interfaces":[{"interface":"back","average":{"1min":0.30399999999999999,"5min":0.34699999999999998,"15min":0.36499999999999999},"min":{"1min":0.23999999999999999,"5min":0.14199999999999999,"15min":0.14199999999999999},"max":{"1min":0.42399999999999999,"5min":1.595,"15min":7.4249999999999998},"last":0.17100000000000001},{"interface":"front","average":{"1min":0.35499999999999998,"5min":0.40899999999999997,"15min":0.376},"min":{"1min":0.23999999999999999,"5min":0.19600000000000001,"15min":0.183},"max":{"1min":0.46999999999999997,"5min":2.6859999999999999,"15min":2.6859999999999999},"last":0.23000000000000001}]},{"osd":2,"last update":"Wed Nov 8 05:32:02 2023","interfaces":[{"interface":"back","average":{"1min":0.252,"5min":0.33000000000000002,"15min":0.36499999999999999},"min":{"1min":0.161,"5min":0.161,"15min":0.161},"max":{"1min":0.39600000000000002,"5min":2.2650000000000001,"15min":5.8209999999999997},"last":0.28499999999999998},{"interface":"front","average":{"1min":0.25800000000000001,"5min":0.29899999999999999,"15min":0.35599999999999998},"min":{"1min":0.17000000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.35799999999999998,"5min":1.137,"15min":4.2350000000000003},"last":0.30599999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-08T05:32:24.281 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-11-08T05:32:24.281 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 459, 'reported_epoch': 458, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:30:44.490644+0000', 'last_change': '2023-11-08T05:28:13.557071+0000', 'last_active': '2023-11-08T05:30:44.490644+0000', 'last_peered': '2023-11-08T05:30:44.490644+0000', 'last_clean': '2023-11-08T05:28:01.458439+0000', 'last_became_active': '2023-11-08T05:28:13.557071+0000', 'last_became_peered': '2023-11-08T05:28:13.557071+0000', 'last_unstale': '2023-11-08T05:30:44.490644+0000', 'last_undegraded': '2023-11-08T05:30:44.490644+0000', 'last_fullsized': '2023-11-08T05:28:13.547025+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:32:24.281 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-11-08T05:32:24.282 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 11, 'reported_epoch': 459, 'state': 'active+clean+remapped', 'last_fresh': '2023-11-08T05:30:45.794760+0000', 'last_change': '2023-11-08T05:30:45.794760+0000', 'last_active': '2023-11-08T05:30:45.794760+0000', 'last_peered': '2023-11-08T05:30:45.794760+0000', 'last_clean': '2023-11-08T05:30:45.794760+0000', 'last_became_active': '2023-11-08T05:30:45.794398+0000', 'last_became_peered': '2023-11-08T05:30:45.794398+0000', 'last_unstale': '2023-11-08T05:30:45.794760+0000', 'last_undegraded': '2023-11-08T05:30:45.794760+0000', 'last_fullsized': '2023-11-08T05:30:45.794760+0000', 'mapping_epoch': 458, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 459, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2], 'acting': [2, 1], 'avail_no_missing': ['2', '1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2023-11-08T05:32:24.283 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.1 is not active+clean 2023-11-08T05:32:24.283 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.1', 'version': "0'0", 'reported_seq': 465, 'reported_epoch': 459, 'state': 'active+clean+remapped', 'last_fresh': '2023-11-08T05:30:45.795085+0000', 'last_change': '2023-11-08T05:30:45.795085+0000', 'last_active': '2023-11-08T05:30:45.795085+0000', 'last_peered': '2023-11-08T05:30:45.795085+0000', 'last_clean': '2023-11-08T05:30:45.795085+0000', 'last_became_active': '2023-11-08T05:30:45.794743+0000', 'last_became_peered': '2023-11-08T05:30:45.794743+0000', 'last_unstale': '2023-11-08T05:30:45.795085+0000', 'last_undegraded': '2023-11-08T05:30:45.795085+0000', 'last_fullsized': '2023-11-08T05:30:45.795085+0000', 'mapping_epoch': 458, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 459, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2], 'acting': [2, 1], 'avail_no_missing': ['2', '1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2023-11-08T05:32:24.283 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-11-08T05:32:24.284 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "0'0", 'reported_seq': 460, 'reported_epoch': 458, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:30:44.490523+0000', 'last_change': '2023-11-08T05:28:13.545468+0000', 'last_active': '2023-11-08T05:30:44.490523+0000', 'last_peered': '2023-11-08T05:30:44.490523+0000', 'last_clean': '2023-11-08T05:28:01.456723+0000', 'last_became_active': '2023-11-08T05:28:13.545468+0000', 'last_became_peered': '2023-11-08T05:28:13.545468+0000', 'last_unstale': '2023-11-08T05:30:44.490523+0000', 'last_undegraded': '2023-11-08T05:30:44.490523+0000', 'last_fullsized': '2023-11-08T05:28:13.537207+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 9, 'last_epoch_clean': 10, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:45.528796+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:45.528796+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:45.528796+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:32:24.284 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.0 is not active+clean 2023-11-08T05:32:24.284 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.0', 'version': "0'0", 'reported_seq': 465, 'reported_epoch': 458, 'state': 'active+clean+remapped', 'last_fresh': '2023-11-08T05:30:44.491907+0000', 'last_change': '2023-11-08T05:29:44.395350+0000', 'last_active': '2023-11-08T05:30:44.491907+0000', 'last_peered': '2023-11-08T05:30:44.491907+0000', 'last_clean': '2023-11-08T05:30:44.491907+0000', 'last_became_active': '2023-11-08T05:29:44.395012+0000', 'last_became_peered': '2023-11-08T05:29:44.395012+0000', 'last_unstale': '2023-11-08T05:30:44.491907+0000', 'last_undegraded': '2023-11-08T05:30:44.491907+0000', 'last_fullsized': '2023-11-08T05:30:44.491907+0000', 'mapping_epoch': 455, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 456, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2], 'acting': [2, 1], 'avail_no_missing': ['2', '1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2023-11-08T05:32:24.284 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-11-08T05:32:24.285 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 459, 'reported_epoch': 458, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:30:44.490629+0000', 'last_change': '2023-11-08T05:28:13.563357+0000', 'last_active': '2023-11-08T05:30:44.490629+0000', 'last_peered': '2023-11-08T05:30:44.490629+0000', 'last_clean': '2023-11-08T05:28:01.457691+0000', 'last_became_active': '2023-11-08T05:28:13.563357+0000', 'last_became_peered': '2023-11-08T05:28:13.563357+0000', 'last_unstale': '2023-11-08T05:30:44.490629+0000', 'last_undegraded': '2023-11-08T05:30:44.490629+0000', 'last_fullsized': '2023-11-08T05:28:13.541614+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:32:24.285 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-11-08T05:32:24.285 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 459, 'reported_epoch': 458, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:30:44.490691+0000', 'last_change': '2023-11-08T05:28:13.550672+0000', 'last_active': '2023-11-08T05:30:44.490691+0000', 'last_peered': '2023-11-08T05:30:44.490691+0000', 'last_clean': '2023-11-08T05:28:01.457904+0000', 'last_became_active': '2023-11-08T05:28:13.550672+0000', 'last_became_peered': '2023-11-08T05:28:13.550672+0000', 'last_unstale': '2023-11-08T05:30:44.490691+0000', 'last_undegraded': '2023-11-08T05:30:44.490691+0000', 'last_fullsized': '2023-11-08T05:28:13.539595+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:32:24.286 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-11-08T05:32:24.286 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 459, 'reported_epoch': 458, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:30:44.490805+0000', 'last_change': '2023-11-08T05:28:13.557968+0000', 'last_active': '2023-11-08T05:30:44.490805+0000', 'last_peered': '2023-11-08T05:30:44.490805+0000', 'last_clean': '2023-11-08T05:28:01.458433+0000', 'last_became_active': '2023-11-08T05:28:13.557968+0000', 'last_became_peered': '2023-11-08T05:28:13.557968+0000', 'last_unstale': '2023-11-08T05:30:44.490805+0000', 'last_undegraded': '2023-11-08T05:30:44.490805+0000', 'last_fullsized': '2023-11-08T05:28:13.542832+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:32:24.286 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-11-08T05:32:24.638 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T05:32:24.639 INFO:teuthology.orchestra.run.smithi087.stdout:{"fsid":"ab4cbc06-449d-497d-b513-220d9db2b590","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 5 pgs undersized","count":5},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2387,"monmap":{"epoch":1,"min_mon_release_name":"pacific","num_mons":1},"osdmap":{"epoch":459,"num_osds":3,"num_up_osds":2,"osd_up_since":1699421292,"num_in_osds":3,"osd_in_since":1699419158,"num_remapped_pgs":3},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean+remapped","count":3},{"state_name":"active+clean","count":1}],"num_pgs":9,"num_pools":2,"num_objects":0,"data_bytes":0,"bytes_used":2697592832,"bytes_avail":287212699648,"bytes_total":289910292480},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":41,"modified":"2023-11-08T05:29:32.044544+0000","services":{}},"progress_events":{}} 2023-11-08T05:32:24.652 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-11-08T05:32:24.653 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-11-08T05:32:24.935 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T05:32:24.937 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2023-11-08T05:32:24.948 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":1646,"stamp":"2023-11-08T05:32:24.079956+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":10,"acting":13,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":109,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2634368,"kb_used_data":409012,"kb_used_omap":780,"kb_used_meta":2220403,"kb_avail":280481152,"statfs":{"total":289910292480,"available":287212699648,"internally_reserved":0,"allocated":418828288,"data_stored":410500163,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":799447,"internal_metadata":2273692969},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-536782902,"num_objects":-164,"num_object_clones":0,"num_object_copies":-328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-115,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-164,"num_whiteouts":0,"num_read":-122056,"num_read_kb":-39011942,"num_write":-413642,"num_write_kb":-151705339,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-8,"num_bytes_recovered":-16922624,"num_keys_recovered":-2,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.424817"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490644+0000","last_change":"2023-11-08T05:28:13.557071+0000","last_active":"2023-11-08T05:30:44.490644+0000","last_peered":"2023-11-08T05:30:44.490644+0000","last_clean":"2023-11-08T05:28:01.458439+0000","last_became_active":"2023-11-08T05:28:13.557071+0000","last_became_peered":"2023-11-08T05:28:13.557071+0000","last_unstale":"2023-11-08T05:30:44.490644+0000","last_undegraded":"2023-11-08T05:30:44.490644+0000","last_fullsized":"2023-11-08T05:28:13.547025+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":459,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:45.794760+0000","last_change":"2023-11-08T05:30:45.794760+0000","last_active":"2023-11-08T05:30:45.794760+0000","last_peered":"2023-11-08T05:30:45.794760+0000","last_clean":"2023-11-08T05:30:45.794760+0000","last_became_active":"2023-11-08T05:30:45.794398+0000","last_became_peered":"2023-11-08T05:30:45.794398+0000","last_unstale":"2023-11-08T05:30:45.794760+0000","last_undegraded":"2023-11-08T05:30:45.794760+0000","last_fullsized":"2023-11-08T05:30:45.794760+0000","mapping_epoch":458,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":459,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":465,"reported_epoch":459,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:45.795085+0000","last_change":"2023-11-08T05:30:45.795085+0000","last_active":"2023-11-08T05:30:45.795085+0000","last_peered":"2023-11-08T05:30:45.795085+0000","last_clean":"2023-11-08T05:30:45.795085+0000","last_became_active":"2023-11-08T05:30:45.794743+0000","last_became_peered":"2023-11-08T05:30:45.794743+0000","last_unstale":"2023-11-08T05:30:45.795085+0000","last_undegraded":"2023-11-08T05:30:45.795085+0000","last_fullsized":"2023-11-08T05:30:45.795085+0000","mapping_epoch":458,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":459,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":454,"reported_epoch":458,"state":"active+clean","last_fresh":"2023-11-08T05:30:44.490618+0000","last_change":"2023-11-08T04:52:47.548597+0000","last_active":"2023-11-08T05:30:44.490618+0000","last_peered":"2023-11-08T05:30:44.490618+0000","last_clean":"2023-11-08T05:30:44.490618+0000","last_became_active":"2023-11-08T04:52:47.548294+0000","last_became_peered":"2023-11-08T04:52:47.548294+0000","last_unstale":"2023-11-08T05:30:44.490618+0000","last_undegraded":"2023-11-08T05:30:44.490618+0000","last_fullsized":"2023-11-08T05:30:44.490618+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":460,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490523+0000","last_change":"2023-11-08T05:28:13.545468+0000","last_active":"2023-11-08T05:30:44.490523+0000","last_peered":"2023-11-08T05:30:44.490523+0000","last_clean":"2023-11-08T05:28:01.456723+0000","last_became_active":"2023-11-08T05:28:13.545468+0000","last_became_peered":"2023-11-08T05:28:13.545468+0000","last_unstale":"2023-11-08T05:30:44.490523+0000","last_undegraded":"2023-11-08T05:30:44.490523+0000","last_fullsized":"2023-11-08T05:28:13.537207+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_clean_scrub_stamp":"2023-11-08T04:52:45.528796+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":465,"reported_epoch":458,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:44.491907+0000","last_change":"2023-11-08T05:29:44.395350+0000","last_active":"2023-11-08T05:30:44.491907+0000","last_peered":"2023-11-08T05:30:44.491907+0000","last_clean":"2023-11-08T05:30:44.491907+0000","last_became_active":"2023-11-08T05:29:44.395012+0000","last_became_peered":"2023-11-08T05:29:44.395012+0000","last_unstale":"2023-11-08T05:30:44.491907+0000","last_undegraded":"2023-11-08T05:30:44.491907+0000","last_fullsized":"2023-11-08T05:30:44.491907+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490629+0000","last_change":"2023-11-08T05:28:13.563357+0000","last_active":"2023-11-08T05:30:44.490629+0000","last_peered":"2023-11-08T05:30:44.490629+0000","last_clean":"2023-11-08T05:28:01.457691+0000","last_became_active":"2023-11-08T05:28:13.563357+0000","last_became_peered":"2023-11-08T05:28:13.563357+0000","last_unstale":"2023-11-08T05:30:44.490629+0000","last_undegraded":"2023-11-08T05:30:44.490629+0000","last_fullsized":"2023-11-08T05:28:13.541614+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490691+0000","last_change":"2023-11-08T05:28:13.550672+0000","last_active":"2023-11-08T05:30:44.490691+0000","last_peered":"2023-11-08T05:30:44.490691+0000","last_clean":"2023-11-08T05:28:01.457904+0000","last_became_active":"2023-11-08T05:28:13.550672+0000","last_became_peered":"2023-11-08T05:28:13.550672+0000","last_unstale":"2023-11-08T05:30:44.490691+0000","last_undegraded":"2023-11-08T05:30:44.490691+0000","last_fullsized":"2023-11-08T05:28:13.539595+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490805+0000","last_change":"2023-11-08T05:28:13.557968+0000","last_active":"2023-11-08T05:30:44.490805+0000","last_peered":"2023-11-08T05:30:44.490805+0000","last_clean":"2023-11-08T05:28:01.458433+0000","last_became_active":"2023-11-08T05:28:13.557968+0000","last_became_peered":"2023-11-08T05:28:13.557968+0000","last_unstale":"2023-11-08T05:30:44.490805+0000","last_undegraded":"2023-11-08T05:30:44.490805+0000","last_fullsized":"2023-11-08T05:28:13.542832+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":1,"acting":1,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":12,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738846,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":645040,"kb_used_data":5352,"kb_used_omap":275,"kb_used_meta":639404,"kb_avail":93726800,"statfs":{"total":96636764160,"available":95976243200,"internally_reserved":0,"allocated":5480448,"data_stored":2697647,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":281650,"internal_metadata":654750670},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:59 2023","interfaces":[{"interface":"back","average":{"1min":0.30499999999999999,"5min":0.439,"15min":0.36099999999999999},"min":{"1min":0.17399999999999999,"5min":0.14499999999999999,"15min":0.14499999999999999},"max":{"1min":0.60099999999999998,"5min":4.9379999999999997,"15min":4.9379999999999997},"last":0.379},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.49399999999999999,"15min":0.42399999999999999},"min":{"1min":0.221,"5min":0.19600000000000001,"15min":0.17000000000000001},"max":{"1min":0.63200000000000001,"5min":4.9859999999999998,"15min":4.9859999999999998},"last":0.40899999999999997}]},{"osd":1,"last update":"Wed Nov 8 05:32:06 2023","interfaces":[{"interface":"back","average":{"1min":0.27400000000000002,"5min":0.307,"15min":0.46800000000000003},"min":{"1min":0.17599999999999999,"5min":0.16,"15min":0.153},"max":{"1min":0.39000000000000001,"5min":1.825,"15min":23.242999999999999},"last":0.29199999999999998},{"interface":"front","average":{"1min":0.28000000000000003,"5min":0.34799999999999998,"15min":0.45700000000000002},"min":{"1min":0.20200000000000001,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":0.41699999999999998,"5min":2.0419999999999998,"15min":23.251000000000001},"last":0.311}]}]},{"osd":0,"up_from":8,"seq":34359738798,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1339228,"kb_used_data":398304,"kb_used_omap":291,"kb_used_meta":936476,"kb_avail":93032612,"statfs":{"total":96636764160,"available":95265394688,"internally_reserved":0,"allocated":407863296,"data_stored":405103900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":298462,"internal_metadata":958951970},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.436,"15min":0.36299999999999999},"min":{"1min":0.19700000000000001,"5min":0.14699999999999999,"15min":0.13700000000000001},"max":{"1min":3.222,"5min":6.5499999999999998,"15min":6.9989999999999997},"last":0.245},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.46300000000000002,"15min":0.42299999999999999},"min":{"1min":0.17799999999999999,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":5.0030000000000001,"5min":6.0629999999999997,"15min":9.6869999999999994},"last":0.28599999999999998}]},{"osd":2,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.51600000000000001,"15min":0.47099999999999997},"min":{"1min":0.21099999999999999,"5min":0.153,"15min":0.153},"max":{"1min":5.7439999999999998,"5min":5.8280000000000003,"15min":9.7639999999999993},"last":0.252},{"interface":"front","average":{"1min":0.497,"5min":0.56200000000000006,"15min":0.42299999999999999},"min":{"1min":0.20399999999999999,"5min":0.187,"15min":0.16800000000000001},"max":{"1min":3.9319999999999999,"5min":8.8040000000000003,"15min":8.8040000000000003},"last":0.29899999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738848,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":650100,"kb_used_data":5356,"kb_used_omap":214,"kb_used_meta":644521,"kb_avail":93721740,"statfs":{"total":96636764160,"available":95971061760,"internally_reserved":0,"allocated":5484544,"data_stored":2698616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":219335,"internal_metadata":659990329},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:56 2023","interfaces":[{"interface":"back","average":{"1min":0.30399999999999999,"5min":0.34699999999999998,"15min":0.36499999999999999},"min":{"1min":0.23999999999999999,"5min":0.14199999999999999,"15min":0.14199999999999999},"max":{"1min":0.42399999999999999,"5min":1.595,"15min":7.4249999999999998},"last":0.17100000000000001},{"interface":"front","average":{"1min":0.35499999999999998,"5min":0.40899999999999997,"15min":0.376},"min":{"1min":0.23999999999999999,"5min":0.19600000000000001,"15min":0.183},"max":{"1min":0.46999999999999997,"5min":2.6859999999999999,"15min":2.6859999999999999},"last":0.23000000000000001}]},{"osd":2,"last update":"Wed Nov 8 05:32:02 2023","interfaces":[{"interface":"back","average":{"1min":0.252,"5min":0.33000000000000002,"15min":0.36499999999999999},"min":{"1min":0.161,"5min":0.161,"15min":0.161},"max":{"1min":0.39600000000000002,"5min":2.2650000000000001,"15min":5.8209999999999997},"last":0.28499999999999998},{"interface":"front","average":{"1min":0.25800000000000001,"5min":0.29899999999999999,"15min":0.35599999999999998},"min":{"1min":0.17000000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.35799999999999998,"5min":1.137,"15min":4.2350000000000003},"last":0.30599999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-08T05:32:26.632 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~249s 2023-11-08T05:32:27.949 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-11-08T05:32:28.246 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T05:32:28.248 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2023-11-08T05:32:28.259 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":1648,"stamp":"2023-11-08T05:32:28.080755+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":10,"acting":13,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":109,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2634368,"kb_used_data":409012,"kb_used_omap":780,"kb_used_meta":2220403,"kb_avail":280481152,"statfs":{"total":289910292480,"available":287212699648,"internally_reserved":0,"allocated":418828288,"data_stored":410500163,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":799447,"internal_metadata":2273692969},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-536782902,"num_objects":-164,"num_object_clones":0,"num_object_copies":-328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-115,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-164,"num_whiteouts":0,"num_read":-122056,"num_read_kb":-39011942,"num_write":-413642,"num_write_kb":-151705339,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-8,"num_bytes_recovered":-16922624,"num_keys_recovered":-2,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.424817"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490644+0000","last_change":"2023-11-08T05:28:13.557071+0000","last_active":"2023-11-08T05:30:44.490644+0000","last_peered":"2023-11-08T05:30:44.490644+0000","last_clean":"2023-11-08T05:28:01.458439+0000","last_became_active":"2023-11-08T05:28:13.557071+0000","last_became_peered":"2023-11-08T05:28:13.557071+0000","last_unstale":"2023-11-08T05:30:44.490644+0000","last_undegraded":"2023-11-08T05:30:44.490644+0000","last_fullsized":"2023-11-08T05:28:13.547025+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":459,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:45.794760+0000","last_change":"2023-11-08T05:30:45.794760+0000","last_active":"2023-11-08T05:30:45.794760+0000","last_peered":"2023-11-08T05:30:45.794760+0000","last_clean":"2023-11-08T05:30:45.794760+0000","last_became_active":"2023-11-08T05:30:45.794398+0000","last_became_peered":"2023-11-08T05:30:45.794398+0000","last_unstale":"2023-11-08T05:30:45.794760+0000","last_undegraded":"2023-11-08T05:30:45.794760+0000","last_fullsized":"2023-11-08T05:30:45.794760+0000","mapping_epoch":458,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":459,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":465,"reported_epoch":459,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:45.795085+0000","last_change":"2023-11-08T05:30:45.795085+0000","last_active":"2023-11-08T05:30:45.795085+0000","last_peered":"2023-11-08T05:30:45.795085+0000","last_clean":"2023-11-08T05:30:45.795085+0000","last_became_active":"2023-11-08T05:30:45.794743+0000","last_became_peered":"2023-11-08T05:30:45.794743+0000","last_unstale":"2023-11-08T05:30:45.795085+0000","last_undegraded":"2023-11-08T05:30:45.795085+0000","last_fullsized":"2023-11-08T05:30:45.795085+0000","mapping_epoch":458,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":459,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":454,"reported_epoch":458,"state":"active+clean","last_fresh":"2023-11-08T05:30:44.490618+0000","last_change":"2023-11-08T04:52:47.548597+0000","last_active":"2023-11-08T05:30:44.490618+0000","last_peered":"2023-11-08T05:30:44.490618+0000","last_clean":"2023-11-08T05:30:44.490618+0000","last_became_active":"2023-11-08T04:52:47.548294+0000","last_became_peered":"2023-11-08T04:52:47.548294+0000","last_unstale":"2023-11-08T05:30:44.490618+0000","last_undegraded":"2023-11-08T05:30:44.490618+0000","last_fullsized":"2023-11-08T05:30:44.490618+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":460,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490523+0000","last_change":"2023-11-08T05:28:13.545468+0000","last_active":"2023-11-08T05:30:44.490523+0000","last_peered":"2023-11-08T05:30:44.490523+0000","last_clean":"2023-11-08T05:28:01.456723+0000","last_became_active":"2023-11-08T05:28:13.545468+0000","last_became_peered":"2023-11-08T05:28:13.545468+0000","last_unstale":"2023-11-08T05:30:44.490523+0000","last_undegraded":"2023-11-08T05:30:44.490523+0000","last_fullsized":"2023-11-08T05:28:13.537207+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_clean_scrub_stamp":"2023-11-08T04:52:45.528796+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":465,"reported_epoch":458,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:44.491907+0000","last_change":"2023-11-08T05:29:44.395350+0000","last_active":"2023-11-08T05:30:44.491907+0000","last_peered":"2023-11-08T05:30:44.491907+0000","last_clean":"2023-11-08T05:30:44.491907+0000","last_became_active":"2023-11-08T05:29:44.395012+0000","last_became_peered":"2023-11-08T05:29:44.395012+0000","last_unstale":"2023-11-08T05:30:44.491907+0000","last_undegraded":"2023-11-08T05:30:44.491907+0000","last_fullsized":"2023-11-08T05:30:44.491907+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490629+0000","last_change":"2023-11-08T05:28:13.563357+0000","last_active":"2023-11-08T05:30:44.490629+0000","last_peered":"2023-11-08T05:30:44.490629+0000","last_clean":"2023-11-08T05:28:01.457691+0000","last_became_active":"2023-11-08T05:28:13.563357+0000","last_became_peered":"2023-11-08T05:28:13.563357+0000","last_unstale":"2023-11-08T05:30:44.490629+0000","last_undegraded":"2023-11-08T05:30:44.490629+0000","last_fullsized":"2023-11-08T05:28:13.541614+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490691+0000","last_change":"2023-11-08T05:28:13.550672+0000","last_active":"2023-11-08T05:30:44.490691+0000","last_peered":"2023-11-08T05:30:44.490691+0000","last_clean":"2023-11-08T05:28:01.457904+0000","last_became_active":"2023-11-08T05:28:13.550672+0000","last_became_peered":"2023-11-08T05:28:13.550672+0000","last_unstale":"2023-11-08T05:30:44.490691+0000","last_undegraded":"2023-11-08T05:30:44.490691+0000","last_fullsized":"2023-11-08T05:28:13.539595+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490805+0000","last_change":"2023-11-08T05:28:13.557968+0000","last_active":"2023-11-08T05:30:44.490805+0000","last_peered":"2023-11-08T05:30:44.490805+0000","last_clean":"2023-11-08T05:28:01.458433+0000","last_became_active":"2023-11-08T05:28:13.557968+0000","last_became_peered":"2023-11-08T05:28:13.557968+0000","last_unstale":"2023-11-08T05:30:44.490805+0000","last_undegraded":"2023-11-08T05:30:44.490805+0000","last_fullsized":"2023-11-08T05:28:13.542832+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":1,"acting":1,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":12,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738847,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":645040,"kb_used_data":5352,"kb_used_omap":275,"kb_used_meta":639404,"kb_avail":93726800,"statfs":{"total":96636764160,"available":95976243200,"internally_reserved":0,"allocated":5480448,"data_stored":2697647,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":281650,"internal_metadata":654750670},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:59 2023","interfaces":[{"interface":"back","average":{"1min":0.30499999999999999,"5min":0.439,"15min":0.36099999999999999},"min":{"1min":0.17399999999999999,"5min":0.14499999999999999,"15min":0.14499999999999999},"max":{"1min":0.60099999999999998,"5min":4.9379999999999997,"15min":4.9379999999999997},"last":0.379},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.49399999999999999,"15min":0.42399999999999999},"min":{"1min":0.221,"5min":0.19600000000000001,"15min":0.17000000000000001},"max":{"1min":0.63200000000000001,"5min":4.9859999999999998,"15min":4.9859999999999998},"last":0.40899999999999997}]},{"osd":1,"last update":"Wed Nov 8 05:32:06 2023","interfaces":[{"interface":"back","average":{"1min":0.27400000000000002,"5min":0.307,"15min":0.46800000000000003},"min":{"1min":0.17599999999999999,"5min":0.16,"15min":0.153},"max":{"1min":0.39000000000000001,"5min":1.825,"15min":23.242999999999999},"last":0.23200000000000001},{"interface":"front","average":{"1min":0.28000000000000003,"5min":0.34799999999999998,"15min":0.45700000000000002},"min":{"1min":0.20200000000000001,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":0.41699999999999998,"5min":2.0419999999999998,"15min":23.251000000000001},"last":0.25600000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738798,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1339228,"kb_used_data":398304,"kb_used_omap":291,"kb_used_meta":936476,"kb_avail":93032612,"statfs":{"total":96636764160,"available":95265394688,"internally_reserved":0,"allocated":407863296,"data_stored":405103900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":298462,"internal_metadata":958951970},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.436,"15min":0.36299999999999999},"min":{"1min":0.19700000000000001,"5min":0.14699999999999999,"15min":0.13700000000000001},"max":{"1min":3.222,"5min":6.5499999999999998,"15min":6.9989999999999997},"last":0.245},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.46300000000000002,"15min":0.42299999999999999},"min":{"1min":0.17799999999999999,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":5.0030000000000001,"5min":6.0629999999999997,"15min":9.6869999999999994},"last":0.28599999999999998}]},{"osd":2,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.51600000000000001,"15min":0.47099999999999997},"min":{"1min":0.21099999999999999,"5min":0.153,"15min":0.153},"max":{"1min":5.7439999999999998,"5min":5.8280000000000003,"15min":9.7639999999999993},"last":0.252},{"interface":"front","average":{"1min":0.497,"5min":0.56200000000000006,"15min":0.42299999999999999},"min":{"1min":0.20399999999999999,"5min":0.187,"15min":0.16800000000000001},"max":{"1min":3.9319999999999999,"5min":8.8040000000000003,"15min":8.8040000000000003},"last":0.29899999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738849,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":650100,"kb_used_data":5356,"kb_used_omap":214,"kb_used_meta":644521,"kb_avail":93721740,"statfs":{"total":96636764160,"available":95971061760,"internally_reserved":0,"allocated":5484544,"data_stored":2698616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":219335,"internal_metadata":659990329},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:56 2023","interfaces":[{"interface":"back","average":{"1min":0.30399999999999999,"5min":0.34699999999999998,"15min":0.36499999999999999},"min":{"1min":0.23999999999999999,"5min":0.14199999999999999,"15min":0.14199999999999999},"max":{"1min":0.42399999999999999,"5min":1.595,"15min":7.4249999999999998},"last":0.17100000000000001},{"interface":"front","average":{"1min":0.35499999999999998,"5min":0.40899999999999997,"15min":0.376},"min":{"1min":0.23999999999999999,"5min":0.19600000000000001,"15min":0.183},"max":{"1min":0.46999999999999997,"5min":2.6859999999999999,"15min":2.6859999999999999},"last":0.23000000000000001}]},{"osd":2,"last update":"Wed Nov 8 05:32:02 2023","interfaces":[{"interface":"back","average":{"1min":0.252,"5min":0.33000000000000002,"15min":0.36499999999999999},"min":{"1min":0.161,"5min":0.161,"15min":0.161},"max":{"1min":0.39600000000000002,"5min":2.2650000000000001,"15min":5.8209999999999997},"last":0.23200000000000001},{"interface":"front","average":{"1min":0.25800000000000001,"5min":0.29899999999999999,"15min":0.35599999999999998},"min":{"1min":0.17000000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.35799999999999998,"5min":1.137,"15min":4.2350000000000003},"last":0.25900000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-08T05:32:28.260 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-11-08T05:32:28.545 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T05:32:28.547 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2023-11-08T05:32:28.558 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":1648,"stamp":"2023-11-08T05:32:28.080755+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":10,"acting":13,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":109,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2634368,"kb_used_data":409012,"kb_used_omap":780,"kb_used_meta":2220403,"kb_avail":280481152,"statfs":{"total":289910292480,"available":287212699648,"internally_reserved":0,"allocated":418828288,"data_stored":410500163,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":799447,"internal_metadata":2273692969},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-536782902,"num_objects":-164,"num_object_clones":0,"num_object_copies":-328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-115,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-164,"num_whiteouts":0,"num_read":-122056,"num_read_kb":-39011942,"num_write":-413642,"num_write_kb":-151705339,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-8,"num_bytes_recovered":-16922624,"num_keys_recovered":-2,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.424817"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490644+0000","last_change":"2023-11-08T05:28:13.557071+0000","last_active":"2023-11-08T05:30:44.490644+0000","last_peered":"2023-11-08T05:30:44.490644+0000","last_clean":"2023-11-08T05:28:01.458439+0000","last_became_active":"2023-11-08T05:28:13.557071+0000","last_became_peered":"2023-11-08T05:28:13.557071+0000","last_unstale":"2023-11-08T05:30:44.490644+0000","last_undegraded":"2023-11-08T05:30:44.490644+0000","last_fullsized":"2023-11-08T05:28:13.547025+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":459,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:45.794760+0000","last_change":"2023-11-08T05:30:45.794760+0000","last_active":"2023-11-08T05:30:45.794760+0000","last_peered":"2023-11-08T05:30:45.794760+0000","last_clean":"2023-11-08T05:30:45.794760+0000","last_became_active":"2023-11-08T05:30:45.794398+0000","last_became_peered":"2023-11-08T05:30:45.794398+0000","last_unstale":"2023-11-08T05:30:45.794760+0000","last_undegraded":"2023-11-08T05:30:45.794760+0000","last_fullsized":"2023-11-08T05:30:45.794760+0000","mapping_epoch":458,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":459,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":465,"reported_epoch":459,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:45.795085+0000","last_change":"2023-11-08T05:30:45.795085+0000","last_active":"2023-11-08T05:30:45.795085+0000","last_peered":"2023-11-08T05:30:45.795085+0000","last_clean":"2023-11-08T05:30:45.795085+0000","last_became_active":"2023-11-08T05:30:45.794743+0000","last_became_peered":"2023-11-08T05:30:45.794743+0000","last_unstale":"2023-11-08T05:30:45.795085+0000","last_undegraded":"2023-11-08T05:30:45.795085+0000","last_fullsized":"2023-11-08T05:30:45.795085+0000","mapping_epoch":458,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":459,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":454,"reported_epoch":458,"state":"active+clean","last_fresh":"2023-11-08T05:30:44.490618+0000","last_change":"2023-11-08T04:52:47.548597+0000","last_active":"2023-11-08T05:30:44.490618+0000","last_peered":"2023-11-08T05:30:44.490618+0000","last_clean":"2023-11-08T05:30:44.490618+0000","last_became_active":"2023-11-08T04:52:47.548294+0000","last_became_peered":"2023-11-08T04:52:47.548294+0000","last_unstale":"2023-11-08T05:30:44.490618+0000","last_undegraded":"2023-11-08T05:30:44.490618+0000","last_fullsized":"2023-11-08T05:30:44.490618+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":460,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490523+0000","last_change":"2023-11-08T05:28:13.545468+0000","last_active":"2023-11-08T05:30:44.490523+0000","last_peered":"2023-11-08T05:30:44.490523+0000","last_clean":"2023-11-08T05:28:01.456723+0000","last_became_active":"2023-11-08T05:28:13.545468+0000","last_became_peered":"2023-11-08T05:28:13.545468+0000","last_unstale":"2023-11-08T05:30:44.490523+0000","last_undegraded":"2023-11-08T05:30:44.490523+0000","last_fullsized":"2023-11-08T05:28:13.537207+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_clean_scrub_stamp":"2023-11-08T04:52:45.528796+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":465,"reported_epoch":458,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:44.491907+0000","last_change":"2023-11-08T05:29:44.395350+0000","last_active":"2023-11-08T05:30:44.491907+0000","last_peered":"2023-11-08T05:30:44.491907+0000","last_clean":"2023-11-08T05:30:44.491907+0000","last_became_active":"2023-11-08T05:29:44.395012+0000","last_became_peered":"2023-11-08T05:29:44.395012+0000","last_unstale":"2023-11-08T05:30:44.491907+0000","last_undegraded":"2023-11-08T05:30:44.491907+0000","last_fullsized":"2023-11-08T05:30:44.491907+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490629+0000","last_change":"2023-11-08T05:28:13.563357+0000","last_active":"2023-11-08T05:30:44.490629+0000","last_peered":"2023-11-08T05:30:44.490629+0000","last_clean":"2023-11-08T05:28:01.457691+0000","last_became_active":"2023-11-08T05:28:13.563357+0000","last_became_peered":"2023-11-08T05:28:13.563357+0000","last_unstale":"2023-11-08T05:30:44.490629+0000","last_undegraded":"2023-11-08T05:30:44.490629+0000","last_fullsized":"2023-11-08T05:28:13.541614+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490691+0000","last_change":"2023-11-08T05:28:13.550672+0000","last_active":"2023-11-08T05:30:44.490691+0000","last_peered":"2023-11-08T05:30:44.490691+0000","last_clean":"2023-11-08T05:28:01.457904+0000","last_became_active":"2023-11-08T05:28:13.550672+0000","last_became_peered":"2023-11-08T05:28:13.550672+0000","last_unstale":"2023-11-08T05:30:44.490691+0000","last_undegraded":"2023-11-08T05:30:44.490691+0000","last_fullsized":"2023-11-08T05:28:13.539595+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490805+0000","last_change":"2023-11-08T05:28:13.557968+0000","last_active":"2023-11-08T05:30:44.490805+0000","last_peered":"2023-11-08T05:30:44.490805+0000","last_clean":"2023-11-08T05:28:01.458433+0000","last_became_active":"2023-11-08T05:28:13.557968+0000","last_became_peered":"2023-11-08T05:28:13.557968+0000","last_unstale":"2023-11-08T05:30:44.490805+0000","last_undegraded":"2023-11-08T05:30:44.490805+0000","last_fullsized":"2023-11-08T05:28:13.542832+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":1,"acting":1,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":12,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738847,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":645040,"kb_used_data":5352,"kb_used_omap":275,"kb_used_meta":639404,"kb_avail":93726800,"statfs":{"total":96636764160,"available":95976243200,"internally_reserved":0,"allocated":5480448,"data_stored":2697647,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":281650,"internal_metadata":654750670},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:59 2023","interfaces":[{"interface":"back","average":{"1min":0.30499999999999999,"5min":0.439,"15min":0.36099999999999999},"min":{"1min":0.17399999999999999,"5min":0.14499999999999999,"15min":0.14499999999999999},"max":{"1min":0.60099999999999998,"5min":4.9379999999999997,"15min":4.9379999999999997},"last":0.379},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.49399999999999999,"15min":0.42399999999999999},"min":{"1min":0.221,"5min":0.19600000000000001,"15min":0.17000000000000001},"max":{"1min":0.63200000000000001,"5min":4.9859999999999998,"15min":4.9859999999999998},"last":0.40899999999999997}]},{"osd":1,"last update":"Wed Nov 8 05:32:06 2023","interfaces":[{"interface":"back","average":{"1min":0.27400000000000002,"5min":0.307,"15min":0.46800000000000003},"min":{"1min":0.17599999999999999,"5min":0.16,"15min":0.153},"max":{"1min":0.39000000000000001,"5min":1.825,"15min":23.242999999999999},"last":0.23200000000000001},{"interface":"front","average":{"1min":0.28000000000000003,"5min":0.34799999999999998,"15min":0.45700000000000002},"min":{"1min":0.20200000000000001,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":0.41699999999999998,"5min":2.0419999999999998,"15min":23.251000000000001},"last":0.25600000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738798,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1339228,"kb_used_data":398304,"kb_used_omap":291,"kb_used_meta":936476,"kb_avail":93032612,"statfs":{"total":96636764160,"available":95265394688,"internally_reserved":0,"allocated":407863296,"data_stored":405103900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":298462,"internal_metadata":958951970},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.436,"15min":0.36299999999999999},"min":{"1min":0.19700000000000001,"5min":0.14699999999999999,"15min":0.13700000000000001},"max":{"1min":3.222,"5min":6.5499999999999998,"15min":6.9989999999999997},"last":0.245},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.46300000000000002,"15min":0.42299999999999999},"min":{"1min":0.17799999999999999,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":5.0030000000000001,"5min":6.0629999999999997,"15min":9.6869999999999994},"last":0.28599999999999998}]},{"osd":2,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.51600000000000001,"15min":0.47099999999999997},"min":{"1min":0.21099999999999999,"5min":0.153,"15min":0.153},"max":{"1min":5.7439999999999998,"5min":5.8280000000000003,"15min":9.7639999999999993},"last":0.252},{"interface":"front","average":{"1min":0.497,"5min":0.56200000000000006,"15min":0.42299999999999999},"min":{"1min":0.20399999999999999,"5min":0.187,"15min":0.16800000000000001},"max":{"1min":3.9319999999999999,"5min":8.8040000000000003,"15min":8.8040000000000003},"last":0.29899999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738849,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":650100,"kb_used_data":5356,"kb_used_omap":214,"kb_used_meta":644521,"kb_avail":93721740,"statfs":{"total":96636764160,"available":95971061760,"internally_reserved":0,"allocated":5484544,"data_stored":2698616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":219335,"internal_metadata":659990329},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:56 2023","interfaces":[{"interface":"back","average":{"1min":0.30399999999999999,"5min":0.34699999999999998,"15min":0.36499999999999999},"min":{"1min":0.23999999999999999,"5min":0.14199999999999999,"15min":0.14199999999999999},"max":{"1min":0.42399999999999999,"5min":1.595,"15min":7.4249999999999998},"last":0.17100000000000001},{"interface":"front","average":{"1min":0.35499999999999998,"5min":0.40899999999999997,"15min":0.376},"min":{"1min":0.23999999999999999,"5min":0.19600000000000001,"15min":0.183},"max":{"1min":0.46999999999999997,"5min":2.6859999999999999,"15min":2.6859999999999999},"last":0.23000000000000001}]},{"osd":2,"last update":"Wed Nov 8 05:32:02 2023","interfaces":[{"interface":"back","average":{"1min":0.252,"5min":0.33000000000000002,"15min":0.36499999999999999},"min":{"1min":0.161,"5min":0.161,"15min":0.161},"max":{"1min":0.39600000000000002,"5min":2.2650000000000001,"15min":5.8209999999999997},"last":0.23200000000000001},{"interface":"front","average":{"1min":0.25800000000000001,"5min":0.29899999999999999,"15min":0.35599999999999998},"min":{"1min":0.17000000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.35799999999999998,"5min":1.137,"15min":4.2350000000000003},"last":0.25900000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-08T05:32:28.560 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-11-08T05:32:28.560 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 459, 'reported_epoch': 458, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:30:44.490644+0000', 'last_change': '2023-11-08T05:28:13.557071+0000', 'last_active': '2023-11-08T05:30:44.490644+0000', 'last_peered': '2023-11-08T05:30:44.490644+0000', 'last_clean': '2023-11-08T05:28:01.458439+0000', 'last_became_active': '2023-11-08T05:28:13.557071+0000', 'last_became_peered': '2023-11-08T05:28:13.557071+0000', 'last_unstale': '2023-11-08T05:30:44.490644+0000', 'last_undegraded': '2023-11-08T05:30:44.490644+0000', 'last_fullsized': '2023-11-08T05:28:13.547025+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:32:28.560 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-11-08T05:32:28.560 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 11, 'reported_epoch': 459, 'state': 'active+clean+remapped', 'last_fresh': '2023-11-08T05:30:45.794760+0000', 'last_change': '2023-11-08T05:30:45.794760+0000', 'last_active': '2023-11-08T05:30:45.794760+0000', 'last_peered': '2023-11-08T05:30:45.794760+0000', 'last_clean': '2023-11-08T05:30:45.794760+0000', 'last_became_active': '2023-11-08T05:30:45.794398+0000', 'last_became_peered': '2023-11-08T05:30:45.794398+0000', 'last_unstale': '2023-11-08T05:30:45.794760+0000', 'last_undegraded': '2023-11-08T05:30:45.794760+0000', 'last_fullsized': '2023-11-08T05:30:45.794760+0000', 'mapping_epoch': 458, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 459, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2], 'acting': [2, 1], 'avail_no_missing': ['2', '1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2023-11-08T05:32:28.561 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.1 is not active+clean 2023-11-08T05:32:28.561 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.1', 'version': "0'0", 'reported_seq': 465, 'reported_epoch': 459, 'state': 'active+clean+remapped', 'last_fresh': '2023-11-08T05:30:45.795085+0000', 'last_change': '2023-11-08T05:30:45.795085+0000', 'last_active': '2023-11-08T05:30:45.795085+0000', 'last_peered': '2023-11-08T05:30:45.795085+0000', 'last_clean': '2023-11-08T05:30:45.795085+0000', 'last_became_active': '2023-11-08T05:30:45.794743+0000', 'last_became_peered': '2023-11-08T05:30:45.794743+0000', 'last_unstale': '2023-11-08T05:30:45.795085+0000', 'last_undegraded': '2023-11-08T05:30:45.795085+0000', 'last_fullsized': '2023-11-08T05:30:45.795085+0000', 'mapping_epoch': 458, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 459, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2], 'acting': [2, 1], 'avail_no_missing': ['2', '1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2023-11-08T05:32:28.561 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-11-08T05:32:28.562 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "0'0", 'reported_seq': 460, 'reported_epoch': 458, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:30:44.490523+0000', 'last_change': '2023-11-08T05:28:13.545468+0000', 'last_active': '2023-11-08T05:30:44.490523+0000', 'last_peered': '2023-11-08T05:30:44.490523+0000', 'last_clean': '2023-11-08T05:28:01.456723+0000', 'last_became_active': '2023-11-08T05:28:13.545468+0000', 'last_became_peered': '2023-11-08T05:28:13.545468+0000', 'last_unstale': '2023-11-08T05:30:44.490523+0000', 'last_undegraded': '2023-11-08T05:30:44.490523+0000', 'last_fullsized': '2023-11-08T05:28:13.537207+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 9, 'last_epoch_clean': 10, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:45.528796+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:45.528796+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:45.528796+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:32:28.562 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.0 is not active+clean 2023-11-08T05:32:28.562 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.0', 'version': "0'0", 'reported_seq': 465, 'reported_epoch': 458, 'state': 'active+clean+remapped', 'last_fresh': '2023-11-08T05:30:44.491907+0000', 'last_change': '2023-11-08T05:29:44.395350+0000', 'last_active': '2023-11-08T05:30:44.491907+0000', 'last_peered': '2023-11-08T05:30:44.491907+0000', 'last_clean': '2023-11-08T05:30:44.491907+0000', 'last_became_active': '2023-11-08T05:29:44.395012+0000', 'last_became_peered': '2023-11-08T05:29:44.395012+0000', 'last_unstale': '2023-11-08T05:30:44.491907+0000', 'last_undegraded': '2023-11-08T05:30:44.491907+0000', 'last_fullsized': '2023-11-08T05:30:44.491907+0000', 'mapping_epoch': 455, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 456, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2], 'acting': [2, 1], 'avail_no_missing': ['2', '1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2023-11-08T05:32:28.563 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-11-08T05:32:28.563 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 459, 'reported_epoch': 458, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:30:44.490629+0000', 'last_change': '2023-11-08T05:28:13.563357+0000', 'last_active': '2023-11-08T05:30:44.490629+0000', 'last_peered': '2023-11-08T05:30:44.490629+0000', 'last_clean': '2023-11-08T05:28:01.457691+0000', 'last_became_active': '2023-11-08T05:28:13.563357+0000', 'last_became_peered': '2023-11-08T05:28:13.563357+0000', 'last_unstale': '2023-11-08T05:30:44.490629+0000', 'last_undegraded': '2023-11-08T05:30:44.490629+0000', 'last_fullsized': '2023-11-08T05:28:13.541614+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:32:28.563 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-11-08T05:32:28.564 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 459, 'reported_epoch': 458, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:30:44.490691+0000', 'last_change': '2023-11-08T05:28:13.550672+0000', 'last_active': '2023-11-08T05:30:44.490691+0000', 'last_peered': '2023-11-08T05:30:44.490691+0000', 'last_clean': '2023-11-08T05:28:01.457904+0000', 'last_became_active': '2023-11-08T05:28:13.550672+0000', 'last_became_peered': '2023-11-08T05:28:13.550672+0000', 'last_unstale': '2023-11-08T05:30:44.490691+0000', 'last_undegraded': '2023-11-08T05:30:44.490691+0000', 'last_fullsized': '2023-11-08T05:28:13.539595+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:32:28.564 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-11-08T05:32:28.564 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 459, 'reported_epoch': 458, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:30:44.490805+0000', 'last_change': '2023-11-08T05:28:13.557968+0000', 'last_active': '2023-11-08T05:30:44.490805+0000', 'last_peered': '2023-11-08T05:30:44.490805+0000', 'last_clean': '2023-11-08T05:28:01.458433+0000', 'last_became_active': '2023-11-08T05:28:13.557968+0000', 'last_became_peered': '2023-11-08T05:28:13.557968+0000', 'last_unstale': '2023-11-08T05:30:44.490805+0000', 'last_undegraded': '2023-11-08T05:30:44.490805+0000', 'last_fullsized': '2023-11-08T05:28:13.542832+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:32:28.565 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-11-08T05:32:28.909 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T05:32:28.910 INFO:teuthology.orchestra.run.smithi087.stdout:{"fsid":"ab4cbc06-449d-497d-b513-220d9db2b590","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 5 pgs undersized","count":5},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2391,"monmap":{"epoch":1,"min_mon_release_name":"pacific","num_mons":1},"osdmap":{"epoch":459,"num_osds":3,"num_up_osds":2,"osd_up_since":1699421292,"num_in_osds":3,"osd_in_since":1699419158,"num_remapped_pgs":3},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean+remapped","count":3},{"state_name":"active+clean","count":1}],"num_pgs":9,"num_pools":2,"num_objects":0,"data_bytes":0,"bytes_used":2697592832,"bytes_avail":287212699648,"bytes_total":289910292480},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":41,"modified":"2023-11-08T05:29:32.044544+0000","services":{}},"progress_events":{}} 2023-11-08T05:32:28.923 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-11-08T05:32:28.923 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-11-08T05:32:29.239 INFO:teuthology.orchestra.run.smithi087.stderr:2023-11-08T05:32:29.238+0000 7f7231d9b700 0 --2- 172.21.15.87:0/1881685666 >> v2:172.21.15.87:3300/0 conn(0x7f72340ab030 0x7f7234067d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2023-11-08T05:32:29.406 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T05:32:29.407 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2023-11-08T05:32:29.418 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":1648,"stamp":"2023-11-08T05:32:28.080755+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":10,"acting":13,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":109,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2634368,"kb_used_data":409012,"kb_used_omap":780,"kb_used_meta":2220403,"kb_avail":280481152,"statfs":{"total":289910292480,"available":287212699648,"internally_reserved":0,"allocated":418828288,"data_stored":410500163,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":799447,"internal_metadata":2273692969},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-536782902,"num_objects":-164,"num_object_clones":0,"num_object_copies":-328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-115,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-164,"num_whiteouts":0,"num_read":-122056,"num_read_kb":-39011942,"num_write":-413642,"num_write_kb":-151705339,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-8,"num_bytes_recovered":-16922624,"num_keys_recovered":-2,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.424817"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490644+0000","last_change":"2023-11-08T05:28:13.557071+0000","last_active":"2023-11-08T05:30:44.490644+0000","last_peered":"2023-11-08T05:30:44.490644+0000","last_clean":"2023-11-08T05:28:01.458439+0000","last_became_active":"2023-11-08T05:28:13.557071+0000","last_became_peered":"2023-11-08T05:28:13.557071+0000","last_unstale":"2023-11-08T05:30:44.490644+0000","last_undegraded":"2023-11-08T05:30:44.490644+0000","last_fullsized":"2023-11-08T05:28:13.547025+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":459,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:45.794760+0000","last_change":"2023-11-08T05:30:45.794760+0000","last_active":"2023-11-08T05:30:45.794760+0000","last_peered":"2023-11-08T05:30:45.794760+0000","last_clean":"2023-11-08T05:30:45.794760+0000","last_became_active":"2023-11-08T05:30:45.794398+0000","last_became_peered":"2023-11-08T05:30:45.794398+0000","last_unstale":"2023-11-08T05:30:45.794760+0000","last_undegraded":"2023-11-08T05:30:45.794760+0000","last_fullsized":"2023-11-08T05:30:45.794760+0000","mapping_epoch":458,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":459,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":465,"reported_epoch":459,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:45.795085+0000","last_change":"2023-11-08T05:30:45.795085+0000","last_active":"2023-11-08T05:30:45.795085+0000","last_peered":"2023-11-08T05:30:45.795085+0000","last_clean":"2023-11-08T05:30:45.795085+0000","last_became_active":"2023-11-08T05:30:45.794743+0000","last_became_peered":"2023-11-08T05:30:45.794743+0000","last_unstale":"2023-11-08T05:30:45.795085+0000","last_undegraded":"2023-11-08T05:30:45.795085+0000","last_fullsized":"2023-11-08T05:30:45.795085+0000","mapping_epoch":458,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":459,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":454,"reported_epoch":458,"state":"active+clean","last_fresh":"2023-11-08T05:30:44.490618+0000","last_change":"2023-11-08T04:52:47.548597+0000","last_active":"2023-11-08T05:30:44.490618+0000","last_peered":"2023-11-08T05:30:44.490618+0000","last_clean":"2023-11-08T05:30:44.490618+0000","last_became_active":"2023-11-08T04:52:47.548294+0000","last_became_peered":"2023-11-08T04:52:47.548294+0000","last_unstale":"2023-11-08T05:30:44.490618+0000","last_undegraded":"2023-11-08T05:30:44.490618+0000","last_fullsized":"2023-11-08T05:30:44.490618+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":460,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490523+0000","last_change":"2023-11-08T05:28:13.545468+0000","last_active":"2023-11-08T05:30:44.490523+0000","last_peered":"2023-11-08T05:30:44.490523+0000","last_clean":"2023-11-08T05:28:01.456723+0000","last_became_active":"2023-11-08T05:28:13.545468+0000","last_became_peered":"2023-11-08T05:28:13.545468+0000","last_unstale":"2023-11-08T05:30:44.490523+0000","last_undegraded":"2023-11-08T05:30:44.490523+0000","last_fullsized":"2023-11-08T05:28:13.537207+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_clean_scrub_stamp":"2023-11-08T04:52:45.528796+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":465,"reported_epoch":458,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:44.491907+0000","last_change":"2023-11-08T05:29:44.395350+0000","last_active":"2023-11-08T05:30:44.491907+0000","last_peered":"2023-11-08T05:30:44.491907+0000","last_clean":"2023-11-08T05:30:44.491907+0000","last_became_active":"2023-11-08T05:29:44.395012+0000","last_became_peered":"2023-11-08T05:29:44.395012+0000","last_unstale":"2023-11-08T05:30:44.491907+0000","last_undegraded":"2023-11-08T05:30:44.491907+0000","last_fullsized":"2023-11-08T05:30:44.491907+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490629+0000","last_change":"2023-11-08T05:28:13.563357+0000","last_active":"2023-11-08T05:30:44.490629+0000","last_peered":"2023-11-08T05:30:44.490629+0000","last_clean":"2023-11-08T05:28:01.457691+0000","last_became_active":"2023-11-08T05:28:13.563357+0000","last_became_peered":"2023-11-08T05:28:13.563357+0000","last_unstale":"2023-11-08T05:30:44.490629+0000","last_undegraded":"2023-11-08T05:30:44.490629+0000","last_fullsized":"2023-11-08T05:28:13.541614+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490691+0000","last_change":"2023-11-08T05:28:13.550672+0000","last_active":"2023-11-08T05:30:44.490691+0000","last_peered":"2023-11-08T05:30:44.490691+0000","last_clean":"2023-11-08T05:28:01.457904+0000","last_became_active":"2023-11-08T05:28:13.550672+0000","last_became_peered":"2023-11-08T05:28:13.550672+0000","last_unstale":"2023-11-08T05:30:44.490691+0000","last_undegraded":"2023-11-08T05:30:44.490691+0000","last_fullsized":"2023-11-08T05:28:13.539595+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490805+0000","last_change":"2023-11-08T05:28:13.557968+0000","last_active":"2023-11-08T05:30:44.490805+0000","last_peered":"2023-11-08T05:30:44.490805+0000","last_clean":"2023-11-08T05:28:01.458433+0000","last_became_active":"2023-11-08T05:28:13.557968+0000","last_became_peered":"2023-11-08T05:28:13.557968+0000","last_unstale":"2023-11-08T05:30:44.490805+0000","last_undegraded":"2023-11-08T05:30:44.490805+0000","last_fullsized":"2023-11-08T05:28:13.542832+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":1,"acting":1,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":12,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738847,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":645040,"kb_used_data":5352,"kb_used_omap":275,"kb_used_meta":639404,"kb_avail":93726800,"statfs":{"total":96636764160,"available":95976243200,"internally_reserved":0,"allocated":5480448,"data_stored":2697647,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":281650,"internal_metadata":654750670},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:59 2023","interfaces":[{"interface":"back","average":{"1min":0.30499999999999999,"5min":0.439,"15min":0.36099999999999999},"min":{"1min":0.17399999999999999,"5min":0.14499999999999999,"15min":0.14499999999999999},"max":{"1min":0.60099999999999998,"5min":4.9379999999999997,"15min":4.9379999999999997},"last":0.379},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.49399999999999999,"15min":0.42399999999999999},"min":{"1min":0.221,"5min":0.19600000000000001,"15min":0.17000000000000001},"max":{"1min":0.63200000000000001,"5min":4.9859999999999998,"15min":4.9859999999999998},"last":0.40899999999999997}]},{"osd":1,"last update":"Wed Nov 8 05:32:06 2023","interfaces":[{"interface":"back","average":{"1min":0.27400000000000002,"5min":0.307,"15min":0.46800000000000003},"min":{"1min":0.17599999999999999,"5min":0.16,"15min":0.153},"max":{"1min":0.39000000000000001,"5min":1.825,"15min":23.242999999999999},"last":0.23200000000000001},{"interface":"front","average":{"1min":0.28000000000000003,"5min":0.34799999999999998,"15min":0.45700000000000002},"min":{"1min":0.20200000000000001,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":0.41699999999999998,"5min":2.0419999999999998,"15min":23.251000000000001},"last":0.25600000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738798,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1339228,"kb_used_data":398304,"kb_used_omap":291,"kb_used_meta":936476,"kb_avail":93032612,"statfs":{"total":96636764160,"available":95265394688,"internally_reserved":0,"allocated":407863296,"data_stored":405103900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":298462,"internal_metadata":958951970},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.436,"15min":0.36299999999999999},"min":{"1min":0.19700000000000001,"5min":0.14699999999999999,"15min":0.13700000000000001},"max":{"1min":3.222,"5min":6.5499999999999998,"15min":6.9989999999999997},"last":0.245},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.46300000000000002,"15min":0.42299999999999999},"min":{"1min":0.17799999999999999,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":5.0030000000000001,"5min":6.0629999999999997,"15min":9.6869999999999994},"last":0.28599999999999998}]},{"osd":2,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.51600000000000001,"15min":0.47099999999999997},"min":{"1min":0.21099999999999999,"5min":0.153,"15min":0.153},"max":{"1min":5.7439999999999998,"5min":5.8280000000000003,"15min":9.7639999999999993},"last":0.252},{"interface":"front","average":{"1min":0.497,"5min":0.56200000000000006,"15min":0.42299999999999999},"min":{"1min":0.20399999999999999,"5min":0.187,"15min":0.16800000000000001},"max":{"1min":3.9319999999999999,"5min":8.8040000000000003,"15min":8.8040000000000003},"last":0.29899999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738849,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":650100,"kb_used_data":5356,"kb_used_omap":214,"kb_used_meta":644521,"kb_avail":93721740,"statfs":{"total":96636764160,"available":95971061760,"internally_reserved":0,"allocated":5484544,"data_stored":2698616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":219335,"internal_metadata":659990329},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:56 2023","interfaces":[{"interface":"back","average":{"1min":0.30399999999999999,"5min":0.34699999999999998,"15min":0.36499999999999999},"min":{"1min":0.23999999999999999,"5min":0.14199999999999999,"15min":0.14199999999999999},"max":{"1min":0.42399999999999999,"5min":1.595,"15min":7.4249999999999998},"last":0.17100000000000001},{"interface":"front","average":{"1min":0.35499999999999998,"5min":0.40899999999999997,"15min":0.376},"min":{"1min":0.23999999999999999,"5min":0.19600000000000001,"15min":0.183},"max":{"1min":0.46999999999999997,"5min":2.6859999999999999,"15min":2.6859999999999999},"last":0.23000000000000001}]},{"osd":2,"last update":"Wed Nov 8 05:32:02 2023","interfaces":[{"interface":"back","average":{"1min":0.252,"5min":0.33000000000000002,"15min":0.36499999999999999},"min":{"1min":0.161,"5min":0.161,"15min":0.161},"max":{"1min":0.39600000000000002,"5min":2.2650000000000001,"15min":5.8209999999999997},"last":0.23200000000000001},{"interface":"front","average":{"1min":0.25800000000000001,"5min":0.29899999999999999,"15min":0.35599999999999998},"min":{"1min":0.17000000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.35799999999999998,"5min":1.137,"15min":4.2350000000000003},"last":0.25900000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-08T05:32:32.036 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~254s 2023-11-08T05:32:32.420 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-11-08T05:32:32.717 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T05:32:32.719 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2023-11-08T05:32:32.730 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":1650,"stamp":"2023-11-08T05:32:32.081573+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":10,"acting":13,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":109,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2634368,"kb_used_data":409012,"kb_used_omap":780,"kb_used_meta":2220403,"kb_avail":280481152,"statfs":{"total":289910292480,"available":287212699648,"internally_reserved":0,"allocated":418828288,"data_stored":410500163,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":799447,"internal_metadata":2273692969},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-536782902,"num_objects":-164,"num_object_clones":0,"num_object_copies":-328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-115,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-164,"num_whiteouts":0,"num_read":-122056,"num_read_kb":-39011942,"num_write":-413642,"num_write_kb":-151705339,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-8,"num_bytes_recovered":-16922624,"num_keys_recovered":-2,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.424817"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490644+0000","last_change":"2023-11-08T05:28:13.557071+0000","last_active":"2023-11-08T05:30:44.490644+0000","last_peered":"2023-11-08T05:30:44.490644+0000","last_clean":"2023-11-08T05:28:01.458439+0000","last_became_active":"2023-11-08T05:28:13.557071+0000","last_became_peered":"2023-11-08T05:28:13.557071+0000","last_unstale":"2023-11-08T05:30:44.490644+0000","last_undegraded":"2023-11-08T05:30:44.490644+0000","last_fullsized":"2023-11-08T05:28:13.547025+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":459,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:45.794760+0000","last_change":"2023-11-08T05:30:45.794760+0000","last_active":"2023-11-08T05:30:45.794760+0000","last_peered":"2023-11-08T05:30:45.794760+0000","last_clean":"2023-11-08T05:30:45.794760+0000","last_became_active":"2023-11-08T05:30:45.794398+0000","last_became_peered":"2023-11-08T05:30:45.794398+0000","last_unstale":"2023-11-08T05:30:45.794760+0000","last_undegraded":"2023-11-08T05:30:45.794760+0000","last_fullsized":"2023-11-08T05:30:45.794760+0000","mapping_epoch":458,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":459,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":465,"reported_epoch":459,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:45.795085+0000","last_change":"2023-11-08T05:30:45.795085+0000","last_active":"2023-11-08T05:30:45.795085+0000","last_peered":"2023-11-08T05:30:45.795085+0000","last_clean":"2023-11-08T05:30:45.795085+0000","last_became_active":"2023-11-08T05:30:45.794743+0000","last_became_peered":"2023-11-08T05:30:45.794743+0000","last_unstale":"2023-11-08T05:30:45.795085+0000","last_undegraded":"2023-11-08T05:30:45.795085+0000","last_fullsized":"2023-11-08T05:30:45.795085+0000","mapping_epoch":458,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":459,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":454,"reported_epoch":458,"state":"active+clean","last_fresh":"2023-11-08T05:30:44.490618+0000","last_change":"2023-11-08T04:52:47.548597+0000","last_active":"2023-11-08T05:30:44.490618+0000","last_peered":"2023-11-08T05:30:44.490618+0000","last_clean":"2023-11-08T05:30:44.490618+0000","last_became_active":"2023-11-08T04:52:47.548294+0000","last_became_peered":"2023-11-08T04:52:47.548294+0000","last_unstale":"2023-11-08T05:30:44.490618+0000","last_undegraded":"2023-11-08T05:30:44.490618+0000","last_fullsized":"2023-11-08T05:30:44.490618+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":460,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490523+0000","last_change":"2023-11-08T05:28:13.545468+0000","last_active":"2023-11-08T05:30:44.490523+0000","last_peered":"2023-11-08T05:30:44.490523+0000","last_clean":"2023-11-08T05:28:01.456723+0000","last_became_active":"2023-11-08T05:28:13.545468+0000","last_became_peered":"2023-11-08T05:28:13.545468+0000","last_unstale":"2023-11-08T05:30:44.490523+0000","last_undegraded":"2023-11-08T05:30:44.490523+0000","last_fullsized":"2023-11-08T05:28:13.537207+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_clean_scrub_stamp":"2023-11-08T04:52:45.528796+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":465,"reported_epoch":458,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:44.491907+0000","last_change":"2023-11-08T05:29:44.395350+0000","last_active":"2023-11-08T05:30:44.491907+0000","last_peered":"2023-11-08T05:30:44.491907+0000","last_clean":"2023-11-08T05:30:44.491907+0000","last_became_active":"2023-11-08T05:29:44.395012+0000","last_became_peered":"2023-11-08T05:29:44.395012+0000","last_unstale":"2023-11-08T05:30:44.491907+0000","last_undegraded":"2023-11-08T05:30:44.491907+0000","last_fullsized":"2023-11-08T05:30:44.491907+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490629+0000","last_change":"2023-11-08T05:28:13.563357+0000","last_active":"2023-11-08T05:30:44.490629+0000","last_peered":"2023-11-08T05:30:44.490629+0000","last_clean":"2023-11-08T05:28:01.457691+0000","last_became_active":"2023-11-08T05:28:13.563357+0000","last_became_peered":"2023-11-08T05:28:13.563357+0000","last_unstale":"2023-11-08T05:30:44.490629+0000","last_undegraded":"2023-11-08T05:30:44.490629+0000","last_fullsized":"2023-11-08T05:28:13.541614+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490691+0000","last_change":"2023-11-08T05:28:13.550672+0000","last_active":"2023-11-08T05:30:44.490691+0000","last_peered":"2023-11-08T05:30:44.490691+0000","last_clean":"2023-11-08T05:28:01.457904+0000","last_became_active":"2023-11-08T05:28:13.550672+0000","last_became_peered":"2023-11-08T05:28:13.550672+0000","last_unstale":"2023-11-08T05:30:44.490691+0000","last_undegraded":"2023-11-08T05:30:44.490691+0000","last_fullsized":"2023-11-08T05:28:13.539595+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490805+0000","last_change":"2023-11-08T05:28:13.557968+0000","last_active":"2023-11-08T05:30:44.490805+0000","last_peered":"2023-11-08T05:30:44.490805+0000","last_clean":"2023-11-08T05:28:01.458433+0000","last_became_active":"2023-11-08T05:28:13.557968+0000","last_became_peered":"2023-11-08T05:28:13.557968+0000","last_unstale":"2023-11-08T05:30:44.490805+0000","last_undegraded":"2023-11-08T05:30:44.490805+0000","last_fullsized":"2023-11-08T05:28:13.542832+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":1,"acting":1,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":12,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738848,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":645040,"kb_used_data":5352,"kb_used_omap":275,"kb_used_meta":639404,"kb_avail":93726800,"statfs":{"total":96636764160,"available":95976243200,"internally_reserved":0,"allocated":5480448,"data_stored":2697647,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":281650,"internal_metadata":654750670},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:59 2023","interfaces":[{"interface":"back","average":{"1min":0.30499999999999999,"5min":0.439,"15min":0.36099999999999999},"min":{"1min":0.17399999999999999,"5min":0.14499999999999999,"15min":0.14499999999999999},"max":{"1min":0.60099999999999998,"5min":4.9379999999999997,"15min":4.9379999999999997},"last":0.379},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.49399999999999999,"15min":0.42399999999999999},"min":{"1min":0.221,"5min":0.19600000000000001,"15min":0.17000000000000001},"max":{"1min":0.63200000000000001,"5min":4.9859999999999998,"15min":4.9859999999999998},"last":0.40899999999999997}]},{"osd":1,"last update":"Wed Nov 8 05:32:06 2023","interfaces":[{"interface":"back","average":{"1min":0.27400000000000002,"5min":0.307,"15min":0.46800000000000003},"min":{"1min":0.17599999999999999,"5min":0.16,"15min":0.153},"max":{"1min":0.39000000000000001,"5min":1.825,"15min":23.242999999999999},"last":0.22900000000000001},{"interface":"front","average":{"1min":0.28000000000000003,"5min":0.34799999999999998,"15min":0.45700000000000002},"min":{"1min":0.20200000000000001,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":0.41699999999999998,"5min":2.0419999999999998,"15min":23.251000000000001},"last":0.247}]}]},{"osd":0,"up_from":8,"seq":34359738798,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1339228,"kb_used_data":398304,"kb_used_omap":291,"kb_used_meta":936476,"kb_avail":93032612,"statfs":{"total":96636764160,"available":95265394688,"internally_reserved":0,"allocated":407863296,"data_stored":405103900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":298462,"internal_metadata":958951970},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.436,"15min":0.36299999999999999},"min":{"1min":0.19700000000000001,"5min":0.14699999999999999,"15min":0.13700000000000001},"max":{"1min":3.222,"5min":6.5499999999999998,"15min":6.9989999999999997},"last":0.245},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.46300000000000002,"15min":0.42299999999999999},"min":{"1min":0.17799999999999999,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":5.0030000000000001,"5min":6.0629999999999997,"15min":9.6869999999999994},"last":0.28599999999999998}]},{"osd":2,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.51600000000000001,"15min":0.47099999999999997},"min":{"1min":0.21099999999999999,"5min":0.153,"15min":0.153},"max":{"1min":5.7439999999999998,"5min":5.8280000000000003,"15min":9.7639999999999993},"last":0.252},{"interface":"front","average":{"1min":0.497,"5min":0.56200000000000006,"15min":0.42299999999999999},"min":{"1min":0.20399999999999999,"5min":0.187,"15min":0.16800000000000001},"max":{"1min":3.9319999999999999,"5min":8.8040000000000003,"15min":8.8040000000000003},"last":0.29899999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738850,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":650100,"kb_used_data":5356,"kb_used_omap":214,"kb_used_meta":644521,"kb_avail":93721740,"statfs":{"total":96636764160,"available":95971061760,"internally_reserved":0,"allocated":5484544,"data_stored":2698616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":219335,"internal_metadata":659990329},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:56 2023","interfaces":[{"interface":"back","average":{"1min":0.30399999999999999,"5min":0.34699999999999998,"15min":0.36499999999999999},"min":{"1min":0.23999999999999999,"5min":0.14199999999999999,"15min":0.14199999999999999},"max":{"1min":0.42399999999999999,"5min":1.595,"15min":7.4249999999999998},"last":0.17100000000000001},{"interface":"front","average":{"1min":0.35499999999999998,"5min":0.40899999999999997,"15min":0.376},"min":{"1min":0.23999999999999999,"5min":0.19600000000000001,"15min":0.183},"max":{"1min":0.46999999999999997,"5min":2.6859999999999999,"15min":2.6859999999999999},"last":0.23000000000000001}]},{"osd":2,"last update":"Wed Nov 8 05:32:02 2023","interfaces":[{"interface":"back","average":{"1min":0.252,"5min":0.33000000000000002,"15min":0.36499999999999999},"min":{"1min":0.161,"5min":0.161,"15min":0.161},"max":{"1min":0.39600000000000002,"5min":2.2650000000000001,"15min":5.8209999999999997},"last":0.38100000000000001},{"interface":"front","average":{"1min":0.25800000000000001,"5min":0.29899999999999999,"15min":0.35599999999999998},"min":{"1min":0.17000000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.35799999999999998,"5min":1.137,"15min":4.2350000000000003},"last":0.40100000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-08T05:32:32.732 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-11-08T05:32:33.014 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T05:32:33.017 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2023-11-08T05:32:33.027 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":1650,"stamp":"2023-11-08T05:32:32.081573+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":10,"acting":13,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":109,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2634368,"kb_used_data":409012,"kb_used_omap":780,"kb_used_meta":2220403,"kb_avail":280481152,"statfs":{"total":289910292480,"available":287212699648,"internally_reserved":0,"allocated":418828288,"data_stored":410500163,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":799447,"internal_metadata":2273692969},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-536782902,"num_objects":-164,"num_object_clones":0,"num_object_copies":-328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-115,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-164,"num_whiteouts":0,"num_read":-122056,"num_read_kb":-39011942,"num_write":-413642,"num_write_kb":-151705339,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-8,"num_bytes_recovered":-16922624,"num_keys_recovered":-2,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.424817"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490644+0000","last_change":"2023-11-08T05:28:13.557071+0000","last_active":"2023-11-08T05:30:44.490644+0000","last_peered":"2023-11-08T05:30:44.490644+0000","last_clean":"2023-11-08T05:28:01.458439+0000","last_became_active":"2023-11-08T05:28:13.557071+0000","last_became_peered":"2023-11-08T05:28:13.557071+0000","last_unstale":"2023-11-08T05:30:44.490644+0000","last_undegraded":"2023-11-08T05:30:44.490644+0000","last_fullsized":"2023-11-08T05:28:13.547025+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":459,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:45.794760+0000","last_change":"2023-11-08T05:30:45.794760+0000","last_active":"2023-11-08T05:30:45.794760+0000","last_peered":"2023-11-08T05:30:45.794760+0000","last_clean":"2023-11-08T05:30:45.794760+0000","last_became_active":"2023-11-08T05:30:45.794398+0000","last_became_peered":"2023-11-08T05:30:45.794398+0000","last_unstale":"2023-11-08T05:30:45.794760+0000","last_undegraded":"2023-11-08T05:30:45.794760+0000","last_fullsized":"2023-11-08T05:30:45.794760+0000","mapping_epoch":458,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":459,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":465,"reported_epoch":459,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:45.795085+0000","last_change":"2023-11-08T05:30:45.795085+0000","last_active":"2023-11-08T05:30:45.795085+0000","last_peered":"2023-11-08T05:30:45.795085+0000","last_clean":"2023-11-08T05:30:45.795085+0000","last_became_active":"2023-11-08T05:30:45.794743+0000","last_became_peered":"2023-11-08T05:30:45.794743+0000","last_unstale":"2023-11-08T05:30:45.795085+0000","last_undegraded":"2023-11-08T05:30:45.795085+0000","last_fullsized":"2023-11-08T05:30:45.795085+0000","mapping_epoch":458,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":459,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":454,"reported_epoch":458,"state":"active+clean","last_fresh":"2023-11-08T05:30:44.490618+0000","last_change":"2023-11-08T04:52:47.548597+0000","last_active":"2023-11-08T05:30:44.490618+0000","last_peered":"2023-11-08T05:30:44.490618+0000","last_clean":"2023-11-08T05:30:44.490618+0000","last_became_active":"2023-11-08T04:52:47.548294+0000","last_became_peered":"2023-11-08T04:52:47.548294+0000","last_unstale":"2023-11-08T05:30:44.490618+0000","last_undegraded":"2023-11-08T05:30:44.490618+0000","last_fullsized":"2023-11-08T05:30:44.490618+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":460,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490523+0000","last_change":"2023-11-08T05:28:13.545468+0000","last_active":"2023-11-08T05:30:44.490523+0000","last_peered":"2023-11-08T05:30:44.490523+0000","last_clean":"2023-11-08T05:28:01.456723+0000","last_became_active":"2023-11-08T05:28:13.545468+0000","last_became_peered":"2023-11-08T05:28:13.545468+0000","last_unstale":"2023-11-08T05:30:44.490523+0000","last_undegraded":"2023-11-08T05:30:44.490523+0000","last_fullsized":"2023-11-08T05:28:13.537207+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_clean_scrub_stamp":"2023-11-08T04:52:45.528796+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":465,"reported_epoch":458,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:44.491907+0000","last_change":"2023-11-08T05:29:44.395350+0000","last_active":"2023-11-08T05:30:44.491907+0000","last_peered":"2023-11-08T05:30:44.491907+0000","last_clean":"2023-11-08T05:30:44.491907+0000","last_became_active":"2023-11-08T05:29:44.395012+0000","last_became_peered":"2023-11-08T05:29:44.395012+0000","last_unstale":"2023-11-08T05:30:44.491907+0000","last_undegraded":"2023-11-08T05:30:44.491907+0000","last_fullsized":"2023-11-08T05:30:44.491907+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490629+0000","last_change":"2023-11-08T05:28:13.563357+0000","last_active":"2023-11-08T05:30:44.490629+0000","last_peered":"2023-11-08T05:30:44.490629+0000","last_clean":"2023-11-08T05:28:01.457691+0000","last_became_active":"2023-11-08T05:28:13.563357+0000","last_became_peered":"2023-11-08T05:28:13.563357+0000","last_unstale":"2023-11-08T05:30:44.490629+0000","last_undegraded":"2023-11-08T05:30:44.490629+0000","last_fullsized":"2023-11-08T05:28:13.541614+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490691+0000","last_change":"2023-11-08T05:28:13.550672+0000","last_active":"2023-11-08T05:30:44.490691+0000","last_peered":"2023-11-08T05:30:44.490691+0000","last_clean":"2023-11-08T05:28:01.457904+0000","last_became_active":"2023-11-08T05:28:13.550672+0000","last_became_peered":"2023-11-08T05:28:13.550672+0000","last_unstale":"2023-11-08T05:30:44.490691+0000","last_undegraded":"2023-11-08T05:30:44.490691+0000","last_fullsized":"2023-11-08T05:28:13.539595+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490805+0000","last_change":"2023-11-08T05:28:13.557968+0000","last_active":"2023-11-08T05:30:44.490805+0000","last_peered":"2023-11-08T05:30:44.490805+0000","last_clean":"2023-11-08T05:28:01.458433+0000","last_became_active":"2023-11-08T05:28:13.557968+0000","last_became_peered":"2023-11-08T05:28:13.557968+0000","last_unstale":"2023-11-08T05:30:44.490805+0000","last_undegraded":"2023-11-08T05:30:44.490805+0000","last_fullsized":"2023-11-08T05:28:13.542832+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":1,"acting":1,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":12,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738848,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":645040,"kb_used_data":5352,"kb_used_omap":275,"kb_used_meta":639404,"kb_avail":93726800,"statfs":{"total":96636764160,"available":95976243200,"internally_reserved":0,"allocated":5480448,"data_stored":2697647,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":281650,"internal_metadata":654750670},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:59 2023","interfaces":[{"interface":"back","average":{"1min":0.30499999999999999,"5min":0.439,"15min":0.36099999999999999},"min":{"1min":0.17399999999999999,"5min":0.14499999999999999,"15min":0.14499999999999999},"max":{"1min":0.60099999999999998,"5min":4.9379999999999997,"15min":4.9379999999999997},"last":0.379},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.49399999999999999,"15min":0.42399999999999999},"min":{"1min":0.221,"5min":0.19600000000000001,"15min":0.17000000000000001},"max":{"1min":0.63200000000000001,"5min":4.9859999999999998,"15min":4.9859999999999998},"last":0.40899999999999997}]},{"osd":1,"last update":"Wed Nov 8 05:32:06 2023","interfaces":[{"interface":"back","average":{"1min":0.27400000000000002,"5min":0.307,"15min":0.46800000000000003},"min":{"1min":0.17599999999999999,"5min":0.16,"15min":0.153},"max":{"1min":0.39000000000000001,"5min":1.825,"15min":23.242999999999999},"last":0.22900000000000001},{"interface":"front","average":{"1min":0.28000000000000003,"5min":0.34799999999999998,"15min":0.45700000000000002},"min":{"1min":0.20200000000000001,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":0.41699999999999998,"5min":2.0419999999999998,"15min":23.251000000000001},"last":0.247}]}]},{"osd":0,"up_from":8,"seq":34359738798,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1339228,"kb_used_data":398304,"kb_used_omap":291,"kb_used_meta":936476,"kb_avail":93032612,"statfs":{"total":96636764160,"available":95265394688,"internally_reserved":0,"allocated":407863296,"data_stored":405103900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":298462,"internal_metadata":958951970},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.436,"15min":0.36299999999999999},"min":{"1min":0.19700000000000001,"5min":0.14699999999999999,"15min":0.13700000000000001},"max":{"1min":3.222,"5min":6.5499999999999998,"15min":6.9989999999999997},"last":0.245},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.46300000000000002,"15min":0.42299999999999999},"min":{"1min":0.17799999999999999,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":5.0030000000000001,"5min":6.0629999999999997,"15min":9.6869999999999994},"last":0.28599999999999998}]},{"osd":2,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.51600000000000001,"15min":0.47099999999999997},"min":{"1min":0.21099999999999999,"5min":0.153,"15min":0.153},"max":{"1min":5.7439999999999998,"5min":5.8280000000000003,"15min":9.7639999999999993},"last":0.252},{"interface":"front","average":{"1min":0.497,"5min":0.56200000000000006,"15min":0.42299999999999999},"min":{"1min":0.20399999999999999,"5min":0.187,"15min":0.16800000000000001},"max":{"1min":3.9319999999999999,"5min":8.8040000000000003,"15min":8.8040000000000003},"last":0.29899999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738850,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":650100,"kb_used_data":5356,"kb_used_omap":214,"kb_used_meta":644521,"kb_avail":93721740,"statfs":{"total":96636764160,"available":95971061760,"internally_reserved":0,"allocated":5484544,"data_stored":2698616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":219335,"internal_metadata":659990329},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:56 2023","interfaces":[{"interface":"back","average":{"1min":0.30399999999999999,"5min":0.34699999999999998,"15min":0.36499999999999999},"min":{"1min":0.23999999999999999,"5min":0.14199999999999999,"15min":0.14199999999999999},"max":{"1min":0.42399999999999999,"5min":1.595,"15min":7.4249999999999998},"last":0.17100000000000001},{"interface":"front","average":{"1min":0.35499999999999998,"5min":0.40899999999999997,"15min":0.376},"min":{"1min":0.23999999999999999,"5min":0.19600000000000001,"15min":0.183},"max":{"1min":0.46999999999999997,"5min":2.6859999999999999,"15min":2.6859999999999999},"last":0.23000000000000001}]},{"osd":2,"last update":"Wed Nov 8 05:32:02 2023","interfaces":[{"interface":"back","average":{"1min":0.252,"5min":0.33000000000000002,"15min":0.36499999999999999},"min":{"1min":0.161,"5min":0.161,"15min":0.161},"max":{"1min":0.39600000000000002,"5min":2.2650000000000001,"15min":5.8209999999999997},"last":0.38100000000000001},{"interface":"front","average":{"1min":0.25800000000000001,"5min":0.29899999999999999,"15min":0.35599999999999998},"min":{"1min":0.17000000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.35799999999999998,"5min":1.137,"15min":4.2350000000000003},"last":0.40100000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-08T05:32:33.028 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-11-08T05:32:33.028 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 459, 'reported_epoch': 458, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:30:44.490644+0000', 'last_change': '2023-11-08T05:28:13.557071+0000', 'last_active': '2023-11-08T05:30:44.490644+0000', 'last_peered': '2023-11-08T05:30:44.490644+0000', 'last_clean': '2023-11-08T05:28:01.458439+0000', 'last_became_active': '2023-11-08T05:28:13.557071+0000', 'last_became_peered': '2023-11-08T05:28:13.557071+0000', 'last_unstale': '2023-11-08T05:30:44.490644+0000', 'last_undegraded': '2023-11-08T05:30:44.490644+0000', 'last_fullsized': '2023-11-08T05:28:13.547025+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:32:33.029 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-11-08T05:32:33.029 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 11, 'reported_epoch': 459, 'state': 'active+clean+remapped', 'last_fresh': '2023-11-08T05:30:45.794760+0000', 'last_change': '2023-11-08T05:30:45.794760+0000', 'last_active': '2023-11-08T05:30:45.794760+0000', 'last_peered': '2023-11-08T05:30:45.794760+0000', 'last_clean': '2023-11-08T05:30:45.794760+0000', 'last_became_active': '2023-11-08T05:30:45.794398+0000', 'last_became_peered': '2023-11-08T05:30:45.794398+0000', 'last_unstale': '2023-11-08T05:30:45.794760+0000', 'last_undegraded': '2023-11-08T05:30:45.794760+0000', 'last_fullsized': '2023-11-08T05:30:45.794760+0000', 'mapping_epoch': 458, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 459, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2], 'acting': [2, 1], 'avail_no_missing': ['2', '1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2023-11-08T05:32:33.029 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.1 is not active+clean 2023-11-08T05:32:33.030 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.1', 'version': "0'0", 'reported_seq': 465, 'reported_epoch': 459, 'state': 'active+clean+remapped', 'last_fresh': '2023-11-08T05:30:45.795085+0000', 'last_change': '2023-11-08T05:30:45.795085+0000', 'last_active': '2023-11-08T05:30:45.795085+0000', 'last_peered': '2023-11-08T05:30:45.795085+0000', 'last_clean': '2023-11-08T05:30:45.795085+0000', 'last_became_active': '2023-11-08T05:30:45.794743+0000', 'last_became_peered': '2023-11-08T05:30:45.794743+0000', 'last_unstale': '2023-11-08T05:30:45.795085+0000', 'last_undegraded': '2023-11-08T05:30:45.795085+0000', 'last_fullsized': '2023-11-08T05:30:45.795085+0000', 'mapping_epoch': 458, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 459, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2], 'acting': [2, 1], 'avail_no_missing': ['2', '1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2023-11-08T05:32:33.030 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-11-08T05:32:33.030 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "0'0", 'reported_seq': 460, 'reported_epoch': 458, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:30:44.490523+0000', 'last_change': '2023-11-08T05:28:13.545468+0000', 'last_active': '2023-11-08T05:30:44.490523+0000', 'last_peered': '2023-11-08T05:30:44.490523+0000', 'last_clean': '2023-11-08T05:28:01.456723+0000', 'last_became_active': '2023-11-08T05:28:13.545468+0000', 'last_became_peered': '2023-11-08T05:28:13.545468+0000', 'last_unstale': '2023-11-08T05:30:44.490523+0000', 'last_undegraded': '2023-11-08T05:30:44.490523+0000', 'last_fullsized': '2023-11-08T05:28:13.537207+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 9, 'last_epoch_clean': 10, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:45.528796+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:45.528796+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:45.528796+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:32:33.031 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.0 is not active+clean 2023-11-08T05:32:33.031 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.0', 'version': "0'0", 'reported_seq': 465, 'reported_epoch': 458, 'state': 'active+clean+remapped', 'last_fresh': '2023-11-08T05:30:44.491907+0000', 'last_change': '2023-11-08T05:29:44.395350+0000', 'last_active': '2023-11-08T05:30:44.491907+0000', 'last_peered': '2023-11-08T05:30:44.491907+0000', 'last_clean': '2023-11-08T05:30:44.491907+0000', 'last_became_active': '2023-11-08T05:29:44.395012+0000', 'last_became_peered': '2023-11-08T05:29:44.395012+0000', 'last_unstale': '2023-11-08T05:30:44.491907+0000', 'last_undegraded': '2023-11-08T05:30:44.491907+0000', 'last_fullsized': '2023-11-08T05:30:44.491907+0000', 'mapping_epoch': 455, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 456, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2], 'acting': [2, 1], 'avail_no_missing': ['2', '1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2023-11-08T05:32:33.031 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-11-08T05:32:33.032 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 459, 'reported_epoch': 458, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:30:44.490629+0000', 'last_change': '2023-11-08T05:28:13.563357+0000', 'last_active': '2023-11-08T05:30:44.490629+0000', 'last_peered': '2023-11-08T05:30:44.490629+0000', 'last_clean': '2023-11-08T05:28:01.457691+0000', 'last_became_active': '2023-11-08T05:28:13.563357+0000', 'last_became_peered': '2023-11-08T05:28:13.563357+0000', 'last_unstale': '2023-11-08T05:30:44.490629+0000', 'last_undegraded': '2023-11-08T05:30:44.490629+0000', 'last_fullsized': '2023-11-08T05:28:13.541614+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:32:33.033 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-11-08T05:32:33.033 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 459, 'reported_epoch': 458, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:30:44.490691+0000', 'last_change': '2023-11-08T05:28:13.550672+0000', 'last_active': '2023-11-08T05:30:44.490691+0000', 'last_peered': '2023-11-08T05:30:44.490691+0000', 'last_clean': '2023-11-08T05:28:01.457904+0000', 'last_became_active': '2023-11-08T05:28:13.550672+0000', 'last_became_peered': '2023-11-08T05:28:13.550672+0000', 'last_unstale': '2023-11-08T05:30:44.490691+0000', 'last_undegraded': '2023-11-08T05:30:44.490691+0000', 'last_fullsized': '2023-11-08T05:28:13.539595+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:32:33.034 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-11-08T05:32:33.034 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 459, 'reported_epoch': 458, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:30:44.490805+0000', 'last_change': '2023-11-08T05:28:13.557968+0000', 'last_active': '2023-11-08T05:30:44.490805+0000', 'last_peered': '2023-11-08T05:30:44.490805+0000', 'last_clean': '2023-11-08T05:28:01.458433+0000', 'last_became_active': '2023-11-08T05:28:13.557968+0000', 'last_became_peered': '2023-11-08T05:28:13.557968+0000', 'last_unstale': '2023-11-08T05:30:44.490805+0000', 'last_undegraded': '2023-11-08T05:30:44.490805+0000', 'last_fullsized': '2023-11-08T05:28:13.542832+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:32:33.035 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-11-08T05:32:33.381 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T05:32:33.381 INFO:teuthology.orchestra.run.smithi087.stdout:{"fsid":"ab4cbc06-449d-497d-b513-220d9db2b590","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 5 pgs undersized","count":5},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2396,"monmap":{"epoch":1,"min_mon_release_name":"pacific","num_mons":1},"osdmap":{"epoch":459,"num_osds":3,"num_up_osds":2,"osd_up_since":1699421292,"num_in_osds":3,"osd_in_since":1699419158,"num_remapped_pgs":3},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean+remapped","count":3},{"state_name":"active+clean","count":1}],"num_pgs":9,"num_pools":2,"num_objects":0,"data_bytes":0,"bytes_used":2697592832,"bytes_avail":287212699648,"bytes_total":289910292480},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":41,"modified":"2023-11-08T05:29:32.044544+0000","services":{}},"progress_events":{}} 2023-11-08T05:32:33.394 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-11-08T05:32:33.395 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-11-08T05:32:33.678 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T05:32:33.679 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2023-11-08T05:32:33.690 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":1650,"stamp":"2023-11-08T05:32:32.081573+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":10,"acting":13,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":109,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2634368,"kb_used_data":409012,"kb_used_omap":780,"kb_used_meta":2220403,"kb_avail":280481152,"statfs":{"total":289910292480,"available":287212699648,"internally_reserved":0,"allocated":418828288,"data_stored":410500163,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":799447,"internal_metadata":2273692969},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-536782902,"num_objects":-164,"num_object_clones":0,"num_object_copies":-328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-115,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-164,"num_whiteouts":0,"num_read":-122056,"num_read_kb":-39011942,"num_write":-413642,"num_write_kb":-151705339,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-8,"num_bytes_recovered":-16922624,"num_keys_recovered":-2,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.424817"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490644+0000","last_change":"2023-11-08T05:28:13.557071+0000","last_active":"2023-11-08T05:30:44.490644+0000","last_peered":"2023-11-08T05:30:44.490644+0000","last_clean":"2023-11-08T05:28:01.458439+0000","last_became_active":"2023-11-08T05:28:13.557071+0000","last_became_peered":"2023-11-08T05:28:13.557071+0000","last_unstale":"2023-11-08T05:30:44.490644+0000","last_undegraded":"2023-11-08T05:30:44.490644+0000","last_fullsized":"2023-11-08T05:28:13.547025+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":459,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:45.794760+0000","last_change":"2023-11-08T05:30:45.794760+0000","last_active":"2023-11-08T05:30:45.794760+0000","last_peered":"2023-11-08T05:30:45.794760+0000","last_clean":"2023-11-08T05:30:45.794760+0000","last_became_active":"2023-11-08T05:30:45.794398+0000","last_became_peered":"2023-11-08T05:30:45.794398+0000","last_unstale":"2023-11-08T05:30:45.794760+0000","last_undegraded":"2023-11-08T05:30:45.794760+0000","last_fullsized":"2023-11-08T05:30:45.794760+0000","mapping_epoch":458,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":459,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":465,"reported_epoch":459,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:45.795085+0000","last_change":"2023-11-08T05:30:45.795085+0000","last_active":"2023-11-08T05:30:45.795085+0000","last_peered":"2023-11-08T05:30:45.795085+0000","last_clean":"2023-11-08T05:30:45.795085+0000","last_became_active":"2023-11-08T05:30:45.794743+0000","last_became_peered":"2023-11-08T05:30:45.794743+0000","last_unstale":"2023-11-08T05:30:45.795085+0000","last_undegraded":"2023-11-08T05:30:45.795085+0000","last_fullsized":"2023-11-08T05:30:45.795085+0000","mapping_epoch":458,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":459,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":454,"reported_epoch":458,"state":"active+clean","last_fresh":"2023-11-08T05:30:44.490618+0000","last_change":"2023-11-08T04:52:47.548597+0000","last_active":"2023-11-08T05:30:44.490618+0000","last_peered":"2023-11-08T05:30:44.490618+0000","last_clean":"2023-11-08T05:30:44.490618+0000","last_became_active":"2023-11-08T04:52:47.548294+0000","last_became_peered":"2023-11-08T04:52:47.548294+0000","last_unstale":"2023-11-08T05:30:44.490618+0000","last_undegraded":"2023-11-08T05:30:44.490618+0000","last_fullsized":"2023-11-08T05:30:44.490618+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":460,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490523+0000","last_change":"2023-11-08T05:28:13.545468+0000","last_active":"2023-11-08T05:30:44.490523+0000","last_peered":"2023-11-08T05:30:44.490523+0000","last_clean":"2023-11-08T05:28:01.456723+0000","last_became_active":"2023-11-08T05:28:13.545468+0000","last_became_peered":"2023-11-08T05:28:13.545468+0000","last_unstale":"2023-11-08T05:30:44.490523+0000","last_undegraded":"2023-11-08T05:30:44.490523+0000","last_fullsized":"2023-11-08T05:28:13.537207+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_clean_scrub_stamp":"2023-11-08T04:52:45.528796+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":465,"reported_epoch":458,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:44.491907+0000","last_change":"2023-11-08T05:29:44.395350+0000","last_active":"2023-11-08T05:30:44.491907+0000","last_peered":"2023-11-08T05:30:44.491907+0000","last_clean":"2023-11-08T05:30:44.491907+0000","last_became_active":"2023-11-08T05:29:44.395012+0000","last_became_peered":"2023-11-08T05:29:44.395012+0000","last_unstale":"2023-11-08T05:30:44.491907+0000","last_undegraded":"2023-11-08T05:30:44.491907+0000","last_fullsized":"2023-11-08T05:30:44.491907+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490629+0000","last_change":"2023-11-08T05:28:13.563357+0000","last_active":"2023-11-08T05:30:44.490629+0000","last_peered":"2023-11-08T05:30:44.490629+0000","last_clean":"2023-11-08T05:28:01.457691+0000","last_became_active":"2023-11-08T05:28:13.563357+0000","last_became_peered":"2023-11-08T05:28:13.563357+0000","last_unstale":"2023-11-08T05:30:44.490629+0000","last_undegraded":"2023-11-08T05:30:44.490629+0000","last_fullsized":"2023-11-08T05:28:13.541614+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490691+0000","last_change":"2023-11-08T05:28:13.550672+0000","last_active":"2023-11-08T05:30:44.490691+0000","last_peered":"2023-11-08T05:30:44.490691+0000","last_clean":"2023-11-08T05:28:01.457904+0000","last_became_active":"2023-11-08T05:28:13.550672+0000","last_became_peered":"2023-11-08T05:28:13.550672+0000","last_unstale":"2023-11-08T05:30:44.490691+0000","last_undegraded":"2023-11-08T05:30:44.490691+0000","last_fullsized":"2023-11-08T05:28:13.539595+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490805+0000","last_change":"2023-11-08T05:28:13.557968+0000","last_active":"2023-11-08T05:30:44.490805+0000","last_peered":"2023-11-08T05:30:44.490805+0000","last_clean":"2023-11-08T05:28:01.458433+0000","last_became_active":"2023-11-08T05:28:13.557968+0000","last_became_peered":"2023-11-08T05:28:13.557968+0000","last_unstale":"2023-11-08T05:30:44.490805+0000","last_undegraded":"2023-11-08T05:30:44.490805+0000","last_fullsized":"2023-11-08T05:28:13.542832+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":1,"acting":1,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":12,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738848,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":645040,"kb_used_data":5352,"kb_used_omap":275,"kb_used_meta":639404,"kb_avail":93726800,"statfs":{"total":96636764160,"available":95976243200,"internally_reserved":0,"allocated":5480448,"data_stored":2697647,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":281650,"internal_metadata":654750670},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:59 2023","interfaces":[{"interface":"back","average":{"1min":0.30499999999999999,"5min":0.439,"15min":0.36099999999999999},"min":{"1min":0.17399999999999999,"5min":0.14499999999999999,"15min":0.14499999999999999},"max":{"1min":0.60099999999999998,"5min":4.9379999999999997,"15min":4.9379999999999997},"last":0.379},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.49399999999999999,"15min":0.42399999999999999},"min":{"1min":0.221,"5min":0.19600000000000001,"15min":0.17000000000000001},"max":{"1min":0.63200000000000001,"5min":4.9859999999999998,"15min":4.9859999999999998},"last":0.40899999999999997}]},{"osd":1,"last update":"Wed Nov 8 05:32:06 2023","interfaces":[{"interface":"back","average":{"1min":0.27400000000000002,"5min":0.307,"15min":0.46800000000000003},"min":{"1min":0.17599999999999999,"5min":0.16,"15min":0.153},"max":{"1min":0.39000000000000001,"5min":1.825,"15min":23.242999999999999},"last":0.22900000000000001},{"interface":"front","average":{"1min":0.28000000000000003,"5min":0.34799999999999998,"15min":0.45700000000000002},"min":{"1min":0.20200000000000001,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":0.41699999999999998,"5min":2.0419999999999998,"15min":23.251000000000001},"last":0.247}]}]},{"osd":0,"up_from":8,"seq":34359738798,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1339228,"kb_used_data":398304,"kb_used_omap":291,"kb_used_meta":936476,"kb_avail":93032612,"statfs":{"total":96636764160,"available":95265394688,"internally_reserved":0,"allocated":407863296,"data_stored":405103900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":298462,"internal_metadata":958951970},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.436,"15min":0.36299999999999999},"min":{"1min":0.19700000000000001,"5min":0.14699999999999999,"15min":0.13700000000000001},"max":{"1min":3.222,"5min":6.5499999999999998,"15min":6.9989999999999997},"last":0.245},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.46300000000000002,"15min":0.42299999999999999},"min":{"1min":0.17799999999999999,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":5.0030000000000001,"5min":6.0629999999999997,"15min":9.6869999999999994},"last":0.28599999999999998}]},{"osd":2,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.51600000000000001,"15min":0.47099999999999997},"min":{"1min":0.21099999999999999,"5min":0.153,"15min":0.153},"max":{"1min":5.7439999999999998,"5min":5.8280000000000003,"15min":9.7639999999999993},"last":0.252},{"interface":"front","average":{"1min":0.497,"5min":0.56200000000000006,"15min":0.42299999999999999},"min":{"1min":0.20399999999999999,"5min":0.187,"15min":0.16800000000000001},"max":{"1min":3.9319999999999999,"5min":8.8040000000000003,"15min":8.8040000000000003},"last":0.29899999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738850,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":650100,"kb_used_data":5356,"kb_used_omap":214,"kb_used_meta":644521,"kb_avail":93721740,"statfs":{"total":96636764160,"available":95971061760,"internally_reserved":0,"allocated":5484544,"data_stored":2698616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":219335,"internal_metadata":659990329},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:56 2023","interfaces":[{"interface":"back","average":{"1min":0.30399999999999999,"5min":0.34699999999999998,"15min":0.36499999999999999},"min":{"1min":0.23999999999999999,"5min":0.14199999999999999,"15min":0.14199999999999999},"max":{"1min":0.42399999999999999,"5min":1.595,"15min":7.4249999999999998},"last":0.17100000000000001},{"interface":"front","average":{"1min":0.35499999999999998,"5min":0.40899999999999997,"15min":0.376},"min":{"1min":0.23999999999999999,"5min":0.19600000000000001,"15min":0.183},"max":{"1min":0.46999999999999997,"5min":2.6859999999999999,"15min":2.6859999999999999},"last":0.23000000000000001}]},{"osd":2,"last update":"Wed Nov 8 05:32:02 2023","interfaces":[{"interface":"back","average":{"1min":0.252,"5min":0.33000000000000002,"15min":0.36499999999999999},"min":{"1min":0.161,"5min":0.161,"15min":0.161},"max":{"1min":0.39600000000000002,"5min":2.2650000000000001,"15min":5.8209999999999997},"last":0.38100000000000001},{"interface":"front","average":{"1min":0.25800000000000001,"5min":0.29899999999999999,"15min":0.35599999999999998},"min":{"1min":0.17000000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.35799999999999998,"5min":1.137,"15min":4.2350000000000003},"last":0.40100000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-08T05:32:36.692 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-11-08T05:32:36.989 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T05:32:36.991 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2023-11-08T05:32:37.002 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":1652,"stamp":"2023-11-08T05:32:36.082498+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":10,"acting":13,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":109,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2634368,"kb_used_data":409012,"kb_used_omap":780,"kb_used_meta":2220403,"kb_avail":280481152,"statfs":{"total":289910292480,"available":287212699648,"internally_reserved":0,"allocated":418828288,"data_stored":410500163,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":799447,"internal_metadata":2273692969},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-536782902,"num_objects":-164,"num_object_clones":0,"num_object_copies":-328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-115,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-164,"num_whiteouts":0,"num_read":-122056,"num_read_kb":-39011942,"num_write":-413642,"num_write_kb":-151705339,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-8,"num_bytes_recovered":-16922624,"num_keys_recovered":-2,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.424817"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490644+0000","last_change":"2023-11-08T05:28:13.557071+0000","last_active":"2023-11-08T05:30:44.490644+0000","last_peered":"2023-11-08T05:30:44.490644+0000","last_clean":"2023-11-08T05:28:01.458439+0000","last_became_active":"2023-11-08T05:28:13.557071+0000","last_became_peered":"2023-11-08T05:28:13.557071+0000","last_unstale":"2023-11-08T05:30:44.490644+0000","last_undegraded":"2023-11-08T05:30:44.490644+0000","last_fullsized":"2023-11-08T05:28:13.547025+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":459,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:45.794760+0000","last_change":"2023-11-08T05:30:45.794760+0000","last_active":"2023-11-08T05:30:45.794760+0000","last_peered":"2023-11-08T05:30:45.794760+0000","last_clean":"2023-11-08T05:30:45.794760+0000","last_became_active":"2023-11-08T05:30:45.794398+0000","last_became_peered":"2023-11-08T05:30:45.794398+0000","last_unstale":"2023-11-08T05:30:45.794760+0000","last_undegraded":"2023-11-08T05:30:45.794760+0000","last_fullsized":"2023-11-08T05:30:45.794760+0000","mapping_epoch":458,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":459,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":465,"reported_epoch":459,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:45.795085+0000","last_change":"2023-11-08T05:30:45.795085+0000","last_active":"2023-11-08T05:30:45.795085+0000","last_peered":"2023-11-08T05:30:45.795085+0000","last_clean":"2023-11-08T05:30:45.795085+0000","last_became_active":"2023-11-08T05:30:45.794743+0000","last_became_peered":"2023-11-08T05:30:45.794743+0000","last_unstale":"2023-11-08T05:30:45.795085+0000","last_undegraded":"2023-11-08T05:30:45.795085+0000","last_fullsized":"2023-11-08T05:30:45.795085+0000","mapping_epoch":458,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":459,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":454,"reported_epoch":458,"state":"active+clean","last_fresh":"2023-11-08T05:30:44.490618+0000","last_change":"2023-11-08T04:52:47.548597+0000","last_active":"2023-11-08T05:30:44.490618+0000","last_peered":"2023-11-08T05:30:44.490618+0000","last_clean":"2023-11-08T05:30:44.490618+0000","last_became_active":"2023-11-08T04:52:47.548294+0000","last_became_peered":"2023-11-08T04:52:47.548294+0000","last_unstale":"2023-11-08T05:30:44.490618+0000","last_undegraded":"2023-11-08T05:30:44.490618+0000","last_fullsized":"2023-11-08T05:30:44.490618+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":460,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490523+0000","last_change":"2023-11-08T05:28:13.545468+0000","last_active":"2023-11-08T05:30:44.490523+0000","last_peered":"2023-11-08T05:30:44.490523+0000","last_clean":"2023-11-08T05:28:01.456723+0000","last_became_active":"2023-11-08T05:28:13.545468+0000","last_became_peered":"2023-11-08T05:28:13.545468+0000","last_unstale":"2023-11-08T05:30:44.490523+0000","last_undegraded":"2023-11-08T05:30:44.490523+0000","last_fullsized":"2023-11-08T05:28:13.537207+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_clean_scrub_stamp":"2023-11-08T04:52:45.528796+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":465,"reported_epoch":458,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:44.491907+0000","last_change":"2023-11-08T05:29:44.395350+0000","last_active":"2023-11-08T05:30:44.491907+0000","last_peered":"2023-11-08T05:30:44.491907+0000","last_clean":"2023-11-08T05:30:44.491907+0000","last_became_active":"2023-11-08T05:29:44.395012+0000","last_became_peered":"2023-11-08T05:29:44.395012+0000","last_unstale":"2023-11-08T05:30:44.491907+0000","last_undegraded":"2023-11-08T05:30:44.491907+0000","last_fullsized":"2023-11-08T05:30:44.491907+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490629+0000","last_change":"2023-11-08T05:28:13.563357+0000","last_active":"2023-11-08T05:30:44.490629+0000","last_peered":"2023-11-08T05:30:44.490629+0000","last_clean":"2023-11-08T05:28:01.457691+0000","last_became_active":"2023-11-08T05:28:13.563357+0000","last_became_peered":"2023-11-08T05:28:13.563357+0000","last_unstale":"2023-11-08T05:30:44.490629+0000","last_undegraded":"2023-11-08T05:30:44.490629+0000","last_fullsized":"2023-11-08T05:28:13.541614+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490691+0000","last_change":"2023-11-08T05:28:13.550672+0000","last_active":"2023-11-08T05:30:44.490691+0000","last_peered":"2023-11-08T05:30:44.490691+0000","last_clean":"2023-11-08T05:28:01.457904+0000","last_became_active":"2023-11-08T05:28:13.550672+0000","last_became_peered":"2023-11-08T05:28:13.550672+0000","last_unstale":"2023-11-08T05:30:44.490691+0000","last_undegraded":"2023-11-08T05:30:44.490691+0000","last_fullsized":"2023-11-08T05:28:13.539595+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490805+0000","last_change":"2023-11-08T05:28:13.557968+0000","last_active":"2023-11-08T05:30:44.490805+0000","last_peered":"2023-11-08T05:30:44.490805+0000","last_clean":"2023-11-08T05:28:01.458433+0000","last_became_active":"2023-11-08T05:28:13.557968+0000","last_became_peered":"2023-11-08T05:28:13.557968+0000","last_unstale":"2023-11-08T05:30:44.490805+0000","last_undegraded":"2023-11-08T05:30:44.490805+0000","last_fullsized":"2023-11-08T05:28:13.542832+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":1,"acting":1,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":12,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738849,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":645040,"kb_used_data":5352,"kb_used_omap":275,"kb_used_meta":639404,"kb_avail":93726800,"statfs":{"total":96636764160,"available":95976243200,"internally_reserved":0,"allocated":5480448,"data_stored":2697647,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":281650,"internal_metadata":654750670},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:59 2023","interfaces":[{"interface":"back","average":{"1min":0.30499999999999999,"5min":0.439,"15min":0.36099999999999999},"min":{"1min":0.17399999999999999,"5min":0.14499999999999999,"15min":0.14499999999999999},"max":{"1min":0.60099999999999998,"5min":4.9379999999999997,"15min":4.9379999999999997},"last":0.379},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.49399999999999999,"15min":0.42399999999999999},"min":{"1min":0.221,"5min":0.19600000000000001,"15min":0.17000000000000001},"max":{"1min":0.63200000000000001,"5min":4.9859999999999998,"15min":4.9859999999999998},"last":0.40899999999999997}]},{"osd":1,"last update":"Wed Nov 8 05:32:06 2023","interfaces":[{"interface":"back","average":{"1min":0.27400000000000002,"5min":0.307,"15min":0.46800000000000003},"min":{"1min":0.17599999999999999,"5min":0.16,"15min":0.153},"max":{"1min":0.39000000000000001,"5min":1.825,"15min":23.242999999999999},"last":0.29899999999999999},{"interface":"front","average":{"1min":0.28000000000000003,"5min":0.34799999999999998,"15min":0.45700000000000002},"min":{"1min":0.20200000000000001,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":0.41699999999999998,"5min":2.0419999999999998,"15min":23.251000000000001},"last":0.32100000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738798,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1339228,"kb_used_data":398304,"kb_used_omap":291,"kb_used_meta":936476,"kb_avail":93032612,"statfs":{"total":96636764160,"available":95265394688,"internally_reserved":0,"allocated":407863296,"data_stored":405103900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":298462,"internal_metadata":958951970},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.436,"15min":0.36299999999999999},"min":{"1min":0.19700000000000001,"5min":0.14699999999999999,"15min":0.13700000000000001},"max":{"1min":3.222,"5min":6.5499999999999998,"15min":6.9989999999999997},"last":0.245},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.46300000000000002,"15min":0.42299999999999999},"min":{"1min":0.17799999999999999,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":5.0030000000000001,"5min":6.0629999999999997,"15min":9.6869999999999994},"last":0.28599999999999998}]},{"osd":2,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.51600000000000001,"15min":0.47099999999999997},"min":{"1min":0.21099999999999999,"5min":0.153,"15min":0.153},"max":{"1min":5.7439999999999998,"5min":5.8280000000000003,"15min":9.7639999999999993},"last":0.252},{"interface":"front","average":{"1min":0.497,"5min":0.56200000000000006,"15min":0.42299999999999999},"min":{"1min":0.20399999999999999,"5min":0.187,"15min":0.16800000000000001},"max":{"1min":3.9319999999999999,"5min":8.8040000000000003,"15min":8.8040000000000003},"last":0.29899999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738851,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":650100,"kb_used_data":5356,"kb_used_omap":214,"kb_used_meta":644521,"kb_avail":93721740,"statfs":{"total":96636764160,"available":95971061760,"internally_reserved":0,"allocated":5484544,"data_stored":2698616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":219335,"internal_metadata":659990329},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:56 2023","interfaces":[{"interface":"back","average":{"1min":0.30399999999999999,"5min":0.34699999999999998,"15min":0.36499999999999999},"min":{"1min":0.23999999999999999,"5min":0.14199999999999999,"15min":0.14199999999999999},"max":{"1min":0.42399999999999999,"5min":1.595,"15min":7.4249999999999998},"last":0.17100000000000001},{"interface":"front","average":{"1min":0.35499999999999998,"5min":0.40899999999999997,"15min":0.376},"min":{"1min":0.23999999999999999,"5min":0.19600000000000001,"15min":0.183},"max":{"1min":0.46999999999999997,"5min":2.6859999999999999,"15min":2.6859999999999999},"last":0.23000000000000001}]},{"osd":2,"last update":"Wed Nov 8 05:32:02 2023","interfaces":[{"interface":"back","average":{"1min":0.252,"5min":0.33000000000000002,"15min":0.36499999999999999},"min":{"1min":0.161,"5min":0.161,"15min":0.161},"max":{"1min":0.39600000000000002,"5min":2.2650000000000001,"15min":5.8209999999999997},"last":0.214},{"interface":"front","average":{"1min":0.25800000000000001,"5min":0.29899999999999999,"15min":0.35599999999999998},"min":{"1min":0.17000000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.35799999999999998,"5min":1.137,"15min":4.2350000000000003},"last":0.247}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-08T05:32:37.003 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-11-08T05:32:37.287 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T05:32:37.289 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2023-11-08T05:32:37.300 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":1652,"stamp":"2023-11-08T05:32:36.082498+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":10,"acting":13,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":109,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2634368,"kb_used_data":409012,"kb_used_omap":780,"kb_used_meta":2220403,"kb_avail":280481152,"statfs":{"total":289910292480,"available":287212699648,"internally_reserved":0,"allocated":418828288,"data_stored":410500163,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":799447,"internal_metadata":2273692969},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-536782902,"num_objects":-164,"num_object_clones":0,"num_object_copies":-328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-115,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-164,"num_whiteouts":0,"num_read":-122056,"num_read_kb":-39011942,"num_write":-413642,"num_write_kb":-151705339,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-8,"num_bytes_recovered":-16922624,"num_keys_recovered":-2,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.424817"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490644+0000","last_change":"2023-11-08T05:28:13.557071+0000","last_active":"2023-11-08T05:30:44.490644+0000","last_peered":"2023-11-08T05:30:44.490644+0000","last_clean":"2023-11-08T05:28:01.458439+0000","last_became_active":"2023-11-08T05:28:13.557071+0000","last_became_peered":"2023-11-08T05:28:13.557071+0000","last_unstale":"2023-11-08T05:30:44.490644+0000","last_undegraded":"2023-11-08T05:30:44.490644+0000","last_fullsized":"2023-11-08T05:28:13.547025+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":459,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:45.794760+0000","last_change":"2023-11-08T05:30:45.794760+0000","last_active":"2023-11-08T05:30:45.794760+0000","last_peered":"2023-11-08T05:30:45.794760+0000","last_clean":"2023-11-08T05:30:45.794760+0000","last_became_active":"2023-11-08T05:30:45.794398+0000","last_became_peered":"2023-11-08T05:30:45.794398+0000","last_unstale":"2023-11-08T05:30:45.794760+0000","last_undegraded":"2023-11-08T05:30:45.794760+0000","last_fullsized":"2023-11-08T05:30:45.794760+0000","mapping_epoch":458,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":459,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":465,"reported_epoch":459,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:45.795085+0000","last_change":"2023-11-08T05:30:45.795085+0000","last_active":"2023-11-08T05:30:45.795085+0000","last_peered":"2023-11-08T05:30:45.795085+0000","last_clean":"2023-11-08T05:30:45.795085+0000","last_became_active":"2023-11-08T05:30:45.794743+0000","last_became_peered":"2023-11-08T05:30:45.794743+0000","last_unstale":"2023-11-08T05:30:45.795085+0000","last_undegraded":"2023-11-08T05:30:45.795085+0000","last_fullsized":"2023-11-08T05:30:45.795085+0000","mapping_epoch":458,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":459,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":454,"reported_epoch":458,"state":"active+clean","last_fresh":"2023-11-08T05:30:44.490618+0000","last_change":"2023-11-08T04:52:47.548597+0000","last_active":"2023-11-08T05:30:44.490618+0000","last_peered":"2023-11-08T05:30:44.490618+0000","last_clean":"2023-11-08T05:30:44.490618+0000","last_became_active":"2023-11-08T04:52:47.548294+0000","last_became_peered":"2023-11-08T04:52:47.548294+0000","last_unstale":"2023-11-08T05:30:44.490618+0000","last_undegraded":"2023-11-08T05:30:44.490618+0000","last_fullsized":"2023-11-08T05:30:44.490618+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":460,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490523+0000","last_change":"2023-11-08T05:28:13.545468+0000","last_active":"2023-11-08T05:30:44.490523+0000","last_peered":"2023-11-08T05:30:44.490523+0000","last_clean":"2023-11-08T05:28:01.456723+0000","last_became_active":"2023-11-08T05:28:13.545468+0000","last_became_peered":"2023-11-08T05:28:13.545468+0000","last_unstale":"2023-11-08T05:30:44.490523+0000","last_undegraded":"2023-11-08T05:30:44.490523+0000","last_fullsized":"2023-11-08T05:28:13.537207+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_clean_scrub_stamp":"2023-11-08T04:52:45.528796+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":465,"reported_epoch":458,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:44.491907+0000","last_change":"2023-11-08T05:29:44.395350+0000","last_active":"2023-11-08T05:30:44.491907+0000","last_peered":"2023-11-08T05:30:44.491907+0000","last_clean":"2023-11-08T05:30:44.491907+0000","last_became_active":"2023-11-08T05:29:44.395012+0000","last_became_peered":"2023-11-08T05:29:44.395012+0000","last_unstale":"2023-11-08T05:30:44.491907+0000","last_undegraded":"2023-11-08T05:30:44.491907+0000","last_fullsized":"2023-11-08T05:30:44.491907+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490629+0000","last_change":"2023-11-08T05:28:13.563357+0000","last_active":"2023-11-08T05:30:44.490629+0000","last_peered":"2023-11-08T05:30:44.490629+0000","last_clean":"2023-11-08T05:28:01.457691+0000","last_became_active":"2023-11-08T05:28:13.563357+0000","last_became_peered":"2023-11-08T05:28:13.563357+0000","last_unstale":"2023-11-08T05:30:44.490629+0000","last_undegraded":"2023-11-08T05:30:44.490629+0000","last_fullsized":"2023-11-08T05:28:13.541614+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490691+0000","last_change":"2023-11-08T05:28:13.550672+0000","last_active":"2023-11-08T05:30:44.490691+0000","last_peered":"2023-11-08T05:30:44.490691+0000","last_clean":"2023-11-08T05:28:01.457904+0000","last_became_active":"2023-11-08T05:28:13.550672+0000","last_became_peered":"2023-11-08T05:28:13.550672+0000","last_unstale":"2023-11-08T05:30:44.490691+0000","last_undegraded":"2023-11-08T05:30:44.490691+0000","last_fullsized":"2023-11-08T05:28:13.539595+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490805+0000","last_change":"2023-11-08T05:28:13.557968+0000","last_active":"2023-11-08T05:30:44.490805+0000","last_peered":"2023-11-08T05:30:44.490805+0000","last_clean":"2023-11-08T05:28:01.458433+0000","last_became_active":"2023-11-08T05:28:13.557968+0000","last_became_peered":"2023-11-08T05:28:13.557968+0000","last_unstale":"2023-11-08T05:30:44.490805+0000","last_undegraded":"2023-11-08T05:30:44.490805+0000","last_fullsized":"2023-11-08T05:28:13.542832+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":1,"acting":1,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":12,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738849,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":645040,"kb_used_data":5352,"kb_used_omap":275,"kb_used_meta":639404,"kb_avail":93726800,"statfs":{"total":96636764160,"available":95976243200,"internally_reserved":0,"allocated":5480448,"data_stored":2697647,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":281650,"internal_metadata":654750670},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:59 2023","interfaces":[{"interface":"back","average":{"1min":0.30499999999999999,"5min":0.439,"15min":0.36099999999999999},"min":{"1min":0.17399999999999999,"5min":0.14499999999999999,"15min":0.14499999999999999},"max":{"1min":0.60099999999999998,"5min":4.9379999999999997,"15min":4.9379999999999997},"last":0.379},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.49399999999999999,"15min":0.42399999999999999},"min":{"1min":0.221,"5min":0.19600000000000001,"15min":0.17000000000000001},"max":{"1min":0.63200000000000001,"5min":4.9859999999999998,"15min":4.9859999999999998},"last":0.40899999999999997}]},{"osd":1,"last update":"Wed Nov 8 05:32:06 2023","interfaces":[{"interface":"back","average":{"1min":0.27400000000000002,"5min":0.307,"15min":0.46800000000000003},"min":{"1min":0.17599999999999999,"5min":0.16,"15min":0.153},"max":{"1min":0.39000000000000001,"5min":1.825,"15min":23.242999999999999},"last":0.29899999999999999},{"interface":"front","average":{"1min":0.28000000000000003,"5min":0.34799999999999998,"15min":0.45700000000000002},"min":{"1min":0.20200000000000001,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":0.41699999999999998,"5min":2.0419999999999998,"15min":23.251000000000001},"last":0.32100000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738798,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1339228,"kb_used_data":398304,"kb_used_omap":291,"kb_used_meta":936476,"kb_avail":93032612,"statfs":{"total":96636764160,"available":95265394688,"internally_reserved":0,"allocated":407863296,"data_stored":405103900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":298462,"internal_metadata":958951970},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.436,"15min":0.36299999999999999},"min":{"1min":0.19700000000000001,"5min":0.14699999999999999,"15min":0.13700000000000001},"max":{"1min":3.222,"5min":6.5499999999999998,"15min":6.9989999999999997},"last":0.245},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.46300000000000002,"15min":0.42299999999999999},"min":{"1min":0.17799999999999999,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":5.0030000000000001,"5min":6.0629999999999997,"15min":9.6869999999999994},"last":0.28599999999999998}]},{"osd":2,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.51600000000000001,"15min":0.47099999999999997},"min":{"1min":0.21099999999999999,"5min":0.153,"15min":0.153},"max":{"1min":5.7439999999999998,"5min":5.8280000000000003,"15min":9.7639999999999993},"last":0.252},{"interface":"front","average":{"1min":0.497,"5min":0.56200000000000006,"15min":0.42299999999999999},"min":{"1min":0.20399999999999999,"5min":0.187,"15min":0.16800000000000001},"max":{"1min":3.9319999999999999,"5min":8.8040000000000003,"15min":8.8040000000000003},"last":0.29899999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738851,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":650100,"kb_used_data":5356,"kb_used_omap":214,"kb_used_meta":644521,"kb_avail":93721740,"statfs":{"total":96636764160,"available":95971061760,"internally_reserved":0,"allocated":5484544,"data_stored":2698616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":219335,"internal_metadata":659990329},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:56 2023","interfaces":[{"interface":"back","average":{"1min":0.30399999999999999,"5min":0.34699999999999998,"15min":0.36499999999999999},"min":{"1min":0.23999999999999999,"5min":0.14199999999999999,"15min":0.14199999999999999},"max":{"1min":0.42399999999999999,"5min":1.595,"15min":7.4249999999999998},"last":0.17100000000000001},{"interface":"front","average":{"1min":0.35499999999999998,"5min":0.40899999999999997,"15min":0.376},"min":{"1min":0.23999999999999999,"5min":0.19600000000000001,"15min":0.183},"max":{"1min":0.46999999999999997,"5min":2.6859999999999999,"15min":2.6859999999999999},"last":0.23000000000000001}]},{"osd":2,"last update":"Wed Nov 8 05:32:02 2023","interfaces":[{"interface":"back","average":{"1min":0.252,"5min":0.33000000000000002,"15min":0.36499999999999999},"min":{"1min":0.161,"5min":0.161,"15min":0.161},"max":{"1min":0.39600000000000002,"5min":2.2650000000000001,"15min":5.8209999999999997},"last":0.214},{"interface":"front","average":{"1min":0.25800000000000001,"5min":0.29899999999999999,"15min":0.35599999999999998},"min":{"1min":0.17000000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.35799999999999998,"5min":1.137,"15min":4.2350000000000003},"last":0.247}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-08T05:32:37.301 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-11-08T05:32:37.301 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 459, 'reported_epoch': 458, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:30:44.490644+0000', 'last_change': '2023-11-08T05:28:13.557071+0000', 'last_active': '2023-11-08T05:30:44.490644+0000', 'last_peered': '2023-11-08T05:30:44.490644+0000', 'last_clean': '2023-11-08T05:28:01.458439+0000', 'last_became_active': '2023-11-08T05:28:13.557071+0000', 'last_became_peered': '2023-11-08T05:28:13.557071+0000', 'last_unstale': '2023-11-08T05:30:44.490644+0000', 'last_undegraded': '2023-11-08T05:30:44.490644+0000', 'last_fullsized': '2023-11-08T05:28:13.547025+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:32:37.302 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-11-08T05:32:37.302 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 11, 'reported_epoch': 459, 'state': 'active+clean+remapped', 'last_fresh': '2023-11-08T05:30:45.794760+0000', 'last_change': '2023-11-08T05:30:45.794760+0000', 'last_active': '2023-11-08T05:30:45.794760+0000', 'last_peered': '2023-11-08T05:30:45.794760+0000', 'last_clean': '2023-11-08T05:30:45.794760+0000', 'last_became_active': '2023-11-08T05:30:45.794398+0000', 'last_became_peered': '2023-11-08T05:30:45.794398+0000', 'last_unstale': '2023-11-08T05:30:45.794760+0000', 'last_undegraded': '2023-11-08T05:30:45.794760+0000', 'last_fullsized': '2023-11-08T05:30:45.794760+0000', 'mapping_epoch': 458, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 459, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2], 'acting': [2, 1], 'avail_no_missing': ['2', '1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2023-11-08T05:32:37.302 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.1 is not active+clean 2023-11-08T05:32:37.303 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.1', 'version': "0'0", 'reported_seq': 465, 'reported_epoch': 459, 'state': 'active+clean+remapped', 'last_fresh': '2023-11-08T05:30:45.795085+0000', 'last_change': '2023-11-08T05:30:45.795085+0000', 'last_active': '2023-11-08T05:30:45.795085+0000', 'last_peered': '2023-11-08T05:30:45.795085+0000', 'last_clean': '2023-11-08T05:30:45.795085+0000', 'last_became_active': '2023-11-08T05:30:45.794743+0000', 'last_became_peered': '2023-11-08T05:30:45.794743+0000', 'last_unstale': '2023-11-08T05:30:45.795085+0000', 'last_undegraded': '2023-11-08T05:30:45.795085+0000', 'last_fullsized': '2023-11-08T05:30:45.795085+0000', 'mapping_epoch': 458, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 459, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2], 'acting': [2, 1], 'avail_no_missing': ['2', '1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2023-11-08T05:32:37.303 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-11-08T05:32:37.303 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "0'0", 'reported_seq': 460, 'reported_epoch': 458, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:30:44.490523+0000', 'last_change': '2023-11-08T05:28:13.545468+0000', 'last_active': '2023-11-08T05:30:44.490523+0000', 'last_peered': '2023-11-08T05:30:44.490523+0000', 'last_clean': '2023-11-08T05:28:01.456723+0000', 'last_became_active': '2023-11-08T05:28:13.545468+0000', 'last_became_peered': '2023-11-08T05:28:13.545468+0000', 'last_unstale': '2023-11-08T05:30:44.490523+0000', 'last_undegraded': '2023-11-08T05:30:44.490523+0000', 'last_fullsized': '2023-11-08T05:28:13.537207+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 9, 'last_epoch_clean': 10, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:45.528796+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:45.528796+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:45.528796+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:32:37.304 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.0 is not active+clean 2023-11-08T05:32:37.304 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.0', 'version': "0'0", 'reported_seq': 465, 'reported_epoch': 458, 'state': 'active+clean+remapped', 'last_fresh': '2023-11-08T05:30:44.491907+0000', 'last_change': '2023-11-08T05:29:44.395350+0000', 'last_active': '2023-11-08T05:30:44.491907+0000', 'last_peered': '2023-11-08T05:30:44.491907+0000', 'last_clean': '2023-11-08T05:30:44.491907+0000', 'last_became_active': '2023-11-08T05:29:44.395012+0000', 'last_became_peered': '2023-11-08T05:29:44.395012+0000', 'last_unstale': '2023-11-08T05:30:44.491907+0000', 'last_undegraded': '2023-11-08T05:30:44.491907+0000', 'last_fullsized': '2023-11-08T05:30:44.491907+0000', 'mapping_epoch': 455, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 456, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2], 'acting': [2, 1], 'avail_no_missing': ['2', '1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2023-11-08T05:32:37.304 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-11-08T05:32:37.304 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 459, 'reported_epoch': 458, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:30:44.490629+0000', 'last_change': '2023-11-08T05:28:13.563357+0000', 'last_active': '2023-11-08T05:30:44.490629+0000', 'last_peered': '2023-11-08T05:30:44.490629+0000', 'last_clean': '2023-11-08T05:28:01.457691+0000', 'last_became_active': '2023-11-08T05:28:13.563357+0000', 'last_became_peered': '2023-11-08T05:28:13.563357+0000', 'last_unstale': '2023-11-08T05:30:44.490629+0000', 'last_undegraded': '2023-11-08T05:30:44.490629+0000', 'last_fullsized': '2023-11-08T05:28:13.541614+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:32:37.305 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-11-08T05:32:37.305 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 459, 'reported_epoch': 458, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:30:44.490691+0000', 'last_change': '2023-11-08T05:28:13.550672+0000', 'last_active': '2023-11-08T05:30:44.490691+0000', 'last_peered': '2023-11-08T05:30:44.490691+0000', 'last_clean': '2023-11-08T05:28:01.457904+0000', 'last_became_active': '2023-11-08T05:28:13.550672+0000', 'last_became_peered': '2023-11-08T05:28:13.550672+0000', 'last_unstale': '2023-11-08T05:30:44.490691+0000', 'last_undegraded': '2023-11-08T05:30:44.490691+0000', 'last_fullsized': '2023-11-08T05:28:13.539595+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:32:37.305 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-11-08T05:32:37.306 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 459, 'reported_epoch': 458, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:30:44.490805+0000', 'last_change': '2023-11-08T05:28:13.557968+0000', 'last_active': '2023-11-08T05:30:44.490805+0000', 'last_peered': '2023-11-08T05:30:44.490805+0000', 'last_clean': '2023-11-08T05:28:01.458433+0000', 'last_became_active': '2023-11-08T05:28:13.557968+0000', 'last_became_peered': '2023-11-08T05:28:13.557968+0000', 'last_unstale': '2023-11-08T05:30:44.490805+0000', 'last_undegraded': '2023-11-08T05:30:44.490805+0000', 'last_fullsized': '2023-11-08T05:28:13.542832+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:32:37.306 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-11-08T05:32:37.440 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~259s 2023-11-08T05:32:37.654 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T05:32:37.654 INFO:teuthology.orchestra.run.smithi087.stdout:{"fsid":"ab4cbc06-449d-497d-b513-220d9db2b590","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 5 pgs undersized","count":5},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2400,"monmap":{"epoch":1,"min_mon_release_name":"pacific","num_mons":1},"osdmap":{"epoch":459,"num_osds":3,"num_up_osds":2,"osd_up_since":1699421292,"num_in_osds":3,"osd_in_since":1699419158,"num_remapped_pgs":3},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean+remapped","count":3},{"state_name":"active+clean","count":1}],"num_pgs":9,"num_pools":2,"num_objects":0,"data_bytes":0,"bytes_used":2697592832,"bytes_avail":287212699648,"bytes_total":289910292480},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":41,"modified":"2023-11-08T05:29:32.044544+0000","services":{}},"progress_events":{}} 2023-11-08T05:32:37.667 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-11-08T05:32:37.668 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-11-08T05:32:37.954 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T05:32:37.955 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2023-11-08T05:32:37.966 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":1652,"stamp":"2023-11-08T05:32:36.082498+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":10,"acting":13,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":109,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2634368,"kb_used_data":409012,"kb_used_omap":780,"kb_used_meta":2220403,"kb_avail":280481152,"statfs":{"total":289910292480,"available":287212699648,"internally_reserved":0,"allocated":418828288,"data_stored":410500163,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":799447,"internal_metadata":2273692969},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-536782902,"num_objects":-164,"num_object_clones":0,"num_object_copies":-328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-115,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-164,"num_whiteouts":0,"num_read":-122056,"num_read_kb":-39011942,"num_write":-413642,"num_write_kb":-151705339,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-8,"num_bytes_recovered":-16922624,"num_keys_recovered":-2,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.424817"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490644+0000","last_change":"2023-11-08T05:28:13.557071+0000","last_active":"2023-11-08T05:30:44.490644+0000","last_peered":"2023-11-08T05:30:44.490644+0000","last_clean":"2023-11-08T05:28:01.458439+0000","last_became_active":"2023-11-08T05:28:13.557071+0000","last_became_peered":"2023-11-08T05:28:13.557071+0000","last_unstale":"2023-11-08T05:30:44.490644+0000","last_undegraded":"2023-11-08T05:30:44.490644+0000","last_fullsized":"2023-11-08T05:28:13.547025+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":459,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:45.794760+0000","last_change":"2023-11-08T05:30:45.794760+0000","last_active":"2023-11-08T05:30:45.794760+0000","last_peered":"2023-11-08T05:30:45.794760+0000","last_clean":"2023-11-08T05:30:45.794760+0000","last_became_active":"2023-11-08T05:30:45.794398+0000","last_became_peered":"2023-11-08T05:30:45.794398+0000","last_unstale":"2023-11-08T05:30:45.794760+0000","last_undegraded":"2023-11-08T05:30:45.794760+0000","last_fullsized":"2023-11-08T05:30:45.794760+0000","mapping_epoch":458,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":459,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":465,"reported_epoch":459,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:45.795085+0000","last_change":"2023-11-08T05:30:45.795085+0000","last_active":"2023-11-08T05:30:45.795085+0000","last_peered":"2023-11-08T05:30:45.795085+0000","last_clean":"2023-11-08T05:30:45.795085+0000","last_became_active":"2023-11-08T05:30:45.794743+0000","last_became_peered":"2023-11-08T05:30:45.794743+0000","last_unstale":"2023-11-08T05:30:45.795085+0000","last_undegraded":"2023-11-08T05:30:45.795085+0000","last_fullsized":"2023-11-08T05:30:45.795085+0000","mapping_epoch":458,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":459,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":454,"reported_epoch":458,"state":"active+clean","last_fresh":"2023-11-08T05:30:44.490618+0000","last_change":"2023-11-08T04:52:47.548597+0000","last_active":"2023-11-08T05:30:44.490618+0000","last_peered":"2023-11-08T05:30:44.490618+0000","last_clean":"2023-11-08T05:30:44.490618+0000","last_became_active":"2023-11-08T04:52:47.548294+0000","last_became_peered":"2023-11-08T04:52:47.548294+0000","last_unstale":"2023-11-08T05:30:44.490618+0000","last_undegraded":"2023-11-08T05:30:44.490618+0000","last_fullsized":"2023-11-08T05:30:44.490618+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":460,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490523+0000","last_change":"2023-11-08T05:28:13.545468+0000","last_active":"2023-11-08T05:30:44.490523+0000","last_peered":"2023-11-08T05:30:44.490523+0000","last_clean":"2023-11-08T05:28:01.456723+0000","last_became_active":"2023-11-08T05:28:13.545468+0000","last_became_peered":"2023-11-08T05:28:13.545468+0000","last_unstale":"2023-11-08T05:30:44.490523+0000","last_undegraded":"2023-11-08T05:30:44.490523+0000","last_fullsized":"2023-11-08T05:28:13.537207+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_clean_scrub_stamp":"2023-11-08T04:52:45.528796+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":465,"reported_epoch":458,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:44.491907+0000","last_change":"2023-11-08T05:29:44.395350+0000","last_active":"2023-11-08T05:30:44.491907+0000","last_peered":"2023-11-08T05:30:44.491907+0000","last_clean":"2023-11-08T05:30:44.491907+0000","last_became_active":"2023-11-08T05:29:44.395012+0000","last_became_peered":"2023-11-08T05:29:44.395012+0000","last_unstale":"2023-11-08T05:30:44.491907+0000","last_undegraded":"2023-11-08T05:30:44.491907+0000","last_fullsized":"2023-11-08T05:30:44.491907+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490629+0000","last_change":"2023-11-08T05:28:13.563357+0000","last_active":"2023-11-08T05:30:44.490629+0000","last_peered":"2023-11-08T05:30:44.490629+0000","last_clean":"2023-11-08T05:28:01.457691+0000","last_became_active":"2023-11-08T05:28:13.563357+0000","last_became_peered":"2023-11-08T05:28:13.563357+0000","last_unstale":"2023-11-08T05:30:44.490629+0000","last_undegraded":"2023-11-08T05:30:44.490629+0000","last_fullsized":"2023-11-08T05:28:13.541614+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490691+0000","last_change":"2023-11-08T05:28:13.550672+0000","last_active":"2023-11-08T05:30:44.490691+0000","last_peered":"2023-11-08T05:30:44.490691+0000","last_clean":"2023-11-08T05:28:01.457904+0000","last_became_active":"2023-11-08T05:28:13.550672+0000","last_became_peered":"2023-11-08T05:28:13.550672+0000","last_unstale":"2023-11-08T05:30:44.490691+0000","last_undegraded":"2023-11-08T05:30:44.490691+0000","last_fullsized":"2023-11-08T05:28:13.539595+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490805+0000","last_change":"2023-11-08T05:28:13.557968+0000","last_active":"2023-11-08T05:30:44.490805+0000","last_peered":"2023-11-08T05:30:44.490805+0000","last_clean":"2023-11-08T05:28:01.458433+0000","last_became_active":"2023-11-08T05:28:13.557968+0000","last_became_peered":"2023-11-08T05:28:13.557968+0000","last_unstale":"2023-11-08T05:30:44.490805+0000","last_undegraded":"2023-11-08T05:30:44.490805+0000","last_fullsized":"2023-11-08T05:28:13.542832+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":1,"acting":1,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":12,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738849,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":645040,"kb_used_data":5352,"kb_used_omap":275,"kb_used_meta":639404,"kb_avail":93726800,"statfs":{"total":96636764160,"available":95976243200,"internally_reserved":0,"allocated":5480448,"data_stored":2697647,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":281650,"internal_metadata":654750670},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:59 2023","interfaces":[{"interface":"back","average":{"1min":0.30499999999999999,"5min":0.439,"15min":0.36099999999999999},"min":{"1min":0.17399999999999999,"5min":0.14499999999999999,"15min":0.14499999999999999},"max":{"1min":0.60099999999999998,"5min":4.9379999999999997,"15min":4.9379999999999997},"last":0.379},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.49399999999999999,"15min":0.42399999999999999},"min":{"1min":0.221,"5min":0.19600000000000001,"15min":0.17000000000000001},"max":{"1min":0.63200000000000001,"5min":4.9859999999999998,"15min":4.9859999999999998},"last":0.40899999999999997}]},{"osd":1,"last update":"Wed Nov 8 05:32:06 2023","interfaces":[{"interface":"back","average":{"1min":0.27400000000000002,"5min":0.307,"15min":0.46800000000000003},"min":{"1min":0.17599999999999999,"5min":0.16,"15min":0.153},"max":{"1min":0.39000000000000001,"5min":1.825,"15min":23.242999999999999},"last":0.29899999999999999},{"interface":"front","average":{"1min":0.28000000000000003,"5min":0.34799999999999998,"15min":0.45700000000000002},"min":{"1min":0.20200000000000001,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":0.41699999999999998,"5min":2.0419999999999998,"15min":23.251000000000001},"last":0.32100000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738798,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1339228,"kb_used_data":398304,"kb_used_omap":291,"kb_used_meta":936476,"kb_avail":93032612,"statfs":{"total":96636764160,"available":95265394688,"internally_reserved":0,"allocated":407863296,"data_stored":405103900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":298462,"internal_metadata":958951970},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.436,"15min":0.36299999999999999},"min":{"1min":0.19700000000000001,"5min":0.14699999999999999,"15min":0.13700000000000001},"max":{"1min":3.222,"5min":6.5499999999999998,"15min":6.9989999999999997},"last":0.245},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.46300000000000002,"15min":0.42299999999999999},"min":{"1min":0.17799999999999999,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":5.0030000000000001,"5min":6.0629999999999997,"15min":9.6869999999999994},"last":0.28599999999999998}]},{"osd":2,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.51600000000000001,"15min":0.47099999999999997},"min":{"1min":0.21099999999999999,"5min":0.153,"15min":0.153},"max":{"1min":5.7439999999999998,"5min":5.8280000000000003,"15min":9.7639999999999993},"last":0.252},{"interface":"front","average":{"1min":0.497,"5min":0.56200000000000006,"15min":0.42299999999999999},"min":{"1min":0.20399999999999999,"5min":0.187,"15min":0.16800000000000001},"max":{"1min":3.9319999999999999,"5min":8.8040000000000003,"15min":8.8040000000000003},"last":0.29899999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738851,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":650100,"kb_used_data":5356,"kb_used_omap":214,"kb_used_meta":644521,"kb_avail":93721740,"statfs":{"total":96636764160,"available":95971061760,"internally_reserved":0,"allocated":5484544,"data_stored":2698616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":219335,"internal_metadata":659990329},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:56 2023","interfaces":[{"interface":"back","average":{"1min":0.30399999999999999,"5min":0.34699999999999998,"15min":0.36499999999999999},"min":{"1min":0.23999999999999999,"5min":0.14199999999999999,"15min":0.14199999999999999},"max":{"1min":0.42399999999999999,"5min":1.595,"15min":7.4249999999999998},"last":0.17100000000000001},{"interface":"front","average":{"1min":0.35499999999999998,"5min":0.40899999999999997,"15min":0.376},"min":{"1min":0.23999999999999999,"5min":0.19600000000000001,"15min":0.183},"max":{"1min":0.46999999999999997,"5min":2.6859999999999999,"15min":2.6859999999999999},"last":0.23000000000000001}]},{"osd":2,"last update":"Wed Nov 8 05:32:02 2023","interfaces":[{"interface":"back","average":{"1min":0.252,"5min":0.33000000000000002,"15min":0.36499999999999999},"min":{"1min":0.161,"5min":0.161,"15min":0.161},"max":{"1min":0.39600000000000002,"5min":2.2650000000000001,"15min":5.8209999999999997},"last":0.214},{"interface":"front","average":{"1min":0.25800000000000001,"5min":0.29899999999999999,"15min":0.35599999999999998},"min":{"1min":0.17000000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.35799999999999998,"5min":1.137,"15min":4.2350000000000003},"last":0.247}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-08T05:32:40.969 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-11-08T05:32:41.265 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T05:32:41.267 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2023-11-08T05:32:41.278 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":1654,"stamp":"2023-11-08T05:32:40.083348+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":10,"acting":13,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":109,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2634368,"kb_used_data":409012,"kb_used_omap":780,"kb_used_meta":2220403,"kb_avail":280481152,"statfs":{"total":289910292480,"available":287212699648,"internally_reserved":0,"allocated":418828288,"data_stored":410500163,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":799447,"internal_metadata":2273692969},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-536782902,"num_objects":-164,"num_object_clones":0,"num_object_copies":-328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-115,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-164,"num_whiteouts":0,"num_read":-122056,"num_read_kb":-39011942,"num_write":-413642,"num_write_kb":-151705339,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-8,"num_bytes_recovered":-16922624,"num_keys_recovered":-2,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.424817"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490644+0000","last_change":"2023-11-08T05:28:13.557071+0000","last_active":"2023-11-08T05:30:44.490644+0000","last_peered":"2023-11-08T05:30:44.490644+0000","last_clean":"2023-11-08T05:28:01.458439+0000","last_became_active":"2023-11-08T05:28:13.557071+0000","last_became_peered":"2023-11-08T05:28:13.557071+0000","last_unstale":"2023-11-08T05:30:44.490644+0000","last_undegraded":"2023-11-08T05:30:44.490644+0000","last_fullsized":"2023-11-08T05:28:13.547025+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":459,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:45.794760+0000","last_change":"2023-11-08T05:30:45.794760+0000","last_active":"2023-11-08T05:30:45.794760+0000","last_peered":"2023-11-08T05:30:45.794760+0000","last_clean":"2023-11-08T05:30:45.794760+0000","last_became_active":"2023-11-08T05:30:45.794398+0000","last_became_peered":"2023-11-08T05:30:45.794398+0000","last_unstale":"2023-11-08T05:30:45.794760+0000","last_undegraded":"2023-11-08T05:30:45.794760+0000","last_fullsized":"2023-11-08T05:30:45.794760+0000","mapping_epoch":458,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":459,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":465,"reported_epoch":459,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:45.795085+0000","last_change":"2023-11-08T05:30:45.795085+0000","last_active":"2023-11-08T05:30:45.795085+0000","last_peered":"2023-11-08T05:30:45.795085+0000","last_clean":"2023-11-08T05:30:45.795085+0000","last_became_active":"2023-11-08T05:30:45.794743+0000","last_became_peered":"2023-11-08T05:30:45.794743+0000","last_unstale":"2023-11-08T05:30:45.795085+0000","last_undegraded":"2023-11-08T05:30:45.795085+0000","last_fullsized":"2023-11-08T05:30:45.795085+0000","mapping_epoch":458,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":459,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":454,"reported_epoch":458,"state":"active+clean","last_fresh":"2023-11-08T05:30:44.490618+0000","last_change":"2023-11-08T04:52:47.548597+0000","last_active":"2023-11-08T05:30:44.490618+0000","last_peered":"2023-11-08T05:30:44.490618+0000","last_clean":"2023-11-08T05:30:44.490618+0000","last_became_active":"2023-11-08T04:52:47.548294+0000","last_became_peered":"2023-11-08T04:52:47.548294+0000","last_unstale":"2023-11-08T05:30:44.490618+0000","last_undegraded":"2023-11-08T05:30:44.490618+0000","last_fullsized":"2023-11-08T05:30:44.490618+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":460,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490523+0000","last_change":"2023-11-08T05:28:13.545468+0000","last_active":"2023-11-08T05:30:44.490523+0000","last_peered":"2023-11-08T05:30:44.490523+0000","last_clean":"2023-11-08T05:28:01.456723+0000","last_became_active":"2023-11-08T05:28:13.545468+0000","last_became_peered":"2023-11-08T05:28:13.545468+0000","last_unstale":"2023-11-08T05:30:44.490523+0000","last_undegraded":"2023-11-08T05:30:44.490523+0000","last_fullsized":"2023-11-08T05:28:13.537207+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_clean_scrub_stamp":"2023-11-08T04:52:45.528796+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":465,"reported_epoch":458,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:44.491907+0000","last_change":"2023-11-08T05:29:44.395350+0000","last_active":"2023-11-08T05:30:44.491907+0000","last_peered":"2023-11-08T05:30:44.491907+0000","last_clean":"2023-11-08T05:30:44.491907+0000","last_became_active":"2023-11-08T05:29:44.395012+0000","last_became_peered":"2023-11-08T05:29:44.395012+0000","last_unstale":"2023-11-08T05:30:44.491907+0000","last_undegraded":"2023-11-08T05:30:44.491907+0000","last_fullsized":"2023-11-08T05:30:44.491907+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490629+0000","last_change":"2023-11-08T05:28:13.563357+0000","last_active":"2023-11-08T05:30:44.490629+0000","last_peered":"2023-11-08T05:30:44.490629+0000","last_clean":"2023-11-08T05:28:01.457691+0000","last_became_active":"2023-11-08T05:28:13.563357+0000","last_became_peered":"2023-11-08T05:28:13.563357+0000","last_unstale":"2023-11-08T05:30:44.490629+0000","last_undegraded":"2023-11-08T05:30:44.490629+0000","last_fullsized":"2023-11-08T05:28:13.541614+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490691+0000","last_change":"2023-11-08T05:28:13.550672+0000","last_active":"2023-11-08T05:30:44.490691+0000","last_peered":"2023-11-08T05:30:44.490691+0000","last_clean":"2023-11-08T05:28:01.457904+0000","last_became_active":"2023-11-08T05:28:13.550672+0000","last_became_peered":"2023-11-08T05:28:13.550672+0000","last_unstale":"2023-11-08T05:30:44.490691+0000","last_undegraded":"2023-11-08T05:30:44.490691+0000","last_fullsized":"2023-11-08T05:28:13.539595+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490805+0000","last_change":"2023-11-08T05:28:13.557968+0000","last_active":"2023-11-08T05:30:44.490805+0000","last_peered":"2023-11-08T05:30:44.490805+0000","last_clean":"2023-11-08T05:28:01.458433+0000","last_became_active":"2023-11-08T05:28:13.557968+0000","last_became_peered":"2023-11-08T05:28:13.557968+0000","last_unstale":"2023-11-08T05:30:44.490805+0000","last_undegraded":"2023-11-08T05:30:44.490805+0000","last_fullsized":"2023-11-08T05:28:13.542832+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":1,"acting":1,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":12,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738850,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":645040,"kb_used_data":5352,"kb_used_omap":275,"kb_used_meta":639404,"kb_avail":93726800,"statfs":{"total":96636764160,"available":95976243200,"internally_reserved":0,"allocated":5480448,"data_stored":2697647,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":281650,"internal_metadata":654750670},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:59 2023","interfaces":[{"interface":"back","average":{"1min":0.30499999999999999,"5min":0.439,"15min":0.36099999999999999},"min":{"1min":0.17399999999999999,"5min":0.14499999999999999,"15min":0.14499999999999999},"max":{"1min":0.60099999999999998,"5min":4.9379999999999997,"15min":4.9379999999999997},"last":0.379},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.49399999999999999,"15min":0.42399999999999999},"min":{"1min":0.221,"5min":0.19600000000000001,"15min":0.17000000000000001},"max":{"1min":0.63200000000000001,"5min":4.9859999999999998,"15min":4.9859999999999998},"last":0.40899999999999997}]},{"osd":1,"last update":"Wed Nov 8 05:32:06 2023","interfaces":[{"interface":"back","average":{"1min":0.27400000000000002,"5min":0.307,"15min":0.46800000000000003},"min":{"1min":0.17599999999999999,"5min":0.16,"15min":0.153},"max":{"1min":0.39000000000000001,"5min":1.825,"15min":23.242999999999999},"last":0.34899999999999998},{"interface":"front","average":{"1min":0.28000000000000003,"5min":0.34799999999999998,"15min":0.45700000000000002},"min":{"1min":0.20200000000000001,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":0.41699999999999998,"5min":2.0419999999999998,"15min":23.251000000000001},"last":0.35899999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738798,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1339228,"kb_used_data":398304,"kb_used_omap":291,"kb_used_meta":936476,"kb_avail":93032612,"statfs":{"total":96636764160,"available":95265394688,"internally_reserved":0,"allocated":407863296,"data_stored":405103900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":298462,"internal_metadata":958951970},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.436,"15min":0.36299999999999999},"min":{"1min":0.19700000000000001,"5min":0.14699999999999999,"15min":0.13700000000000001},"max":{"1min":3.222,"5min":6.5499999999999998,"15min":6.9989999999999997},"last":0.245},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.46300000000000002,"15min":0.42299999999999999},"min":{"1min":0.17799999999999999,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":5.0030000000000001,"5min":6.0629999999999997,"15min":9.6869999999999994},"last":0.28599999999999998}]},{"osd":2,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.51600000000000001,"15min":0.47099999999999997},"min":{"1min":0.21099999999999999,"5min":0.153,"15min":0.153},"max":{"1min":5.7439999999999998,"5min":5.8280000000000003,"15min":9.7639999999999993},"last":0.252},{"interface":"front","average":{"1min":0.497,"5min":0.56200000000000006,"15min":0.42299999999999999},"min":{"1min":0.20399999999999999,"5min":0.187,"15min":0.16800000000000001},"max":{"1min":3.9319999999999999,"5min":8.8040000000000003,"15min":8.8040000000000003},"last":0.29899999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738851,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":650100,"kb_used_data":5356,"kb_used_omap":214,"kb_used_meta":644521,"kb_avail":93721740,"statfs":{"total":96636764160,"available":95971061760,"internally_reserved":0,"allocated":5484544,"data_stored":2698616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":219335,"internal_metadata":659990329},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:56 2023","interfaces":[{"interface":"back","average":{"1min":0.30399999999999999,"5min":0.34699999999999998,"15min":0.36499999999999999},"min":{"1min":0.23999999999999999,"5min":0.14199999999999999,"15min":0.14199999999999999},"max":{"1min":0.42399999999999999,"5min":1.595,"15min":7.4249999999999998},"last":0.17100000000000001},{"interface":"front","average":{"1min":0.35499999999999998,"5min":0.40899999999999997,"15min":0.376},"min":{"1min":0.23999999999999999,"5min":0.19600000000000001,"15min":0.183},"max":{"1min":0.46999999999999997,"5min":2.6859999999999999,"15min":2.6859999999999999},"last":0.23000000000000001}]},{"osd":2,"last update":"Wed Nov 8 05:32:02 2023","interfaces":[{"interface":"back","average":{"1min":0.252,"5min":0.33000000000000002,"15min":0.36499999999999999},"min":{"1min":0.161,"5min":0.161,"15min":0.161},"max":{"1min":0.39600000000000002,"5min":2.2650000000000001,"15min":5.8209999999999997},"last":0.214},{"interface":"front","average":{"1min":0.25800000000000001,"5min":0.29899999999999999,"15min":0.35599999999999998},"min":{"1min":0.17000000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.35799999999999998,"5min":1.137,"15min":4.2350000000000003},"last":0.247}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-08T05:32:41.280 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-11-08T05:32:41.562 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T05:32:41.563 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2023-11-08T05:32:41.574 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":1654,"stamp":"2023-11-08T05:32:40.083348+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":10,"acting":13,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":109,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2634368,"kb_used_data":409012,"kb_used_omap":780,"kb_used_meta":2220403,"kb_avail":280481152,"statfs":{"total":289910292480,"available":287212699648,"internally_reserved":0,"allocated":418828288,"data_stored":410500163,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":799447,"internal_metadata":2273692969},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-536782902,"num_objects":-164,"num_object_clones":0,"num_object_copies":-328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-115,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-164,"num_whiteouts":0,"num_read":-122056,"num_read_kb":-39011942,"num_write":-413642,"num_write_kb":-151705339,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-8,"num_bytes_recovered":-16922624,"num_keys_recovered":-2,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.424817"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490644+0000","last_change":"2023-11-08T05:28:13.557071+0000","last_active":"2023-11-08T05:30:44.490644+0000","last_peered":"2023-11-08T05:30:44.490644+0000","last_clean":"2023-11-08T05:28:01.458439+0000","last_became_active":"2023-11-08T05:28:13.557071+0000","last_became_peered":"2023-11-08T05:28:13.557071+0000","last_unstale":"2023-11-08T05:30:44.490644+0000","last_undegraded":"2023-11-08T05:30:44.490644+0000","last_fullsized":"2023-11-08T05:28:13.547025+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":459,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:45.794760+0000","last_change":"2023-11-08T05:30:45.794760+0000","last_active":"2023-11-08T05:30:45.794760+0000","last_peered":"2023-11-08T05:30:45.794760+0000","last_clean":"2023-11-08T05:30:45.794760+0000","last_became_active":"2023-11-08T05:30:45.794398+0000","last_became_peered":"2023-11-08T05:30:45.794398+0000","last_unstale":"2023-11-08T05:30:45.794760+0000","last_undegraded":"2023-11-08T05:30:45.794760+0000","last_fullsized":"2023-11-08T05:30:45.794760+0000","mapping_epoch":458,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":459,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":465,"reported_epoch":459,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:45.795085+0000","last_change":"2023-11-08T05:30:45.795085+0000","last_active":"2023-11-08T05:30:45.795085+0000","last_peered":"2023-11-08T05:30:45.795085+0000","last_clean":"2023-11-08T05:30:45.795085+0000","last_became_active":"2023-11-08T05:30:45.794743+0000","last_became_peered":"2023-11-08T05:30:45.794743+0000","last_unstale":"2023-11-08T05:30:45.795085+0000","last_undegraded":"2023-11-08T05:30:45.795085+0000","last_fullsized":"2023-11-08T05:30:45.795085+0000","mapping_epoch":458,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":459,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":454,"reported_epoch":458,"state":"active+clean","last_fresh":"2023-11-08T05:30:44.490618+0000","last_change":"2023-11-08T04:52:47.548597+0000","last_active":"2023-11-08T05:30:44.490618+0000","last_peered":"2023-11-08T05:30:44.490618+0000","last_clean":"2023-11-08T05:30:44.490618+0000","last_became_active":"2023-11-08T04:52:47.548294+0000","last_became_peered":"2023-11-08T04:52:47.548294+0000","last_unstale":"2023-11-08T05:30:44.490618+0000","last_undegraded":"2023-11-08T05:30:44.490618+0000","last_fullsized":"2023-11-08T05:30:44.490618+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":460,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490523+0000","last_change":"2023-11-08T05:28:13.545468+0000","last_active":"2023-11-08T05:30:44.490523+0000","last_peered":"2023-11-08T05:30:44.490523+0000","last_clean":"2023-11-08T05:28:01.456723+0000","last_became_active":"2023-11-08T05:28:13.545468+0000","last_became_peered":"2023-11-08T05:28:13.545468+0000","last_unstale":"2023-11-08T05:30:44.490523+0000","last_undegraded":"2023-11-08T05:30:44.490523+0000","last_fullsized":"2023-11-08T05:28:13.537207+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_clean_scrub_stamp":"2023-11-08T04:52:45.528796+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":465,"reported_epoch":458,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:44.491907+0000","last_change":"2023-11-08T05:29:44.395350+0000","last_active":"2023-11-08T05:30:44.491907+0000","last_peered":"2023-11-08T05:30:44.491907+0000","last_clean":"2023-11-08T05:30:44.491907+0000","last_became_active":"2023-11-08T05:29:44.395012+0000","last_became_peered":"2023-11-08T05:29:44.395012+0000","last_unstale":"2023-11-08T05:30:44.491907+0000","last_undegraded":"2023-11-08T05:30:44.491907+0000","last_fullsized":"2023-11-08T05:30:44.491907+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490629+0000","last_change":"2023-11-08T05:28:13.563357+0000","last_active":"2023-11-08T05:30:44.490629+0000","last_peered":"2023-11-08T05:30:44.490629+0000","last_clean":"2023-11-08T05:28:01.457691+0000","last_became_active":"2023-11-08T05:28:13.563357+0000","last_became_peered":"2023-11-08T05:28:13.563357+0000","last_unstale":"2023-11-08T05:30:44.490629+0000","last_undegraded":"2023-11-08T05:30:44.490629+0000","last_fullsized":"2023-11-08T05:28:13.541614+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490691+0000","last_change":"2023-11-08T05:28:13.550672+0000","last_active":"2023-11-08T05:30:44.490691+0000","last_peered":"2023-11-08T05:30:44.490691+0000","last_clean":"2023-11-08T05:28:01.457904+0000","last_became_active":"2023-11-08T05:28:13.550672+0000","last_became_peered":"2023-11-08T05:28:13.550672+0000","last_unstale":"2023-11-08T05:30:44.490691+0000","last_undegraded":"2023-11-08T05:30:44.490691+0000","last_fullsized":"2023-11-08T05:28:13.539595+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490805+0000","last_change":"2023-11-08T05:28:13.557968+0000","last_active":"2023-11-08T05:30:44.490805+0000","last_peered":"2023-11-08T05:30:44.490805+0000","last_clean":"2023-11-08T05:28:01.458433+0000","last_became_active":"2023-11-08T05:28:13.557968+0000","last_became_peered":"2023-11-08T05:28:13.557968+0000","last_unstale":"2023-11-08T05:30:44.490805+0000","last_undegraded":"2023-11-08T05:30:44.490805+0000","last_fullsized":"2023-11-08T05:28:13.542832+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":1,"acting":1,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":12,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738850,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":645040,"kb_used_data":5352,"kb_used_omap":275,"kb_used_meta":639404,"kb_avail":93726800,"statfs":{"total":96636764160,"available":95976243200,"internally_reserved":0,"allocated":5480448,"data_stored":2697647,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":281650,"internal_metadata":654750670},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:59 2023","interfaces":[{"interface":"back","average":{"1min":0.30499999999999999,"5min":0.439,"15min":0.36099999999999999},"min":{"1min":0.17399999999999999,"5min":0.14499999999999999,"15min":0.14499999999999999},"max":{"1min":0.60099999999999998,"5min":4.9379999999999997,"15min":4.9379999999999997},"last":0.379},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.49399999999999999,"15min":0.42399999999999999},"min":{"1min":0.221,"5min":0.19600000000000001,"15min":0.17000000000000001},"max":{"1min":0.63200000000000001,"5min":4.9859999999999998,"15min":4.9859999999999998},"last":0.40899999999999997}]},{"osd":1,"last update":"Wed Nov 8 05:32:06 2023","interfaces":[{"interface":"back","average":{"1min":0.27400000000000002,"5min":0.307,"15min":0.46800000000000003},"min":{"1min":0.17599999999999999,"5min":0.16,"15min":0.153},"max":{"1min":0.39000000000000001,"5min":1.825,"15min":23.242999999999999},"last":0.34899999999999998},{"interface":"front","average":{"1min":0.28000000000000003,"5min":0.34799999999999998,"15min":0.45700000000000002},"min":{"1min":0.20200000000000001,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":0.41699999999999998,"5min":2.0419999999999998,"15min":23.251000000000001},"last":0.35899999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738798,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1339228,"kb_used_data":398304,"kb_used_omap":291,"kb_used_meta":936476,"kb_avail":93032612,"statfs":{"total":96636764160,"available":95265394688,"internally_reserved":0,"allocated":407863296,"data_stored":405103900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":298462,"internal_metadata":958951970},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.436,"15min":0.36299999999999999},"min":{"1min":0.19700000000000001,"5min":0.14699999999999999,"15min":0.13700000000000001},"max":{"1min":3.222,"5min":6.5499999999999998,"15min":6.9989999999999997},"last":0.245},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.46300000000000002,"15min":0.42299999999999999},"min":{"1min":0.17799999999999999,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":5.0030000000000001,"5min":6.0629999999999997,"15min":9.6869999999999994},"last":0.28599999999999998}]},{"osd":2,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.51600000000000001,"15min":0.47099999999999997},"min":{"1min":0.21099999999999999,"5min":0.153,"15min":0.153},"max":{"1min":5.7439999999999998,"5min":5.8280000000000003,"15min":9.7639999999999993},"last":0.252},{"interface":"front","average":{"1min":0.497,"5min":0.56200000000000006,"15min":0.42299999999999999},"min":{"1min":0.20399999999999999,"5min":0.187,"15min":0.16800000000000001},"max":{"1min":3.9319999999999999,"5min":8.8040000000000003,"15min":8.8040000000000003},"last":0.29899999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738851,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":650100,"kb_used_data":5356,"kb_used_omap":214,"kb_used_meta":644521,"kb_avail":93721740,"statfs":{"total":96636764160,"available":95971061760,"internally_reserved":0,"allocated":5484544,"data_stored":2698616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":219335,"internal_metadata":659990329},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:56 2023","interfaces":[{"interface":"back","average":{"1min":0.30399999999999999,"5min":0.34699999999999998,"15min":0.36499999999999999},"min":{"1min":0.23999999999999999,"5min":0.14199999999999999,"15min":0.14199999999999999},"max":{"1min":0.42399999999999999,"5min":1.595,"15min":7.4249999999999998},"last":0.17100000000000001},{"interface":"front","average":{"1min":0.35499999999999998,"5min":0.40899999999999997,"15min":0.376},"min":{"1min":0.23999999999999999,"5min":0.19600000000000001,"15min":0.183},"max":{"1min":0.46999999999999997,"5min":2.6859999999999999,"15min":2.6859999999999999},"last":0.23000000000000001}]},{"osd":2,"last update":"Wed Nov 8 05:32:02 2023","interfaces":[{"interface":"back","average":{"1min":0.252,"5min":0.33000000000000002,"15min":0.36499999999999999},"min":{"1min":0.161,"5min":0.161,"15min":0.161},"max":{"1min":0.39600000000000002,"5min":2.2650000000000001,"15min":5.8209999999999997},"last":0.214},{"interface":"front","average":{"1min":0.25800000000000001,"5min":0.29899999999999999,"15min":0.35599999999999998},"min":{"1min":0.17000000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.35799999999999998,"5min":1.137,"15min":4.2350000000000003},"last":0.247}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-08T05:32:41.576 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-11-08T05:32:41.576 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 459, 'reported_epoch': 458, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:30:44.490644+0000', 'last_change': '2023-11-08T05:28:13.557071+0000', 'last_active': '2023-11-08T05:30:44.490644+0000', 'last_peered': '2023-11-08T05:30:44.490644+0000', 'last_clean': '2023-11-08T05:28:01.458439+0000', 'last_became_active': '2023-11-08T05:28:13.557071+0000', 'last_became_peered': '2023-11-08T05:28:13.557071+0000', 'last_unstale': '2023-11-08T05:30:44.490644+0000', 'last_undegraded': '2023-11-08T05:30:44.490644+0000', 'last_fullsized': '2023-11-08T05:28:13.547025+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:32:41.576 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-11-08T05:32:41.576 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 11, 'reported_epoch': 459, 'state': 'active+clean+remapped', 'last_fresh': '2023-11-08T05:30:45.794760+0000', 'last_change': '2023-11-08T05:30:45.794760+0000', 'last_active': '2023-11-08T05:30:45.794760+0000', 'last_peered': '2023-11-08T05:30:45.794760+0000', 'last_clean': '2023-11-08T05:30:45.794760+0000', 'last_became_active': '2023-11-08T05:30:45.794398+0000', 'last_became_peered': '2023-11-08T05:30:45.794398+0000', 'last_unstale': '2023-11-08T05:30:45.794760+0000', 'last_undegraded': '2023-11-08T05:30:45.794760+0000', 'last_fullsized': '2023-11-08T05:30:45.794760+0000', 'mapping_epoch': 458, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 459, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2], 'acting': [2, 1], 'avail_no_missing': ['2', '1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2023-11-08T05:32:41.577 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.1 is not active+clean 2023-11-08T05:32:41.577 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.1', 'version': "0'0", 'reported_seq': 465, 'reported_epoch': 459, 'state': 'active+clean+remapped', 'last_fresh': '2023-11-08T05:30:45.795085+0000', 'last_change': '2023-11-08T05:30:45.795085+0000', 'last_active': '2023-11-08T05:30:45.795085+0000', 'last_peered': '2023-11-08T05:30:45.795085+0000', 'last_clean': '2023-11-08T05:30:45.795085+0000', 'last_became_active': '2023-11-08T05:30:45.794743+0000', 'last_became_peered': '2023-11-08T05:30:45.794743+0000', 'last_unstale': '2023-11-08T05:30:45.795085+0000', 'last_undegraded': '2023-11-08T05:30:45.795085+0000', 'last_fullsized': '2023-11-08T05:30:45.795085+0000', 'mapping_epoch': 458, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 459, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2], 'acting': [2, 1], 'avail_no_missing': ['2', '1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2023-11-08T05:32:41.577 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-11-08T05:32:41.578 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "0'0", 'reported_seq': 460, 'reported_epoch': 458, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:30:44.490523+0000', 'last_change': '2023-11-08T05:28:13.545468+0000', 'last_active': '2023-11-08T05:30:44.490523+0000', 'last_peered': '2023-11-08T05:30:44.490523+0000', 'last_clean': '2023-11-08T05:28:01.456723+0000', 'last_became_active': '2023-11-08T05:28:13.545468+0000', 'last_became_peered': '2023-11-08T05:28:13.545468+0000', 'last_unstale': '2023-11-08T05:30:44.490523+0000', 'last_undegraded': '2023-11-08T05:30:44.490523+0000', 'last_fullsized': '2023-11-08T05:28:13.537207+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 9, 'last_epoch_clean': 10, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:45.528796+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:45.528796+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:45.528796+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:32:41.578 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.0 is not active+clean 2023-11-08T05:32:41.578 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.0', 'version': "0'0", 'reported_seq': 465, 'reported_epoch': 458, 'state': 'active+clean+remapped', 'last_fresh': '2023-11-08T05:30:44.491907+0000', 'last_change': '2023-11-08T05:29:44.395350+0000', 'last_active': '2023-11-08T05:30:44.491907+0000', 'last_peered': '2023-11-08T05:30:44.491907+0000', 'last_clean': '2023-11-08T05:30:44.491907+0000', 'last_became_active': '2023-11-08T05:29:44.395012+0000', 'last_became_peered': '2023-11-08T05:29:44.395012+0000', 'last_unstale': '2023-11-08T05:30:44.491907+0000', 'last_undegraded': '2023-11-08T05:30:44.491907+0000', 'last_fullsized': '2023-11-08T05:30:44.491907+0000', 'mapping_epoch': 455, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 456, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2], 'acting': [2, 1], 'avail_no_missing': ['2', '1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2023-11-08T05:32:41.579 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-11-08T05:32:41.579 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 459, 'reported_epoch': 458, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:30:44.490629+0000', 'last_change': '2023-11-08T05:28:13.563357+0000', 'last_active': '2023-11-08T05:30:44.490629+0000', 'last_peered': '2023-11-08T05:30:44.490629+0000', 'last_clean': '2023-11-08T05:28:01.457691+0000', 'last_became_active': '2023-11-08T05:28:13.563357+0000', 'last_became_peered': '2023-11-08T05:28:13.563357+0000', 'last_unstale': '2023-11-08T05:30:44.490629+0000', 'last_undegraded': '2023-11-08T05:30:44.490629+0000', 'last_fullsized': '2023-11-08T05:28:13.541614+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:32:41.579 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-11-08T05:32:41.580 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 459, 'reported_epoch': 458, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:30:44.490691+0000', 'last_change': '2023-11-08T05:28:13.550672+0000', 'last_active': '2023-11-08T05:30:44.490691+0000', 'last_peered': '2023-11-08T05:30:44.490691+0000', 'last_clean': '2023-11-08T05:28:01.457904+0000', 'last_became_active': '2023-11-08T05:28:13.550672+0000', 'last_became_peered': '2023-11-08T05:28:13.550672+0000', 'last_unstale': '2023-11-08T05:30:44.490691+0000', 'last_undegraded': '2023-11-08T05:30:44.490691+0000', 'last_fullsized': '2023-11-08T05:28:13.539595+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:32:41.580 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-11-08T05:32:41.580 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 459, 'reported_epoch': 458, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:30:44.490805+0000', 'last_change': '2023-11-08T05:28:13.557968+0000', 'last_active': '2023-11-08T05:30:44.490805+0000', 'last_peered': '2023-11-08T05:30:44.490805+0000', 'last_clean': '2023-11-08T05:28:01.458433+0000', 'last_became_active': '2023-11-08T05:28:13.557968+0000', 'last_became_peered': '2023-11-08T05:28:13.557968+0000', 'last_unstale': '2023-11-08T05:30:44.490805+0000', 'last_undegraded': '2023-11-08T05:30:44.490805+0000', 'last_fullsized': '2023-11-08T05:28:13.542832+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:32:41.581 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-11-08T05:32:41.928 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T05:32:41.929 INFO:teuthology.orchestra.run.smithi087.stdout:{"fsid":"ab4cbc06-449d-497d-b513-220d9db2b590","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 5 pgs undersized","count":5},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2404,"monmap":{"epoch":1,"min_mon_release_name":"pacific","num_mons":1},"osdmap":{"epoch":459,"num_osds":3,"num_up_osds":2,"osd_up_since":1699421292,"num_in_osds":3,"osd_in_since":1699419158,"num_remapped_pgs":3},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean+remapped","count":3},{"state_name":"active+clean","count":1}],"num_pgs":9,"num_pools":2,"num_objects":0,"data_bytes":0,"bytes_used":2697592832,"bytes_avail":287212699648,"bytes_total":289910292480},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":41,"modified":"2023-11-08T05:29:32.044544+0000","services":{}},"progress_events":{}} 2023-11-08T05:32:41.942 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-11-08T05:32:41.942 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-11-08T05:32:42.226 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T05:32:42.226 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2023-11-08T05:32:42.237 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":1655,"stamp":"2023-11-08T05:32:42.083886+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":10,"acting":13,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":109,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2634368,"kb_used_data":409012,"kb_used_omap":780,"kb_used_meta":2220403,"kb_avail":280481152,"statfs":{"total":289910292480,"available":287212699648,"internally_reserved":0,"allocated":418828288,"data_stored":410500163,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":799447,"internal_metadata":2273692969},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-536782902,"num_objects":-164,"num_object_clones":0,"num_object_copies":-328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-115,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-164,"num_whiteouts":0,"num_read":-122056,"num_read_kb":-39011942,"num_write":-413642,"num_write_kb":-151705339,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-8,"num_bytes_recovered":-16922624,"num_keys_recovered":-2,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.424817"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490644+0000","last_change":"2023-11-08T05:28:13.557071+0000","last_active":"2023-11-08T05:30:44.490644+0000","last_peered":"2023-11-08T05:30:44.490644+0000","last_clean":"2023-11-08T05:28:01.458439+0000","last_became_active":"2023-11-08T05:28:13.557071+0000","last_became_peered":"2023-11-08T05:28:13.557071+0000","last_unstale":"2023-11-08T05:30:44.490644+0000","last_undegraded":"2023-11-08T05:30:44.490644+0000","last_fullsized":"2023-11-08T05:28:13.547025+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":459,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:45.794760+0000","last_change":"2023-11-08T05:30:45.794760+0000","last_active":"2023-11-08T05:30:45.794760+0000","last_peered":"2023-11-08T05:30:45.794760+0000","last_clean":"2023-11-08T05:30:45.794760+0000","last_became_active":"2023-11-08T05:30:45.794398+0000","last_became_peered":"2023-11-08T05:30:45.794398+0000","last_unstale":"2023-11-08T05:30:45.794760+0000","last_undegraded":"2023-11-08T05:30:45.794760+0000","last_fullsized":"2023-11-08T05:30:45.794760+0000","mapping_epoch":458,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":459,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":465,"reported_epoch":459,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:45.795085+0000","last_change":"2023-11-08T05:30:45.795085+0000","last_active":"2023-11-08T05:30:45.795085+0000","last_peered":"2023-11-08T05:30:45.795085+0000","last_clean":"2023-11-08T05:30:45.795085+0000","last_became_active":"2023-11-08T05:30:45.794743+0000","last_became_peered":"2023-11-08T05:30:45.794743+0000","last_unstale":"2023-11-08T05:30:45.795085+0000","last_undegraded":"2023-11-08T05:30:45.795085+0000","last_fullsized":"2023-11-08T05:30:45.795085+0000","mapping_epoch":458,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":459,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":454,"reported_epoch":458,"state":"active+clean","last_fresh":"2023-11-08T05:30:44.490618+0000","last_change":"2023-11-08T04:52:47.548597+0000","last_active":"2023-11-08T05:30:44.490618+0000","last_peered":"2023-11-08T05:30:44.490618+0000","last_clean":"2023-11-08T05:30:44.490618+0000","last_became_active":"2023-11-08T04:52:47.548294+0000","last_became_peered":"2023-11-08T04:52:47.548294+0000","last_unstale":"2023-11-08T05:30:44.490618+0000","last_undegraded":"2023-11-08T05:30:44.490618+0000","last_fullsized":"2023-11-08T05:30:44.490618+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":460,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490523+0000","last_change":"2023-11-08T05:28:13.545468+0000","last_active":"2023-11-08T05:30:44.490523+0000","last_peered":"2023-11-08T05:30:44.490523+0000","last_clean":"2023-11-08T05:28:01.456723+0000","last_became_active":"2023-11-08T05:28:13.545468+0000","last_became_peered":"2023-11-08T05:28:13.545468+0000","last_unstale":"2023-11-08T05:30:44.490523+0000","last_undegraded":"2023-11-08T05:30:44.490523+0000","last_fullsized":"2023-11-08T05:28:13.537207+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_clean_scrub_stamp":"2023-11-08T04:52:45.528796+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":465,"reported_epoch":458,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:44.491907+0000","last_change":"2023-11-08T05:29:44.395350+0000","last_active":"2023-11-08T05:30:44.491907+0000","last_peered":"2023-11-08T05:30:44.491907+0000","last_clean":"2023-11-08T05:30:44.491907+0000","last_became_active":"2023-11-08T05:29:44.395012+0000","last_became_peered":"2023-11-08T05:29:44.395012+0000","last_unstale":"2023-11-08T05:30:44.491907+0000","last_undegraded":"2023-11-08T05:30:44.491907+0000","last_fullsized":"2023-11-08T05:30:44.491907+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490629+0000","last_change":"2023-11-08T05:28:13.563357+0000","last_active":"2023-11-08T05:30:44.490629+0000","last_peered":"2023-11-08T05:30:44.490629+0000","last_clean":"2023-11-08T05:28:01.457691+0000","last_became_active":"2023-11-08T05:28:13.563357+0000","last_became_peered":"2023-11-08T05:28:13.563357+0000","last_unstale":"2023-11-08T05:30:44.490629+0000","last_undegraded":"2023-11-08T05:30:44.490629+0000","last_fullsized":"2023-11-08T05:28:13.541614+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490691+0000","last_change":"2023-11-08T05:28:13.550672+0000","last_active":"2023-11-08T05:30:44.490691+0000","last_peered":"2023-11-08T05:30:44.490691+0000","last_clean":"2023-11-08T05:28:01.457904+0000","last_became_active":"2023-11-08T05:28:13.550672+0000","last_became_peered":"2023-11-08T05:28:13.550672+0000","last_unstale":"2023-11-08T05:30:44.490691+0000","last_undegraded":"2023-11-08T05:30:44.490691+0000","last_fullsized":"2023-11-08T05:28:13.539595+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490805+0000","last_change":"2023-11-08T05:28:13.557968+0000","last_active":"2023-11-08T05:30:44.490805+0000","last_peered":"2023-11-08T05:30:44.490805+0000","last_clean":"2023-11-08T05:28:01.458433+0000","last_became_active":"2023-11-08T05:28:13.557968+0000","last_became_peered":"2023-11-08T05:28:13.557968+0000","last_unstale":"2023-11-08T05:30:44.490805+0000","last_undegraded":"2023-11-08T05:30:44.490805+0000","last_fullsized":"2023-11-08T05:28:13.542832+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":1,"acting":1,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":12,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738850,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":645040,"kb_used_data":5352,"kb_used_omap":275,"kb_used_meta":639404,"kb_avail":93726800,"statfs":{"total":96636764160,"available":95976243200,"internally_reserved":0,"allocated":5480448,"data_stored":2697647,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":281650,"internal_metadata":654750670},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:59 2023","interfaces":[{"interface":"back","average":{"1min":0.30499999999999999,"5min":0.439,"15min":0.36099999999999999},"min":{"1min":0.17399999999999999,"5min":0.14499999999999999,"15min":0.14499999999999999},"max":{"1min":0.60099999999999998,"5min":4.9379999999999997,"15min":4.9379999999999997},"last":0.379},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.49399999999999999,"15min":0.42399999999999999},"min":{"1min":0.221,"5min":0.19600000000000001,"15min":0.17000000000000001},"max":{"1min":0.63200000000000001,"5min":4.9859999999999998,"15min":4.9859999999999998},"last":0.40899999999999997}]},{"osd":1,"last update":"Wed Nov 8 05:32:06 2023","interfaces":[{"interface":"back","average":{"1min":0.27400000000000002,"5min":0.307,"15min":0.46800000000000003},"min":{"1min":0.17599999999999999,"5min":0.16,"15min":0.153},"max":{"1min":0.39000000000000001,"5min":1.825,"15min":23.242999999999999},"last":0.34899999999999998},{"interface":"front","average":{"1min":0.28000000000000003,"5min":0.34799999999999998,"15min":0.45700000000000002},"min":{"1min":0.20200000000000001,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":0.41699999999999998,"5min":2.0419999999999998,"15min":23.251000000000001},"last":0.35899999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738798,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1339228,"kb_used_data":398304,"kb_used_omap":291,"kb_used_meta":936476,"kb_avail":93032612,"statfs":{"total":96636764160,"available":95265394688,"internally_reserved":0,"allocated":407863296,"data_stored":405103900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":298462,"internal_metadata":958951970},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.436,"15min":0.36299999999999999},"min":{"1min":0.19700000000000001,"5min":0.14699999999999999,"15min":0.13700000000000001},"max":{"1min":3.222,"5min":6.5499999999999998,"15min":6.9989999999999997},"last":0.245},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.46300000000000002,"15min":0.42299999999999999},"min":{"1min":0.17799999999999999,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":5.0030000000000001,"5min":6.0629999999999997,"15min":9.6869999999999994},"last":0.28599999999999998}]},{"osd":2,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.51600000000000001,"15min":0.47099999999999997},"min":{"1min":0.21099999999999999,"5min":0.153,"15min":0.153},"max":{"1min":5.7439999999999998,"5min":5.8280000000000003,"15min":9.7639999999999993},"last":0.252},{"interface":"front","average":{"1min":0.497,"5min":0.56200000000000006,"15min":0.42299999999999999},"min":{"1min":0.20399999999999999,"5min":0.187,"15min":0.16800000000000001},"max":{"1min":3.9319999999999999,"5min":8.8040000000000003,"15min":8.8040000000000003},"last":0.29899999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738852,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":650100,"kb_used_data":5356,"kb_used_omap":214,"kb_used_meta":644521,"kb_avail":93721740,"statfs":{"total":96636764160,"available":95971061760,"internally_reserved":0,"allocated":5484544,"data_stored":2698616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":219335,"internal_metadata":659990329},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:56 2023","interfaces":[{"interface":"back","average":{"1min":0.30399999999999999,"5min":0.34699999999999998,"15min":0.36499999999999999},"min":{"1min":0.23999999999999999,"5min":0.14199999999999999,"15min":0.14199999999999999},"max":{"1min":0.42399999999999999,"5min":1.595,"15min":7.4249999999999998},"last":0.17100000000000001},{"interface":"front","average":{"1min":0.35499999999999998,"5min":0.40899999999999997,"15min":0.376},"min":{"1min":0.23999999999999999,"5min":0.19600000000000001,"15min":0.183},"max":{"1min":0.46999999999999997,"5min":2.6859999999999999,"15min":2.6859999999999999},"last":0.23000000000000001}]},{"osd":2,"last update":"Wed Nov 8 05:32:02 2023","interfaces":[{"interface":"back","average":{"1min":0.252,"5min":0.33000000000000002,"15min":0.36499999999999999},"min":{"1min":0.161,"5min":0.161,"15min":0.161},"max":{"1min":0.39600000000000002,"5min":2.2650000000000001,"15min":5.8209999999999997},"last":0.23300000000000001},{"interface":"front","average":{"1min":0.25800000000000001,"5min":0.29899999999999999,"15min":0.35599999999999998},"min":{"1min":0.17000000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.35799999999999998,"5min":1.137,"15min":4.2350000000000003},"last":0.19600000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-08T05:32:42.844 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~265s 2023-11-08T05:32:43.488 INFO:tasks.ceph.cluster2.mgr.x.smithi077.stderr:2023-11-08T05:32:43.486+0000 7f16c17cf700 -1 calc_pg_upmaps abort due to max <= 0 2023-11-08T05:32:45.238 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-11-08T05:32:45.535 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T05:32:45.537 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2023-11-08T05:32:45.547 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":1656,"stamp":"2023-11-08T05:32:44.084262+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":10,"acting":13,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":109,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2634368,"kb_used_data":409012,"kb_used_omap":780,"kb_used_meta":2220403,"kb_avail":280481152,"statfs":{"total":289910292480,"available":287212699648,"internally_reserved":0,"allocated":418828288,"data_stored":410500163,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":799447,"internal_metadata":2273692969},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-536782902,"num_objects":-164,"num_object_clones":0,"num_object_copies":-328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-115,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-164,"num_whiteouts":0,"num_read":-122056,"num_read_kb":-39011942,"num_write":-413642,"num_write_kb":-151705339,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-8,"num_bytes_recovered":-16922624,"num_keys_recovered":-2,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.424817"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490644+0000","last_change":"2023-11-08T05:28:13.557071+0000","last_active":"2023-11-08T05:30:44.490644+0000","last_peered":"2023-11-08T05:30:44.490644+0000","last_clean":"2023-11-08T05:28:01.458439+0000","last_became_active":"2023-11-08T05:28:13.557071+0000","last_became_peered":"2023-11-08T05:28:13.557071+0000","last_unstale":"2023-11-08T05:30:44.490644+0000","last_undegraded":"2023-11-08T05:30:44.490644+0000","last_fullsized":"2023-11-08T05:28:13.547025+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":459,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:45.794760+0000","last_change":"2023-11-08T05:30:45.794760+0000","last_active":"2023-11-08T05:30:45.794760+0000","last_peered":"2023-11-08T05:30:45.794760+0000","last_clean":"2023-11-08T05:30:45.794760+0000","last_became_active":"2023-11-08T05:30:45.794398+0000","last_became_peered":"2023-11-08T05:30:45.794398+0000","last_unstale":"2023-11-08T05:30:45.794760+0000","last_undegraded":"2023-11-08T05:30:45.794760+0000","last_fullsized":"2023-11-08T05:30:45.794760+0000","mapping_epoch":458,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":459,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":465,"reported_epoch":459,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:45.795085+0000","last_change":"2023-11-08T05:30:45.795085+0000","last_active":"2023-11-08T05:30:45.795085+0000","last_peered":"2023-11-08T05:30:45.795085+0000","last_clean":"2023-11-08T05:30:45.795085+0000","last_became_active":"2023-11-08T05:30:45.794743+0000","last_became_peered":"2023-11-08T05:30:45.794743+0000","last_unstale":"2023-11-08T05:30:45.795085+0000","last_undegraded":"2023-11-08T05:30:45.795085+0000","last_fullsized":"2023-11-08T05:30:45.795085+0000","mapping_epoch":458,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":459,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":454,"reported_epoch":458,"state":"active+clean","last_fresh":"2023-11-08T05:30:44.490618+0000","last_change":"2023-11-08T04:52:47.548597+0000","last_active":"2023-11-08T05:30:44.490618+0000","last_peered":"2023-11-08T05:30:44.490618+0000","last_clean":"2023-11-08T05:30:44.490618+0000","last_became_active":"2023-11-08T04:52:47.548294+0000","last_became_peered":"2023-11-08T04:52:47.548294+0000","last_unstale":"2023-11-08T05:30:44.490618+0000","last_undegraded":"2023-11-08T05:30:44.490618+0000","last_fullsized":"2023-11-08T05:30:44.490618+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":460,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490523+0000","last_change":"2023-11-08T05:28:13.545468+0000","last_active":"2023-11-08T05:30:44.490523+0000","last_peered":"2023-11-08T05:30:44.490523+0000","last_clean":"2023-11-08T05:28:01.456723+0000","last_became_active":"2023-11-08T05:28:13.545468+0000","last_became_peered":"2023-11-08T05:28:13.545468+0000","last_unstale":"2023-11-08T05:30:44.490523+0000","last_undegraded":"2023-11-08T05:30:44.490523+0000","last_fullsized":"2023-11-08T05:28:13.537207+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_clean_scrub_stamp":"2023-11-08T04:52:45.528796+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":465,"reported_epoch":458,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:44.491907+0000","last_change":"2023-11-08T05:29:44.395350+0000","last_active":"2023-11-08T05:30:44.491907+0000","last_peered":"2023-11-08T05:30:44.491907+0000","last_clean":"2023-11-08T05:30:44.491907+0000","last_became_active":"2023-11-08T05:29:44.395012+0000","last_became_peered":"2023-11-08T05:29:44.395012+0000","last_unstale":"2023-11-08T05:30:44.491907+0000","last_undegraded":"2023-11-08T05:30:44.491907+0000","last_fullsized":"2023-11-08T05:30:44.491907+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490629+0000","last_change":"2023-11-08T05:28:13.563357+0000","last_active":"2023-11-08T05:30:44.490629+0000","last_peered":"2023-11-08T05:30:44.490629+0000","last_clean":"2023-11-08T05:28:01.457691+0000","last_became_active":"2023-11-08T05:28:13.563357+0000","last_became_peered":"2023-11-08T05:28:13.563357+0000","last_unstale":"2023-11-08T05:30:44.490629+0000","last_undegraded":"2023-11-08T05:30:44.490629+0000","last_fullsized":"2023-11-08T05:28:13.541614+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490691+0000","last_change":"2023-11-08T05:28:13.550672+0000","last_active":"2023-11-08T05:30:44.490691+0000","last_peered":"2023-11-08T05:30:44.490691+0000","last_clean":"2023-11-08T05:28:01.457904+0000","last_became_active":"2023-11-08T05:28:13.550672+0000","last_became_peered":"2023-11-08T05:28:13.550672+0000","last_unstale":"2023-11-08T05:30:44.490691+0000","last_undegraded":"2023-11-08T05:30:44.490691+0000","last_fullsized":"2023-11-08T05:28:13.539595+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490805+0000","last_change":"2023-11-08T05:28:13.557968+0000","last_active":"2023-11-08T05:30:44.490805+0000","last_peered":"2023-11-08T05:30:44.490805+0000","last_clean":"2023-11-08T05:28:01.458433+0000","last_became_active":"2023-11-08T05:28:13.557968+0000","last_became_peered":"2023-11-08T05:28:13.557968+0000","last_unstale":"2023-11-08T05:30:44.490805+0000","last_undegraded":"2023-11-08T05:30:44.490805+0000","last_fullsized":"2023-11-08T05:28:13.542832+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":1,"acting":1,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":12,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738850,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":645040,"kb_used_data":5352,"kb_used_omap":275,"kb_used_meta":639404,"kb_avail":93726800,"statfs":{"total":96636764160,"available":95976243200,"internally_reserved":0,"allocated":5480448,"data_stored":2697647,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":281650,"internal_metadata":654750670},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:59 2023","interfaces":[{"interface":"back","average":{"1min":0.30499999999999999,"5min":0.439,"15min":0.36099999999999999},"min":{"1min":0.17399999999999999,"5min":0.14499999999999999,"15min":0.14499999999999999},"max":{"1min":0.60099999999999998,"5min":4.9379999999999997,"15min":4.9379999999999997},"last":0.379},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.49399999999999999,"15min":0.42399999999999999},"min":{"1min":0.221,"5min":0.19600000000000001,"15min":0.17000000000000001},"max":{"1min":0.63200000000000001,"5min":4.9859999999999998,"15min":4.9859999999999998},"last":0.40899999999999997}]},{"osd":1,"last update":"Wed Nov 8 05:32:06 2023","interfaces":[{"interface":"back","average":{"1min":0.27400000000000002,"5min":0.307,"15min":0.46800000000000003},"min":{"1min":0.17599999999999999,"5min":0.16,"15min":0.153},"max":{"1min":0.39000000000000001,"5min":1.825,"15min":23.242999999999999},"last":0.34899999999999998},{"interface":"front","average":{"1min":0.28000000000000003,"5min":0.34799999999999998,"15min":0.45700000000000002},"min":{"1min":0.20200000000000001,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":0.41699999999999998,"5min":2.0419999999999998,"15min":23.251000000000001},"last":0.35899999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738798,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1339228,"kb_used_data":398304,"kb_used_omap":291,"kb_used_meta":936476,"kb_avail":93032612,"statfs":{"total":96636764160,"available":95265394688,"internally_reserved":0,"allocated":407863296,"data_stored":405103900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":298462,"internal_metadata":958951970},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.436,"15min":0.36299999999999999},"min":{"1min":0.19700000000000001,"5min":0.14699999999999999,"15min":0.13700000000000001},"max":{"1min":3.222,"5min":6.5499999999999998,"15min":6.9989999999999997},"last":0.245},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.46300000000000002,"15min":0.42299999999999999},"min":{"1min":0.17799999999999999,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":5.0030000000000001,"5min":6.0629999999999997,"15min":9.6869999999999994},"last":0.28599999999999998}]},{"osd":2,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.51600000000000001,"15min":0.47099999999999997},"min":{"1min":0.21099999999999999,"5min":0.153,"15min":0.153},"max":{"1min":5.7439999999999998,"5min":5.8280000000000003,"15min":9.7639999999999993},"last":0.252},{"interface":"front","average":{"1min":0.497,"5min":0.56200000000000006,"15min":0.42299999999999999},"min":{"1min":0.20399999999999999,"5min":0.187,"15min":0.16800000000000001},"max":{"1min":3.9319999999999999,"5min":8.8040000000000003,"15min":8.8040000000000003},"last":0.29899999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738852,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":650100,"kb_used_data":5356,"kb_used_omap":214,"kb_used_meta":644521,"kb_avail":93721740,"statfs":{"total":96636764160,"available":95971061760,"internally_reserved":0,"allocated":5484544,"data_stored":2698616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":219335,"internal_metadata":659990329},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:56 2023","interfaces":[{"interface":"back","average":{"1min":0.30399999999999999,"5min":0.34699999999999998,"15min":0.36499999999999999},"min":{"1min":0.23999999999999999,"5min":0.14199999999999999,"15min":0.14199999999999999},"max":{"1min":0.42399999999999999,"5min":1.595,"15min":7.4249999999999998},"last":0.17100000000000001},{"interface":"front","average":{"1min":0.35499999999999998,"5min":0.40899999999999997,"15min":0.376},"min":{"1min":0.23999999999999999,"5min":0.19600000000000001,"15min":0.183},"max":{"1min":0.46999999999999997,"5min":2.6859999999999999,"15min":2.6859999999999999},"last":0.23000000000000001}]},{"osd":2,"last update":"Wed Nov 8 05:32:02 2023","interfaces":[{"interface":"back","average":{"1min":0.252,"5min":0.33000000000000002,"15min":0.36499999999999999},"min":{"1min":0.161,"5min":0.161,"15min":0.161},"max":{"1min":0.39600000000000002,"5min":2.2650000000000001,"15min":5.8209999999999997},"last":0.23300000000000001},{"interface":"front","average":{"1min":0.25800000000000001,"5min":0.29899999999999999,"15min":0.35599999999999998},"min":{"1min":0.17000000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.35799999999999998,"5min":1.137,"15min":4.2350000000000003},"last":0.19600000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-08T05:32:45.549 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-11-08T05:32:45.832 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T05:32:45.834 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2023-11-08T05:32:45.845 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":1656,"stamp":"2023-11-08T05:32:44.084262+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":10,"acting":13,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":109,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2634368,"kb_used_data":409012,"kb_used_omap":780,"kb_used_meta":2220403,"kb_avail":280481152,"statfs":{"total":289910292480,"available":287212699648,"internally_reserved":0,"allocated":418828288,"data_stored":410500163,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":799447,"internal_metadata":2273692969},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-536782902,"num_objects":-164,"num_object_clones":0,"num_object_copies":-328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-115,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-164,"num_whiteouts":0,"num_read":-122056,"num_read_kb":-39011942,"num_write":-413642,"num_write_kb":-151705339,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-8,"num_bytes_recovered":-16922624,"num_keys_recovered":-2,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.424817"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490644+0000","last_change":"2023-11-08T05:28:13.557071+0000","last_active":"2023-11-08T05:30:44.490644+0000","last_peered":"2023-11-08T05:30:44.490644+0000","last_clean":"2023-11-08T05:28:01.458439+0000","last_became_active":"2023-11-08T05:28:13.557071+0000","last_became_peered":"2023-11-08T05:28:13.557071+0000","last_unstale":"2023-11-08T05:30:44.490644+0000","last_undegraded":"2023-11-08T05:30:44.490644+0000","last_fullsized":"2023-11-08T05:28:13.547025+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":459,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:45.794760+0000","last_change":"2023-11-08T05:30:45.794760+0000","last_active":"2023-11-08T05:30:45.794760+0000","last_peered":"2023-11-08T05:30:45.794760+0000","last_clean":"2023-11-08T05:30:45.794760+0000","last_became_active":"2023-11-08T05:30:45.794398+0000","last_became_peered":"2023-11-08T05:30:45.794398+0000","last_unstale":"2023-11-08T05:30:45.794760+0000","last_undegraded":"2023-11-08T05:30:45.794760+0000","last_fullsized":"2023-11-08T05:30:45.794760+0000","mapping_epoch":458,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":459,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":465,"reported_epoch":459,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:45.795085+0000","last_change":"2023-11-08T05:30:45.795085+0000","last_active":"2023-11-08T05:30:45.795085+0000","last_peered":"2023-11-08T05:30:45.795085+0000","last_clean":"2023-11-08T05:30:45.795085+0000","last_became_active":"2023-11-08T05:30:45.794743+0000","last_became_peered":"2023-11-08T05:30:45.794743+0000","last_unstale":"2023-11-08T05:30:45.795085+0000","last_undegraded":"2023-11-08T05:30:45.795085+0000","last_fullsized":"2023-11-08T05:30:45.795085+0000","mapping_epoch":458,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":459,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":454,"reported_epoch":458,"state":"active+clean","last_fresh":"2023-11-08T05:30:44.490618+0000","last_change":"2023-11-08T04:52:47.548597+0000","last_active":"2023-11-08T05:30:44.490618+0000","last_peered":"2023-11-08T05:30:44.490618+0000","last_clean":"2023-11-08T05:30:44.490618+0000","last_became_active":"2023-11-08T04:52:47.548294+0000","last_became_peered":"2023-11-08T04:52:47.548294+0000","last_unstale":"2023-11-08T05:30:44.490618+0000","last_undegraded":"2023-11-08T05:30:44.490618+0000","last_fullsized":"2023-11-08T05:30:44.490618+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":460,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490523+0000","last_change":"2023-11-08T05:28:13.545468+0000","last_active":"2023-11-08T05:30:44.490523+0000","last_peered":"2023-11-08T05:30:44.490523+0000","last_clean":"2023-11-08T05:28:01.456723+0000","last_became_active":"2023-11-08T05:28:13.545468+0000","last_became_peered":"2023-11-08T05:28:13.545468+0000","last_unstale":"2023-11-08T05:30:44.490523+0000","last_undegraded":"2023-11-08T05:30:44.490523+0000","last_fullsized":"2023-11-08T05:28:13.537207+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_clean_scrub_stamp":"2023-11-08T04:52:45.528796+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":465,"reported_epoch":458,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:44.491907+0000","last_change":"2023-11-08T05:29:44.395350+0000","last_active":"2023-11-08T05:30:44.491907+0000","last_peered":"2023-11-08T05:30:44.491907+0000","last_clean":"2023-11-08T05:30:44.491907+0000","last_became_active":"2023-11-08T05:29:44.395012+0000","last_became_peered":"2023-11-08T05:29:44.395012+0000","last_unstale":"2023-11-08T05:30:44.491907+0000","last_undegraded":"2023-11-08T05:30:44.491907+0000","last_fullsized":"2023-11-08T05:30:44.491907+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490629+0000","last_change":"2023-11-08T05:28:13.563357+0000","last_active":"2023-11-08T05:30:44.490629+0000","last_peered":"2023-11-08T05:30:44.490629+0000","last_clean":"2023-11-08T05:28:01.457691+0000","last_became_active":"2023-11-08T05:28:13.563357+0000","last_became_peered":"2023-11-08T05:28:13.563357+0000","last_unstale":"2023-11-08T05:30:44.490629+0000","last_undegraded":"2023-11-08T05:30:44.490629+0000","last_fullsized":"2023-11-08T05:28:13.541614+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490691+0000","last_change":"2023-11-08T05:28:13.550672+0000","last_active":"2023-11-08T05:30:44.490691+0000","last_peered":"2023-11-08T05:30:44.490691+0000","last_clean":"2023-11-08T05:28:01.457904+0000","last_became_active":"2023-11-08T05:28:13.550672+0000","last_became_peered":"2023-11-08T05:28:13.550672+0000","last_unstale":"2023-11-08T05:30:44.490691+0000","last_undegraded":"2023-11-08T05:30:44.490691+0000","last_fullsized":"2023-11-08T05:28:13.539595+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490805+0000","last_change":"2023-11-08T05:28:13.557968+0000","last_active":"2023-11-08T05:30:44.490805+0000","last_peered":"2023-11-08T05:30:44.490805+0000","last_clean":"2023-11-08T05:28:01.458433+0000","last_became_active":"2023-11-08T05:28:13.557968+0000","last_became_peered":"2023-11-08T05:28:13.557968+0000","last_unstale":"2023-11-08T05:30:44.490805+0000","last_undegraded":"2023-11-08T05:30:44.490805+0000","last_fullsized":"2023-11-08T05:28:13.542832+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":1,"acting":1,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":12,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738850,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":645040,"kb_used_data":5352,"kb_used_omap":275,"kb_used_meta":639404,"kb_avail":93726800,"statfs":{"total":96636764160,"available":95976243200,"internally_reserved":0,"allocated":5480448,"data_stored":2697647,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":281650,"internal_metadata":654750670},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:59 2023","interfaces":[{"interface":"back","average":{"1min":0.30499999999999999,"5min":0.439,"15min":0.36099999999999999},"min":{"1min":0.17399999999999999,"5min":0.14499999999999999,"15min":0.14499999999999999},"max":{"1min":0.60099999999999998,"5min":4.9379999999999997,"15min":4.9379999999999997},"last":0.379},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.49399999999999999,"15min":0.42399999999999999},"min":{"1min":0.221,"5min":0.19600000000000001,"15min":0.17000000000000001},"max":{"1min":0.63200000000000001,"5min":4.9859999999999998,"15min":4.9859999999999998},"last":0.40899999999999997}]},{"osd":1,"last update":"Wed Nov 8 05:32:06 2023","interfaces":[{"interface":"back","average":{"1min":0.27400000000000002,"5min":0.307,"15min":0.46800000000000003},"min":{"1min":0.17599999999999999,"5min":0.16,"15min":0.153},"max":{"1min":0.39000000000000001,"5min":1.825,"15min":23.242999999999999},"last":0.34899999999999998},{"interface":"front","average":{"1min":0.28000000000000003,"5min":0.34799999999999998,"15min":0.45700000000000002},"min":{"1min":0.20200000000000001,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":0.41699999999999998,"5min":2.0419999999999998,"15min":23.251000000000001},"last":0.35899999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738798,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1339228,"kb_used_data":398304,"kb_used_omap":291,"kb_used_meta":936476,"kb_avail":93032612,"statfs":{"total":96636764160,"available":95265394688,"internally_reserved":0,"allocated":407863296,"data_stored":405103900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":298462,"internal_metadata":958951970},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.436,"15min":0.36299999999999999},"min":{"1min":0.19700000000000001,"5min":0.14699999999999999,"15min":0.13700000000000001},"max":{"1min":3.222,"5min":6.5499999999999998,"15min":6.9989999999999997},"last":0.245},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.46300000000000002,"15min":0.42299999999999999},"min":{"1min":0.17799999999999999,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":5.0030000000000001,"5min":6.0629999999999997,"15min":9.6869999999999994},"last":0.28599999999999998}]},{"osd":2,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.51600000000000001,"15min":0.47099999999999997},"min":{"1min":0.21099999999999999,"5min":0.153,"15min":0.153},"max":{"1min":5.7439999999999998,"5min":5.8280000000000003,"15min":9.7639999999999993},"last":0.252},{"interface":"front","average":{"1min":0.497,"5min":0.56200000000000006,"15min":0.42299999999999999},"min":{"1min":0.20399999999999999,"5min":0.187,"15min":0.16800000000000001},"max":{"1min":3.9319999999999999,"5min":8.8040000000000003,"15min":8.8040000000000003},"last":0.29899999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738852,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":650100,"kb_used_data":5356,"kb_used_omap":214,"kb_used_meta":644521,"kb_avail":93721740,"statfs":{"total":96636764160,"available":95971061760,"internally_reserved":0,"allocated":5484544,"data_stored":2698616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":219335,"internal_metadata":659990329},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:56 2023","interfaces":[{"interface":"back","average":{"1min":0.30399999999999999,"5min":0.34699999999999998,"15min":0.36499999999999999},"min":{"1min":0.23999999999999999,"5min":0.14199999999999999,"15min":0.14199999999999999},"max":{"1min":0.42399999999999999,"5min":1.595,"15min":7.4249999999999998},"last":0.17100000000000001},{"interface":"front","average":{"1min":0.35499999999999998,"5min":0.40899999999999997,"15min":0.376},"min":{"1min":0.23999999999999999,"5min":0.19600000000000001,"15min":0.183},"max":{"1min":0.46999999999999997,"5min":2.6859999999999999,"15min":2.6859999999999999},"last":0.23000000000000001}]},{"osd":2,"last update":"Wed Nov 8 05:32:02 2023","interfaces":[{"interface":"back","average":{"1min":0.252,"5min":0.33000000000000002,"15min":0.36499999999999999},"min":{"1min":0.161,"5min":0.161,"15min":0.161},"max":{"1min":0.39600000000000002,"5min":2.2650000000000001,"15min":5.8209999999999997},"last":0.23300000000000001},{"interface":"front","average":{"1min":0.25800000000000001,"5min":0.29899999999999999,"15min":0.35599999999999998},"min":{"1min":0.17000000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.35799999999999998,"5min":1.137,"15min":4.2350000000000003},"last":0.19600000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-08T05:32:45.846 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-11-08T05:32:45.846 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 459, 'reported_epoch': 458, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:30:44.490644+0000', 'last_change': '2023-11-08T05:28:13.557071+0000', 'last_active': '2023-11-08T05:30:44.490644+0000', 'last_peered': '2023-11-08T05:30:44.490644+0000', 'last_clean': '2023-11-08T05:28:01.458439+0000', 'last_became_active': '2023-11-08T05:28:13.557071+0000', 'last_became_peered': '2023-11-08T05:28:13.557071+0000', 'last_unstale': '2023-11-08T05:30:44.490644+0000', 'last_undegraded': '2023-11-08T05:30:44.490644+0000', 'last_fullsized': '2023-11-08T05:28:13.547025+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:32:45.847 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-11-08T05:32:45.847 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 11, 'reported_epoch': 459, 'state': 'active+clean+remapped', 'last_fresh': '2023-11-08T05:30:45.794760+0000', 'last_change': '2023-11-08T05:30:45.794760+0000', 'last_active': '2023-11-08T05:30:45.794760+0000', 'last_peered': '2023-11-08T05:30:45.794760+0000', 'last_clean': '2023-11-08T05:30:45.794760+0000', 'last_became_active': '2023-11-08T05:30:45.794398+0000', 'last_became_peered': '2023-11-08T05:30:45.794398+0000', 'last_unstale': '2023-11-08T05:30:45.794760+0000', 'last_undegraded': '2023-11-08T05:30:45.794760+0000', 'last_fullsized': '2023-11-08T05:30:45.794760+0000', 'mapping_epoch': 458, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 459, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2], 'acting': [2, 1], 'avail_no_missing': ['2', '1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2023-11-08T05:32:45.847 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.1 is not active+clean 2023-11-08T05:32:45.848 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.1', 'version': "0'0", 'reported_seq': 465, 'reported_epoch': 459, 'state': 'active+clean+remapped', 'last_fresh': '2023-11-08T05:30:45.795085+0000', 'last_change': '2023-11-08T05:30:45.795085+0000', 'last_active': '2023-11-08T05:30:45.795085+0000', 'last_peered': '2023-11-08T05:30:45.795085+0000', 'last_clean': '2023-11-08T05:30:45.795085+0000', 'last_became_active': '2023-11-08T05:30:45.794743+0000', 'last_became_peered': '2023-11-08T05:30:45.794743+0000', 'last_unstale': '2023-11-08T05:30:45.795085+0000', 'last_undegraded': '2023-11-08T05:30:45.795085+0000', 'last_fullsized': '2023-11-08T05:30:45.795085+0000', 'mapping_epoch': 458, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 459, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2], 'acting': [2, 1], 'avail_no_missing': ['2', '1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2023-11-08T05:32:45.848 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-11-08T05:32:45.848 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "0'0", 'reported_seq': 460, 'reported_epoch': 458, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:30:44.490523+0000', 'last_change': '2023-11-08T05:28:13.545468+0000', 'last_active': '2023-11-08T05:30:44.490523+0000', 'last_peered': '2023-11-08T05:30:44.490523+0000', 'last_clean': '2023-11-08T05:28:01.456723+0000', 'last_became_active': '2023-11-08T05:28:13.545468+0000', 'last_became_peered': '2023-11-08T05:28:13.545468+0000', 'last_unstale': '2023-11-08T05:30:44.490523+0000', 'last_undegraded': '2023-11-08T05:30:44.490523+0000', 'last_fullsized': '2023-11-08T05:28:13.537207+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 9, 'last_epoch_clean': 10, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:45.528796+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:45.528796+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:45.528796+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:32:45.849 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.0 is not active+clean 2023-11-08T05:32:45.849 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.0', 'version': "0'0", 'reported_seq': 465, 'reported_epoch': 458, 'state': 'active+clean+remapped', 'last_fresh': '2023-11-08T05:30:44.491907+0000', 'last_change': '2023-11-08T05:29:44.395350+0000', 'last_active': '2023-11-08T05:30:44.491907+0000', 'last_peered': '2023-11-08T05:30:44.491907+0000', 'last_clean': '2023-11-08T05:30:44.491907+0000', 'last_became_active': '2023-11-08T05:29:44.395012+0000', 'last_became_peered': '2023-11-08T05:29:44.395012+0000', 'last_unstale': '2023-11-08T05:30:44.491907+0000', 'last_undegraded': '2023-11-08T05:30:44.491907+0000', 'last_fullsized': '2023-11-08T05:30:44.491907+0000', 'mapping_epoch': 455, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 456, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2], 'acting': [2, 1], 'avail_no_missing': ['2', '1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2023-11-08T05:32:45.849 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-11-08T05:32:45.849 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 459, 'reported_epoch': 458, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:30:44.490629+0000', 'last_change': '2023-11-08T05:28:13.563357+0000', 'last_active': '2023-11-08T05:30:44.490629+0000', 'last_peered': '2023-11-08T05:30:44.490629+0000', 'last_clean': '2023-11-08T05:28:01.457691+0000', 'last_became_active': '2023-11-08T05:28:13.563357+0000', 'last_became_peered': '2023-11-08T05:28:13.563357+0000', 'last_unstale': '2023-11-08T05:30:44.490629+0000', 'last_undegraded': '2023-11-08T05:30:44.490629+0000', 'last_fullsized': '2023-11-08T05:28:13.541614+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:32:45.850 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-11-08T05:32:45.850 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 459, 'reported_epoch': 458, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:30:44.490691+0000', 'last_change': '2023-11-08T05:28:13.550672+0000', 'last_active': '2023-11-08T05:30:44.490691+0000', 'last_peered': '2023-11-08T05:30:44.490691+0000', 'last_clean': '2023-11-08T05:28:01.457904+0000', 'last_became_active': '2023-11-08T05:28:13.550672+0000', 'last_became_peered': '2023-11-08T05:28:13.550672+0000', 'last_unstale': '2023-11-08T05:30:44.490691+0000', 'last_undegraded': '2023-11-08T05:30:44.490691+0000', 'last_fullsized': '2023-11-08T05:28:13.539595+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:32:45.850 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-11-08T05:32:45.851 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 459, 'reported_epoch': 458, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:30:44.490805+0000', 'last_change': '2023-11-08T05:28:13.557968+0000', 'last_active': '2023-11-08T05:30:44.490805+0000', 'last_peered': '2023-11-08T05:30:44.490805+0000', 'last_clean': '2023-11-08T05:28:01.458433+0000', 'last_became_active': '2023-11-08T05:28:13.557968+0000', 'last_became_peered': '2023-11-08T05:28:13.557968+0000', 'last_unstale': '2023-11-08T05:30:44.490805+0000', 'last_undegraded': '2023-11-08T05:30:44.490805+0000', 'last_fullsized': '2023-11-08T05:28:13.542832+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:32:45.851 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-11-08T05:32:46.200 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T05:32:46.201 INFO:teuthology.orchestra.run.smithi087.stdout:{"fsid":"ab4cbc06-449d-497d-b513-220d9db2b590","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 5 pgs undersized","count":5},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2408,"monmap":{"epoch":1,"min_mon_release_name":"pacific","num_mons":1},"osdmap":{"epoch":459,"num_osds":3,"num_up_osds":2,"osd_up_since":1699421292,"num_in_osds":3,"osd_in_since":1699419158,"num_remapped_pgs":3},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean+remapped","count":3},{"state_name":"active+clean","count":1}],"num_pgs":9,"num_pools":2,"num_objects":0,"data_bytes":0,"bytes_used":2697592832,"bytes_avail":287212699648,"bytes_total":289910292480},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":41,"modified":"2023-11-08T05:29:32.044544+0000","services":{}},"progress_events":{}} 2023-11-08T05:32:46.214 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-11-08T05:32:46.214 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-11-08T05:32:46.495 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T05:32:46.496 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2023-11-08T05:32:46.507 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":1657,"stamp":"2023-11-08T05:32:46.084677+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":10,"acting":13,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":109,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2634368,"kb_used_data":409012,"kb_used_omap":780,"kb_used_meta":2220403,"kb_avail":280481152,"statfs":{"total":289910292480,"available":287212699648,"internally_reserved":0,"allocated":418828288,"data_stored":410500163,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":799447,"internal_metadata":2273692969},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-536782902,"num_objects":-164,"num_object_clones":0,"num_object_copies":-328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-115,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-164,"num_whiteouts":0,"num_read":-122056,"num_read_kb":-39011942,"num_write":-413642,"num_write_kb":-151705339,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-8,"num_bytes_recovered":-16922624,"num_keys_recovered":-2,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.424817"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490644+0000","last_change":"2023-11-08T05:28:13.557071+0000","last_active":"2023-11-08T05:30:44.490644+0000","last_peered":"2023-11-08T05:30:44.490644+0000","last_clean":"2023-11-08T05:28:01.458439+0000","last_became_active":"2023-11-08T05:28:13.557071+0000","last_became_peered":"2023-11-08T05:28:13.557071+0000","last_unstale":"2023-11-08T05:30:44.490644+0000","last_undegraded":"2023-11-08T05:30:44.490644+0000","last_fullsized":"2023-11-08T05:28:13.547025+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":459,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:45.794760+0000","last_change":"2023-11-08T05:30:45.794760+0000","last_active":"2023-11-08T05:30:45.794760+0000","last_peered":"2023-11-08T05:30:45.794760+0000","last_clean":"2023-11-08T05:30:45.794760+0000","last_became_active":"2023-11-08T05:30:45.794398+0000","last_became_peered":"2023-11-08T05:30:45.794398+0000","last_unstale":"2023-11-08T05:30:45.794760+0000","last_undegraded":"2023-11-08T05:30:45.794760+0000","last_fullsized":"2023-11-08T05:30:45.794760+0000","mapping_epoch":458,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":459,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":465,"reported_epoch":459,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:45.795085+0000","last_change":"2023-11-08T05:30:45.795085+0000","last_active":"2023-11-08T05:30:45.795085+0000","last_peered":"2023-11-08T05:30:45.795085+0000","last_clean":"2023-11-08T05:30:45.795085+0000","last_became_active":"2023-11-08T05:30:45.794743+0000","last_became_peered":"2023-11-08T05:30:45.794743+0000","last_unstale":"2023-11-08T05:30:45.795085+0000","last_undegraded":"2023-11-08T05:30:45.795085+0000","last_fullsized":"2023-11-08T05:30:45.795085+0000","mapping_epoch":458,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":459,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":454,"reported_epoch":458,"state":"active+clean","last_fresh":"2023-11-08T05:30:44.490618+0000","last_change":"2023-11-08T04:52:47.548597+0000","last_active":"2023-11-08T05:30:44.490618+0000","last_peered":"2023-11-08T05:30:44.490618+0000","last_clean":"2023-11-08T05:30:44.490618+0000","last_became_active":"2023-11-08T04:52:47.548294+0000","last_became_peered":"2023-11-08T04:52:47.548294+0000","last_unstale":"2023-11-08T05:30:44.490618+0000","last_undegraded":"2023-11-08T05:30:44.490618+0000","last_fullsized":"2023-11-08T05:30:44.490618+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":460,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490523+0000","last_change":"2023-11-08T05:28:13.545468+0000","last_active":"2023-11-08T05:30:44.490523+0000","last_peered":"2023-11-08T05:30:44.490523+0000","last_clean":"2023-11-08T05:28:01.456723+0000","last_became_active":"2023-11-08T05:28:13.545468+0000","last_became_peered":"2023-11-08T05:28:13.545468+0000","last_unstale":"2023-11-08T05:30:44.490523+0000","last_undegraded":"2023-11-08T05:30:44.490523+0000","last_fullsized":"2023-11-08T05:28:13.537207+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_clean_scrub_stamp":"2023-11-08T04:52:45.528796+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":465,"reported_epoch":458,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:44.491907+0000","last_change":"2023-11-08T05:29:44.395350+0000","last_active":"2023-11-08T05:30:44.491907+0000","last_peered":"2023-11-08T05:30:44.491907+0000","last_clean":"2023-11-08T05:30:44.491907+0000","last_became_active":"2023-11-08T05:29:44.395012+0000","last_became_peered":"2023-11-08T05:29:44.395012+0000","last_unstale":"2023-11-08T05:30:44.491907+0000","last_undegraded":"2023-11-08T05:30:44.491907+0000","last_fullsized":"2023-11-08T05:30:44.491907+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490629+0000","last_change":"2023-11-08T05:28:13.563357+0000","last_active":"2023-11-08T05:30:44.490629+0000","last_peered":"2023-11-08T05:30:44.490629+0000","last_clean":"2023-11-08T05:28:01.457691+0000","last_became_active":"2023-11-08T05:28:13.563357+0000","last_became_peered":"2023-11-08T05:28:13.563357+0000","last_unstale":"2023-11-08T05:30:44.490629+0000","last_undegraded":"2023-11-08T05:30:44.490629+0000","last_fullsized":"2023-11-08T05:28:13.541614+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490691+0000","last_change":"2023-11-08T05:28:13.550672+0000","last_active":"2023-11-08T05:30:44.490691+0000","last_peered":"2023-11-08T05:30:44.490691+0000","last_clean":"2023-11-08T05:28:01.457904+0000","last_became_active":"2023-11-08T05:28:13.550672+0000","last_became_peered":"2023-11-08T05:28:13.550672+0000","last_unstale":"2023-11-08T05:30:44.490691+0000","last_undegraded":"2023-11-08T05:30:44.490691+0000","last_fullsized":"2023-11-08T05:28:13.539595+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490805+0000","last_change":"2023-11-08T05:28:13.557968+0000","last_active":"2023-11-08T05:30:44.490805+0000","last_peered":"2023-11-08T05:30:44.490805+0000","last_clean":"2023-11-08T05:28:01.458433+0000","last_became_active":"2023-11-08T05:28:13.557968+0000","last_became_peered":"2023-11-08T05:28:13.557968+0000","last_unstale":"2023-11-08T05:30:44.490805+0000","last_undegraded":"2023-11-08T05:30:44.490805+0000","last_fullsized":"2023-11-08T05:28:13.542832+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":1,"acting":1,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":12,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738851,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":645040,"kb_used_data":5352,"kb_used_omap":275,"kb_used_meta":639404,"kb_avail":93726800,"statfs":{"total":96636764160,"available":95976243200,"internally_reserved":0,"allocated":5480448,"data_stored":2697647,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":281650,"internal_metadata":654750670},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:59 2023","interfaces":[{"interface":"back","average":{"1min":0.30499999999999999,"5min":0.439,"15min":0.36099999999999999},"min":{"1min":0.17399999999999999,"5min":0.14499999999999999,"15min":0.14499999999999999},"max":{"1min":0.60099999999999998,"5min":4.9379999999999997,"15min":4.9379999999999997},"last":0.379},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.49399999999999999,"15min":0.42399999999999999},"min":{"1min":0.221,"5min":0.19600000000000001,"15min":0.17000000000000001},"max":{"1min":0.63200000000000001,"5min":4.9859999999999998,"15min":4.9859999999999998},"last":0.40899999999999997}]},{"osd":1,"last update":"Wed Nov 8 05:32:06 2023","interfaces":[{"interface":"back","average":{"1min":0.27400000000000002,"5min":0.307,"15min":0.46800000000000003},"min":{"1min":0.17599999999999999,"5min":0.16,"15min":0.153},"max":{"1min":0.39000000000000001,"5min":1.825,"15min":23.242999999999999},"last":0.187},{"interface":"front","average":{"1min":0.28000000000000003,"5min":0.34799999999999998,"15min":0.45700000000000002},"min":{"1min":0.20200000000000001,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":0.41699999999999998,"5min":2.0419999999999998,"15min":23.251000000000001},"last":0.20399999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738798,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1339228,"kb_used_data":398304,"kb_used_omap":291,"kb_used_meta":936476,"kb_avail":93032612,"statfs":{"total":96636764160,"available":95265394688,"internally_reserved":0,"allocated":407863296,"data_stored":405103900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":298462,"internal_metadata":958951970},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.436,"15min":0.36299999999999999},"min":{"1min":0.19700000000000001,"5min":0.14699999999999999,"15min":0.13700000000000001},"max":{"1min":3.222,"5min":6.5499999999999998,"15min":6.9989999999999997},"last":0.245},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.46300000000000002,"15min":0.42299999999999999},"min":{"1min":0.17799999999999999,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":5.0030000000000001,"5min":6.0629999999999997,"15min":9.6869999999999994},"last":0.28599999999999998}]},{"osd":2,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.51600000000000001,"15min":0.47099999999999997},"min":{"1min":0.21099999999999999,"5min":0.153,"15min":0.153},"max":{"1min":5.7439999999999998,"5min":5.8280000000000003,"15min":9.7639999999999993},"last":0.252},{"interface":"front","average":{"1min":0.497,"5min":0.56200000000000006,"15min":0.42299999999999999},"min":{"1min":0.20399999999999999,"5min":0.187,"15min":0.16800000000000001},"max":{"1min":3.9319999999999999,"5min":8.8040000000000003,"15min":8.8040000000000003},"last":0.29899999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738853,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":650100,"kb_used_data":5356,"kb_used_omap":214,"kb_used_meta":644521,"kb_avail":93721740,"statfs":{"total":96636764160,"available":95971061760,"internally_reserved":0,"allocated":5484544,"data_stored":2698616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":219335,"internal_metadata":659990329},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:56 2023","interfaces":[{"interface":"back","average":{"1min":0.30399999999999999,"5min":0.34699999999999998,"15min":0.36499999999999999},"min":{"1min":0.23999999999999999,"5min":0.14199999999999999,"15min":0.14199999999999999},"max":{"1min":0.42399999999999999,"5min":1.595,"15min":7.4249999999999998},"last":0.17100000000000001},{"interface":"front","average":{"1min":0.35499999999999998,"5min":0.40899999999999997,"15min":0.376},"min":{"1min":0.23999999999999999,"5min":0.19600000000000001,"15min":0.183},"max":{"1min":0.46999999999999997,"5min":2.6859999999999999,"15min":2.6859999999999999},"last":0.23000000000000001}]},{"osd":2,"last update":"Wed Nov 8 05:32:02 2023","interfaces":[{"interface":"back","average":{"1min":0.252,"5min":0.33000000000000002,"15min":0.36499999999999999},"min":{"1min":0.161,"5min":0.161,"15min":0.161},"max":{"1min":0.39600000000000002,"5min":2.2650000000000001,"15min":5.8209999999999997},"last":0.23999999999999999},{"interface":"front","average":{"1min":0.25800000000000001,"5min":0.29899999999999999,"15min":0.35599999999999998},"min":{"1min":0.17000000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.35799999999999998,"5min":1.137,"15min":4.2350000000000003},"last":0.22800000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-08T05:32:48.248 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~270s 2023-11-08T05:32:49.509 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-11-08T05:32:49.804 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T05:32:49.805 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2023-11-08T05:32:49.817 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":1658,"stamp":"2023-11-08T05:32:48.085073+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":10,"acting":13,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":109,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2634368,"kb_used_data":409012,"kb_used_omap":780,"kb_used_meta":2220403,"kb_avail":280481152,"statfs":{"total":289910292480,"available":287212699648,"internally_reserved":0,"allocated":418828288,"data_stored":410500163,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":799447,"internal_metadata":2273692969},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-536782902,"num_objects":-164,"num_object_clones":0,"num_object_copies":-328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-115,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-164,"num_whiteouts":0,"num_read":-122056,"num_read_kb":-39011942,"num_write":-413642,"num_write_kb":-151705339,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-8,"num_bytes_recovered":-16922624,"num_keys_recovered":-2,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.424817"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490644+0000","last_change":"2023-11-08T05:28:13.557071+0000","last_active":"2023-11-08T05:30:44.490644+0000","last_peered":"2023-11-08T05:30:44.490644+0000","last_clean":"2023-11-08T05:28:01.458439+0000","last_became_active":"2023-11-08T05:28:13.557071+0000","last_became_peered":"2023-11-08T05:28:13.557071+0000","last_unstale":"2023-11-08T05:30:44.490644+0000","last_undegraded":"2023-11-08T05:30:44.490644+0000","last_fullsized":"2023-11-08T05:28:13.547025+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":459,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:45.794760+0000","last_change":"2023-11-08T05:30:45.794760+0000","last_active":"2023-11-08T05:30:45.794760+0000","last_peered":"2023-11-08T05:30:45.794760+0000","last_clean":"2023-11-08T05:30:45.794760+0000","last_became_active":"2023-11-08T05:30:45.794398+0000","last_became_peered":"2023-11-08T05:30:45.794398+0000","last_unstale":"2023-11-08T05:30:45.794760+0000","last_undegraded":"2023-11-08T05:30:45.794760+0000","last_fullsized":"2023-11-08T05:30:45.794760+0000","mapping_epoch":458,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":459,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":465,"reported_epoch":459,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:45.795085+0000","last_change":"2023-11-08T05:30:45.795085+0000","last_active":"2023-11-08T05:30:45.795085+0000","last_peered":"2023-11-08T05:30:45.795085+0000","last_clean":"2023-11-08T05:30:45.795085+0000","last_became_active":"2023-11-08T05:30:45.794743+0000","last_became_peered":"2023-11-08T05:30:45.794743+0000","last_unstale":"2023-11-08T05:30:45.795085+0000","last_undegraded":"2023-11-08T05:30:45.795085+0000","last_fullsized":"2023-11-08T05:30:45.795085+0000","mapping_epoch":458,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":459,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":454,"reported_epoch":458,"state":"active+clean","last_fresh":"2023-11-08T05:30:44.490618+0000","last_change":"2023-11-08T04:52:47.548597+0000","last_active":"2023-11-08T05:30:44.490618+0000","last_peered":"2023-11-08T05:30:44.490618+0000","last_clean":"2023-11-08T05:30:44.490618+0000","last_became_active":"2023-11-08T04:52:47.548294+0000","last_became_peered":"2023-11-08T04:52:47.548294+0000","last_unstale":"2023-11-08T05:30:44.490618+0000","last_undegraded":"2023-11-08T05:30:44.490618+0000","last_fullsized":"2023-11-08T05:30:44.490618+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":460,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490523+0000","last_change":"2023-11-08T05:28:13.545468+0000","last_active":"2023-11-08T05:30:44.490523+0000","last_peered":"2023-11-08T05:30:44.490523+0000","last_clean":"2023-11-08T05:28:01.456723+0000","last_became_active":"2023-11-08T05:28:13.545468+0000","last_became_peered":"2023-11-08T05:28:13.545468+0000","last_unstale":"2023-11-08T05:30:44.490523+0000","last_undegraded":"2023-11-08T05:30:44.490523+0000","last_fullsized":"2023-11-08T05:28:13.537207+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_clean_scrub_stamp":"2023-11-08T04:52:45.528796+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":465,"reported_epoch":458,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:44.491907+0000","last_change":"2023-11-08T05:29:44.395350+0000","last_active":"2023-11-08T05:30:44.491907+0000","last_peered":"2023-11-08T05:30:44.491907+0000","last_clean":"2023-11-08T05:30:44.491907+0000","last_became_active":"2023-11-08T05:29:44.395012+0000","last_became_peered":"2023-11-08T05:29:44.395012+0000","last_unstale":"2023-11-08T05:30:44.491907+0000","last_undegraded":"2023-11-08T05:30:44.491907+0000","last_fullsized":"2023-11-08T05:30:44.491907+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490629+0000","last_change":"2023-11-08T05:28:13.563357+0000","last_active":"2023-11-08T05:30:44.490629+0000","last_peered":"2023-11-08T05:30:44.490629+0000","last_clean":"2023-11-08T05:28:01.457691+0000","last_became_active":"2023-11-08T05:28:13.563357+0000","last_became_peered":"2023-11-08T05:28:13.563357+0000","last_unstale":"2023-11-08T05:30:44.490629+0000","last_undegraded":"2023-11-08T05:30:44.490629+0000","last_fullsized":"2023-11-08T05:28:13.541614+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490691+0000","last_change":"2023-11-08T05:28:13.550672+0000","last_active":"2023-11-08T05:30:44.490691+0000","last_peered":"2023-11-08T05:30:44.490691+0000","last_clean":"2023-11-08T05:28:01.457904+0000","last_became_active":"2023-11-08T05:28:13.550672+0000","last_became_peered":"2023-11-08T05:28:13.550672+0000","last_unstale":"2023-11-08T05:30:44.490691+0000","last_undegraded":"2023-11-08T05:30:44.490691+0000","last_fullsized":"2023-11-08T05:28:13.539595+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490805+0000","last_change":"2023-11-08T05:28:13.557968+0000","last_active":"2023-11-08T05:30:44.490805+0000","last_peered":"2023-11-08T05:30:44.490805+0000","last_clean":"2023-11-08T05:28:01.458433+0000","last_became_active":"2023-11-08T05:28:13.557968+0000","last_became_peered":"2023-11-08T05:28:13.557968+0000","last_unstale":"2023-11-08T05:30:44.490805+0000","last_undegraded":"2023-11-08T05:30:44.490805+0000","last_fullsized":"2023-11-08T05:28:13.542832+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":1,"acting":1,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":12,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738851,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":645040,"kb_used_data":5352,"kb_used_omap":275,"kb_used_meta":639404,"kb_avail":93726800,"statfs":{"total":96636764160,"available":95976243200,"internally_reserved":0,"allocated":5480448,"data_stored":2697647,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":281650,"internal_metadata":654750670},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:59 2023","interfaces":[{"interface":"back","average":{"1min":0.30499999999999999,"5min":0.439,"15min":0.36099999999999999},"min":{"1min":0.17399999999999999,"5min":0.14499999999999999,"15min":0.14499999999999999},"max":{"1min":0.60099999999999998,"5min":4.9379999999999997,"15min":4.9379999999999997},"last":0.379},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.49399999999999999,"15min":0.42399999999999999},"min":{"1min":0.221,"5min":0.19600000000000001,"15min":0.17000000000000001},"max":{"1min":0.63200000000000001,"5min":4.9859999999999998,"15min":4.9859999999999998},"last":0.40899999999999997}]},{"osd":1,"last update":"Wed Nov 8 05:32:06 2023","interfaces":[{"interface":"back","average":{"1min":0.27400000000000002,"5min":0.307,"15min":0.46800000000000003},"min":{"1min":0.17599999999999999,"5min":0.16,"15min":0.153},"max":{"1min":0.39000000000000001,"5min":1.825,"15min":23.242999999999999},"last":0.187},{"interface":"front","average":{"1min":0.28000000000000003,"5min":0.34799999999999998,"15min":0.45700000000000002},"min":{"1min":0.20200000000000001,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":0.41699999999999998,"5min":2.0419999999999998,"15min":23.251000000000001},"last":0.20399999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738798,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1339228,"kb_used_data":398304,"kb_used_omap":291,"kb_used_meta":936476,"kb_avail":93032612,"statfs":{"total":96636764160,"available":95265394688,"internally_reserved":0,"allocated":407863296,"data_stored":405103900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":298462,"internal_metadata":958951970},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.436,"15min":0.36299999999999999},"min":{"1min":0.19700000000000001,"5min":0.14699999999999999,"15min":0.13700000000000001},"max":{"1min":3.222,"5min":6.5499999999999998,"15min":6.9989999999999997},"last":0.245},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.46300000000000002,"15min":0.42299999999999999},"min":{"1min":0.17799999999999999,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":5.0030000000000001,"5min":6.0629999999999997,"15min":9.6869999999999994},"last":0.28599999999999998}]},{"osd":2,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.51600000000000001,"15min":0.47099999999999997},"min":{"1min":0.21099999999999999,"5min":0.153,"15min":0.153},"max":{"1min":5.7439999999999998,"5min":5.8280000000000003,"15min":9.7639999999999993},"last":0.252},{"interface":"front","average":{"1min":0.497,"5min":0.56200000000000006,"15min":0.42299999999999999},"min":{"1min":0.20399999999999999,"5min":0.187,"15min":0.16800000000000001},"max":{"1min":3.9319999999999999,"5min":8.8040000000000003,"15min":8.8040000000000003},"last":0.29899999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738853,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":650100,"kb_used_data":5356,"kb_used_omap":214,"kb_used_meta":644521,"kb_avail":93721740,"statfs":{"total":96636764160,"available":95971061760,"internally_reserved":0,"allocated":5484544,"data_stored":2698616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":219335,"internal_metadata":659990329},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:56 2023","interfaces":[{"interface":"back","average":{"1min":0.30399999999999999,"5min":0.34699999999999998,"15min":0.36499999999999999},"min":{"1min":0.23999999999999999,"5min":0.14199999999999999,"15min":0.14199999999999999},"max":{"1min":0.42399999999999999,"5min":1.595,"15min":7.4249999999999998},"last":0.17100000000000001},{"interface":"front","average":{"1min":0.35499999999999998,"5min":0.40899999999999997,"15min":0.376},"min":{"1min":0.23999999999999999,"5min":0.19600000000000001,"15min":0.183},"max":{"1min":0.46999999999999997,"5min":2.6859999999999999,"15min":2.6859999999999999},"last":0.23000000000000001}]},{"osd":2,"last update":"Wed Nov 8 05:32:02 2023","interfaces":[{"interface":"back","average":{"1min":0.252,"5min":0.33000000000000002,"15min":0.36499999999999999},"min":{"1min":0.161,"5min":0.161,"15min":0.161},"max":{"1min":0.39600000000000002,"5min":2.2650000000000001,"15min":5.8209999999999997},"last":0.23999999999999999},{"interface":"front","average":{"1min":0.25800000000000001,"5min":0.29899999999999999,"15min":0.35599999999999998},"min":{"1min":0.17000000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.35799999999999998,"5min":1.137,"15min":4.2350000000000003},"last":0.22800000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-08T05:32:49.818 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-11-08T05:32:50.099 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T05:32:50.101 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2023-11-08T05:32:50.112 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":1659,"stamp":"2023-11-08T05:32:50.085548+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":10,"acting":13,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":109,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2634368,"kb_used_data":409012,"kb_used_omap":780,"kb_used_meta":2220403,"kb_avail":280481152,"statfs":{"total":289910292480,"available":287212699648,"internally_reserved":0,"allocated":418828288,"data_stored":410500163,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":799447,"internal_metadata":2273692969},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-536782902,"num_objects":-164,"num_object_clones":0,"num_object_copies":-328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-115,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-164,"num_whiteouts":0,"num_read":-122056,"num_read_kb":-39011942,"num_write":-413642,"num_write_kb":-151705339,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-8,"num_bytes_recovered":-16922624,"num_keys_recovered":-2,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.424817"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490644+0000","last_change":"2023-11-08T05:28:13.557071+0000","last_active":"2023-11-08T05:30:44.490644+0000","last_peered":"2023-11-08T05:30:44.490644+0000","last_clean":"2023-11-08T05:28:01.458439+0000","last_became_active":"2023-11-08T05:28:13.557071+0000","last_became_peered":"2023-11-08T05:28:13.557071+0000","last_unstale":"2023-11-08T05:30:44.490644+0000","last_undegraded":"2023-11-08T05:30:44.490644+0000","last_fullsized":"2023-11-08T05:28:13.547025+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":459,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:45.794760+0000","last_change":"2023-11-08T05:30:45.794760+0000","last_active":"2023-11-08T05:30:45.794760+0000","last_peered":"2023-11-08T05:30:45.794760+0000","last_clean":"2023-11-08T05:30:45.794760+0000","last_became_active":"2023-11-08T05:30:45.794398+0000","last_became_peered":"2023-11-08T05:30:45.794398+0000","last_unstale":"2023-11-08T05:30:45.794760+0000","last_undegraded":"2023-11-08T05:30:45.794760+0000","last_fullsized":"2023-11-08T05:30:45.794760+0000","mapping_epoch":458,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":459,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":465,"reported_epoch":459,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:45.795085+0000","last_change":"2023-11-08T05:30:45.795085+0000","last_active":"2023-11-08T05:30:45.795085+0000","last_peered":"2023-11-08T05:30:45.795085+0000","last_clean":"2023-11-08T05:30:45.795085+0000","last_became_active":"2023-11-08T05:30:45.794743+0000","last_became_peered":"2023-11-08T05:30:45.794743+0000","last_unstale":"2023-11-08T05:30:45.795085+0000","last_undegraded":"2023-11-08T05:30:45.795085+0000","last_fullsized":"2023-11-08T05:30:45.795085+0000","mapping_epoch":458,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":459,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":454,"reported_epoch":458,"state":"active+clean","last_fresh":"2023-11-08T05:30:44.490618+0000","last_change":"2023-11-08T04:52:47.548597+0000","last_active":"2023-11-08T05:30:44.490618+0000","last_peered":"2023-11-08T05:30:44.490618+0000","last_clean":"2023-11-08T05:30:44.490618+0000","last_became_active":"2023-11-08T04:52:47.548294+0000","last_became_peered":"2023-11-08T04:52:47.548294+0000","last_unstale":"2023-11-08T05:30:44.490618+0000","last_undegraded":"2023-11-08T05:30:44.490618+0000","last_fullsized":"2023-11-08T05:30:44.490618+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":460,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490523+0000","last_change":"2023-11-08T05:28:13.545468+0000","last_active":"2023-11-08T05:30:44.490523+0000","last_peered":"2023-11-08T05:30:44.490523+0000","last_clean":"2023-11-08T05:28:01.456723+0000","last_became_active":"2023-11-08T05:28:13.545468+0000","last_became_peered":"2023-11-08T05:28:13.545468+0000","last_unstale":"2023-11-08T05:30:44.490523+0000","last_undegraded":"2023-11-08T05:30:44.490523+0000","last_fullsized":"2023-11-08T05:28:13.537207+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_clean_scrub_stamp":"2023-11-08T04:52:45.528796+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":465,"reported_epoch":458,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:44.491907+0000","last_change":"2023-11-08T05:29:44.395350+0000","last_active":"2023-11-08T05:30:44.491907+0000","last_peered":"2023-11-08T05:30:44.491907+0000","last_clean":"2023-11-08T05:30:44.491907+0000","last_became_active":"2023-11-08T05:29:44.395012+0000","last_became_peered":"2023-11-08T05:29:44.395012+0000","last_unstale":"2023-11-08T05:30:44.491907+0000","last_undegraded":"2023-11-08T05:30:44.491907+0000","last_fullsized":"2023-11-08T05:30:44.491907+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490629+0000","last_change":"2023-11-08T05:28:13.563357+0000","last_active":"2023-11-08T05:30:44.490629+0000","last_peered":"2023-11-08T05:30:44.490629+0000","last_clean":"2023-11-08T05:28:01.457691+0000","last_became_active":"2023-11-08T05:28:13.563357+0000","last_became_peered":"2023-11-08T05:28:13.563357+0000","last_unstale":"2023-11-08T05:30:44.490629+0000","last_undegraded":"2023-11-08T05:30:44.490629+0000","last_fullsized":"2023-11-08T05:28:13.541614+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490691+0000","last_change":"2023-11-08T05:28:13.550672+0000","last_active":"2023-11-08T05:30:44.490691+0000","last_peered":"2023-11-08T05:30:44.490691+0000","last_clean":"2023-11-08T05:28:01.457904+0000","last_became_active":"2023-11-08T05:28:13.550672+0000","last_became_peered":"2023-11-08T05:28:13.550672+0000","last_unstale":"2023-11-08T05:30:44.490691+0000","last_undegraded":"2023-11-08T05:30:44.490691+0000","last_fullsized":"2023-11-08T05:28:13.539595+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490805+0000","last_change":"2023-11-08T05:28:13.557968+0000","last_active":"2023-11-08T05:30:44.490805+0000","last_peered":"2023-11-08T05:30:44.490805+0000","last_clean":"2023-11-08T05:28:01.458433+0000","last_became_active":"2023-11-08T05:28:13.557968+0000","last_became_peered":"2023-11-08T05:28:13.557968+0000","last_unstale":"2023-11-08T05:30:44.490805+0000","last_undegraded":"2023-11-08T05:30:44.490805+0000","last_fullsized":"2023-11-08T05:28:13.542832+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":1,"acting":1,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":12,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738852,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":645040,"kb_used_data":5352,"kb_used_omap":275,"kb_used_meta":639404,"kb_avail":93726800,"statfs":{"total":96636764160,"available":95976243200,"internally_reserved":0,"allocated":5480448,"data_stored":2697647,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":281650,"internal_metadata":654750670},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:59 2023","interfaces":[{"interface":"back","average":{"1min":0.30499999999999999,"5min":0.439,"15min":0.36099999999999999},"min":{"1min":0.17399999999999999,"5min":0.14499999999999999,"15min":0.14499999999999999},"max":{"1min":0.60099999999999998,"5min":4.9379999999999997,"15min":4.9379999999999997},"last":0.379},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.49399999999999999,"15min":0.42399999999999999},"min":{"1min":0.221,"5min":0.19600000000000001,"15min":0.17000000000000001},"max":{"1min":0.63200000000000001,"5min":4.9859999999999998,"15min":4.9859999999999998},"last":0.40899999999999997}]},{"osd":1,"last update":"Wed Nov 8 05:32:06 2023","interfaces":[{"interface":"back","average":{"1min":0.27400000000000002,"5min":0.307,"15min":0.46800000000000003},"min":{"1min":0.17599999999999999,"5min":0.16,"15min":0.153},"max":{"1min":0.39000000000000001,"5min":1.825,"15min":23.242999999999999},"last":0.28899999999999998},{"interface":"front","average":{"1min":0.28000000000000003,"5min":0.34799999999999998,"15min":0.45700000000000002},"min":{"1min":0.20200000000000001,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":0.41699999999999998,"5min":2.0419999999999998,"15min":23.251000000000001},"last":0.26100000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738798,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1339228,"kb_used_data":398304,"kb_used_omap":291,"kb_used_meta":936476,"kb_avail":93032612,"statfs":{"total":96636764160,"available":95265394688,"internally_reserved":0,"allocated":407863296,"data_stored":405103900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":298462,"internal_metadata":958951970},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.436,"15min":0.36299999999999999},"min":{"1min":0.19700000000000001,"5min":0.14699999999999999,"15min":0.13700000000000001},"max":{"1min":3.222,"5min":6.5499999999999998,"15min":6.9989999999999997},"last":0.245},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.46300000000000002,"15min":0.42299999999999999},"min":{"1min":0.17799999999999999,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":5.0030000000000001,"5min":6.0629999999999997,"15min":9.6869999999999994},"last":0.28599999999999998}]},{"osd":2,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.51600000000000001,"15min":0.47099999999999997},"min":{"1min":0.21099999999999999,"5min":0.153,"15min":0.153},"max":{"1min":5.7439999999999998,"5min":5.8280000000000003,"15min":9.7639999999999993},"last":0.252},{"interface":"front","average":{"1min":0.497,"5min":0.56200000000000006,"15min":0.42299999999999999},"min":{"1min":0.20399999999999999,"5min":0.187,"15min":0.16800000000000001},"max":{"1min":3.9319999999999999,"5min":8.8040000000000003,"15min":8.8040000000000003},"last":0.29899999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738853,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":650100,"kb_used_data":5356,"kb_used_omap":214,"kb_used_meta":644521,"kb_avail":93721740,"statfs":{"total":96636764160,"available":95971061760,"internally_reserved":0,"allocated":5484544,"data_stored":2698616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":219335,"internal_metadata":659990329},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:56 2023","interfaces":[{"interface":"back","average":{"1min":0.30399999999999999,"5min":0.34699999999999998,"15min":0.36499999999999999},"min":{"1min":0.23999999999999999,"5min":0.14199999999999999,"15min":0.14199999999999999},"max":{"1min":0.42399999999999999,"5min":1.595,"15min":7.4249999999999998},"last":0.17100000000000001},{"interface":"front","average":{"1min":0.35499999999999998,"5min":0.40899999999999997,"15min":0.376},"min":{"1min":0.23999999999999999,"5min":0.19600000000000001,"15min":0.183},"max":{"1min":0.46999999999999997,"5min":2.6859999999999999,"15min":2.6859999999999999},"last":0.23000000000000001}]},{"osd":2,"last update":"Wed Nov 8 05:32:02 2023","interfaces":[{"interface":"back","average":{"1min":0.252,"5min":0.33000000000000002,"15min":0.36499999999999999},"min":{"1min":0.161,"5min":0.161,"15min":0.161},"max":{"1min":0.39600000000000002,"5min":2.2650000000000001,"15min":5.8209999999999997},"last":0.23999999999999999},{"interface":"front","average":{"1min":0.25800000000000001,"5min":0.29899999999999999,"15min":0.35599999999999998},"min":{"1min":0.17000000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.35799999999999998,"5min":1.137,"15min":4.2350000000000003},"last":0.22800000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-08T05:32:50.114 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-11-08T05:32:50.114 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 459, 'reported_epoch': 458, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:30:44.490644+0000', 'last_change': '2023-11-08T05:28:13.557071+0000', 'last_active': '2023-11-08T05:30:44.490644+0000', 'last_peered': '2023-11-08T05:30:44.490644+0000', 'last_clean': '2023-11-08T05:28:01.458439+0000', 'last_became_active': '2023-11-08T05:28:13.557071+0000', 'last_became_peered': '2023-11-08T05:28:13.557071+0000', 'last_unstale': '2023-11-08T05:30:44.490644+0000', 'last_undegraded': '2023-11-08T05:30:44.490644+0000', 'last_fullsized': '2023-11-08T05:28:13.547025+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:32:50.114 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-11-08T05:32:50.114 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 11, 'reported_epoch': 459, 'state': 'active+clean+remapped', 'last_fresh': '2023-11-08T05:30:45.794760+0000', 'last_change': '2023-11-08T05:30:45.794760+0000', 'last_active': '2023-11-08T05:30:45.794760+0000', 'last_peered': '2023-11-08T05:30:45.794760+0000', 'last_clean': '2023-11-08T05:30:45.794760+0000', 'last_became_active': '2023-11-08T05:30:45.794398+0000', 'last_became_peered': '2023-11-08T05:30:45.794398+0000', 'last_unstale': '2023-11-08T05:30:45.794760+0000', 'last_undegraded': '2023-11-08T05:30:45.794760+0000', 'last_fullsized': '2023-11-08T05:30:45.794760+0000', 'mapping_epoch': 458, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 459, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2], 'acting': [2, 1], 'avail_no_missing': ['2', '1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2023-11-08T05:32:50.115 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.1 is not active+clean 2023-11-08T05:32:50.115 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.1', 'version': "0'0", 'reported_seq': 465, 'reported_epoch': 459, 'state': 'active+clean+remapped', 'last_fresh': '2023-11-08T05:30:45.795085+0000', 'last_change': '2023-11-08T05:30:45.795085+0000', 'last_active': '2023-11-08T05:30:45.795085+0000', 'last_peered': '2023-11-08T05:30:45.795085+0000', 'last_clean': '2023-11-08T05:30:45.795085+0000', 'last_became_active': '2023-11-08T05:30:45.794743+0000', 'last_became_peered': '2023-11-08T05:30:45.794743+0000', 'last_unstale': '2023-11-08T05:30:45.795085+0000', 'last_undegraded': '2023-11-08T05:30:45.795085+0000', 'last_fullsized': '2023-11-08T05:30:45.795085+0000', 'mapping_epoch': 458, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 459, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2], 'acting': [2, 1], 'avail_no_missing': ['2', '1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2023-11-08T05:32:50.115 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-11-08T05:32:50.116 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "0'0", 'reported_seq': 460, 'reported_epoch': 458, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:30:44.490523+0000', 'last_change': '2023-11-08T05:28:13.545468+0000', 'last_active': '2023-11-08T05:30:44.490523+0000', 'last_peered': '2023-11-08T05:30:44.490523+0000', 'last_clean': '2023-11-08T05:28:01.456723+0000', 'last_became_active': '2023-11-08T05:28:13.545468+0000', 'last_became_peered': '2023-11-08T05:28:13.545468+0000', 'last_unstale': '2023-11-08T05:30:44.490523+0000', 'last_undegraded': '2023-11-08T05:30:44.490523+0000', 'last_fullsized': '2023-11-08T05:28:13.537207+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 9, 'last_epoch_clean': 10, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:45.528796+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:45.528796+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:45.528796+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:32:50.116 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.0 is not active+clean 2023-11-08T05:32:50.116 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.0', 'version': "0'0", 'reported_seq': 465, 'reported_epoch': 458, 'state': 'active+clean+remapped', 'last_fresh': '2023-11-08T05:30:44.491907+0000', 'last_change': '2023-11-08T05:29:44.395350+0000', 'last_active': '2023-11-08T05:30:44.491907+0000', 'last_peered': '2023-11-08T05:30:44.491907+0000', 'last_clean': '2023-11-08T05:30:44.491907+0000', 'last_became_active': '2023-11-08T05:29:44.395012+0000', 'last_became_peered': '2023-11-08T05:29:44.395012+0000', 'last_unstale': '2023-11-08T05:30:44.491907+0000', 'last_undegraded': '2023-11-08T05:30:44.491907+0000', 'last_fullsized': '2023-11-08T05:30:44.491907+0000', 'mapping_epoch': 455, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 456, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2], 'acting': [2, 1], 'avail_no_missing': ['2', '1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2023-11-08T05:32:50.116 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-11-08T05:32:50.117 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 459, 'reported_epoch': 458, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:30:44.490629+0000', 'last_change': '2023-11-08T05:28:13.563357+0000', 'last_active': '2023-11-08T05:30:44.490629+0000', 'last_peered': '2023-11-08T05:30:44.490629+0000', 'last_clean': '2023-11-08T05:28:01.457691+0000', 'last_became_active': '2023-11-08T05:28:13.563357+0000', 'last_became_peered': '2023-11-08T05:28:13.563357+0000', 'last_unstale': '2023-11-08T05:30:44.490629+0000', 'last_undegraded': '2023-11-08T05:30:44.490629+0000', 'last_fullsized': '2023-11-08T05:28:13.541614+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:32:50.117 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-11-08T05:32:50.117 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 459, 'reported_epoch': 458, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:30:44.490691+0000', 'last_change': '2023-11-08T05:28:13.550672+0000', 'last_active': '2023-11-08T05:30:44.490691+0000', 'last_peered': '2023-11-08T05:30:44.490691+0000', 'last_clean': '2023-11-08T05:28:01.457904+0000', 'last_became_active': '2023-11-08T05:28:13.550672+0000', 'last_became_peered': '2023-11-08T05:28:13.550672+0000', 'last_unstale': '2023-11-08T05:30:44.490691+0000', 'last_undegraded': '2023-11-08T05:30:44.490691+0000', 'last_fullsized': '2023-11-08T05:28:13.539595+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:32:50.118 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-11-08T05:32:50.118 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 459, 'reported_epoch': 458, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:30:44.490805+0000', 'last_change': '2023-11-08T05:28:13.557968+0000', 'last_active': '2023-11-08T05:30:44.490805+0000', 'last_peered': '2023-11-08T05:30:44.490805+0000', 'last_clean': '2023-11-08T05:28:01.458433+0000', 'last_became_active': '2023-11-08T05:28:13.557968+0000', 'last_became_peered': '2023-11-08T05:28:13.557968+0000', 'last_unstale': '2023-11-08T05:30:44.490805+0000', 'last_undegraded': '2023-11-08T05:30:44.490805+0000', 'last_fullsized': '2023-11-08T05:28:13.542832+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:32:50.118 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-11-08T05:32:50.463 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T05:32:50.463 INFO:teuthology.orchestra.run.smithi087.stdout:{"fsid":"ab4cbc06-449d-497d-b513-220d9db2b590","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 5 pgs undersized","count":5},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2413,"monmap":{"epoch":1,"min_mon_release_name":"pacific","num_mons":1},"osdmap":{"epoch":459,"num_osds":3,"num_up_osds":2,"osd_up_since":1699421292,"num_in_osds":3,"osd_in_since":1699419158,"num_remapped_pgs":3},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean+remapped","count":3},{"state_name":"active+clean","count":1}],"num_pgs":9,"num_pools":2,"num_objects":0,"data_bytes":0,"bytes_used":2697592832,"bytes_avail":287212699648,"bytes_total":289910292480},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":41,"modified":"2023-11-08T05:29:32.044544+0000","services":{}},"progress_events":{}} 2023-11-08T05:32:50.476 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-11-08T05:32:50.476 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-11-08T05:32:50.759 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T05:32:50.760 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2023-11-08T05:32:50.771 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":1659,"stamp":"2023-11-08T05:32:50.085548+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":10,"acting":13,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":109,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2634368,"kb_used_data":409012,"kb_used_omap":780,"kb_used_meta":2220403,"kb_avail":280481152,"statfs":{"total":289910292480,"available":287212699648,"internally_reserved":0,"allocated":418828288,"data_stored":410500163,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":799447,"internal_metadata":2273692969},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-536782902,"num_objects":-164,"num_object_clones":0,"num_object_copies":-328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-115,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-164,"num_whiteouts":0,"num_read":-122056,"num_read_kb":-39011942,"num_write":-413642,"num_write_kb":-151705339,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-8,"num_bytes_recovered":-16922624,"num_keys_recovered":-2,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.424817"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490644+0000","last_change":"2023-11-08T05:28:13.557071+0000","last_active":"2023-11-08T05:30:44.490644+0000","last_peered":"2023-11-08T05:30:44.490644+0000","last_clean":"2023-11-08T05:28:01.458439+0000","last_became_active":"2023-11-08T05:28:13.557071+0000","last_became_peered":"2023-11-08T05:28:13.557071+0000","last_unstale":"2023-11-08T05:30:44.490644+0000","last_undegraded":"2023-11-08T05:30:44.490644+0000","last_fullsized":"2023-11-08T05:28:13.547025+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":459,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:45.794760+0000","last_change":"2023-11-08T05:30:45.794760+0000","last_active":"2023-11-08T05:30:45.794760+0000","last_peered":"2023-11-08T05:30:45.794760+0000","last_clean":"2023-11-08T05:30:45.794760+0000","last_became_active":"2023-11-08T05:30:45.794398+0000","last_became_peered":"2023-11-08T05:30:45.794398+0000","last_unstale":"2023-11-08T05:30:45.794760+0000","last_undegraded":"2023-11-08T05:30:45.794760+0000","last_fullsized":"2023-11-08T05:30:45.794760+0000","mapping_epoch":458,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":459,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":465,"reported_epoch":459,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:45.795085+0000","last_change":"2023-11-08T05:30:45.795085+0000","last_active":"2023-11-08T05:30:45.795085+0000","last_peered":"2023-11-08T05:30:45.795085+0000","last_clean":"2023-11-08T05:30:45.795085+0000","last_became_active":"2023-11-08T05:30:45.794743+0000","last_became_peered":"2023-11-08T05:30:45.794743+0000","last_unstale":"2023-11-08T05:30:45.795085+0000","last_undegraded":"2023-11-08T05:30:45.795085+0000","last_fullsized":"2023-11-08T05:30:45.795085+0000","mapping_epoch":458,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":459,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":454,"reported_epoch":458,"state":"active+clean","last_fresh":"2023-11-08T05:30:44.490618+0000","last_change":"2023-11-08T04:52:47.548597+0000","last_active":"2023-11-08T05:30:44.490618+0000","last_peered":"2023-11-08T05:30:44.490618+0000","last_clean":"2023-11-08T05:30:44.490618+0000","last_became_active":"2023-11-08T04:52:47.548294+0000","last_became_peered":"2023-11-08T04:52:47.548294+0000","last_unstale":"2023-11-08T05:30:44.490618+0000","last_undegraded":"2023-11-08T05:30:44.490618+0000","last_fullsized":"2023-11-08T05:30:44.490618+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":460,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490523+0000","last_change":"2023-11-08T05:28:13.545468+0000","last_active":"2023-11-08T05:30:44.490523+0000","last_peered":"2023-11-08T05:30:44.490523+0000","last_clean":"2023-11-08T05:28:01.456723+0000","last_became_active":"2023-11-08T05:28:13.545468+0000","last_became_peered":"2023-11-08T05:28:13.545468+0000","last_unstale":"2023-11-08T05:30:44.490523+0000","last_undegraded":"2023-11-08T05:30:44.490523+0000","last_fullsized":"2023-11-08T05:28:13.537207+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_clean_scrub_stamp":"2023-11-08T04:52:45.528796+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":465,"reported_epoch":458,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:44.491907+0000","last_change":"2023-11-08T05:29:44.395350+0000","last_active":"2023-11-08T05:30:44.491907+0000","last_peered":"2023-11-08T05:30:44.491907+0000","last_clean":"2023-11-08T05:30:44.491907+0000","last_became_active":"2023-11-08T05:29:44.395012+0000","last_became_peered":"2023-11-08T05:29:44.395012+0000","last_unstale":"2023-11-08T05:30:44.491907+0000","last_undegraded":"2023-11-08T05:30:44.491907+0000","last_fullsized":"2023-11-08T05:30:44.491907+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490629+0000","last_change":"2023-11-08T05:28:13.563357+0000","last_active":"2023-11-08T05:30:44.490629+0000","last_peered":"2023-11-08T05:30:44.490629+0000","last_clean":"2023-11-08T05:28:01.457691+0000","last_became_active":"2023-11-08T05:28:13.563357+0000","last_became_peered":"2023-11-08T05:28:13.563357+0000","last_unstale":"2023-11-08T05:30:44.490629+0000","last_undegraded":"2023-11-08T05:30:44.490629+0000","last_fullsized":"2023-11-08T05:28:13.541614+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490691+0000","last_change":"2023-11-08T05:28:13.550672+0000","last_active":"2023-11-08T05:30:44.490691+0000","last_peered":"2023-11-08T05:30:44.490691+0000","last_clean":"2023-11-08T05:28:01.457904+0000","last_became_active":"2023-11-08T05:28:13.550672+0000","last_became_peered":"2023-11-08T05:28:13.550672+0000","last_unstale":"2023-11-08T05:30:44.490691+0000","last_undegraded":"2023-11-08T05:30:44.490691+0000","last_fullsized":"2023-11-08T05:28:13.539595+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490805+0000","last_change":"2023-11-08T05:28:13.557968+0000","last_active":"2023-11-08T05:30:44.490805+0000","last_peered":"2023-11-08T05:30:44.490805+0000","last_clean":"2023-11-08T05:28:01.458433+0000","last_became_active":"2023-11-08T05:28:13.557968+0000","last_became_peered":"2023-11-08T05:28:13.557968+0000","last_unstale":"2023-11-08T05:30:44.490805+0000","last_undegraded":"2023-11-08T05:30:44.490805+0000","last_fullsized":"2023-11-08T05:28:13.542832+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":1,"acting":1,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":12,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738852,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":645040,"kb_used_data":5352,"kb_used_omap":275,"kb_used_meta":639404,"kb_avail":93726800,"statfs":{"total":96636764160,"available":95976243200,"internally_reserved":0,"allocated":5480448,"data_stored":2697647,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":281650,"internal_metadata":654750670},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:59 2023","interfaces":[{"interface":"back","average":{"1min":0.30499999999999999,"5min":0.439,"15min":0.36099999999999999},"min":{"1min":0.17399999999999999,"5min":0.14499999999999999,"15min":0.14499999999999999},"max":{"1min":0.60099999999999998,"5min":4.9379999999999997,"15min":4.9379999999999997},"last":0.379},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.49399999999999999,"15min":0.42399999999999999},"min":{"1min":0.221,"5min":0.19600000000000001,"15min":0.17000000000000001},"max":{"1min":0.63200000000000001,"5min":4.9859999999999998,"15min":4.9859999999999998},"last":0.40899999999999997}]},{"osd":1,"last update":"Wed Nov 8 05:32:06 2023","interfaces":[{"interface":"back","average":{"1min":0.27400000000000002,"5min":0.307,"15min":0.46800000000000003},"min":{"1min":0.17599999999999999,"5min":0.16,"15min":0.153},"max":{"1min":0.39000000000000001,"5min":1.825,"15min":23.242999999999999},"last":0.28899999999999998},{"interface":"front","average":{"1min":0.28000000000000003,"5min":0.34799999999999998,"15min":0.45700000000000002},"min":{"1min":0.20200000000000001,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":0.41699999999999998,"5min":2.0419999999999998,"15min":23.251000000000001},"last":0.26100000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738798,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1339228,"kb_used_data":398304,"kb_used_omap":291,"kb_used_meta":936476,"kb_avail":93032612,"statfs":{"total":96636764160,"available":95265394688,"internally_reserved":0,"allocated":407863296,"data_stored":405103900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":298462,"internal_metadata":958951970},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.436,"15min":0.36299999999999999},"min":{"1min":0.19700000000000001,"5min":0.14699999999999999,"15min":0.13700000000000001},"max":{"1min":3.222,"5min":6.5499999999999998,"15min":6.9989999999999997},"last":0.245},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.46300000000000002,"15min":0.42299999999999999},"min":{"1min":0.17799999999999999,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":5.0030000000000001,"5min":6.0629999999999997,"15min":9.6869999999999994},"last":0.28599999999999998}]},{"osd":2,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.51600000000000001,"15min":0.47099999999999997},"min":{"1min":0.21099999999999999,"5min":0.153,"15min":0.153},"max":{"1min":5.7439999999999998,"5min":5.8280000000000003,"15min":9.7639999999999993},"last":0.252},{"interface":"front","average":{"1min":0.497,"5min":0.56200000000000006,"15min":0.42299999999999999},"min":{"1min":0.20399999999999999,"5min":0.187,"15min":0.16800000000000001},"max":{"1min":3.9319999999999999,"5min":8.8040000000000003,"15min":8.8040000000000003},"last":0.29899999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738853,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":650100,"kb_used_data":5356,"kb_used_omap":214,"kb_used_meta":644521,"kb_avail":93721740,"statfs":{"total":96636764160,"available":95971061760,"internally_reserved":0,"allocated":5484544,"data_stored":2698616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":219335,"internal_metadata":659990329},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:56 2023","interfaces":[{"interface":"back","average":{"1min":0.30399999999999999,"5min":0.34699999999999998,"15min":0.36499999999999999},"min":{"1min":0.23999999999999999,"5min":0.14199999999999999,"15min":0.14199999999999999},"max":{"1min":0.42399999999999999,"5min":1.595,"15min":7.4249999999999998},"last":0.17100000000000001},{"interface":"front","average":{"1min":0.35499999999999998,"5min":0.40899999999999997,"15min":0.376},"min":{"1min":0.23999999999999999,"5min":0.19600000000000001,"15min":0.183},"max":{"1min":0.46999999999999997,"5min":2.6859999999999999,"15min":2.6859999999999999},"last":0.23000000000000001}]},{"osd":2,"last update":"Wed Nov 8 05:32:02 2023","interfaces":[{"interface":"back","average":{"1min":0.252,"5min":0.33000000000000002,"15min":0.36499999999999999},"min":{"1min":0.161,"5min":0.161,"15min":0.161},"max":{"1min":0.39600000000000002,"5min":2.2650000000000001,"15min":5.8209999999999997},"last":0.23999999999999999},{"interface":"front","average":{"1min":0.25800000000000001,"5min":0.29899999999999999,"15min":0.35599999999999998},"min":{"1min":0.17000000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.35799999999999998,"5min":1.137,"15min":4.2350000000000003},"last":0.22800000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-08T05:32:53.652 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~276s 2023-11-08T05:32:53.773 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-11-08T05:32:54.070 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T05:32:54.071 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2023-11-08T05:32:54.082 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":1660,"stamp":"2023-11-08T05:32:52.085922+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":10,"acting":13,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":109,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2634368,"kb_used_data":409012,"kb_used_omap":780,"kb_used_meta":2220403,"kb_avail":280481152,"statfs":{"total":289910292480,"available":287212699648,"internally_reserved":0,"allocated":418828288,"data_stored":410500163,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":799447,"internal_metadata":2273692969},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-536782902,"num_objects":-164,"num_object_clones":0,"num_object_copies":-328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-115,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-164,"num_whiteouts":0,"num_read":-122056,"num_read_kb":-39011942,"num_write":-413642,"num_write_kb":-151705339,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-8,"num_bytes_recovered":-16922624,"num_keys_recovered":-2,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.424817"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490644+0000","last_change":"2023-11-08T05:28:13.557071+0000","last_active":"2023-11-08T05:30:44.490644+0000","last_peered":"2023-11-08T05:30:44.490644+0000","last_clean":"2023-11-08T05:28:01.458439+0000","last_became_active":"2023-11-08T05:28:13.557071+0000","last_became_peered":"2023-11-08T05:28:13.557071+0000","last_unstale":"2023-11-08T05:30:44.490644+0000","last_undegraded":"2023-11-08T05:30:44.490644+0000","last_fullsized":"2023-11-08T05:28:13.547025+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":459,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:45.794760+0000","last_change":"2023-11-08T05:30:45.794760+0000","last_active":"2023-11-08T05:30:45.794760+0000","last_peered":"2023-11-08T05:30:45.794760+0000","last_clean":"2023-11-08T05:30:45.794760+0000","last_became_active":"2023-11-08T05:30:45.794398+0000","last_became_peered":"2023-11-08T05:30:45.794398+0000","last_unstale":"2023-11-08T05:30:45.794760+0000","last_undegraded":"2023-11-08T05:30:45.794760+0000","last_fullsized":"2023-11-08T05:30:45.794760+0000","mapping_epoch":458,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":459,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":465,"reported_epoch":459,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:45.795085+0000","last_change":"2023-11-08T05:30:45.795085+0000","last_active":"2023-11-08T05:30:45.795085+0000","last_peered":"2023-11-08T05:30:45.795085+0000","last_clean":"2023-11-08T05:30:45.795085+0000","last_became_active":"2023-11-08T05:30:45.794743+0000","last_became_peered":"2023-11-08T05:30:45.794743+0000","last_unstale":"2023-11-08T05:30:45.795085+0000","last_undegraded":"2023-11-08T05:30:45.795085+0000","last_fullsized":"2023-11-08T05:30:45.795085+0000","mapping_epoch":458,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":459,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":454,"reported_epoch":458,"state":"active+clean","last_fresh":"2023-11-08T05:30:44.490618+0000","last_change":"2023-11-08T04:52:47.548597+0000","last_active":"2023-11-08T05:30:44.490618+0000","last_peered":"2023-11-08T05:30:44.490618+0000","last_clean":"2023-11-08T05:30:44.490618+0000","last_became_active":"2023-11-08T04:52:47.548294+0000","last_became_peered":"2023-11-08T04:52:47.548294+0000","last_unstale":"2023-11-08T05:30:44.490618+0000","last_undegraded":"2023-11-08T05:30:44.490618+0000","last_fullsized":"2023-11-08T05:30:44.490618+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":460,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490523+0000","last_change":"2023-11-08T05:28:13.545468+0000","last_active":"2023-11-08T05:30:44.490523+0000","last_peered":"2023-11-08T05:30:44.490523+0000","last_clean":"2023-11-08T05:28:01.456723+0000","last_became_active":"2023-11-08T05:28:13.545468+0000","last_became_peered":"2023-11-08T05:28:13.545468+0000","last_unstale":"2023-11-08T05:30:44.490523+0000","last_undegraded":"2023-11-08T05:30:44.490523+0000","last_fullsized":"2023-11-08T05:28:13.537207+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_clean_scrub_stamp":"2023-11-08T04:52:45.528796+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":465,"reported_epoch":458,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:44.491907+0000","last_change":"2023-11-08T05:29:44.395350+0000","last_active":"2023-11-08T05:30:44.491907+0000","last_peered":"2023-11-08T05:30:44.491907+0000","last_clean":"2023-11-08T05:30:44.491907+0000","last_became_active":"2023-11-08T05:29:44.395012+0000","last_became_peered":"2023-11-08T05:29:44.395012+0000","last_unstale":"2023-11-08T05:30:44.491907+0000","last_undegraded":"2023-11-08T05:30:44.491907+0000","last_fullsized":"2023-11-08T05:30:44.491907+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490629+0000","last_change":"2023-11-08T05:28:13.563357+0000","last_active":"2023-11-08T05:30:44.490629+0000","last_peered":"2023-11-08T05:30:44.490629+0000","last_clean":"2023-11-08T05:28:01.457691+0000","last_became_active":"2023-11-08T05:28:13.563357+0000","last_became_peered":"2023-11-08T05:28:13.563357+0000","last_unstale":"2023-11-08T05:30:44.490629+0000","last_undegraded":"2023-11-08T05:30:44.490629+0000","last_fullsized":"2023-11-08T05:28:13.541614+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490691+0000","last_change":"2023-11-08T05:28:13.550672+0000","last_active":"2023-11-08T05:30:44.490691+0000","last_peered":"2023-11-08T05:30:44.490691+0000","last_clean":"2023-11-08T05:28:01.457904+0000","last_became_active":"2023-11-08T05:28:13.550672+0000","last_became_peered":"2023-11-08T05:28:13.550672+0000","last_unstale":"2023-11-08T05:30:44.490691+0000","last_undegraded":"2023-11-08T05:30:44.490691+0000","last_fullsized":"2023-11-08T05:28:13.539595+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490805+0000","last_change":"2023-11-08T05:28:13.557968+0000","last_active":"2023-11-08T05:30:44.490805+0000","last_peered":"2023-11-08T05:30:44.490805+0000","last_clean":"2023-11-08T05:28:01.458433+0000","last_became_active":"2023-11-08T05:28:13.557968+0000","last_became_peered":"2023-11-08T05:28:13.557968+0000","last_unstale":"2023-11-08T05:30:44.490805+0000","last_undegraded":"2023-11-08T05:30:44.490805+0000","last_fullsized":"2023-11-08T05:28:13.542832+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":1,"acting":1,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":12,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738852,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":645040,"kb_used_data":5352,"kb_used_omap":275,"kb_used_meta":639404,"kb_avail":93726800,"statfs":{"total":96636764160,"available":95976243200,"internally_reserved":0,"allocated":5480448,"data_stored":2697647,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":281650,"internal_metadata":654750670},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:59 2023","interfaces":[{"interface":"back","average":{"1min":0.30499999999999999,"5min":0.439,"15min":0.36099999999999999},"min":{"1min":0.17399999999999999,"5min":0.14499999999999999,"15min":0.14499999999999999},"max":{"1min":0.60099999999999998,"5min":4.9379999999999997,"15min":4.9379999999999997},"last":0.379},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.49399999999999999,"15min":0.42399999999999999},"min":{"1min":0.221,"5min":0.19600000000000001,"15min":0.17000000000000001},"max":{"1min":0.63200000000000001,"5min":4.9859999999999998,"15min":4.9859999999999998},"last":0.40899999999999997}]},{"osd":1,"last update":"Wed Nov 8 05:32:06 2023","interfaces":[{"interface":"back","average":{"1min":0.27400000000000002,"5min":0.307,"15min":0.46800000000000003},"min":{"1min":0.17599999999999999,"5min":0.16,"15min":0.153},"max":{"1min":0.39000000000000001,"5min":1.825,"15min":23.242999999999999},"last":0.28899999999999998},{"interface":"front","average":{"1min":0.28000000000000003,"5min":0.34799999999999998,"15min":0.45700000000000002},"min":{"1min":0.20200000000000001,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":0.41699999999999998,"5min":2.0419999999999998,"15min":23.251000000000001},"last":0.26100000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738798,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1339228,"kb_used_data":398304,"kb_used_omap":291,"kb_used_meta":936476,"kb_avail":93032612,"statfs":{"total":96636764160,"available":95265394688,"internally_reserved":0,"allocated":407863296,"data_stored":405103900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":298462,"internal_metadata":958951970},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.436,"15min":0.36299999999999999},"min":{"1min":0.19700000000000001,"5min":0.14699999999999999,"15min":0.13700000000000001},"max":{"1min":3.222,"5min":6.5499999999999998,"15min":6.9989999999999997},"last":0.245},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.46300000000000002,"15min":0.42299999999999999},"min":{"1min":0.17799999999999999,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":5.0030000000000001,"5min":6.0629999999999997,"15min":9.6869999999999994},"last":0.28599999999999998}]},{"osd":2,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.51600000000000001,"15min":0.47099999999999997},"min":{"1min":0.21099999999999999,"5min":0.153,"15min":0.153},"max":{"1min":5.7439999999999998,"5min":5.8280000000000003,"15min":9.7639999999999993},"last":0.252},{"interface":"front","average":{"1min":0.497,"5min":0.56200000000000006,"15min":0.42299999999999999},"min":{"1min":0.20399999999999999,"5min":0.187,"15min":0.16800000000000001},"max":{"1min":3.9319999999999999,"5min":8.8040000000000003,"15min":8.8040000000000003},"last":0.29899999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738854,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":650100,"kb_used_data":5356,"kb_used_omap":214,"kb_used_meta":644521,"kb_avail":93721740,"statfs":{"total":96636764160,"available":95971061760,"internally_reserved":0,"allocated":5484544,"data_stored":2698616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":219335,"internal_metadata":659990329},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:56 2023","interfaces":[{"interface":"back","average":{"1min":0.30399999999999999,"5min":0.34699999999999998,"15min":0.36499999999999999},"min":{"1min":0.23999999999999999,"5min":0.14199999999999999,"15min":0.14199999999999999},"max":{"1min":0.42399999999999999,"5min":1.595,"15min":7.4249999999999998},"last":0.17100000000000001},{"interface":"front","average":{"1min":0.35499999999999998,"5min":0.40899999999999997,"15min":0.376},"min":{"1min":0.23999999999999999,"5min":0.19600000000000001,"15min":0.183},"max":{"1min":0.46999999999999997,"5min":2.6859999999999999,"15min":2.6859999999999999},"last":0.23000000000000001}]},{"osd":2,"last update":"Wed Nov 8 05:32:02 2023","interfaces":[{"interface":"back","average":{"1min":0.252,"5min":0.33000000000000002,"15min":0.36499999999999999},"min":{"1min":0.161,"5min":0.161,"15min":0.161},"max":{"1min":0.39600000000000002,"5min":2.2650000000000001,"15min":5.8209999999999997},"last":0.221},{"interface":"front","average":{"1min":0.25800000000000001,"5min":0.29899999999999999,"15min":0.35599999999999998},"min":{"1min":0.17000000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.35799999999999998,"5min":1.137,"15min":4.2350000000000003},"last":0.254}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-08T05:32:54.084 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-11-08T05:32:54.366 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T05:32:54.367 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2023-11-08T05:32:54.378 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":1661,"stamp":"2023-11-08T05:32:54.086282+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":10,"acting":13,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":109,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2634368,"kb_used_data":409012,"kb_used_omap":780,"kb_used_meta":2220403,"kb_avail":280481152,"statfs":{"total":289910292480,"available":287212699648,"internally_reserved":0,"allocated":418828288,"data_stored":410500163,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":799447,"internal_metadata":2273692969},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-536782902,"num_objects":-164,"num_object_clones":0,"num_object_copies":-328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-115,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-164,"num_whiteouts":0,"num_read":-122056,"num_read_kb":-39011942,"num_write":-413642,"num_write_kb":-151705339,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-8,"num_bytes_recovered":-16922624,"num_keys_recovered":-2,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.424817"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490644+0000","last_change":"2023-11-08T05:28:13.557071+0000","last_active":"2023-11-08T05:30:44.490644+0000","last_peered":"2023-11-08T05:30:44.490644+0000","last_clean":"2023-11-08T05:28:01.458439+0000","last_became_active":"2023-11-08T05:28:13.557071+0000","last_became_peered":"2023-11-08T05:28:13.557071+0000","last_unstale":"2023-11-08T05:30:44.490644+0000","last_undegraded":"2023-11-08T05:30:44.490644+0000","last_fullsized":"2023-11-08T05:28:13.547025+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":459,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:45.794760+0000","last_change":"2023-11-08T05:30:45.794760+0000","last_active":"2023-11-08T05:30:45.794760+0000","last_peered":"2023-11-08T05:30:45.794760+0000","last_clean":"2023-11-08T05:30:45.794760+0000","last_became_active":"2023-11-08T05:30:45.794398+0000","last_became_peered":"2023-11-08T05:30:45.794398+0000","last_unstale":"2023-11-08T05:30:45.794760+0000","last_undegraded":"2023-11-08T05:30:45.794760+0000","last_fullsized":"2023-11-08T05:30:45.794760+0000","mapping_epoch":458,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":459,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":465,"reported_epoch":459,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:45.795085+0000","last_change":"2023-11-08T05:30:45.795085+0000","last_active":"2023-11-08T05:30:45.795085+0000","last_peered":"2023-11-08T05:30:45.795085+0000","last_clean":"2023-11-08T05:30:45.795085+0000","last_became_active":"2023-11-08T05:30:45.794743+0000","last_became_peered":"2023-11-08T05:30:45.794743+0000","last_unstale":"2023-11-08T05:30:45.795085+0000","last_undegraded":"2023-11-08T05:30:45.795085+0000","last_fullsized":"2023-11-08T05:30:45.795085+0000","mapping_epoch":458,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":459,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":454,"reported_epoch":458,"state":"active+clean","last_fresh":"2023-11-08T05:30:44.490618+0000","last_change":"2023-11-08T04:52:47.548597+0000","last_active":"2023-11-08T05:30:44.490618+0000","last_peered":"2023-11-08T05:30:44.490618+0000","last_clean":"2023-11-08T05:30:44.490618+0000","last_became_active":"2023-11-08T04:52:47.548294+0000","last_became_peered":"2023-11-08T04:52:47.548294+0000","last_unstale":"2023-11-08T05:30:44.490618+0000","last_undegraded":"2023-11-08T05:30:44.490618+0000","last_fullsized":"2023-11-08T05:30:44.490618+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":460,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490523+0000","last_change":"2023-11-08T05:28:13.545468+0000","last_active":"2023-11-08T05:30:44.490523+0000","last_peered":"2023-11-08T05:30:44.490523+0000","last_clean":"2023-11-08T05:28:01.456723+0000","last_became_active":"2023-11-08T05:28:13.545468+0000","last_became_peered":"2023-11-08T05:28:13.545468+0000","last_unstale":"2023-11-08T05:30:44.490523+0000","last_undegraded":"2023-11-08T05:30:44.490523+0000","last_fullsized":"2023-11-08T05:28:13.537207+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_clean_scrub_stamp":"2023-11-08T04:52:45.528796+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":465,"reported_epoch":458,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:44.491907+0000","last_change":"2023-11-08T05:29:44.395350+0000","last_active":"2023-11-08T05:30:44.491907+0000","last_peered":"2023-11-08T05:30:44.491907+0000","last_clean":"2023-11-08T05:30:44.491907+0000","last_became_active":"2023-11-08T05:29:44.395012+0000","last_became_peered":"2023-11-08T05:29:44.395012+0000","last_unstale":"2023-11-08T05:30:44.491907+0000","last_undegraded":"2023-11-08T05:30:44.491907+0000","last_fullsized":"2023-11-08T05:30:44.491907+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490629+0000","last_change":"2023-11-08T05:28:13.563357+0000","last_active":"2023-11-08T05:30:44.490629+0000","last_peered":"2023-11-08T05:30:44.490629+0000","last_clean":"2023-11-08T05:28:01.457691+0000","last_became_active":"2023-11-08T05:28:13.563357+0000","last_became_peered":"2023-11-08T05:28:13.563357+0000","last_unstale":"2023-11-08T05:30:44.490629+0000","last_undegraded":"2023-11-08T05:30:44.490629+0000","last_fullsized":"2023-11-08T05:28:13.541614+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490691+0000","last_change":"2023-11-08T05:28:13.550672+0000","last_active":"2023-11-08T05:30:44.490691+0000","last_peered":"2023-11-08T05:30:44.490691+0000","last_clean":"2023-11-08T05:28:01.457904+0000","last_became_active":"2023-11-08T05:28:13.550672+0000","last_became_peered":"2023-11-08T05:28:13.550672+0000","last_unstale":"2023-11-08T05:30:44.490691+0000","last_undegraded":"2023-11-08T05:30:44.490691+0000","last_fullsized":"2023-11-08T05:28:13.539595+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490805+0000","last_change":"2023-11-08T05:28:13.557968+0000","last_active":"2023-11-08T05:30:44.490805+0000","last_peered":"2023-11-08T05:30:44.490805+0000","last_clean":"2023-11-08T05:28:01.458433+0000","last_became_active":"2023-11-08T05:28:13.557968+0000","last_became_peered":"2023-11-08T05:28:13.557968+0000","last_unstale":"2023-11-08T05:30:44.490805+0000","last_undegraded":"2023-11-08T05:30:44.490805+0000","last_fullsized":"2023-11-08T05:28:13.542832+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":1,"acting":1,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":12,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738852,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":645040,"kb_used_data":5352,"kb_used_omap":275,"kb_used_meta":639404,"kb_avail":93726800,"statfs":{"total":96636764160,"available":95976243200,"internally_reserved":0,"allocated":5480448,"data_stored":2697647,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":281650,"internal_metadata":654750670},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:59 2023","interfaces":[{"interface":"back","average":{"1min":0.30499999999999999,"5min":0.439,"15min":0.36099999999999999},"min":{"1min":0.17399999999999999,"5min":0.14499999999999999,"15min":0.14499999999999999},"max":{"1min":0.60099999999999998,"5min":4.9379999999999997,"15min":4.9379999999999997},"last":0.379},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.49399999999999999,"15min":0.42399999999999999},"min":{"1min":0.221,"5min":0.19600000000000001,"15min":0.17000000000000001},"max":{"1min":0.63200000000000001,"5min":4.9859999999999998,"15min":4.9859999999999998},"last":0.40899999999999997}]},{"osd":1,"last update":"Wed Nov 8 05:32:06 2023","interfaces":[{"interface":"back","average":{"1min":0.27400000000000002,"5min":0.307,"15min":0.46800000000000003},"min":{"1min":0.17599999999999999,"5min":0.16,"15min":0.153},"max":{"1min":0.39000000000000001,"5min":1.825,"15min":23.242999999999999},"last":0.28899999999999998},{"interface":"front","average":{"1min":0.28000000000000003,"5min":0.34799999999999998,"15min":0.45700000000000002},"min":{"1min":0.20200000000000001,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":0.41699999999999998,"5min":2.0419999999999998,"15min":23.251000000000001},"last":0.26100000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738798,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1339228,"kb_used_data":398304,"kb_used_omap":291,"kb_used_meta":936476,"kb_avail":93032612,"statfs":{"total":96636764160,"available":95265394688,"internally_reserved":0,"allocated":407863296,"data_stored":405103900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":298462,"internal_metadata":958951970},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.436,"15min":0.36299999999999999},"min":{"1min":0.19700000000000001,"5min":0.14699999999999999,"15min":0.13700000000000001},"max":{"1min":3.222,"5min":6.5499999999999998,"15min":6.9989999999999997},"last":0.245},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.46300000000000002,"15min":0.42299999999999999},"min":{"1min":0.17799999999999999,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":5.0030000000000001,"5min":6.0629999999999997,"15min":9.6869999999999994},"last":0.28599999999999998}]},{"osd":2,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.51600000000000001,"15min":0.47099999999999997},"min":{"1min":0.21099999999999999,"5min":0.153,"15min":0.153},"max":{"1min":5.7439999999999998,"5min":5.8280000000000003,"15min":9.7639999999999993},"last":0.252},{"interface":"front","average":{"1min":0.497,"5min":0.56200000000000006,"15min":0.42299999999999999},"min":{"1min":0.20399999999999999,"5min":0.187,"15min":0.16800000000000001},"max":{"1min":3.9319999999999999,"5min":8.8040000000000003,"15min":8.8040000000000003},"last":0.29899999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738854,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":650100,"kb_used_data":5356,"kb_used_omap":214,"kb_used_meta":644521,"kb_avail":93721740,"statfs":{"total":96636764160,"available":95971061760,"internally_reserved":0,"allocated":5484544,"data_stored":2698616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":219335,"internal_metadata":659990329},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:56 2023","interfaces":[{"interface":"back","average":{"1min":0.30399999999999999,"5min":0.34699999999999998,"15min":0.36499999999999999},"min":{"1min":0.23999999999999999,"5min":0.14199999999999999,"15min":0.14199999999999999},"max":{"1min":0.42399999999999999,"5min":1.595,"15min":7.4249999999999998},"last":0.17100000000000001},{"interface":"front","average":{"1min":0.35499999999999998,"5min":0.40899999999999997,"15min":0.376},"min":{"1min":0.23999999999999999,"5min":0.19600000000000001,"15min":0.183},"max":{"1min":0.46999999999999997,"5min":2.6859999999999999,"15min":2.6859999999999999},"last":0.23000000000000001}]},{"osd":2,"last update":"Wed Nov 8 05:32:02 2023","interfaces":[{"interface":"back","average":{"1min":0.252,"5min":0.33000000000000002,"15min":0.36499999999999999},"min":{"1min":0.161,"5min":0.161,"15min":0.161},"max":{"1min":0.39600000000000002,"5min":2.2650000000000001,"15min":5.8209999999999997},"last":0.221},{"interface":"front","average":{"1min":0.25800000000000001,"5min":0.29899999999999999,"15min":0.35599999999999998},"min":{"1min":0.17000000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.35799999999999998,"5min":1.137,"15min":4.2350000000000003},"last":0.254}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-08T05:32:54.379 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-11-08T05:32:54.380 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 459, 'reported_epoch': 458, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:30:44.490644+0000', 'last_change': '2023-11-08T05:28:13.557071+0000', 'last_active': '2023-11-08T05:30:44.490644+0000', 'last_peered': '2023-11-08T05:30:44.490644+0000', 'last_clean': '2023-11-08T05:28:01.458439+0000', 'last_became_active': '2023-11-08T05:28:13.557071+0000', 'last_became_peered': '2023-11-08T05:28:13.557071+0000', 'last_unstale': '2023-11-08T05:30:44.490644+0000', 'last_undegraded': '2023-11-08T05:30:44.490644+0000', 'last_fullsized': '2023-11-08T05:28:13.547025+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:32:54.380 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-11-08T05:32:54.380 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 11, 'reported_epoch': 459, 'state': 'active+clean+remapped', 'last_fresh': '2023-11-08T05:30:45.794760+0000', 'last_change': '2023-11-08T05:30:45.794760+0000', 'last_active': '2023-11-08T05:30:45.794760+0000', 'last_peered': '2023-11-08T05:30:45.794760+0000', 'last_clean': '2023-11-08T05:30:45.794760+0000', 'last_became_active': '2023-11-08T05:30:45.794398+0000', 'last_became_peered': '2023-11-08T05:30:45.794398+0000', 'last_unstale': '2023-11-08T05:30:45.794760+0000', 'last_undegraded': '2023-11-08T05:30:45.794760+0000', 'last_fullsized': '2023-11-08T05:30:45.794760+0000', 'mapping_epoch': 458, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 459, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2], 'acting': [2, 1], 'avail_no_missing': ['2', '1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2023-11-08T05:32:54.381 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.1 is not active+clean 2023-11-08T05:32:54.381 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.1', 'version': "0'0", 'reported_seq': 465, 'reported_epoch': 459, 'state': 'active+clean+remapped', 'last_fresh': '2023-11-08T05:30:45.795085+0000', 'last_change': '2023-11-08T05:30:45.795085+0000', 'last_active': '2023-11-08T05:30:45.795085+0000', 'last_peered': '2023-11-08T05:30:45.795085+0000', 'last_clean': '2023-11-08T05:30:45.795085+0000', 'last_became_active': '2023-11-08T05:30:45.794743+0000', 'last_became_peered': '2023-11-08T05:30:45.794743+0000', 'last_unstale': '2023-11-08T05:30:45.795085+0000', 'last_undegraded': '2023-11-08T05:30:45.795085+0000', 'last_fullsized': '2023-11-08T05:30:45.795085+0000', 'mapping_epoch': 458, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 459, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2], 'acting': [2, 1], 'avail_no_missing': ['2', '1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2023-11-08T05:32:54.381 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-11-08T05:32:54.382 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "0'0", 'reported_seq': 460, 'reported_epoch': 458, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:30:44.490523+0000', 'last_change': '2023-11-08T05:28:13.545468+0000', 'last_active': '2023-11-08T05:30:44.490523+0000', 'last_peered': '2023-11-08T05:30:44.490523+0000', 'last_clean': '2023-11-08T05:28:01.456723+0000', 'last_became_active': '2023-11-08T05:28:13.545468+0000', 'last_became_peered': '2023-11-08T05:28:13.545468+0000', 'last_unstale': '2023-11-08T05:30:44.490523+0000', 'last_undegraded': '2023-11-08T05:30:44.490523+0000', 'last_fullsized': '2023-11-08T05:28:13.537207+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 9, 'last_epoch_clean': 10, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:45.528796+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:45.528796+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:45.528796+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:32:54.382 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.0 is not active+clean 2023-11-08T05:32:54.382 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.0', 'version': "0'0", 'reported_seq': 465, 'reported_epoch': 458, 'state': 'active+clean+remapped', 'last_fresh': '2023-11-08T05:30:44.491907+0000', 'last_change': '2023-11-08T05:29:44.395350+0000', 'last_active': '2023-11-08T05:30:44.491907+0000', 'last_peered': '2023-11-08T05:30:44.491907+0000', 'last_clean': '2023-11-08T05:30:44.491907+0000', 'last_became_active': '2023-11-08T05:29:44.395012+0000', 'last_became_peered': '2023-11-08T05:29:44.395012+0000', 'last_unstale': '2023-11-08T05:30:44.491907+0000', 'last_undegraded': '2023-11-08T05:30:44.491907+0000', 'last_fullsized': '2023-11-08T05:30:44.491907+0000', 'mapping_epoch': 455, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 456, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2], 'acting': [2, 1], 'avail_no_missing': ['2', '1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2023-11-08T05:32:54.383 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-11-08T05:32:54.383 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 459, 'reported_epoch': 458, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:30:44.490629+0000', 'last_change': '2023-11-08T05:28:13.563357+0000', 'last_active': '2023-11-08T05:30:44.490629+0000', 'last_peered': '2023-11-08T05:30:44.490629+0000', 'last_clean': '2023-11-08T05:28:01.457691+0000', 'last_became_active': '2023-11-08T05:28:13.563357+0000', 'last_became_peered': '2023-11-08T05:28:13.563357+0000', 'last_unstale': '2023-11-08T05:30:44.490629+0000', 'last_undegraded': '2023-11-08T05:30:44.490629+0000', 'last_fullsized': '2023-11-08T05:28:13.541614+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:32:54.383 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-11-08T05:32:54.384 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 459, 'reported_epoch': 458, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:30:44.490691+0000', 'last_change': '2023-11-08T05:28:13.550672+0000', 'last_active': '2023-11-08T05:30:44.490691+0000', 'last_peered': '2023-11-08T05:30:44.490691+0000', 'last_clean': '2023-11-08T05:28:01.457904+0000', 'last_became_active': '2023-11-08T05:28:13.550672+0000', 'last_became_peered': '2023-11-08T05:28:13.550672+0000', 'last_unstale': '2023-11-08T05:30:44.490691+0000', 'last_undegraded': '2023-11-08T05:30:44.490691+0000', 'last_fullsized': '2023-11-08T05:28:13.539595+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:32:54.384 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-11-08T05:32:54.384 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 459, 'reported_epoch': 458, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:30:44.490805+0000', 'last_change': '2023-11-08T05:28:13.557968+0000', 'last_active': '2023-11-08T05:30:44.490805+0000', 'last_peered': '2023-11-08T05:30:44.490805+0000', 'last_clean': '2023-11-08T05:28:01.458433+0000', 'last_became_active': '2023-11-08T05:28:13.557968+0000', 'last_became_peered': '2023-11-08T05:28:13.557968+0000', 'last_unstale': '2023-11-08T05:30:44.490805+0000', 'last_undegraded': '2023-11-08T05:30:44.490805+0000', 'last_fullsized': '2023-11-08T05:28:13.542832+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:32:54.385 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-11-08T05:32:54.734 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T05:32:54.734 INFO:teuthology.orchestra.run.smithi087.stdout:{"fsid":"ab4cbc06-449d-497d-b513-220d9db2b590","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 5 pgs undersized","count":5},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2417,"monmap":{"epoch":1,"min_mon_release_name":"pacific","num_mons":1},"osdmap":{"epoch":459,"num_osds":3,"num_up_osds":2,"osd_up_since":1699421292,"num_in_osds":3,"osd_in_since":1699419158,"num_remapped_pgs":3},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean+remapped","count":3},{"state_name":"active+clean","count":1}],"num_pgs":9,"num_pools":2,"num_objects":0,"data_bytes":0,"bytes_used":2697592832,"bytes_avail":287212699648,"bytes_total":289910292480},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":41,"modified":"2023-11-08T05:29:32.044544+0000","services":{}},"progress_events":{}} 2023-11-08T05:32:54.748 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-11-08T05:32:54.748 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-11-08T05:32:55.032 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T05:32:55.034 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2023-11-08T05:32:55.045 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":1661,"stamp":"2023-11-08T05:32:54.086282+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":10,"acting":13,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":109,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2634368,"kb_used_data":409012,"kb_used_omap":780,"kb_used_meta":2220403,"kb_avail":280481152,"statfs":{"total":289910292480,"available":287212699648,"internally_reserved":0,"allocated":418828288,"data_stored":410500163,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":799447,"internal_metadata":2273692969},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-536782902,"num_objects":-164,"num_object_clones":0,"num_object_copies":-328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-115,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-164,"num_whiteouts":0,"num_read":-122056,"num_read_kb":-39011942,"num_write":-413642,"num_write_kb":-151705339,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-8,"num_bytes_recovered":-16922624,"num_keys_recovered":-2,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.424817"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490644+0000","last_change":"2023-11-08T05:28:13.557071+0000","last_active":"2023-11-08T05:30:44.490644+0000","last_peered":"2023-11-08T05:30:44.490644+0000","last_clean":"2023-11-08T05:28:01.458439+0000","last_became_active":"2023-11-08T05:28:13.557071+0000","last_became_peered":"2023-11-08T05:28:13.557071+0000","last_unstale":"2023-11-08T05:30:44.490644+0000","last_undegraded":"2023-11-08T05:30:44.490644+0000","last_fullsized":"2023-11-08T05:28:13.547025+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":459,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:45.794760+0000","last_change":"2023-11-08T05:30:45.794760+0000","last_active":"2023-11-08T05:30:45.794760+0000","last_peered":"2023-11-08T05:30:45.794760+0000","last_clean":"2023-11-08T05:30:45.794760+0000","last_became_active":"2023-11-08T05:30:45.794398+0000","last_became_peered":"2023-11-08T05:30:45.794398+0000","last_unstale":"2023-11-08T05:30:45.794760+0000","last_undegraded":"2023-11-08T05:30:45.794760+0000","last_fullsized":"2023-11-08T05:30:45.794760+0000","mapping_epoch":458,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":459,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":465,"reported_epoch":459,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:45.795085+0000","last_change":"2023-11-08T05:30:45.795085+0000","last_active":"2023-11-08T05:30:45.795085+0000","last_peered":"2023-11-08T05:30:45.795085+0000","last_clean":"2023-11-08T05:30:45.795085+0000","last_became_active":"2023-11-08T05:30:45.794743+0000","last_became_peered":"2023-11-08T05:30:45.794743+0000","last_unstale":"2023-11-08T05:30:45.795085+0000","last_undegraded":"2023-11-08T05:30:45.795085+0000","last_fullsized":"2023-11-08T05:30:45.795085+0000","mapping_epoch":458,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":459,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":454,"reported_epoch":458,"state":"active+clean","last_fresh":"2023-11-08T05:30:44.490618+0000","last_change":"2023-11-08T04:52:47.548597+0000","last_active":"2023-11-08T05:30:44.490618+0000","last_peered":"2023-11-08T05:30:44.490618+0000","last_clean":"2023-11-08T05:30:44.490618+0000","last_became_active":"2023-11-08T04:52:47.548294+0000","last_became_peered":"2023-11-08T04:52:47.548294+0000","last_unstale":"2023-11-08T05:30:44.490618+0000","last_undegraded":"2023-11-08T05:30:44.490618+0000","last_fullsized":"2023-11-08T05:30:44.490618+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":460,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490523+0000","last_change":"2023-11-08T05:28:13.545468+0000","last_active":"2023-11-08T05:30:44.490523+0000","last_peered":"2023-11-08T05:30:44.490523+0000","last_clean":"2023-11-08T05:28:01.456723+0000","last_became_active":"2023-11-08T05:28:13.545468+0000","last_became_peered":"2023-11-08T05:28:13.545468+0000","last_unstale":"2023-11-08T05:30:44.490523+0000","last_undegraded":"2023-11-08T05:30:44.490523+0000","last_fullsized":"2023-11-08T05:28:13.537207+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_clean_scrub_stamp":"2023-11-08T04:52:45.528796+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":465,"reported_epoch":458,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:44.491907+0000","last_change":"2023-11-08T05:29:44.395350+0000","last_active":"2023-11-08T05:30:44.491907+0000","last_peered":"2023-11-08T05:30:44.491907+0000","last_clean":"2023-11-08T05:30:44.491907+0000","last_became_active":"2023-11-08T05:29:44.395012+0000","last_became_peered":"2023-11-08T05:29:44.395012+0000","last_unstale":"2023-11-08T05:30:44.491907+0000","last_undegraded":"2023-11-08T05:30:44.491907+0000","last_fullsized":"2023-11-08T05:30:44.491907+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490629+0000","last_change":"2023-11-08T05:28:13.563357+0000","last_active":"2023-11-08T05:30:44.490629+0000","last_peered":"2023-11-08T05:30:44.490629+0000","last_clean":"2023-11-08T05:28:01.457691+0000","last_became_active":"2023-11-08T05:28:13.563357+0000","last_became_peered":"2023-11-08T05:28:13.563357+0000","last_unstale":"2023-11-08T05:30:44.490629+0000","last_undegraded":"2023-11-08T05:30:44.490629+0000","last_fullsized":"2023-11-08T05:28:13.541614+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490691+0000","last_change":"2023-11-08T05:28:13.550672+0000","last_active":"2023-11-08T05:30:44.490691+0000","last_peered":"2023-11-08T05:30:44.490691+0000","last_clean":"2023-11-08T05:28:01.457904+0000","last_became_active":"2023-11-08T05:28:13.550672+0000","last_became_peered":"2023-11-08T05:28:13.550672+0000","last_unstale":"2023-11-08T05:30:44.490691+0000","last_undegraded":"2023-11-08T05:30:44.490691+0000","last_fullsized":"2023-11-08T05:28:13.539595+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490805+0000","last_change":"2023-11-08T05:28:13.557968+0000","last_active":"2023-11-08T05:30:44.490805+0000","last_peered":"2023-11-08T05:30:44.490805+0000","last_clean":"2023-11-08T05:28:01.458433+0000","last_became_active":"2023-11-08T05:28:13.557968+0000","last_became_peered":"2023-11-08T05:28:13.557968+0000","last_unstale":"2023-11-08T05:30:44.490805+0000","last_undegraded":"2023-11-08T05:30:44.490805+0000","last_fullsized":"2023-11-08T05:28:13.542832+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":1,"acting":1,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":12,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738852,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":645040,"kb_used_data":5352,"kb_used_omap":275,"kb_used_meta":639404,"kb_avail":93726800,"statfs":{"total":96636764160,"available":95976243200,"internally_reserved":0,"allocated":5480448,"data_stored":2697647,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":281650,"internal_metadata":654750670},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:59 2023","interfaces":[{"interface":"back","average":{"1min":0.30499999999999999,"5min":0.439,"15min":0.36099999999999999},"min":{"1min":0.17399999999999999,"5min":0.14499999999999999,"15min":0.14499999999999999},"max":{"1min":0.60099999999999998,"5min":4.9379999999999997,"15min":4.9379999999999997},"last":0.379},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.49399999999999999,"15min":0.42399999999999999},"min":{"1min":0.221,"5min":0.19600000000000001,"15min":0.17000000000000001},"max":{"1min":0.63200000000000001,"5min":4.9859999999999998,"15min":4.9859999999999998},"last":0.40899999999999997}]},{"osd":1,"last update":"Wed Nov 8 05:32:06 2023","interfaces":[{"interface":"back","average":{"1min":0.27400000000000002,"5min":0.307,"15min":0.46800000000000003},"min":{"1min":0.17599999999999999,"5min":0.16,"15min":0.153},"max":{"1min":0.39000000000000001,"5min":1.825,"15min":23.242999999999999},"last":0.28899999999999998},{"interface":"front","average":{"1min":0.28000000000000003,"5min":0.34799999999999998,"15min":0.45700000000000002},"min":{"1min":0.20200000000000001,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":0.41699999999999998,"5min":2.0419999999999998,"15min":23.251000000000001},"last":0.26100000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738798,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1339228,"kb_used_data":398304,"kb_used_omap":291,"kb_used_meta":936476,"kb_avail":93032612,"statfs":{"total":96636764160,"available":95265394688,"internally_reserved":0,"allocated":407863296,"data_stored":405103900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":298462,"internal_metadata":958951970},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.436,"15min":0.36299999999999999},"min":{"1min":0.19700000000000001,"5min":0.14699999999999999,"15min":0.13700000000000001},"max":{"1min":3.222,"5min":6.5499999999999998,"15min":6.9989999999999997},"last":0.245},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.46300000000000002,"15min":0.42299999999999999},"min":{"1min":0.17799999999999999,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":5.0030000000000001,"5min":6.0629999999999997,"15min":9.6869999999999994},"last":0.28599999999999998}]},{"osd":2,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.51600000000000001,"15min":0.47099999999999997},"min":{"1min":0.21099999999999999,"5min":0.153,"15min":0.153},"max":{"1min":5.7439999999999998,"5min":5.8280000000000003,"15min":9.7639999999999993},"last":0.252},{"interface":"front","average":{"1min":0.497,"5min":0.56200000000000006,"15min":0.42299999999999999},"min":{"1min":0.20399999999999999,"5min":0.187,"15min":0.16800000000000001},"max":{"1min":3.9319999999999999,"5min":8.8040000000000003,"15min":8.8040000000000003},"last":0.29899999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738854,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":650100,"kb_used_data":5356,"kb_used_omap":214,"kb_used_meta":644521,"kb_avail":93721740,"statfs":{"total":96636764160,"available":95971061760,"internally_reserved":0,"allocated":5484544,"data_stored":2698616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":219335,"internal_metadata":659990329},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:56 2023","interfaces":[{"interface":"back","average":{"1min":0.30399999999999999,"5min":0.34699999999999998,"15min":0.36499999999999999},"min":{"1min":0.23999999999999999,"5min":0.14199999999999999,"15min":0.14199999999999999},"max":{"1min":0.42399999999999999,"5min":1.595,"15min":7.4249999999999998},"last":0.17100000000000001},{"interface":"front","average":{"1min":0.35499999999999998,"5min":0.40899999999999997,"15min":0.376},"min":{"1min":0.23999999999999999,"5min":0.19600000000000001,"15min":0.183},"max":{"1min":0.46999999999999997,"5min":2.6859999999999999,"15min":2.6859999999999999},"last":0.23000000000000001}]},{"osd":2,"last update":"Wed Nov 8 05:32:02 2023","interfaces":[{"interface":"back","average":{"1min":0.252,"5min":0.33000000000000002,"15min":0.36499999999999999},"min":{"1min":0.161,"5min":0.161,"15min":0.161},"max":{"1min":0.39600000000000002,"5min":2.2650000000000001,"15min":5.8209999999999997},"last":0.221},{"interface":"front","average":{"1min":0.25800000000000001,"5min":0.29899999999999999,"15min":0.35599999999999998},"min":{"1min":0.17000000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.35799999999999998,"5min":1.137,"15min":4.2350000000000003},"last":0.254}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-08T05:32:58.047 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-11-08T05:32:58.343 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T05:32:58.345 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2023-11-08T05:32:58.356 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":1663,"stamp":"2023-11-08T05:32:58.087014+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":10,"acting":13,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":109,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2634368,"kb_used_data":409012,"kb_used_omap":780,"kb_used_meta":2220403,"kb_avail":280481152,"statfs":{"total":289910292480,"available":287212699648,"internally_reserved":0,"allocated":418828288,"data_stored":410500163,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":799447,"internal_metadata":2273692969},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-536782902,"num_objects":-164,"num_object_clones":0,"num_object_copies":-328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-115,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-164,"num_whiteouts":0,"num_read":-122056,"num_read_kb":-39011942,"num_write":-413642,"num_write_kb":-151705339,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-8,"num_bytes_recovered":-16922624,"num_keys_recovered":-2,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.424817"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490644+0000","last_change":"2023-11-08T05:28:13.557071+0000","last_active":"2023-11-08T05:30:44.490644+0000","last_peered":"2023-11-08T05:30:44.490644+0000","last_clean":"2023-11-08T05:28:01.458439+0000","last_became_active":"2023-11-08T05:28:13.557071+0000","last_became_peered":"2023-11-08T05:28:13.557071+0000","last_unstale":"2023-11-08T05:30:44.490644+0000","last_undegraded":"2023-11-08T05:30:44.490644+0000","last_fullsized":"2023-11-08T05:28:13.547025+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":459,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:45.794760+0000","last_change":"2023-11-08T05:30:45.794760+0000","last_active":"2023-11-08T05:30:45.794760+0000","last_peered":"2023-11-08T05:30:45.794760+0000","last_clean":"2023-11-08T05:30:45.794760+0000","last_became_active":"2023-11-08T05:30:45.794398+0000","last_became_peered":"2023-11-08T05:30:45.794398+0000","last_unstale":"2023-11-08T05:30:45.794760+0000","last_undegraded":"2023-11-08T05:30:45.794760+0000","last_fullsized":"2023-11-08T05:30:45.794760+0000","mapping_epoch":458,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":459,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":465,"reported_epoch":459,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:45.795085+0000","last_change":"2023-11-08T05:30:45.795085+0000","last_active":"2023-11-08T05:30:45.795085+0000","last_peered":"2023-11-08T05:30:45.795085+0000","last_clean":"2023-11-08T05:30:45.795085+0000","last_became_active":"2023-11-08T05:30:45.794743+0000","last_became_peered":"2023-11-08T05:30:45.794743+0000","last_unstale":"2023-11-08T05:30:45.795085+0000","last_undegraded":"2023-11-08T05:30:45.795085+0000","last_fullsized":"2023-11-08T05:30:45.795085+0000","mapping_epoch":458,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":459,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":454,"reported_epoch":458,"state":"active+clean","last_fresh":"2023-11-08T05:30:44.490618+0000","last_change":"2023-11-08T04:52:47.548597+0000","last_active":"2023-11-08T05:30:44.490618+0000","last_peered":"2023-11-08T05:30:44.490618+0000","last_clean":"2023-11-08T05:30:44.490618+0000","last_became_active":"2023-11-08T04:52:47.548294+0000","last_became_peered":"2023-11-08T04:52:47.548294+0000","last_unstale":"2023-11-08T05:30:44.490618+0000","last_undegraded":"2023-11-08T05:30:44.490618+0000","last_fullsized":"2023-11-08T05:30:44.490618+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":460,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490523+0000","last_change":"2023-11-08T05:28:13.545468+0000","last_active":"2023-11-08T05:30:44.490523+0000","last_peered":"2023-11-08T05:30:44.490523+0000","last_clean":"2023-11-08T05:28:01.456723+0000","last_became_active":"2023-11-08T05:28:13.545468+0000","last_became_peered":"2023-11-08T05:28:13.545468+0000","last_unstale":"2023-11-08T05:30:44.490523+0000","last_undegraded":"2023-11-08T05:30:44.490523+0000","last_fullsized":"2023-11-08T05:28:13.537207+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_clean_scrub_stamp":"2023-11-08T04:52:45.528796+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":465,"reported_epoch":458,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:44.491907+0000","last_change":"2023-11-08T05:29:44.395350+0000","last_active":"2023-11-08T05:30:44.491907+0000","last_peered":"2023-11-08T05:30:44.491907+0000","last_clean":"2023-11-08T05:30:44.491907+0000","last_became_active":"2023-11-08T05:29:44.395012+0000","last_became_peered":"2023-11-08T05:29:44.395012+0000","last_unstale":"2023-11-08T05:30:44.491907+0000","last_undegraded":"2023-11-08T05:30:44.491907+0000","last_fullsized":"2023-11-08T05:30:44.491907+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490629+0000","last_change":"2023-11-08T05:28:13.563357+0000","last_active":"2023-11-08T05:30:44.490629+0000","last_peered":"2023-11-08T05:30:44.490629+0000","last_clean":"2023-11-08T05:28:01.457691+0000","last_became_active":"2023-11-08T05:28:13.563357+0000","last_became_peered":"2023-11-08T05:28:13.563357+0000","last_unstale":"2023-11-08T05:30:44.490629+0000","last_undegraded":"2023-11-08T05:30:44.490629+0000","last_fullsized":"2023-11-08T05:28:13.541614+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490691+0000","last_change":"2023-11-08T05:28:13.550672+0000","last_active":"2023-11-08T05:30:44.490691+0000","last_peered":"2023-11-08T05:30:44.490691+0000","last_clean":"2023-11-08T05:28:01.457904+0000","last_became_active":"2023-11-08T05:28:13.550672+0000","last_became_peered":"2023-11-08T05:28:13.550672+0000","last_unstale":"2023-11-08T05:30:44.490691+0000","last_undegraded":"2023-11-08T05:30:44.490691+0000","last_fullsized":"2023-11-08T05:28:13.539595+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490805+0000","last_change":"2023-11-08T05:28:13.557968+0000","last_active":"2023-11-08T05:30:44.490805+0000","last_peered":"2023-11-08T05:30:44.490805+0000","last_clean":"2023-11-08T05:28:01.458433+0000","last_became_active":"2023-11-08T05:28:13.557968+0000","last_became_peered":"2023-11-08T05:28:13.557968+0000","last_unstale":"2023-11-08T05:30:44.490805+0000","last_undegraded":"2023-11-08T05:30:44.490805+0000","last_fullsized":"2023-11-08T05:28:13.542832+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":1,"acting":1,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":12,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738853,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":645040,"kb_used_data":5352,"kb_used_omap":275,"kb_used_meta":639404,"kb_avail":93726800,"statfs":{"total":96636764160,"available":95976243200,"internally_reserved":0,"allocated":5480448,"data_stored":2697647,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":281650,"internal_metadata":654750670},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:59 2023","interfaces":[{"interface":"back","average":{"1min":0.30499999999999999,"5min":0.439,"15min":0.36099999999999999},"min":{"1min":0.17399999999999999,"5min":0.14499999999999999,"15min":0.14499999999999999},"max":{"1min":0.60099999999999998,"5min":4.9379999999999997,"15min":4.9379999999999997},"last":0.379},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.49399999999999999,"15min":0.42399999999999999},"min":{"1min":0.221,"5min":0.19600000000000001,"15min":0.17000000000000001},"max":{"1min":0.63200000000000001,"5min":4.9859999999999998,"15min":4.9859999999999998},"last":0.40899999999999997}]},{"osd":1,"last update":"Wed Nov 8 05:32:06 2023","interfaces":[{"interface":"back","average":{"1min":0.27400000000000002,"5min":0.307,"15min":0.46800000000000003},"min":{"1min":0.17599999999999999,"5min":0.16,"15min":0.153},"max":{"1min":0.39000000000000001,"5min":1.825,"15min":23.242999999999999},"last":0.23300000000000001},{"interface":"front","average":{"1min":0.28000000000000003,"5min":0.34799999999999998,"15min":0.45700000000000002},"min":{"1min":0.20200000000000001,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":0.41699999999999998,"5min":2.0419999999999998,"15min":23.251000000000001},"last":0.26500000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738798,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1339228,"kb_used_data":398304,"kb_used_omap":291,"kb_used_meta":936476,"kb_avail":93032612,"statfs":{"total":96636764160,"available":95265394688,"internally_reserved":0,"allocated":407863296,"data_stored":405103900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":298462,"internal_metadata":958951970},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.436,"15min":0.36299999999999999},"min":{"1min":0.19700000000000001,"5min":0.14699999999999999,"15min":0.13700000000000001},"max":{"1min":3.222,"5min":6.5499999999999998,"15min":6.9989999999999997},"last":0.245},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.46300000000000002,"15min":0.42299999999999999},"min":{"1min":0.17799999999999999,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":5.0030000000000001,"5min":6.0629999999999997,"15min":9.6869999999999994},"last":0.28599999999999998}]},{"osd":2,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.51600000000000001,"15min":0.47099999999999997},"min":{"1min":0.21099999999999999,"5min":0.153,"15min":0.153},"max":{"1min":5.7439999999999998,"5min":5.8280000000000003,"15min":9.7639999999999993},"last":0.252},{"interface":"front","average":{"1min":0.497,"5min":0.56200000000000006,"15min":0.42299999999999999},"min":{"1min":0.20399999999999999,"5min":0.187,"15min":0.16800000000000001},"max":{"1min":3.9319999999999999,"5min":8.8040000000000003,"15min":8.8040000000000003},"last":0.29899999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738855,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":650100,"kb_used_data":5356,"kb_used_omap":214,"kb_used_meta":644521,"kb_avail":93721740,"statfs":{"total":96636764160,"available":95971061760,"internally_reserved":0,"allocated":5484544,"data_stored":2698616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":219335,"internal_metadata":659990329},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:56 2023","interfaces":[{"interface":"back","average":{"1min":0.30399999999999999,"5min":0.34699999999999998,"15min":0.36499999999999999},"min":{"1min":0.23999999999999999,"5min":0.14199999999999999,"15min":0.14199999999999999},"max":{"1min":0.42399999999999999,"5min":1.595,"15min":7.4249999999999998},"last":0.17100000000000001},{"interface":"front","average":{"1min":0.35499999999999998,"5min":0.40899999999999997,"15min":0.376},"min":{"1min":0.23999999999999999,"5min":0.19600000000000001,"15min":0.183},"max":{"1min":0.46999999999999997,"5min":2.6859999999999999,"15min":2.6859999999999999},"last":0.23000000000000001}]},{"osd":2,"last update":"Wed Nov 8 05:32:02 2023","interfaces":[{"interface":"back","average":{"1min":0.252,"5min":0.33000000000000002,"15min":0.36499999999999999},"min":{"1min":0.161,"5min":0.161,"15min":0.161},"max":{"1min":0.39600000000000002,"5min":2.2650000000000001,"15min":5.8209999999999997},"last":0.221},{"interface":"front","average":{"1min":0.25800000000000001,"5min":0.29899999999999999,"15min":0.35599999999999998},"min":{"1min":0.17000000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.35799999999999998,"5min":1.137,"15min":4.2350000000000003},"last":0.254}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-08T05:32:58.357 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-11-08T05:32:58.639 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T05:32:58.641 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2023-11-08T05:32:58.652 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":1663,"stamp":"2023-11-08T05:32:58.087014+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":10,"acting":13,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":109,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2634368,"kb_used_data":409012,"kb_used_omap":780,"kb_used_meta":2220403,"kb_avail":280481152,"statfs":{"total":289910292480,"available":287212699648,"internally_reserved":0,"allocated":418828288,"data_stored":410500163,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":799447,"internal_metadata":2273692969},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-536782902,"num_objects":-164,"num_object_clones":0,"num_object_copies":-328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-115,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-164,"num_whiteouts":0,"num_read":-122056,"num_read_kb":-39011942,"num_write":-413642,"num_write_kb":-151705339,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-8,"num_bytes_recovered":-16922624,"num_keys_recovered":-2,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.424817"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490644+0000","last_change":"2023-11-08T05:28:13.557071+0000","last_active":"2023-11-08T05:30:44.490644+0000","last_peered":"2023-11-08T05:30:44.490644+0000","last_clean":"2023-11-08T05:28:01.458439+0000","last_became_active":"2023-11-08T05:28:13.557071+0000","last_became_peered":"2023-11-08T05:28:13.557071+0000","last_unstale":"2023-11-08T05:30:44.490644+0000","last_undegraded":"2023-11-08T05:30:44.490644+0000","last_fullsized":"2023-11-08T05:28:13.547025+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":459,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:45.794760+0000","last_change":"2023-11-08T05:30:45.794760+0000","last_active":"2023-11-08T05:30:45.794760+0000","last_peered":"2023-11-08T05:30:45.794760+0000","last_clean":"2023-11-08T05:30:45.794760+0000","last_became_active":"2023-11-08T05:30:45.794398+0000","last_became_peered":"2023-11-08T05:30:45.794398+0000","last_unstale":"2023-11-08T05:30:45.794760+0000","last_undegraded":"2023-11-08T05:30:45.794760+0000","last_fullsized":"2023-11-08T05:30:45.794760+0000","mapping_epoch":458,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":459,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":465,"reported_epoch":459,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:45.795085+0000","last_change":"2023-11-08T05:30:45.795085+0000","last_active":"2023-11-08T05:30:45.795085+0000","last_peered":"2023-11-08T05:30:45.795085+0000","last_clean":"2023-11-08T05:30:45.795085+0000","last_became_active":"2023-11-08T05:30:45.794743+0000","last_became_peered":"2023-11-08T05:30:45.794743+0000","last_unstale":"2023-11-08T05:30:45.795085+0000","last_undegraded":"2023-11-08T05:30:45.795085+0000","last_fullsized":"2023-11-08T05:30:45.795085+0000","mapping_epoch":458,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":459,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":454,"reported_epoch":458,"state":"active+clean","last_fresh":"2023-11-08T05:30:44.490618+0000","last_change":"2023-11-08T04:52:47.548597+0000","last_active":"2023-11-08T05:30:44.490618+0000","last_peered":"2023-11-08T05:30:44.490618+0000","last_clean":"2023-11-08T05:30:44.490618+0000","last_became_active":"2023-11-08T04:52:47.548294+0000","last_became_peered":"2023-11-08T04:52:47.548294+0000","last_unstale":"2023-11-08T05:30:44.490618+0000","last_undegraded":"2023-11-08T05:30:44.490618+0000","last_fullsized":"2023-11-08T05:30:44.490618+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":460,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490523+0000","last_change":"2023-11-08T05:28:13.545468+0000","last_active":"2023-11-08T05:30:44.490523+0000","last_peered":"2023-11-08T05:30:44.490523+0000","last_clean":"2023-11-08T05:28:01.456723+0000","last_became_active":"2023-11-08T05:28:13.545468+0000","last_became_peered":"2023-11-08T05:28:13.545468+0000","last_unstale":"2023-11-08T05:30:44.490523+0000","last_undegraded":"2023-11-08T05:30:44.490523+0000","last_fullsized":"2023-11-08T05:28:13.537207+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_clean_scrub_stamp":"2023-11-08T04:52:45.528796+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":465,"reported_epoch":458,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:44.491907+0000","last_change":"2023-11-08T05:29:44.395350+0000","last_active":"2023-11-08T05:30:44.491907+0000","last_peered":"2023-11-08T05:30:44.491907+0000","last_clean":"2023-11-08T05:30:44.491907+0000","last_became_active":"2023-11-08T05:29:44.395012+0000","last_became_peered":"2023-11-08T05:29:44.395012+0000","last_unstale":"2023-11-08T05:30:44.491907+0000","last_undegraded":"2023-11-08T05:30:44.491907+0000","last_fullsized":"2023-11-08T05:30:44.491907+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490629+0000","last_change":"2023-11-08T05:28:13.563357+0000","last_active":"2023-11-08T05:30:44.490629+0000","last_peered":"2023-11-08T05:30:44.490629+0000","last_clean":"2023-11-08T05:28:01.457691+0000","last_became_active":"2023-11-08T05:28:13.563357+0000","last_became_peered":"2023-11-08T05:28:13.563357+0000","last_unstale":"2023-11-08T05:30:44.490629+0000","last_undegraded":"2023-11-08T05:30:44.490629+0000","last_fullsized":"2023-11-08T05:28:13.541614+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490691+0000","last_change":"2023-11-08T05:28:13.550672+0000","last_active":"2023-11-08T05:30:44.490691+0000","last_peered":"2023-11-08T05:30:44.490691+0000","last_clean":"2023-11-08T05:28:01.457904+0000","last_became_active":"2023-11-08T05:28:13.550672+0000","last_became_peered":"2023-11-08T05:28:13.550672+0000","last_unstale":"2023-11-08T05:30:44.490691+0000","last_undegraded":"2023-11-08T05:30:44.490691+0000","last_fullsized":"2023-11-08T05:28:13.539595+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490805+0000","last_change":"2023-11-08T05:28:13.557968+0000","last_active":"2023-11-08T05:30:44.490805+0000","last_peered":"2023-11-08T05:30:44.490805+0000","last_clean":"2023-11-08T05:28:01.458433+0000","last_became_active":"2023-11-08T05:28:13.557968+0000","last_became_peered":"2023-11-08T05:28:13.557968+0000","last_unstale":"2023-11-08T05:30:44.490805+0000","last_undegraded":"2023-11-08T05:30:44.490805+0000","last_fullsized":"2023-11-08T05:28:13.542832+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":1,"acting":1,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":12,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738853,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":645040,"kb_used_data":5352,"kb_used_omap":275,"kb_used_meta":639404,"kb_avail":93726800,"statfs":{"total":96636764160,"available":95976243200,"internally_reserved":0,"allocated":5480448,"data_stored":2697647,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":281650,"internal_metadata":654750670},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:59 2023","interfaces":[{"interface":"back","average":{"1min":0.30499999999999999,"5min":0.439,"15min":0.36099999999999999},"min":{"1min":0.17399999999999999,"5min":0.14499999999999999,"15min":0.14499999999999999},"max":{"1min":0.60099999999999998,"5min":4.9379999999999997,"15min":4.9379999999999997},"last":0.379},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.49399999999999999,"15min":0.42399999999999999},"min":{"1min":0.221,"5min":0.19600000000000001,"15min":0.17000000000000001},"max":{"1min":0.63200000000000001,"5min":4.9859999999999998,"15min":4.9859999999999998},"last":0.40899999999999997}]},{"osd":1,"last update":"Wed Nov 8 05:32:06 2023","interfaces":[{"interface":"back","average":{"1min":0.27400000000000002,"5min":0.307,"15min":0.46800000000000003},"min":{"1min":0.17599999999999999,"5min":0.16,"15min":0.153},"max":{"1min":0.39000000000000001,"5min":1.825,"15min":23.242999999999999},"last":0.23300000000000001},{"interface":"front","average":{"1min":0.28000000000000003,"5min":0.34799999999999998,"15min":0.45700000000000002},"min":{"1min":0.20200000000000001,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":0.41699999999999998,"5min":2.0419999999999998,"15min":23.251000000000001},"last":0.26500000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738798,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1339228,"kb_used_data":398304,"kb_used_omap":291,"kb_used_meta":936476,"kb_avail":93032612,"statfs":{"total":96636764160,"available":95265394688,"internally_reserved":0,"allocated":407863296,"data_stored":405103900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":298462,"internal_metadata":958951970},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.436,"15min":0.36299999999999999},"min":{"1min":0.19700000000000001,"5min":0.14699999999999999,"15min":0.13700000000000001},"max":{"1min":3.222,"5min":6.5499999999999998,"15min":6.9989999999999997},"last":0.245},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.46300000000000002,"15min":0.42299999999999999},"min":{"1min":0.17799999999999999,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":5.0030000000000001,"5min":6.0629999999999997,"15min":9.6869999999999994},"last":0.28599999999999998}]},{"osd":2,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.51600000000000001,"15min":0.47099999999999997},"min":{"1min":0.21099999999999999,"5min":0.153,"15min":0.153},"max":{"1min":5.7439999999999998,"5min":5.8280000000000003,"15min":9.7639999999999993},"last":0.252},{"interface":"front","average":{"1min":0.497,"5min":0.56200000000000006,"15min":0.42299999999999999},"min":{"1min":0.20399999999999999,"5min":0.187,"15min":0.16800000000000001},"max":{"1min":3.9319999999999999,"5min":8.8040000000000003,"15min":8.8040000000000003},"last":0.29899999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738855,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":650100,"kb_used_data":5356,"kb_used_omap":214,"kb_used_meta":644521,"kb_avail":93721740,"statfs":{"total":96636764160,"available":95971061760,"internally_reserved":0,"allocated":5484544,"data_stored":2698616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":219335,"internal_metadata":659990329},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:56 2023","interfaces":[{"interface":"back","average":{"1min":0.30399999999999999,"5min":0.34699999999999998,"15min":0.36499999999999999},"min":{"1min":0.23999999999999999,"5min":0.14199999999999999,"15min":0.14199999999999999},"max":{"1min":0.42399999999999999,"5min":1.595,"15min":7.4249999999999998},"last":0.17100000000000001},{"interface":"front","average":{"1min":0.35499999999999998,"5min":0.40899999999999997,"15min":0.376},"min":{"1min":0.23999999999999999,"5min":0.19600000000000001,"15min":0.183},"max":{"1min":0.46999999999999997,"5min":2.6859999999999999,"15min":2.6859999999999999},"last":0.23000000000000001}]},{"osd":2,"last update":"Wed Nov 8 05:32:02 2023","interfaces":[{"interface":"back","average":{"1min":0.252,"5min":0.33000000000000002,"15min":0.36499999999999999},"min":{"1min":0.161,"5min":0.161,"15min":0.161},"max":{"1min":0.39600000000000002,"5min":2.2650000000000001,"15min":5.8209999999999997},"last":0.221},{"interface":"front","average":{"1min":0.25800000000000001,"5min":0.29899999999999999,"15min":0.35599999999999998},"min":{"1min":0.17000000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.35799999999999998,"5min":1.137,"15min":4.2350000000000003},"last":0.254}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-08T05:32:58.653 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-11-08T05:32:58.653 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 459, 'reported_epoch': 458, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:30:44.490644+0000', 'last_change': '2023-11-08T05:28:13.557071+0000', 'last_active': '2023-11-08T05:30:44.490644+0000', 'last_peered': '2023-11-08T05:30:44.490644+0000', 'last_clean': '2023-11-08T05:28:01.458439+0000', 'last_became_active': '2023-11-08T05:28:13.557071+0000', 'last_became_peered': '2023-11-08T05:28:13.557071+0000', 'last_unstale': '2023-11-08T05:30:44.490644+0000', 'last_undegraded': '2023-11-08T05:30:44.490644+0000', 'last_fullsized': '2023-11-08T05:28:13.547025+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:32:58.654 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-11-08T05:32:58.654 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 11, 'reported_epoch': 459, 'state': 'active+clean+remapped', 'last_fresh': '2023-11-08T05:30:45.794760+0000', 'last_change': '2023-11-08T05:30:45.794760+0000', 'last_active': '2023-11-08T05:30:45.794760+0000', 'last_peered': '2023-11-08T05:30:45.794760+0000', 'last_clean': '2023-11-08T05:30:45.794760+0000', 'last_became_active': '2023-11-08T05:30:45.794398+0000', 'last_became_peered': '2023-11-08T05:30:45.794398+0000', 'last_unstale': '2023-11-08T05:30:45.794760+0000', 'last_undegraded': '2023-11-08T05:30:45.794760+0000', 'last_fullsized': '2023-11-08T05:30:45.794760+0000', 'mapping_epoch': 458, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 459, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2], 'acting': [2, 1], 'avail_no_missing': ['2', '1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2023-11-08T05:32:58.654 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.1 is not active+clean 2023-11-08T05:32:58.655 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.1', 'version': "0'0", 'reported_seq': 465, 'reported_epoch': 459, 'state': 'active+clean+remapped', 'last_fresh': '2023-11-08T05:30:45.795085+0000', 'last_change': '2023-11-08T05:30:45.795085+0000', 'last_active': '2023-11-08T05:30:45.795085+0000', 'last_peered': '2023-11-08T05:30:45.795085+0000', 'last_clean': '2023-11-08T05:30:45.795085+0000', 'last_became_active': '2023-11-08T05:30:45.794743+0000', 'last_became_peered': '2023-11-08T05:30:45.794743+0000', 'last_unstale': '2023-11-08T05:30:45.795085+0000', 'last_undegraded': '2023-11-08T05:30:45.795085+0000', 'last_fullsized': '2023-11-08T05:30:45.795085+0000', 'mapping_epoch': 458, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 459, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2], 'acting': [2, 1], 'avail_no_missing': ['2', '1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2023-11-08T05:32:58.655 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-11-08T05:32:58.655 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "0'0", 'reported_seq': 460, 'reported_epoch': 458, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:30:44.490523+0000', 'last_change': '2023-11-08T05:28:13.545468+0000', 'last_active': '2023-11-08T05:30:44.490523+0000', 'last_peered': '2023-11-08T05:30:44.490523+0000', 'last_clean': '2023-11-08T05:28:01.456723+0000', 'last_became_active': '2023-11-08T05:28:13.545468+0000', 'last_became_peered': '2023-11-08T05:28:13.545468+0000', 'last_unstale': '2023-11-08T05:30:44.490523+0000', 'last_undegraded': '2023-11-08T05:30:44.490523+0000', 'last_fullsized': '2023-11-08T05:28:13.537207+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 9, 'last_epoch_clean': 10, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:45.528796+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:45.528796+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:45.528796+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:32:58.656 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.0 is not active+clean 2023-11-08T05:32:58.656 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.0', 'version': "0'0", 'reported_seq': 465, 'reported_epoch': 458, 'state': 'active+clean+remapped', 'last_fresh': '2023-11-08T05:30:44.491907+0000', 'last_change': '2023-11-08T05:29:44.395350+0000', 'last_active': '2023-11-08T05:30:44.491907+0000', 'last_peered': '2023-11-08T05:30:44.491907+0000', 'last_clean': '2023-11-08T05:30:44.491907+0000', 'last_became_active': '2023-11-08T05:29:44.395012+0000', 'last_became_peered': '2023-11-08T05:29:44.395012+0000', 'last_unstale': '2023-11-08T05:30:44.491907+0000', 'last_undegraded': '2023-11-08T05:30:44.491907+0000', 'last_fullsized': '2023-11-08T05:30:44.491907+0000', 'mapping_epoch': 455, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 456, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2], 'acting': [2, 1], 'avail_no_missing': ['2', '1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2023-11-08T05:32:58.656 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-11-08T05:32:58.656 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 459, 'reported_epoch': 458, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:30:44.490629+0000', 'last_change': '2023-11-08T05:28:13.563357+0000', 'last_active': '2023-11-08T05:30:44.490629+0000', 'last_peered': '2023-11-08T05:30:44.490629+0000', 'last_clean': '2023-11-08T05:28:01.457691+0000', 'last_became_active': '2023-11-08T05:28:13.563357+0000', 'last_became_peered': '2023-11-08T05:28:13.563357+0000', 'last_unstale': '2023-11-08T05:30:44.490629+0000', 'last_undegraded': '2023-11-08T05:30:44.490629+0000', 'last_fullsized': '2023-11-08T05:28:13.541614+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:32:58.657 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-11-08T05:32:58.657 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 459, 'reported_epoch': 458, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:30:44.490691+0000', 'last_change': '2023-11-08T05:28:13.550672+0000', 'last_active': '2023-11-08T05:30:44.490691+0000', 'last_peered': '2023-11-08T05:30:44.490691+0000', 'last_clean': '2023-11-08T05:28:01.457904+0000', 'last_became_active': '2023-11-08T05:28:13.550672+0000', 'last_became_peered': '2023-11-08T05:28:13.550672+0000', 'last_unstale': '2023-11-08T05:30:44.490691+0000', 'last_undegraded': '2023-11-08T05:30:44.490691+0000', 'last_fullsized': '2023-11-08T05:28:13.539595+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:32:58.657 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-11-08T05:32:58.658 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 459, 'reported_epoch': 458, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:30:44.490805+0000', 'last_change': '2023-11-08T05:28:13.557968+0000', 'last_active': '2023-11-08T05:30:44.490805+0000', 'last_peered': '2023-11-08T05:30:44.490805+0000', 'last_clean': '2023-11-08T05:28:01.458433+0000', 'last_became_active': '2023-11-08T05:28:13.557968+0000', 'last_became_peered': '2023-11-08T05:28:13.557968+0000', 'last_unstale': '2023-11-08T05:30:44.490805+0000', 'last_undegraded': '2023-11-08T05:30:44.490805+0000', 'last_fullsized': '2023-11-08T05:28:13.542832+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:32:58.658 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-11-08T05:32:59.004 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T05:32:59.004 INFO:teuthology.orchestra.run.smithi087.stdout:{"fsid":"ab4cbc06-449d-497d-b513-220d9db2b590","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 5 pgs undersized","count":5},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2421,"monmap":{"epoch":1,"min_mon_release_name":"pacific","num_mons":1},"osdmap":{"epoch":459,"num_osds":3,"num_up_osds":2,"osd_up_since":1699421292,"num_in_osds":3,"osd_in_since":1699419158,"num_remapped_pgs":3},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean+remapped","count":3},{"state_name":"active+clean","count":1}],"num_pgs":9,"num_pools":2,"num_objects":0,"data_bytes":0,"bytes_used":2697592832,"bytes_avail":287212699648,"bytes_total":289910292480},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":41,"modified":"2023-11-08T05:29:32.044544+0000","services":{}},"progress_events":{}} 2023-11-08T05:32:59.018 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-11-08T05:32:59.018 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-11-08T05:32:59.062 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~281s 2023-11-08T05:32:59.297 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T05:32:59.299 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2023-11-08T05:32:59.310 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":1663,"stamp":"2023-11-08T05:32:58.087014+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":10,"acting":13,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":109,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2634368,"kb_used_data":409012,"kb_used_omap":780,"kb_used_meta":2220403,"kb_avail":280481152,"statfs":{"total":289910292480,"available":287212699648,"internally_reserved":0,"allocated":418828288,"data_stored":410500163,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":799447,"internal_metadata":2273692969},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-536782902,"num_objects":-164,"num_object_clones":0,"num_object_copies":-328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-115,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-164,"num_whiteouts":0,"num_read":-122056,"num_read_kb":-39011942,"num_write":-413642,"num_write_kb":-151705339,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-8,"num_bytes_recovered":-16922624,"num_keys_recovered":-2,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.424817"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490644+0000","last_change":"2023-11-08T05:28:13.557071+0000","last_active":"2023-11-08T05:30:44.490644+0000","last_peered":"2023-11-08T05:30:44.490644+0000","last_clean":"2023-11-08T05:28:01.458439+0000","last_became_active":"2023-11-08T05:28:13.557071+0000","last_became_peered":"2023-11-08T05:28:13.557071+0000","last_unstale":"2023-11-08T05:30:44.490644+0000","last_undegraded":"2023-11-08T05:30:44.490644+0000","last_fullsized":"2023-11-08T05:28:13.547025+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":459,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:45.794760+0000","last_change":"2023-11-08T05:30:45.794760+0000","last_active":"2023-11-08T05:30:45.794760+0000","last_peered":"2023-11-08T05:30:45.794760+0000","last_clean":"2023-11-08T05:30:45.794760+0000","last_became_active":"2023-11-08T05:30:45.794398+0000","last_became_peered":"2023-11-08T05:30:45.794398+0000","last_unstale":"2023-11-08T05:30:45.794760+0000","last_undegraded":"2023-11-08T05:30:45.794760+0000","last_fullsized":"2023-11-08T05:30:45.794760+0000","mapping_epoch":458,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":459,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":465,"reported_epoch":459,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:45.795085+0000","last_change":"2023-11-08T05:30:45.795085+0000","last_active":"2023-11-08T05:30:45.795085+0000","last_peered":"2023-11-08T05:30:45.795085+0000","last_clean":"2023-11-08T05:30:45.795085+0000","last_became_active":"2023-11-08T05:30:45.794743+0000","last_became_peered":"2023-11-08T05:30:45.794743+0000","last_unstale":"2023-11-08T05:30:45.795085+0000","last_undegraded":"2023-11-08T05:30:45.795085+0000","last_fullsized":"2023-11-08T05:30:45.795085+0000","mapping_epoch":458,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":459,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":454,"reported_epoch":458,"state":"active+clean","last_fresh":"2023-11-08T05:30:44.490618+0000","last_change":"2023-11-08T04:52:47.548597+0000","last_active":"2023-11-08T05:30:44.490618+0000","last_peered":"2023-11-08T05:30:44.490618+0000","last_clean":"2023-11-08T05:30:44.490618+0000","last_became_active":"2023-11-08T04:52:47.548294+0000","last_became_peered":"2023-11-08T04:52:47.548294+0000","last_unstale":"2023-11-08T05:30:44.490618+0000","last_undegraded":"2023-11-08T05:30:44.490618+0000","last_fullsized":"2023-11-08T05:30:44.490618+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":460,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490523+0000","last_change":"2023-11-08T05:28:13.545468+0000","last_active":"2023-11-08T05:30:44.490523+0000","last_peered":"2023-11-08T05:30:44.490523+0000","last_clean":"2023-11-08T05:28:01.456723+0000","last_became_active":"2023-11-08T05:28:13.545468+0000","last_became_peered":"2023-11-08T05:28:13.545468+0000","last_unstale":"2023-11-08T05:30:44.490523+0000","last_undegraded":"2023-11-08T05:30:44.490523+0000","last_fullsized":"2023-11-08T05:28:13.537207+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_clean_scrub_stamp":"2023-11-08T04:52:45.528796+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":465,"reported_epoch":458,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:44.491907+0000","last_change":"2023-11-08T05:29:44.395350+0000","last_active":"2023-11-08T05:30:44.491907+0000","last_peered":"2023-11-08T05:30:44.491907+0000","last_clean":"2023-11-08T05:30:44.491907+0000","last_became_active":"2023-11-08T05:29:44.395012+0000","last_became_peered":"2023-11-08T05:29:44.395012+0000","last_unstale":"2023-11-08T05:30:44.491907+0000","last_undegraded":"2023-11-08T05:30:44.491907+0000","last_fullsized":"2023-11-08T05:30:44.491907+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490629+0000","last_change":"2023-11-08T05:28:13.563357+0000","last_active":"2023-11-08T05:30:44.490629+0000","last_peered":"2023-11-08T05:30:44.490629+0000","last_clean":"2023-11-08T05:28:01.457691+0000","last_became_active":"2023-11-08T05:28:13.563357+0000","last_became_peered":"2023-11-08T05:28:13.563357+0000","last_unstale":"2023-11-08T05:30:44.490629+0000","last_undegraded":"2023-11-08T05:30:44.490629+0000","last_fullsized":"2023-11-08T05:28:13.541614+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490691+0000","last_change":"2023-11-08T05:28:13.550672+0000","last_active":"2023-11-08T05:30:44.490691+0000","last_peered":"2023-11-08T05:30:44.490691+0000","last_clean":"2023-11-08T05:28:01.457904+0000","last_became_active":"2023-11-08T05:28:13.550672+0000","last_became_peered":"2023-11-08T05:28:13.550672+0000","last_unstale":"2023-11-08T05:30:44.490691+0000","last_undegraded":"2023-11-08T05:30:44.490691+0000","last_fullsized":"2023-11-08T05:28:13.539595+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490805+0000","last_change":"2023-11-08T05:28:13.557968+0000","last_active":"2023-11-08T05:30:44.490805+0000","last_peered":"2023-11-08T05:30:44.490805+0000","last_clean":"2023-11-08T05:28:01.458433+0000","last_became_active":"2023-11-08T05:28:13.557968+0000","last_became_peered":"2023-11-08T05:28:13.557968+0000","last_unstale":"2023-11-08T05:30:44.490805+0000","last_undegraded":"2023-11-08T05:30:44.490805+0000","last_fullsized":"2023-11-08T05:28:13.542832+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":1,"acting":1,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":12,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738853,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":645040,"kb_used_data":5352,"kb_used_omap":275,"kb_used_meta":639404,"kb_avail":93726800,"statfs":{"total":96636764160,"available":95976243200,"internally_reserved":0,"allocated":5480448,"data_stored":2697647,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":281650,"internal_metadata":654750670},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:59 2023","interfaces":[{"interface":"back","average":{"1min":0.30499999999999999,"5min":0.439,"15min":0.36099999999999999},"min":{"1min":0.17399999999999999,"5min":0.14499999999999999,"15min":0.14499999999999999},"max":{"1min":0.60099999999999998,"5min":4.9379999999999997,"15min":4.9379999999999997},"last":0.379},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.49399999999999999,"15min":0.42399999999999999},"min":{"1min":0.221,"5min":0.19600000000000001,"15min":0.17000000000000001},"max":{"1min":0.63200000000000001,"5min":4.9859999999999998,"15min":4.9859999999999998},"last":0.40899999999999997}]},{"osd":1,"last update":"Wed Nov 8 05:32:06 2023","interfaces":[{"interface":"back","average":{"1min":0.27400000000000002,"5min":0.307,"15min":0.46800000000000003},"min":{"1min":0.17599999999999999,"5min":0.16,"15min":0.153},"max":{"1min":0.39000000000000001,"5min":1.825,"15min":23.242999999999999},"last":0.23300000000000001},{"interface":"front","average":{"1min":0.28000000000000003,"5min":0.34799999999999998,"15min":0.45700000000000002},"min":{"1min":0.20200000000000001,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":0.41699999999999998,"5min":2.0419999999999998,"15min":23.251000000000001},"last":0.26500000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738798,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1339228,"kb_used_data":398304,"kb_used_omap":291,"kb_used_meta":936476,"kb_avail":93032612,"statfs":{"total":96636764160,"available":95265394688,"internally_reserved":0,"allocated":407863296,"data_stored":405103900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":298462,"internal_metadata":958951970},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.436,"15min":0.36299999999999999},"min":{"1min":0.19700000000000001,"5min":0.14699999999999999,"15min":0.13700000000000001},"max":{"1min":3.222,"5min":6.5499999999999998,"15min":6.9989999999999997},"last":0.245},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.46300000000000002,"15min":0.42299999999999999},"min":{"1min":0.17799999999999999,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":5.0030000000000001,"5min":6.0629999999999997,"15min":9.6869999999999994},"last":0.28599999999999998}]},{"osd":2,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.51600000000000001,"15min":0.47099999999999997},"min":{"1min":0.21099999999999999,"5min":0.153,"15min":0.153},"max":{"1min":5.7439999999999998,"5min":5.8280000000000003,"15min":9.7639999999999993},"last":0.252},{"interface":"front","average":{"1min":0.497,"5min":0.56200000000000006,"15min":0.42299999999999999},"min":{"1min":0.20399999999999999,"5min":0.187,"15min":0.16800000000000001},"max":{"1min":3.9319999999999999,"5min":8.8040000000000003,"15min":8.8040000000000003},"last":0.29899999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738855,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":650100,"kb_used_data":5356,"kb_used_omap":214,"kb_used_meta":644521,"kb_avail":93721740,"statfs":{"total":96636764160,"available":95971061760,"internally_reserved":0,"allocated":5484544,"data_stored":2698616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":219335,"internal_metadata":659990329},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:56 2023","interfaces":[{"interface":"back","average":{"1min":0.30399999999999999,"5min":0.34699999999999998,"15min":0.36499999999999999},"min":{"1min":0.23999999999999999,"5min":0.14199999999999999,"15min":0.14199999999999999},"max":{"1min":0.42399999999999999,"5min":1.595,"15min":7.4249999999999998},"last":0.17100000000000001},{"interface":"front","average":{"1min":0.35499999999999998,"5min":0.40899999999999997,"15min":0.376},"min":{"1min":0.23999999999999999,"5min":0.19600000000000001,"15min":0.183},"max":{"1min":0.46999999999999997,"5min":2.6859999999999999,"15min":2.6859999999999999},"last":0.23000000000000001}]},{"osd":2,"last update":"Wed Nov 8 05:32:02 2023","interfaces":[{"interface":"back","average":{"1min":0.252,"5min":0.33000000000000002,"15min":0.36499999999999999},"min":{"1min":0.161,"5min":0.161,"15min":0.161},"max":{"1min":0.39600000000000002,"5min":2.2650000000000001,"15min":5.8209999999999997},"last":0.221},{"interface":"front","average":{"1min":0.25800000000000001,"5min":0.29899999999999999,"15min":0.35599999999999998},"min":{"1min":0.17000000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.35799999999999998,"5min":1.137,"15min":4.2350000000000003},"last":0.254}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-08T05:33:02.312 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-11-08T05:33:02.607 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T05:33:02.609 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2023-11-08T05:33:02.620 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":1665,"stamp":"2023-11-08T05:33:02.087739+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":10,"acting":13,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":109,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2634368,"kb_used_data":409012,"kb_used_omap":780,"kb_used_meta":2220403,"kb_avail":280481152,"statfs":{"total":289910292480,"available":287212699648,"internally_reserved":0,"allocated":418828288,"data_stored":410500163,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":799447,"internal_metadata":2273692969},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-536782902,"num_objects":-164,"num_object_clones":0,"num_object_copies":-328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-115,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-164,"num_whiteouts":0,"num_read":-122056,"num_read_kb":-39011942,"num_write":-413642,"num_write_kb":-151705339,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-8,"num_bytes_recovered":-16922624,"num_keys_recovered":-2,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.424817"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490644+0000","last_change":"2023-11-08T05:28:13.557071+0000","last_active":"2023-11-08T05:30:44.490644+0000","last_peered":"2023-11-08T05:30:44.490644+0000","last_clean":"2023-11-08T05:28:01.458439+0000","last_became_active":"2023-11-08T05:28:13.557071+0000","last_became_peered":"2023-11-08T05:28:13.557071+0000","last_unstale":"2023-11-08T05:30:44.490644+0000","last_undegraded":"2023-11-08T05:30:44.490644+0000","last_fullsized":"2023-11-08T05:28:13.547025+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":459,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:45.794760+0000","last_change":"2023-11-08T05:30:45.794760+0000","last_active":"2023-11-08T05:30:45.794760+0000","last_peered":"2023-11-08T05:30:45.794760+0000","last_clean":"2023-11-08T05:30:45.794760+0000","last_became_active":"2023-11-08T05:30:45.794398+0000","last_became_peered":"2023-11-08T05:30:45.794398+0000","last_unstale":"2023-11-08T05:30:45.794760+0000","last_undegraded":"2023-11-08T05:30:45.794760+0000","last_fullsized":"2023-11-08T05:30:45.794760+0000","mapping_epoch":458,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":459,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":465,"reported_epoch":459,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:45.795085+0000","last_change":"2023-11-08T05:30:45.795085+0000","last_active":"2023-11-08T05:30:45.795085+0000","last_peered":"2023-11-08T05:30:45.795085+0000","last_clean":"2023-11-08T05:30:45.795085+0000","last_became_active":"2023-11-08T05:30:45.794743+0000","last_became_peered":"2023-11-08T05:30:45.794743+0000","last_unstale":"2023-11-08T05:30:45.795085+0000","last_undegraded":"2023-11-08T05:30:45.795085+0000","last_fullsized":"2023-11-08T05:30:45.795085+0000","mapping_epoch":458,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":459,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":454,"reported_epoch":458,"state":"active+clean","last_fresh":"2023-11-08T05:30:44.490618+0000","last_change":"2023-11-08T04:52:47.548597+0000","last_active":"2023-11-08T05:30:44.490618+0000","last_peered":"2023-11-08T05:30:44.490618+0000","last_clean":"2023-11-08T05:30:44.490618+0000","last_became_active":"2023-11-08T04:52:47.548294+0000","last_became_peered":"2023-11-08T04:52:47.548294+0000","last_unstale":"2023-11-08T05:30:44.490618+0000","last_undegraded":"2023-11-08T05:30:44.490618+0000","last_fullsized":"2023-11-08T05:30:44.490618+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":460,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490523+0000","last_change":"2023-11-08T05:28:13.545468+0000","last_active":"2023-11-08T05:30:44.490523+0000","last_peered":"2023-11-08T05:30:44.490523+0000","last_clean":"2023-11-08T05:28:01.456723+0000","last_became_active":"2023-11-08T05:28:13.545468+0000","last_became_peered":"2023-11-08T05:28:13.545468+0000","last_unstale":"2023-11-08T05:30:44.490523+0000","last_undegraded":"2023-11-08T05:30:44.490523+0000","last_fullsized":"2023-11-08T05:28:13.537207+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_clean_scrub_stamp":"2023-11-08T04:52:45.528796+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":465,"reported_epoch":458,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:44.491907+0000","last_change":"2023-11-08T05:29:44.395350+0000","last_active":"2023-11-08T05:30:44.491907+0000","last_peered":"2023-11-08T05:30:44.491907+0000","last_clean":"2023-11-08T05:30:44.491907+0000","last_became_active":"2023-11-08T05:29:44.395012+0000","last_became_peered":"2023-11-08T05:29:44.395012+0000","last_unstale":"2023-11-08T05:30:44.491907+0000","last_undegraded":"2023-11-08T05:30:44.491907+0000","last_fullsized":"2023-11-08T05:30:44.491907+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490629+0000","last_change":"2023-11-08T05:28:13.563357+0000","last_active":"2023-11-08T05:30:44.490629+0000","last_peered":"2023-11-08T05:30:44.490629+0000","last_clean":"2023-11-08T05:28:01.457691+0000","last_became_active":"2023-11-08T05:28:13.563357+0000","last_became_peered":"2023-11-08T05:28:13.563357+0000","last_unstale":"2023-11-08T05:30:44.490629+0000","last_undegraded":"2023-11-08T05:30:44.490629+0000","last_fullsized":"2023-11-08T05:28:13.541614+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490691+0000","last_change":"2023-11-08T05:28:13.550672+0000","last_active":"2023-11-08T05:30:44.490691+0000","last_peered":"2023-11-08T05:30:44.490691+0000","last_clean":"2023-11-08T05:28:01.457904+0000","last_became_active":"2023-11-08T05:28:13.550672+0000","last_became_peered":"2023-11-08T05:28:13.550672+0000","last_unstale":"2023-11-08T05:30:44.490691+0000","last_undegraded":"2023-11-08T05:30:44.490691+0000","last_fullsized":"2023-11-08T05:28:13.539595+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490805+0000","last_change":"2023-11-08T05:28:13.557968+0000","last_active":"2023-11-08T05:30:44.490805+0000","last_peered":"2023-11-08T05:30:44.490805+0000","last_clean":"2023-11-08T05:28:01.458433+0000","last_became_active":"2023-11-08T05:28:13.557968+0000","last_became_peered":"2023-11-08T05:28:13.557968+0000","last_unstale":"2023-11-08T05:30:44.490805+0000","last_undegraded":"2023-11-08T05:30:44.490805+0000","last_fullsized":"2023-11-08T05:28:13.542832+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":1,"acting":1,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":12,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738854,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":645040,"kb_used_data":5352,"kb_used_omap":275,"kb_used_meta":639404,"kb_avail":93726800,"statfs":{"total":96636764160,"available":95976243200,"internally_reserved":0,"allocated":5480448,"data_stored":2697647,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":281650,"internal_metadata":654750670},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:59 2023","interfaces":[{"interface":"back","average":{"1min":0.30499999999999999,"5min":0.439,"15min":0.36099999999999999},"min":{"1min":0.17399999999999999,"5min":0.14499999999999999,"15min":0.14499999999999999},"max":{"1min":0.60099999999999998,"5min":4.9379999999999997,"15min":4.9379999999999997},"last":0.379},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.49399999999999999,"15min":0.42399999999999999},"min":{"1min":0.221,"5min":0.19600000000000001,"15min":0.17000000000000001},"max":{"1min":0.63200000000000001,"5min":4.9859999999999998,"15min":4.9859999999999998},"last":0.40899999999999997}]},{"osd":1,"last update":"Wed Nov 8 05:32:06 2023","interfaces":[{"interface":"back","average":{"1min":0.27400000000000002,"5min":0.307,"15min":0.46800000000000003},"min":{"1min":0.17599999999999999,"5min":0.16,"15min":0.153},"max":{"1min":0.39000000000000001,"5min":1.825,"15min":23.242999999999999},"last":0.221},{"interface":"front","average":{"1min":0.28000000000000003,"5min":0.34799999999999998,"15min":0.45700000000000002},"min":{"1min":0.20200000000000001,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":0.41699999999999998,"5min":2.0419999999999998,"15min":23.251000000000001},"last":0.23300000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738798,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1339228,"kb_used_data":398304,"kb_used_omap":291,"kb_used_meta":936476,"kb_avail":93032612,"statfs":{"total":96636764160,"available":95265394688,"internally_reserved":0,"allocated":407863296,"data_stored":405103900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":298462,"internal_metadata":958951970},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.436,"15min":0.36299999999999999},"min":{"1min":0.19700000000000001,"5min":0.14699999999999999,"15min":0.13700000000000001},"max":{"1min":3.222,"5min":6.5499999999999998,"15min":6.9989999999999997},"last":0.245},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.46300000000000002,"15min":0.42299999999999999},"min":{"1min":0.17799999999999999,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":5.0030000000000001,"5min":6.0629999999999997,"15min":9.6869999999999994},"last":0.28599999999999998}]},{"osd":2,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.51600000000000001,"15min":0.47099999999999997},"min":{"1min":0.21099999999999999,"5min":0.153,"15min":0.153},"max":{"1min":5.7439999999999998,"5min":5.8280000000000003,"15min":9.7639999999999993},"last":0.252},{"interface":"front","average":{"1min":0.497,"5min":0.56200000000000006,"15min":0.42299999999999999},"min":{"1min":0.20399999999999999,"5min":0.187,"15min":0.16800000000000001},"max":{"1min":3.9319999999999999,"5min":8.8040000000000003,"15min":8.8040000000000003},"last":0.29899999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738856,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":650100,"kb_used_data":5356,"kb_used_omap":214,"kb_used_meta":644521,"kb_avail":93721740,"statfs":{"total":96636764160,"available":95971061760,"internally_reserved":0,"allocated":5484544,"data_stored":2698616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":219335,"internal_metadata":659990329},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:56 2023","interfaces":[{"interface":"back","average":{"1min":0.30399999999999999,"5min":0.34699999999999998,"15min":0.36499999999999999},"min":{"1min":0.23999999999999999,"5min":0.14199999999999999,"15min":0.14199999999999999},"max":{"1min":0.42399999999999999,"5min":1.595,"15min":7.4249999999999998},"last":0.17100000000000001},{"interface":"front","average":{"1min":0.35499999999999998,"5min":0.40899999999999997,"15min":0.376},"min":{"1min":0.23999999999999999,"5min":0.19600000000000001,"15min":0.183},"max":{"1min":0.46999999999999997,"5min":2.6859999999999999,"15min":2.6859999999999999},"last":0.23000000000000001}]},{"osd":2,"last update":"Wed Nov 8 05:32:02 2023","interfaces":[{"interface":"back","average":{"1min":0.252,"5min":0.33000000000000002,"15min":0.36499999999999999},"min":{"1min":0.161,"5min":0.161,"15min":0.161},"max":{"1min":0.39600000000000002,"5min":2.2650000000000001,"15min":5.8209999999999997},"last":0.34399999999999997},{"interface":"front","average":{"1min":0.25800000000000001,"5min":0.29899999999999999,"15min":0.35599999999999998},"min":{"1min":0.17000000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.35799999999999998,"5min":1.137,"15min":4.2350000000000003},"last":0.35999999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-08T05:33:02.621 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-11-08T05:33:02.904 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T05:33:02.906 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2023-11-08T05:33:02.917 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":1665,"stamp":"2023-11-08T05:33:02.087739+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":10,"acting":13,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":109,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2634368,"kb_used_data":409012,"kb_used_omap":780,"kb_used_meta":2220403,"kb_avail":280481152,"statfs":{"total":289910292480,"available":287212699648,"internally_reserved":0,"allocated":418828288,"data_stored":410500163,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":799447,"internal_metadata":2273692969},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-536782902,"num_objects":-164,"num_object_clones":0,"num_object_copies":-328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-115,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-164,"num_whiteouts":0,"num_read":-122056,"num_read_kb":-39011942,"num_write":-413642,"num_write_kb":-151705339,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-8,"num_bytes_recovered":-16922624,"num_keys_recovered":-2,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.424817"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490644+0000","last_change":"2023-11-08T05:28:13.557071+0000","last_active":"2023-11-08T05:30:44.490644+0000","last_peered":"2023-11-08T05:30:44.490644+0000","last_clean":"2023-11-08T05:28:01.458439+0000","last_became_active":"2023-11-08T05:28:13.557071+0000","last_became_peered":"2023-11-08T05:28:13.557071+0000","last_unstale":"2023-11-08T05:30:44.490644+0000","last_undegraded":"2023-11-08T05:30:44.490644+0000","last_fullsized":"2023-11-08T05:28:13.547025+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":459,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:45.794760+0000","last_change":"2023-11-08T05:30:45.794760+0000","last_active":"2023-11-08T05:30:45.794760+0000","last_peered":"2023-11-08T05:30:45.794760+0000","last_clean":"2023-11-08T05:30:45.794760+0000","last_became_active":"2023-11-08T05:30:45.794398+0000","last_became_peered":"2023-11-08T05:30:45.794398+0000","last_unstale":"2023-11-08T05:30:45.794760+0000","last_undegraded":"2023-11-08T05:30:45.794760+0000","last_fullsized":"2023-11-08T05:30:45.794760+0000","mapping_epoch":458,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":459,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":465,"reported_epoch":459,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:45.795085+0000","last_change":"2023-11-08T05:30:45.795085+0000","last_active":"2023-11-08T05:30:45.795085+0000","last_peered":"2023-11-08T05:30:45.795085+0000","last_clean":"2023-11-08T05:30:45.795085+0000","last_became_active":"2023-11-08T05:30:45.794743+0000","last_became_peered":"2023-11-08T05:30:45.794743+0000","last_unstale":"2023-11-08T05:30:45.795085+0000","last_undegraded":"2023-11-08T05:30:45.795085+0000","last_fullsized":"2023-11-08T05:30:45.795085+0000","mapping_epoch":458,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":459,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":454,"reported_epoch":458,"state":"active+clean","last_fresh":"2023-11-08T05:30:44.490618+0000","last_change":"2023-11-08T04:52:47.548597+0000","last_active":"2023-11-08T05:30:44.490618+0000","last_peered":"2023-11-08T05:30:44.490618+0000","last_clean":"2023-11-08T05:30:44.490618+0000","last_became_active":"2023-11-08T04:52:47.548294+0000","last_became_peered":"2023-11-08T04:52:47.548294+0000","last_unstale":"2023-11-08T05:30:44.490618+0000","last_undegraded":"2023-11-08T05:30:44.490618+0000","last_fullsized":"2023-11-08T05:30:44.490618+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":460,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490523+0000","last_change":"2023-11-08T05:28:13.545468+0000","last_active":"2023-11-08T05:30:44.490523+0000","last_peered":"2023-11-08T05:30:44.490523+0000","last_clean":"2023-11-08T05:28:01.456723+0000","last_became_active":"2023-11-08T05:28:13.545468+0000","last_became_peered":"2023-11-08T05:28:13.545468+0000","last_unstale":"2023-11-08T05:30:44.490523+0000","last_undegraded":"2023-11-08T05:30:44.490523+0000","last_fullsized":"2023-11-08T05:28:13.537207+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_clean_scrub_stamp":"2023-11-08T04:52:45.528796+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":465,"reported_epoch":458,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:44.491907+0000","last_change":"2023-11-08T05:29:44.395350+0000","last_active":"2023-11-08T05:30:44.491907+0000","last_peered":"2023-11-08T05:30:44.491907+0000","last_clean":"2023-11-08T05:30:44.491907+0000","last_became_active":"2023-11-08T05:29:44.395012+0000","last_became_peered":"2023-11-08T05:29:44.395012+0000","last_unstale":"2023-11-08T05:30:44.491907+0000","last_undegraded":"2023-11-08T05:30:44.491907+0000","last_fullsized":"2023-11-08T05:30:44.491907+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490629+0000","last_change":"2023-11-08T05:28:13.563357+0000","last_active":"2023-11-08T05:30:44.490629+0000","last_peered":"2023-11-08T05:30:44.490629+0000","last_clean":"2023-11-08T05:28:01.457691+0000","last_became_active":"2023-11-08T05:28:13.563357+0000","last_became_peered":"2023-11-08T05:28:13.563357+0000","last_unstale":"2023-11-08T05:30:44.490629+0000","last_undegraded":"2023-11-08T05:30:44.490629+0000","last_fullsized":"2023-11-08T05:28:13.541614+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490691+0000","last_change":"2023-11-08T05:28:13.550672+0000","last_active":"2023-11-08T05:30:44.490691+0000","last_peered":"2023-11-08T05:30:44.490691+0000","last_clean":"2023-11-08T05:28:01.457904+0000","last_became_active":"2023-11-08T05:28:13.550672+0000","last_became_peered":"2023-11-08T05:28:13.550672+0000","last_unstale":"2023-11-08T05:30:44.490691+0000","last_undegraded":"2023-11-08T05:30:44.490691+0000","last_fullsized":"2023-11-08T05:28:13.539595+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490805+0000","last_change":"2023-11-08T05:28:13.557968+0000","last_active":"2023-11-08T05:30:44.490805+0000","last_peered":"2023-11-08T05:30:44.490805+0000","last_clean":"2023-11-08T05:28:01.458433+0000","last_became_active":"2023-11-08T05:28:13.557968+0000","last_became_peered":"2023-11-08T05:28:13.557968+0000","last_unstale":"2023-11-08T05:30:44.490805+0000","last_undegraded":"2023-11-08T05:30:44.490805+0000","last_fullsized":"2023-11-08T05:28:13.542832+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":1,"acting":1,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":12,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738854,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":645040,"kb_used_data":5352,"kb_used_omap":275,"kb_used_meta":639404,"kb_avail":93726800,"statfs":{"total":96636764160,"available":95976243200,"internally_reserved":0,"allocated":5480448,"data_stored":2697647,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":281650,"internal_metadata":654750670},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:59 2023","interfaces":[{"interface":"back","average":{"1min":0.30499999999999999,"5min":0.439,"15min":0.36099999999999999},"min":{"1min":0.17399999999999999,"5min":0.14499999999999999,"15min":0.14499999999999999},"max":{"1min":0.60099999999999998,"5min":4.9379999999999997,"15min":4.9379999999999997},"last":0.379},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.49399999999999999,"15min":0.42399999999999999},"min":{"1min":0.221,"5min":0.19600000000000001,"15min":0.17000000000000001},"max":{"1min":0.63200000000000001,"5min":4.9859999999999998,"15min":4.9859999999999998},"last":0.40899999999999997}]},{"osd":1,"last update":"Wed Nov 8 05:32:06 2023","interfaces":[{"interface":"back","average":{"1min":0.27400000000000002,"5min":0.307,"15min":0.46800000000000003},"min":{"1min":0.17599999999999999,"5min":0.16,"15min":0.153},"max":{"1min":0.39000000000000001,"5min":1.825,"15min":23.242999999999999},"last":0.221},{"interface":"front","average":{"1min":0.28000000000000003,"5min":0.34799999999999998,"15min":0.45700000000000002},"min":{"1min":0.20200000000000001,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":0.41699999999999998,"5min":2.0419999999999998,"15min":23.251000000000001},"last":0.23300000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738798,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1339228,"kb_used_data":398304,"kb_used_omap":291,"kb_used_meta":936476,"kb_avail":93032612,"statfs":{"total":96636764160,"available":95265394688,"internally_reserved":0,"allocated":407863296,"data_stored":405103900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":298462,"internal_metadata":958951970},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.436,"15min":0.36299999999999999},"min":{"1min":0.19700000000000001,"5min":0.14699999999999999,"15min":0.13700000000000001},"max":{"1min":3.222,"5min":6.5499999999999998,"15min":6.9989999999999997},"last":0.245},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.46300000000000002,"15min":0.42299999999999999},"min":{"1min":0.17799999999999999,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":5.0030000000000001,"5min":6.0629999999999997,"15min":9.6869999999999994},"last":0.28599999999999998}]},{"osd":2,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.51600000000000001,"15min":0.47099999999999997},"min":{"1min":0.21099999999999999,"5min":0.153,"15min":0.153},"max":{"1min":5.7439999999999998,"5min":5.8280000000000003,"15min":9.7639999999999993},"last":0.252},{"interface":"front","average":{"1min":0.497,"5min":0.56200000000000006,"15min":0.42299999999999999},"min":{"1min":0.20399999999999999,"5min":0.187,"15min":0.16800000000000001},"max":{"1min":3.9319999999999999,"5min":8.8040000000000003,"15min":8.8040000000000003},"last":0.29899999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738856,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":650100,"kb_used_data":5356,"kb_used_omap":214,"kb_used_meta":644521,"kb_avail":93721740,"statfs":{"total":96636764160,"available":95971061760,"internally_reserved":0,"allocated":5484544,"data_stored":2698616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":219335,"internal_metadata":659990329},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:56 2023","interfaces":[{"interface":"back","average":{"1min":0.30399999999999999,"5min":0.34699999999999998,"15min":0.36499999999999999},"min":{"1min":0.23999999999999999,"5min":0.14199999999999999,"15min":0.14199999999999999},"max":{"1min":0.42399999999999999,"5min":1.595,"15min":7.4249999999999998},"last":0.17100000000000001},{"interface":"front","average":{"1min":0.35499999999999998,"5min":0.40899999999999997,"15min":0.376},"min":{"1min":0.23999999999999999,"5min":0.19600000000000001,"15min":0.183},"max":{"1min":0.46999999999999997,"5min":2.6859999999999999,"15min":2.6859999999999999},"last":0.23000000000000001}]},{"osd":2,"last update":"Wed Nov 8 05:32:02 2023","interfaces":[{"interface":"back","average":{"1min":0.252,"5min":0.33000000000000002,"15min":0.36499999999999999},"min":{"1min":0.161,"5min":0.161,"15min":0.161},"max":{"1min":0.39600000000000002,"5min":2.2650000000000001,"15min":5.8209999999999997},"last":0.34399999999999997},{"interface":"front","average":{"1min":0.25800000000000001,"5min":0.29899999999999999,"15min":0.35599999999999998},"min":{"1min":0.17000000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.35799999999999998,"5min":1.137,"15min":4.2350000000000003},"last":0.35999999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-08T05:33:02.918 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-11-08T05:33:02.918 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 459, 'reported_epoch': 458, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:30:44.490644+0000', 'last_change': '2023-11-08T05:28:13.557071+0000', 'last_active': '2023-11-08T05:30:44.490644+0000', 'last_peered': '2023-11-08T05:30:44.490644+0000', 'last_clean': '2023-11-08T05:28:01.458439+0000', 'last_became_active': '2023-11-08T05:28:13.557071+0000', 'last_became_peered': '2023-11-08T05:28:13.557071+0000', 'last_unstale': '2023-11-08T05:30:44.490644+0000', 'last_undegraded': '2023-11-08T05:30:44.490644+0000', 'last_fullsized': '2023-11-08T05:28:13.547025+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:33:02.918 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-11-08T05:33:02.919 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 11, 'reported_epoch': 459, 'state': 'active+clean+remapped', 'last_fresh': '2023-11-08T05:30:45.794760+0000', 'last_change': '2023-11-08T05:30:45.794760+0000', 'last_active': '2023-11-08T05:30:45.794760+0000', 'last_peered': '2023-11-08T05:30:45.794760+0000', 'last_clean': '2023-11-08T05:30:45.794760+0000', 'last_became_active': '2023-11-08T05:30:45.794398+0000', 'last_became_peered': '2023-11-08T05:30:45.794398+0000', 'last_unstale': '2023-11-08T05:30:45.794760+0000', 'last_undegraded': '2023-11-08T05:30:45.794760+0000', 'last_fullsized': '2023-11-08T05:30:45.794760+0000', 'mapping_epoch': 458, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 459, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2], 'acting': [2, 1], 'avail_no_missing': ['2', '1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2023-11-08T05:33:02.919 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.1 is not active+clean 2023-11-08T05:33:02.919 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.1', 'version': "0'0", 'reported_seq': 465, 'reported_epoch': 459, 'state': 'active+clean+remapped', 'last_fresh': '2023-11-08T05:30:45.795085+0000', 'last_change': '2023-11-08T05:30:45.795085+0000', 'last_active': '2023-11-08T05:30:45.795085+0000', 'last_peered': '2023-11-08T05:30:45.795085+0000', 'last_clean': '2023-11-08T05:30:45.795085+0000', 'last_became_active': '2023-11-08T05:30:45.794743+0000', 'last_became_peered': '2023-11-08T05:30:45.794743+0000', 'last_unstale': '2023-11-08T05:30:45.795085+0000', 'last_undegraded': '2023-11-08T05:30:45.795085+0000', 'last_fullsized': '2023-11-08T05:30:45.795085+0000', 'mapping_epoch': 458, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 459, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2], 'acting': [2, 1], 'avail_no_missing': ['2', '1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2023-11-08T05:33:02.919 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-11-08T05:33:02.920 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "0'0", 'reported_seq': 460, 'reported_epoch': 458, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:30:44.490523+0000', 'last_change': '2023-11-08T05:28:13.545468+0000', 'last_active': '2023-11-08T05:30:44.490523+0000', 'last_peered': '2023-11-08T05:30:44.490523+0000', 'last_clean': '2023-11-08T05:28:01.456723+0000', 'last_became_active': '2023-11-08T05:28:13.545468+0000', 'last_became_peered': '2023-11-08T05:28:13.545468+0000', 'last_unstale': '2023-11-08T05:30:44.490523+0000', 'last_undegraded': '2023-11-08T05:30:44.490523+0000', 'last_fullsized': '2023-11-08T05:28:13.537207+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 9, 'last_epoch_clean': 10, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:45.528796+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:45.528796+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:45.528796+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:33:02.920 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.0 is not active+clean 2023-11-08T05:33:02.920 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.0', 'version': "0'0", 'reported_seq': 465, 'reported_epoch': 458, 'state': 'active+clean+remapped', 'last_fresh': '2023-11-08T05:30:44.491907+0000', 'last_change': '2023-11-08T05:29:44.395350+0000', 'last_active': '2023-11-08T05:30:44.491907+0000', 'last_peered': '2023-11-08T05:30:44.491907+0000', 'last_clean': '2023-11-08T05:30:44.491907+0000', 'last_became_active': '2023-11-08T05:29:44.395012+0000', 'last_became_peered': '2023-11-08T05:29:44.395012+0000', 'last_unstale': '2023-11-08T05:30:44.491907+0000', 'last_undegraded': '2023-11-08T05:30:44.491907+0000', 'last_fullsized': '2023-11-08T05:30:44.491907+0000', 'mapping_epoch': 455, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 456, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2], 'acting': [2, 1], 'avail_no_missing': ['2', '1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2023-11-08T05:33:02.921 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-11-08T05:33:02.921 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 459, 'reported_epoch': 458, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:30:44.490629+0000', 'last_change': '2023-11-08T05:28:13.563357+0000', 'last_active': '2023-11-08T05:30:44.490629+0000', 'last_peered': '2023-11-08T05:30:44.490629+0000', 'last_clean': '2023-11-08T05:28:01.457691+0000', 'last_became_active': '2023-11-08T05:28:13.563357+0000', 'last_became_peered': '2023-11-08T05:28:13.563357+0000', 'last_unstale': '2023-11-08T05:30:44.490629+0000', 'last_undegraded': '2023-11-08T05:30:44.490629+0000', 'last_fullsized': '2023-11-08T05:28:13.541614+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:33:02.921 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-11-08T05:33:02.921 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 459, 'reported_epoch': 458, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:30:44.490691+0000', 'last_change': '2023-11-08T05:28:13.550672+0000', 'last_active': '2023-11-08T05:30:44.490691+0000', 'last_peered': '2023-11-08T05:30:44.490691+0000', 'last_clean': '2023-11-08T05:28:01.457904+0000', 'last_became_active': '2023-11-08T05:28:13.550672+0000', 'last_became_peered': '2023-11-08T05:28:13.550672+0000', 'last_unstale': '2023-11-08T05:30:44.490691+0000', 'last_undegraded': '2023-11-08T05:30:44.490691+0000', 'last_fullsized': '2023-11-08T05:28:13.539595+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:33:02.922 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-11-08T05:33:02.922 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 459, 'reported_epoch': 458, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:30:44.490805+0000', 'last_change': '2023-11-08T05:28:13.557968+0000', 'last_active': '2023-11-08T05:30:44.490805+0000', 'last_peered': '2023-11-08T05:30:44.490805+0000', 'last_clean': '2023-11-08T05:28:01.458433+0000', 'last_became_active': '2023-11-08T05:28:13.557968+0000', 'last_became_peered': '2023-11-08T05:28:13.557968+0000', 'last_unstale': '2023-11-08T05:30:44.490805+0000', 'last_undegraded': '2023-11-08T05:30:44.490805+0000', 'last_fullsized': '2023-11-08T05:28:13.542832+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:33:02.922 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-11-08T05:33:03.268 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T05:33:03.269 INFO:teuthology.orchestra.run.smithi087.stdout:{"fsid":"ab4cbc06-449d-497d-b513-220d9db2b590","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 5 pgs undersized","count":5},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2425,"monmap":{"epoch":1,"min_mon_release_name":"pacific","num_mons":1},"osdmap":{"epoch":459,"num_osds":3,"num_up_osds":2,"osd_up_since":1699421292,"num_in_osds":3,"osd_in_since":1699419158,"num_remapped_pgs":3},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean+remapped","count":3},{"state_name":"active+clean","count":1}],"num_pgs":9,"num_pools":2,"num_objects":0,"data_bytes":0,"bytes_used":2697592832,"bytes_avail":287212699648,"bytes_total":289910292480},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":41,"modified":"2023-11-08T05:29:32.044544+0000","services":{}},"progress_events":{}} 2023-11-08T05:33:03.281 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-11-08T05:33:03.282 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-11-08T05:33:03.566 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T05:33:03.568 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2023-11-08T05:33:03.579 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":1665,"stamp":"2023-11-08T05:33:02.087739+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":10,"acting":13,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":109,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2634368,"kb_used_data":409012,"kb_used_omap":780,"kb_used_meta":2220403,"kb_avail":280481152,"statfs":{"total":289910292480,"available":287212699648,"internally_reserved":0,"allocated":418828288,"data_stored":410500163,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":799447,"internal_metadata":2273692969},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-536782902,"num_objects":-164,"num_object_clones":0,"num_object_copies":-328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-115,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-164,"num_whiteouts":0,"num_read":-122056,"num_read_kb":-39011942,"num_write":-413642,"num_write_kb":-151705339,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-8,"num_bytes_recovered":-16922624,"num_keys_recovered":-2,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.424817"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490644+0000","last_change":"2023-11-08T05:28:13.557071+0000","last_active":"2023-11-08T05:30:44.490644+0000","last_peered":"2023-11-08T05:30:44.490644+0000","last_clean":"2023-11-08T05:28:01.458439+0000","last_became_active":"2023-11-08T05:28:13.557071+0000","last_became_peered":"2023-11-08T05:28:13.557071+0000","last_unstale":"2023-11-08T05:30:44.490644+0000","last_undegraded":"2023-11-08T05:30:44.490644+0000","last_fullsized":"2023-11-08T05:28:13.547025+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":459,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:45.794760+0000","last_change":"2023-11-08T05:30:45.794760+0000","last_active":"2023-11-08T05:30:45.794760+0000","last_peered":"2023-11-08T05:30:45.794760+0000","last_clean":"2023-11-08T05:30:45.794760+0000","last_became_active":"2023-11-08T05:30:45.794398+0000","last_became_peered":"2023-11-08T05:30:45.794398+0000","last_unstale":"2023-11-08T05:30:45.794760+0000","last_undegraded":"2023-11-08T05:30:45.794760+0000","last_fullsized":"2023-11-08T05:30:45.794760+0000","mapping_epoch":458,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":459,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":465,"reported_epoch":459,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:45.795085+0000","last_change":"2023-11-08T05:30:45.795085+0000","last_active":"2023-11-08T05:30:45.795085+0000","last_peered":"2023-11-08T05:30:45.795085+0000","last_clean":"2023-11-08T05:30:45.795085+0000","last_became_active":"2023-11-08T05:30:45.794743+0000","last_became_peered":"2023-11-08T05:30:45.794743+0000","last_unstale":"2023-11-08T05:30:45.795085+0000","last_undegraded":"2023-11-08T05:30:45.795085+0000","last_fullsized":"2023-11-08T05:30:45.795085+0000","mapping_epoch":458,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":459,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":454,"reported_epoch":458,"state":"active+clean","last_fresh":"2023-11-08T05:30:44.490618+0000","last_change":"2023-11-08T04:52:47.548597+0000","last_active":"2023-11-08T05:30:44.490618+0000","last_peered":"2023-11-08T05:30:44.490618+0000","last_clean":"2023-11-08T05:30:44.490618+0000","last_became_active":"2023-11-08T04:52:47.548294+0000","last_became_peered":"2023-11-08T04:52:47.548294+0000","last_unstale":"2023-11-08T05:30:44.490618+0000","last_undegraded":"2023-11-08T05:30:44.490618+0000","last_fullsized":"2023-11-08T05:30:44.490618+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":460,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490523+0000","last_change":"2023-11-08T05:28:13.545468+0000","last_active":"2023-11-08T05:30:44.490523+0000","last_peered":"2023-11-08T05:30:44.490523+0000","last_clean":"2023-11-08T05:28:01.456723+0000","last_became_active":"2023-11-08T05:28:13.545468+0000","last_became_peered":"2023-11-08T05:28:13.545468+0000","last_unstale":"2023-11-08T05:30:44.490523+0000","last_undegraded":"2023-11-08T05:30:44.490523+0000","last_fullsized":"2023-11-08T05:28:13.537207+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_clean_scrub_stamp":"2023-11-08T04:52:45.528796+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":465,"reported_epoch":458,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:44.491907+0000","last_change":"2023-11-08T05:29:44.395350+0000","last_active":"2023-11-08T05:30:44.491907+0000","last_peered":"2023-11-08T05:30:44.491907+0000","last_clean":"2023-11-08T05:30:44.491907+0000","last_became_active":"2023-11-08T05:29:44.395012+0000","last_became_peered":"2023-11-08T05:29:44.395012+0000","last_unstale":"2023-11-08T05:30:44.491907+0000","last_undegraded":"2023-11-08T05:30:44.491907+0000","last_fullsized":"2023-11-08T05:30:44.491907+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490629+0000","last_change":"2023-11-08T05:28:13.563357+0000","last_active":"2023-11-08T05:30:44.490629+0000","last_peered":"2023-11-08T05:30:44.490629+0000","last_clean":"2023-11-08T05:28:01.457691+0000","last_became_active":"2023-11-08T05:28:13.563357+0000","last_became_peered":"2023-11-08T05:28:13.563357+0000","last_unstale":"2023-11-08T05:30:44.490629+0000","last_undegraded":"2023-11-08T05:30:44.490629+0000","last_fullsized":"2023-11-08T05:28:13.541614+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490691+0000","last_change":"2023-11-08T05:28:13.550672+0000","last_active":"2023-11-08T05:30:44.490691+0000","last_peered":"2023-11-08T05:30:44.490691+0000","last_clean":"2023-11-08T05:28:01.457904+0000","last_became_active":"2023-11-08T05:28:13.550672+0000","last_became_peered":"2023-11-08T05:28:13.550672+0000","last_unstale":"2023-11-08T05:30:44.490691+0000","last_undegraded":"2023-11-08T05:30:44.490691+0000","last_fullsized":"2023-11-08T05:28:13.539595+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490805+0000","last_change":"2023-11-08T05:28:13.557968+0000","last_active":"2023-11-08T05:30:44.490805+0000","last_peered":"2023-11-08T05:30:44.490805+0000","last_clean":"2023-11-08T05:28:01.458433+0000","last_became_active":"2023-11-08T05:28:13.557968+0000","last_became_peered":"2023-11-08T05:28:13.557968+0000","last_unstale":"2023-11-08T05:30:44.490805+0000","last_undegraded":"2023-11-08T05:30:44.490805+0000","last_fullsized":"2023-11-08T05:28:13.542832+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":1,"acting":1,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":12,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738854,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":645040,"kb_used_data":5352,"kb_used_omap":275,"kb_used_meta":639404,"kb_avail":93726800,"statfs":{"total":96636764160,"available":95976243200,"internally_reserved":0,"allocated":5480448,"data_stored":2697647,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":281650,"internal_metadata":654750670},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:59 2023","interfaces":[{"interface":"back","average":{"1min":0.30499999999999999,"5min":0.439,"15min":0.36099999999999999},"min":{"1min":0.17399999999999999,"5min":0.14499999999999999,"15min":0.14499999999999999},"max":{"1min":0.60099999999999998,"5min":4.9379999999999997,"15min":4.9379999999999997},"last":0.379},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.49399999999999999,"15min":0.42399999999999999},"min":{"1min":0.221,"5min":0.19600000000000001,"15min":0.17000000000000001},"max":{"1min":0.63200000000000001,"5min":4.9859999999999998,"15min":4.9859999999999998},"last":0.40899999999999997}]},{"osd":1,"last update":"Wed Nov 8 05:32:06 2023","interfaces":[{"interface":"back","average":{"1min":0.27400000000000002,"5min":0.307,"15min":0.46800000000000003},"min":{"1min":0.17599999999999999,"5min":0.16,"15min":0.153},"max":{"1min":0.39000000000000001,"5min":1.825,"15min":23.242999999999999},"last":0.221},{"interface":"front","average":{"1min":0.28000000000000003,"5min":0.34799999999999998,"15min":0.45700000000000002},"min":{"1min":0.20200000000000001,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":0.41699999999999998,"5min":2.0419999999999998,"15min":23.251000000000001},"last":0.23300000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738798,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1339228,"kb_used_data":398304,"kb_used_omap":291,"kb_used_meta":936476,"kb_avail":93032612,"statfs":{"total":96636764160,"available":95265394688,"internally_reserved":0,"allocated":407863296,"data_stored":405103900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":298462,"internal_metadata":958951970},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.436,"15min":0.36299999999999999},"min":{"1min":0.19700000000000001,"5min":0.14699999999999999,"15min":0.13700000000000001},"max":{"1min":3.222,"5min":6.5499999999999998,"15min":6.9989999999999997},"last":0.245},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.46300000000000002,"15min":0.42299999999999999},"min":{"1min":0.17799999999999999,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":5.0030000000000001,"5min":6.0629999999999997,"15min":9.6869999999999994},"last":0.28599999999999998}]},{"osd":2,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.51600000000000001,"15min":0.47099999999999997},"min":{"1min":0.21099999999999999,"5min":0.153,"15min":0.153},"max":{"1min":5.7439999999999998,"5min":5.8280000000000003,"15min":9.7639999999999993},"last":0.252},{"interface":"front","average":{"1min":0.497,"5min":0.56200000000000006,"15min":0.42299999999999999},"min":{"1min":0.20399999999999999,"5min":0.187,"15min":0.16800000000000001},"max":{"1min":3.9319999999999999,"5min":8.8040000000000003,"15min":8.8040000000000003},"last":0.29899999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738856,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":650100,"kb_used_data":5356,"kb_used_omap":214,"kb_used_meta":644521,"kb_avail":93721740,"statfs":{"total":96636764160,"available":95971061760,"internally_reserved":0,"allocated":5484544,"data_stored":2698616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":219335,"internal_metadata":659990329},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:56 2023","interfaces":[{"interface":"back","average":{"1min":0.30399999999999999,"5min":0.34699999999999998,"15min":0.36499999999999999},"min":{"1min":0.23999999999999999,"5min":0.14199999999999999,"15min":0.14199999999999999},"max":{"1min":0.42399999999999999,"5min":1.595,"15min":7.4249999999999998},"last":0.17100000000000001},{"interface":"front","average":{"1min":0.35499999999999998,"5min":0.40899999999999997,"15min":0.376},"min":{"1min":0.23999999999999999,"5min":0.19600000000000001,"15min":0.183},"max":{"1min":0.46999999999999997,"5min":2.6859999999999999,"15min":2.6859999999999999},"last":0.23000000000000001}]},{"osd":2,"last update":"Wed Nov 8 05:32:02 2023","interfaces":[{"interface":"back","average":{"1min":0.252,"5min":0.33000000000000002,"15min":0.36499999999999999},"min":{"1min":0.161,"5min":0.161,"15min":0.161},"max":{"1min":0.39600000000000002,"5min":2.2650000000000001,"15min":5.8209999999999997},"last":0.34399999999999997},{"interface":"front","average":{"1min":0.25800000000000001,"5min":0.29899999999999999,"15min":0.35599999999999998},"min":{"1min":0.17000000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.35799999999999998,"5min":1.137,"15min":4.2350000000000003},"last":0.35999999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-08T05:33:04.465 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~286s 2023-11-08T05:33:06.581 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-11-08T05:33:06.878 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T05:33:06.879 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2023-11-08T05:33:06.890 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":1667,"stamp":"2023-11-08T05:33:06.088674+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":10,"acting":13,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":109,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2634368,"kb_used_data":409012,"kb_used_omap":780,"kb_used_meta":2220403,"kb_avail":280481152,"statfs":{"total":289910292480,"available":287212699648,"internally_reserved":0,"allocated":418828288,"data_stored":410500163,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":799447,"internal_metadata":2273692969},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-536782902,"num_objects":-164,"num_object_clones":0,"num_object_copies":-328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-115,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-164,"num_whiteouts":0,"num_read":-122056,"num_read_kb":-39011942,"num_write":-413642,"num_write_kb":-151705339,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-8,"num_bytes_recovered":-16922624,"num_keys_recovered":-2,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.424817"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490644+0000","last_change":"2023-11-08T05:28:13.557071+0000","last_active":"2023-11-08T05:30:44.490644+0000","last_peered":"2023-11-08T05:30:44.490644+0000","last_clean":"2023-11-08T05:28:01.458439+0000","last_became_active":"2023-11-08T05:28:13.557071+0000","last_became_peered":"2023-11-08T05:28:13.557071+0000","last_unstale":"2023-11-08T05:30:44.490644+0000","last_undegraded":"2023-11-08T05:30:44.490644+0000","last_fullsized":"2023-11-08T05:28:13.547025+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":459,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:45.794760+0000","last_change":"2023-11-08T05:30:45.794760+0000","last_active":"2023-11-08T05:30:45.794760+0000","last_peered":"2023-11-08T05:30:45.794760+0000","last_clean":"2023-11-08T05:30:45.794760+0000","last_became_active":"2023-11-08T05:30:45.794398+0000","last_became_peered":"2023-11-08T05:30:45.794398+0000","last_unstale":"2023-11-08T05:30:45.794760+0000","last_undegraded":"2023-11-08T05:30:45.794760+0000","last_fullsized":"2023-11-08T05:30:45.794760+0000","mapping_epoch":458,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":459,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":465,"reported_epoch":459,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:45.795085+0000","last_change":"2023-11-08T05:30:45.795085+0000","last_active":"2023-11-08T05:30:45.795085+0000","last_peered":"2023-11-08T05:30:45.795085+0000","last_clean":"2023-11-08T05:30:45.795085+0000","last_became_active":"2023-11-08T05:30:45.794743+0000","last_became_peered":"2023-11-08T05:30:45.794743+0000","last_unstale":"2023-11-08T05:30:45.795085+0000","last_undegraded":"2023-11-08T05:30:45.795085+0000","last_fullsized":"2023-11-08T05:30:45.795085+0000","mapping_epoch":458,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":459,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":454,"reported_epoch":458,"state":"active+clean","last_fresh":"2023-11-08T05:30:44.490618+0000","last_change":"2023-11-08T04:52:47.548597+0000","last_active":"2023-11-08T05:30:44.490618+0000","last_peered":"2023-11-08T05:30:44.490618+0000","last_clean":"2023-11-08T05:30:44.490618+0000","last_became_active":"2023-11-08T04:52:47.548294+0000","last_became_peered":"2023-11-08T04:52:47.548294+0000","last_unstale":"2023-11-08T05:30:44.490618+0000","last_undegraded":"2023-11-08T05:30:44.490618+0000","last_fullsized":"2023-11-08T05:30:44.490618+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":460,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490523+0000","last_change":"2023-11-08T05:28:13.545468+0000","last_active":"2023-11-08T05:30:44.490523+0000","last_peered":"2023-11-08T05:30:44.490523+0000","last_clean":"2023-11-08T05:28:01.456723+0000","last_became_active":"2023-11-08T05:28:13.545468+0000","last_became_peered":"2023-11-08T05:28:13.545468+0000","last_unstale":"2023-11-08T05:30:44.490523+0000","last_undegraded":"2023-11-08T05:30:44.490523+0000","last_fullsized":"2023-11-08T05:28:13.537207+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_clean_scrub_stamp":"2023-11-08T04:52:45.528796+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":465,"reported_epoch":458,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:44.491907+0000","last_change":"2023-11-08T05:29:44.395350+0000","last_active":"2023-11-08T05:30:44.491907+0000","last_peered":"2023-11-08T05:30:44.491907+0000","last_clean":"2023-11-08T05:30:44.491907+0000","last_became_active":"2023-11-08T05:29:44.395012+0000","last_became_peered":"2023-11-08T05:29:44.395012+0000","last_unstale":"2023-11-08T05:30:44.491907+0000","last_undegraded":"2023-11-08T05:30:44.491907+0000","last_fullsized":"2023-11-08T05:30:44.491907+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490629+0000","last_change":"2023-11-08T05:28:13.563357+0000","last_active":"2023-11-08T05:30:44.490629+0000","last_peered":"2023-11-08T05:30:44.490629+0000","last_clean":"2023-11-08T05:28:01.457691+0000","last_became_active":"2023-11-08T05:28:13.563357+0000","last_became_peered":"2023-11-08T05:28:13.563357+0000","last_unstale":"2023-11-08T05:30:44.490629+0000","last_undegraded":"2023-11-08T05:30:44.490629+0000","last_fullsized":"2023-11-08T05:28:13.541614+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490691+0000","last_change":"2023-11-08T05:28:13.550672+0000","last_active":"2023-11-08T05:30:44.490691+0000","last_peered":"2023-11-08T05:30:44.490691+0000","last_clean":"2023-11-08T05:28:01.457904+0000","last_became_active":"2023-11-08T05:28:13.550672+0000","last_became_peered":"2023-11-08T05:28:13.550672+0000","last_unstale":"2023-11-08T05:30:44.490691+0000","last_undegraded":"2023-11-08T05:30:44.490691+0000","last_fullsized":"2023-11-08T05:28:13.539595+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490805+0000","last_change":"2023-11-08T05:28:13.557968+0000","last_active":"2023-11-08T05:30:44.490805+0000","last_peered":"2023-11-08T05:30:44.490805+0000","last_clean":"2023-11-08T05:28:01.458433+0000","last_became_active":"2023-11-08T05:28:13.557968+0000","last_became_peered":"2023-11-08T05:28:13.557968+0000","last_unstale":"2023-11-08T05:30:44.490805+0000","last_undegraded":"2023-11-08T05:30:44.490805+0000","last_fullsized":"2023-11-08T05:28:13.542832+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":1,"acting":1,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":12,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738855,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":645040,"kb_used_data":5352,"kb_used_omap":275,"kb_used_meta":639404,"kb_avail":93726800,"statfs":{"total":96636764160,"available":95976243200,"internally_reserved":0,"allocated":5480448,"data_stored":2697647,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":281650,"internal_metadata":654750670},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:59 2023","interfaces":[{"interface":"back","average":{"1min":0.30499999999999999,"5min":0.439,"15min":0.36099999999999999},"min":{"1min":0.17399999999999999,"5min":0.14499999999999999,"15min":0.14499999999999999},"max":{"1min":0.60099999999999998,"5min":4.9379999999999997,"15min":4.9379999999999997},"last":0.379},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.49399999999999999,"15min":0.42399999999999999},"min":{"1min":0.221,"5min":0.19600000000000001,"15min":0.17000000000000001},"max":{"1min":0.63200000000000001,"5min":4.9859999999999998,"15min":4.9859999999999998},"last":0.40899999999999997}]},{"osd":1,"last update":"Wed Nov 8 05:32:06 2023","interfaces":[{"interface":"back","average":{"1min":0.27400000000000002,"5min":0.307,"15min":0.46800000000000003},"min":{"1min":0.17599999999999999,"5min":0.16,"15min":0.153},"max":{"1min":0.39000000000000001,"5min":1.825,"15min":23.242999999999999},"last":0.372},{"interface":"front","average":{"1min":0.28000000000000003,"5min":0.34799999999999998,"15min":0.45700000000000002},"min":{"1min":0.20200000000000001,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":0.41699999999999998,"5min":2.0419999999999998,"15min":23.251000000000001},"last":0.39500000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738798,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1339228,"kb_used_data":398304,"kb_used_omap":291,"kb_used_meta":936476,"kb_avail":93032612,"statfs":{"total":96636764160,"available":95265394688,"internally_reserved":0,"allocated":407863296,"data_stored":405103900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":298462,"internal_metadata":958951970},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.436,"15min":0.36299999999999999},"min":{"1min":0.19700000000000001,"5min":0.14699999999999999,"15min":0.13700000000000001},"max":{"1min":3.222,"5min":6.5499999999999998,"15min":6.9989999999999997},"last":0.245},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.46300000000000002,"15min":0.42299999999999999},"min":{"1min":0.17799999999999999,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":5.0030000000000001,"5min":6.0629999999999997,"15min":9.6869999999999994},"last":0.28599999999999998}]},{"osd":2,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.51600000000000001,"15min":0.47099999999999997},"min":{"1min":0.21099999999999999,"5min":0.153,"15min":0.153},"max":{"1min":5.7439999999999998,"5min":5.8280000000000003,"15min":9.7639999999999993},"last":0.252},{"interface":"front","average":{"1min":0.497,"5min":0.56200000000000006,"15min":0.42299999999999999},"min":{"1min":0.20399999999999999,"5min":0.187,"15min":0.16800000000000001},"max":{"1min":3.9319999999999999,"5min":8.8040000000000003,"15min":8.8040000000000003},"last":0.29899999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738857,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":650100,"kb_used_data":5356,"kb_used_omap":214,"kb_used_meta":644521,"kb_avail":93721740,"statfs":{"total":96636764160,"available":95971061760,"internally_reserved":0,"allocated":5484544,"data_stored":2698616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":219335,"internal_metadata":659990329},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:56 2023","interfaces":[{"interface":"back","average":{"1min":0.30399999999999999,"5min":0.34699999999999998,"15min":0.36499999999999999},"min":{"1min":0.23999999999999999,"5min":0.14199999999999999,"15min":0.14199999999999999},"max":{"1min":0.42399999999999999,"5min":1.595,"15min":7.4249999999999998},"last":0.17100000000000001},{"interface":"front","average":{"1min":0.35499999999999998,"5min":0.40899999999999997,"15min":0.376},"min":{"1min":0.23999999999999999,"5min":0.19600000000000001,"15min":0.183},"max":{"1min":0.46999999999999997,"5min":2.6859999999999999,"15min":2.6859999999999999},"last":0.23000000000000001}]},{"osd":2,"last update":"Wed Nov 8 05:33:03 2023","interfaces":[{"interface":"back","average":{"1min":0.26000000000000001,"5min":0.317,"15min":0.35999999999999999},"min":{"1min":0.18099999999999999,"5min":0.161,"15min":0.161},"max":{"1min":0.40799999999999997,"5min":2.2650000000000001,"15min":5.8209999999999997},"last":0.24399999999999999},{"interface":"front","average":{"1min":0.26900000000000002,"5min":0.28699999999999998,"15min":0.35199999999999998},"min":{"1min":0.192,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.41999999999999998,"5min":1.137,"15min":4.2350000000000003},"last":0.254}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-08T05:33:06.891 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-11-08T05:33:07.173 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T05:33:07.175 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2023-11-08T05:33:07.186 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":1667,"stamp":"2023-11-08T05:33:06.088674+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":10,"acting":13,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":109,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2634368,"kb_used_data":409012,"kb_used_omap":780,"kb_used_meta":2220403,"kb_avail":280481152,"statfs":{"total":289910292480,"available":287212699648,"internally_reserved":0,"allocated":418828288,"data_stored":410500163,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":799447,"internal_metadata":2273692969},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-536782902,"num_objects":-164,"num_object_clones":0,"num_object_copies":-328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-115,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-164,"num_whiteouts":0,"num_read":-122056,"num_read_kb":-39011942,"num_write":-413642,"num_write_kb":-151705339,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-8,"num_bytes_recovered":-16922624,"num_keys_recovered":-2,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.424817"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490644+0000","last_change":"2023-11-08T05:28:13.557071+0000","last_active":"2023-11-08T05:30:44.490644+0000","last_peered":"2023-11-08T05:30:44.490644+0000","last_clean":"2023-11-08T05:28:01.458439+0000","last_became_active":"2023-11-08T05:28:13.557071+0000","last_became_peered":"2023-11-08T05:28:13.557071+0000","last_unstale":"2023-11-08T05:30:44.490644+0000","last_undegraded":"2023-11-08T05:30:44.490644+0000","last_fullsized":"2023-11-08T05:28:13.547025+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":459,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:45.794760+0000","last_change":"2023-11-08T05:30:45.794760+0000","last_active":"2023-11-08T05:30:45.794760+0000","last_peered":"2023-11-08T05:30:45.794760+0000","last_clean":"2023-11-08T05:30:45.794760+0000","last_became_active":"2023-11-08T05:30:45.794398+0000","last_became_peered":"2023-11-08T05:30:45.794398+0000","last_unstale":"2023-11-08T05:30:45.794760+0000","last_undegraded":"2023-11-08T05:30:45.794760+0000","last_fullsized":"2023-11-08T05:30:45.794760+0000","mapping_epoch":458,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":459,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":465,"reported_epoch":459,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:45.795085+0000","last_change":"2023-11-08T05:30:45.795085+0000","last_active":"2023-11-08T05:30:45.795085+0000","last_peered":"2023-11-08T05:30:45.795085+0000","last_clean":"2023-11-08T05:30:45.795085+0000","last_became_active":"2023-11-08T05:30:45.794743+0000","last_became_peered":"2023-11-08T05:30:45.794743+0000","last_unstale":"2023-11-08T05:30:45.795085+0000","last_undegraded":"2023-11-08T05:30:45.795085+0000","last_fullsized":"2023-11-08T05:30:45.795085+0000","mapping_epoch":458,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":459,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":454,"reported_epoch":458,"state":"active+clean","last_fresh":"2023-11-08T05:30:44.490618+0000","last_change":"2023-11-08T04:52:47.548597+0000","last_active":"2023-11-08T05:30:44.490618+0000","last_peered":"2023-11-08T05:30:44.490618+0000","last_clean":"2023-11-08T05:30:44.490618+0000","last_became_active":"2023-11-08T04:52:47.548294+0000","last_became_peered":"2023-11-08T04:52:47.548294+0000","last_unstale":"2023-11-08T05:30:44.490618+0000","last_undegraded":"2023-11-08T05:30:44.490618+0000","last_fullsized":"2023-11-08T05:30:44.490618+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":460,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490523+0000","last_change":"2023-11-08T05:28:13.545468+0000","last_active":"2023-11-08T05:30:44.490523+0000","last_peered":"2023-11-08T05:30:44.490523+0000","last_clean":"2023-11-08T05:28:01.456723+0000","last_became_active":"2023-11-08T05:28:13.545468+0000","last_became_peered":"2023-11-08T05:28:13.545468+0000","last_unstale":"2023-11-08T05:30:44.490523+0000","last_undegraded":"2023-11-08T05:30:44.490523+0000","last_fullsized":"2023-11-08T05:28:13.537207+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_clean_scrub_stamp":"2023-11-08T04:52:45.528796+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":465,"reported_epoch":458,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:44.491907+0000","last_change":"2023-11-08T05:29:44.395350+0000","last_active":"2023-11-08T05:30:44.491907+0000","last_peered":"2023-11-08T05:30:44.491907+0000","last_clean":"2023-11-08T05:30:44.491907+0000","last_became_active":"2023-11-08T05:29:44.395012+0000","last_became_peered":"2023-11-08T05:29:44.395012+0000","last_unstale":"2023-11-08T05:30:44.491907+0000","last_undegraded":"2023-11-08T05:30:44.491907+0000","last_fullsized":"2023-11-08T05:30:44.491907+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490629+0000","last_change":"2023-11-08T05:28:13.563357+0000","last_active":"2023-11-08T05:30:44.490629+0000","last_peered":"2023-11-08T05:30:44.490629+0000","last_clean":"2023-11-08T05:28:01.457691+0000","last_became_active":"2023-11-08T05:28:13.563357+0000","last_became_peered":"2023-11-08T05:28:13.563357+0000","last_unstale":"2023-11-08T05:30:44.490629+0000","last_undegraded":"2023-11-08T05:30:44.490629+0000","last_fullsized":"2023-11-08T05:28:13.541614+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490691+0000","last_change":"2023-11-08T05:28:13.550672+0000","last_active":"2023-11-08T05:30:44.490691+0000","last_peered":"2023-11-08T05:30:44.490691+0000","last_clean":"2023-11-08T05:28:01.457904+0000","last_became_active":"2023-11-08T05:28:13.550672+0000","last_became_peered":"2023-11-08T05:28:13.550672+0000","last_unstale":"2023-11-08T05:30:44.490691+0000","last_undegraded":"2023-11-08T05:30:44.490691+0000","last_fullsized":"2023-11-08T05:28:13.539595+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490805+0000","last_change":"2023-11-08T05:28:13.557968+0000","last_active":"2023-11-08T05:30:44.490805+0000","last_peered":"2023-11-08T05:30:44.490805+0000","last_clean":"2023-11-08T05:28:01.458433+0000","last_became_active":"2023-11-08T05:28:13.557968+0000","last_became_peered":"2023-11-08T05:28:13.557968+0000","last_unstale":"2023-11-08T05:30:44.490805+0000","last_undegraded":"2023-11-08T05:30:44.490805+0000","last_fullsized":"2023-11-08T05:28:13.542832+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":1,"acting":1,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":12,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738855,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":645040,"kb_used_data":5352,"kb_used_omap":275,"kb_used_meta":639404,"kb_avail":93726800,"statfs":{"total":96636764160,"available":95976243200,"internally_reserved":0,"allocated":5480448,"data_stored":2697647,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":281650,"internal_metadata":654750670},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:59 2023","interfaces":[{"interface":"back","average":{"1min":0.30499999999999999,"5min":0.439,"15min":0.36099999999999999},"min":{"1min":0.17399999999999999,"5min":0.14499999999999999,"15min":0.14499999999999999},"max":{"1min":0.60099999999999998,"5min":4.9379999999999997,"15min":4.9379999999999997},"last":0.379},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.49399999999999999,"15min":0.42399999999999999},"min":{"1min":0.221,"5min":0.19600000000000001,"15min":0.17000000000000001},"max":{"1min":0.63200000000000001,"5min":4.9859999999999998,"15min":4.9859999999999998},"last":0.40899999999999997}]},{"osd":1,"last update":"Wed Nov 8 05:32:06 2023","interfaces":[{"interface":"back","average":{"1min":0.27400000000000002,"5min":0.307,"15min":0.46800000000000003},"min":{"1min":0.17599999999999999,"5min":0.16,"15min":0.153},"max":{"1min":0.39000000000000001,"5min":1.825,"15min":23.242999999999999},"last":0.372},{"interface":"front","average":{"1min":0.28000000000000003,"5min":0.34799999999999998,"15min":0.45700000000000002},"min":{"1min":0.20200000000000001,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":0.41699999999999998,"5min":2.0419999999999998,"15min":23.251000000000001},"last":0.39500000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738798,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1339228,"kb_used_data":398304,"kb_used_omap":291,"kb_used_meta":936476,"kb_avail":93032612,"statfs":{"total":96636764160,"available":95265394688,"internally_reserved":0,"allocated":407863296,"data_stored":405103900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":298462,"internal_metadata":958951970},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.436,"15min":0.36299999999999999},"min":{"1min":0.19700000000000001,"5min":0.14699999999999999,"15min":0.13700000000000001},"max":{"1min":3.222,"5min":6.5499999999999998,"15min":6.9989999999999997},"last":0.245},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.46300000000000002,"15min":0.42299999999999999},"min":{"1min":0.17799999999999999,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":5.0030000000000001,"5min":6.0629999999999997,"15min":9.6869999999999994},"last":0.28599999999999998}]},{"osd":2,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.51600000000000001,"15min":0.47099999999999997},"min":{"1min":0.21099999999999999,"5min":0.153,"15min":0.153},"max":{"1min":5.7439999999999998,"5min":5.8280000000000003,"15min":9.7639999999999993},"last":0.252},{"interface":"front","average":{"1min":0.497,"5min":0.56200000000000006,"15min":0.42299999999999999},"min":{"1min":0.20399999999999999,"5min":0.187,"15min":0.16800000000000001},"max":{"1min":3.9319999999999999,"5min":8.8040000000000003,"15min":8.8040000000000003},"last":0.29899999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738857,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":650100,"kb_used_data":5356,"kb_used_omap":214,"kb_used_meta":644521,"kb_avail":93721740,"statfs":{"total":96636764160,"available":95971061760,"internally_reserved":0,"allocated":5484544,"data_stored":2698616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":219335,"internal_metadata":659990329},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:56 2023","interfaces":[{"interface":"back","average":{"1min":0.30399999999999999,"5min":0.34699999999999998,"15min":0.36499999999999999},"min":{"1min":0.23999999999999999,"5min":0.14199999999999999,"15min":0.14199999999999999},"max":{"1min":0.42399999999999999,"5min":1.595,"15min":7.4249999999999998},"last":0.17100000000000001},{"interface":"front","average":{"1min":0.35499999999999998,"5min":0.40899999999999997,"15min":0.376},"min":{"1min":0.23999999999999999,"5min":0.19600000000000001,"15min":0.183},"max":{"1min":0.46999999999999997,"5min":2.6859999999999999,"15min":2.6859999999999999},"last":0.23000000000000001}]},{"osd":2,"last update":"Wed Nov 8 05:33:03 2023","interfaces":[{"interface":"back","average":{"1min":0.26000000000000001,"5min":0.317,"15min":0.35999999999999999},"min":{"1min":0.18099999999999999,"5min":0.161,"15min":0.161},"max":{"1min":0.40799999999999997,"5min":2.2650000000000001,"15min":5.8209999999999997},"last":0.24399999999999999},{"interface":"front","average":{"1min":0.26900000000000002,"5min":0.28699999999999998,"15min":0.35199999999999998},"min":{"1min":0.192,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.41999999999999998,"5min":1.137,"15min":4.2350000000000003},"last":0.254}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-08T05:33:07.187 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-11-08T05:33:07.188 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 459, 'reported_epoch': 458, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:30:44.490644+0000', 'last_change': '2023-11-08T05:28:13.557071+0000', 'last_active': '2023-11-08T05:30:44.490644+0000', 'last_peered': '2023-11-08T05:30:44.490644+0000', 'last_clean': '2023-11-08T05:28:01.458439+0000', 'last_became_active': '2023-11-08T05:28:13.557071+0000', 'last_became_peered': '2023-11-08T05:28:13.557071+0000', 'last_unstale': '2023-11-08T05:30:44.490644+0000', 'last_undegraded': '2023-11-08T05:30:44.490644+0000', 'last_fullsized': '2023-11-08T05:28:13.547025+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:33:07.188 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-11-08T05:33:07.188 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 11, 'reported_epoch': 459, 'state': 'active+clean+remapped', 'last_fresh': '2023-11-08T05:30:45.794760+0000', 'last_change': '2023-11-08T05:30:45.794760+0000', 'last_active': '2023-11-08T05:30:45.794760+0000', 'last_peered': '2023-11-08T05:30:45.794760+0000', 'last_clean': '2023-11-08T05:30:45.794760+0000', 'last_became_active': '2023-11-08T05:30:45.794398+0000', 'last_became_peered': '2023-11-08T05:30:45.794398+0000', 'last_unstale': '2023-11-08T05:30:45.794760+0000', 'last_undegraded': '2023-11-08T05:30:45.794760+0000', 'last_fullsized': '2023-11-08T05:30:45.794760+0000', 'mapping_epoch': 458, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 459, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2], 'acting': [2, 1], 'avail_no_missing': ['2', '1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2023-11-08T05:33:07.189 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.1 is not active+clean 2023-11-08T05:33:07.189 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.1', 'version': "0'0", 'reported_seq': 465, 'reported_epoch': 459, 'state': 'active+clean+remapped', 'last_fresh': '2023-11-08T05:30:45.795085+0000', 'last_change': '2023-11-08T05:30:45.795085+0000', 'last_active': '2023-11-08T05:30:45.795085+0000', 'last_peered': '2023-11-08T05:30:45.795085+0000', 'last_clean': '2023-11-08T05:30:45.795085+0000', 'last_became_active': '2023-11-08T05:30:45.794743+0000', 'last_became_peered': '2023-11-08T05:30:45.794743+0000', 'last_unstale': '2023-11-08T05:30:45.795085+0000', 'last_undegraded': '2023-11-08T05:30:45.795085+0000', 'last_fullsized': '2023-11-08T05:30:45.795085+0000', 'mapping_epoch': 458, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 459, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2], 'acting': [2, 1], 'avail_no_missing': ['2', '1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2023-11-08T05:33:07.189 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-11-08T05:33:07.190 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "0'0", 'reported_seq': 460, 'reported_epoch': 458, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:30:44.490523+0000', 'last_change': '2023-11-08T05:28:13.545468+0000', 'last_active': '2023-11-08T05:30:44.490523+0000', 'last_peered': '2023-11-08T05:30:44.490523+0000', 'last_clean': '2023-11-08T05:28:01.456723+0000', 'last_became_active': '2023-11-08T05:28:13.545468+0000', 'last_became_peered': '2023-11-08T05:28:13.545468+0000', 'last_unstale': '2023-11-08T05:30:44.490523+0000', 'last_undegraded': '2023-11-08T05:30:44.490523+0000', 'last_fullsized': '2023-11-08T05:28:13.537207+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 9, 'last_epoch_clean': 10, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:45.528796+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:45.528796+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:45.528796+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:33:07.190 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.0 is not active+clean 2023-11-08T05:33:07.190 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.0', 'version': "0'0", 'reported_seq': 465, 'reported_epoch': 458, 'state': 'active+clean+remapped', 'last_fresh': '2023-11-08T05:30:44.491907+0000', 'last_change': '2023-11-08T05:29:44.395350+0000', 'last_active': '2023-11-08T05:30:44.491907+0000', 'last_peered': '2023-11-08T05:30:44.491907+0000', 'last_clean': '2023-11-08T05:30:44.491907+0000', 'last_became_active': '2023-11-08T05:29:44.395012+0000', 'last_became_peered': '2023-11-08T05:29:44.395012+0000', 'last_unstale': '2023-11-08T05:30:44.491907+0000', 'last_undegraded': '2023-11-08T05:30:44.491907+0000', 'last_fullsized': '2023-11-08T05:30:44.491907+0000', 'mapping_epoch': 455, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 456, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2], 'acting': [2, 1], 'avail_no_missing': ['2', '1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2023-11-08T05:33:07.191 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-11-08T05:33:07.191 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 459, 'reported_epoch': 458, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:30:44.490629+0000', 'last_change': '2023-11-08T05:28:13.563357+0000', 'last_active': '2023-11-08T05:30:44.490629+0000', 'last_peered': '2023-11-08T05:30:44.490629+0000', 'last_clean': '2023-11-08T05:28:01.457691+0000', 'last_became_active': '2023-11-08T05:28:13.563357+0000', 'last_became_peered': '2023-11-08T05:28:13.563357+0000', 'last_unstale': '2023-11-08T05:30:44.490629+0000', 'last_undegraded': '2023-11-08T05:30:44.490629+0000', 'last_fullsized': '2023-11-08T05:28:13.541614+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:33:07.191 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-11-08T05:33:07.191 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 459, 'reported_epoch': 458, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:30:44.490691+0000', 'last_change': '2023-11-08T05:28:13.550672+0000', 'last_active': '2023-11-08T05:30:44.490691+0000', 'last_peered': '2023-11-08T05:30:44.490691+0000', 'last_clean': '2023-11-08T05:28:01.457904+0000', 'last_became_active': '2023-11-08T05:28:13.550672+0000', 'last_became_peered': '2023-11-08T05:28:13.550672+0000', 'last_unstale': '2023-11-08T05:30:44.490691+0000', 'last_undegraded': '2023-11-08T05:30:44.490691+0000', 'last_fullsized': '2023-11-08T05:28:13.539595+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:33:07.192 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-11-08T05:33:07.192 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 459, 'reported_epoch': 458, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:30:44.490805+0000', 'last_change': '2023-11-08T05:28:13.557968+0000', 'last_active': '2023-11-08T05:30:44.490805+0000', 'last_peered': '2023-11-08T05:30:44.490805+0000', 'last_clean': '2023-11-08T05:28:01.458433+0000', 'last_became_active': '2023-11-08T05:28:13.557968+0000', 'last_became_peered': '2023-11-08T05:28:13.557968+0000', 'last_unstale': '2023-11-08T05:30:44.490805+0000', 'last_undegraded': '2023-11-08T05:30:44.490805+0000', 'last_fullsized': '2023-11-08T05:28:13.542832+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:33:07.192 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-11-08T05:33:07.539 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T05:33:07.540 INFO:teuthology.orchestra.run.smithi087.stdout:{"fsid":"ab4cbc06-449d-497d-b513-220d9db2b590","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 5 pgs undersized","count":5},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2430,"monmap":{"epoch":1,"min_mon_release_name":"pacific","num_mons":1},"osdmap":{"epoch":459,"num_osds":3,"num_up_osds":2,"osd_up_since":1699421292,"num_in_osds":3,"osd_in_since":1699419158,"num_remapped_pgs":3},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean+remapped","count":3},{"state_name":"active+clean","count":1}],"num_pgs":9,"num_pools":2,"num_objects":0,"data_bytes":0,"bytes_used":2697592832,"bytes_avail":287212699648,"bytes_total":289910292480},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":41,"modified":"2023-11-08T05:29:32.044544+0000","services":{}},"progress_events":{}} 2023-11-08T05:33:07.553 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-11-08T05:33:07.554 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-11-08T05:33:07.837 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T05:33:07.838 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2023-11-08T05:33:07.849 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":1667,"stamp":"2023-11-08T05:33:06.088674+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":10,"acting":13,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":109,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2634368,"kb_used_data":409012,"kb_used_omap":780,"kb_used_meta":2220403,"kb_avail":280481152,"statfs":{"total":289910292480,"available":287212699648,"internally_reserved":0,"allocated":418828288,"data_stored":410500163,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":799447,"internal_metadata":2273692969},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-536782902,"num_objects":-164,"num_object_clones":0,"num_object_copies":-328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-115,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-164,"num_whiteouts":0,"num_read":-122056,"num_read_kb":-39011942,"num_write":-413642,"num_write_kb":-151705339,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-8,"num_bytes_recovered":-16922624,"num_keys_recovered":-2,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.424817"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490644+0000","last_change":"2023-11-08T05:28:13.557071+0000","last_active":"2023-11-08T05:30:44.490644+0000","last_peered":"2023-11-08T05:30:44.490644+0000","last_clean":"2023-11-08T05:28:01.458439+0000","last_became_active":"2023-11-08T05:28:13.557071+0000","last_became_peered":"2023-11-08T05:28:13.557071+0000","last_unstale":"2023-11-08T05:30:44.490644+0000","last_undegraded":"2023-11-08T05:30:44.490644+0000","last_fullsized":"2023-11-08T05:28:13.547025+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":459,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:45.794760+0000","last_change":"2023-11-08T05:30:45.794760+0000","last_active":"2023-11-08T05:30:45.794760+0000","last_peered":"2023-11-08T05:30:45.794760+0000","last_clean":"2023-11-08T05:30:45.794760+0000","last_became_active":"2023-11-08T05:30:45.794398+0000","last_became_peered":"2023-11-08T05:30:45.794398+0000","last_unstale":"2023-11-08T05:30:45.794760+0000","last_undegraded":"2023-11-08T05:30:45.794760+0000","last_fullsized":"2023-11-08T05:30:45.794760+0000","mapping_epoch":458,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":459,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":465,"reported_epoch":459,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:45.795085+0000","last_change":"2023-11-08T05:30:45.795085+0000","last_active":"2023-11-08T05:30:45.795085+0000","last_peered":"2023-11-08T05:30:45.795085+0000","last_clean":"2023-11-08T05:30:45.795085+0000","last_became_active":"2023-11-08T05:30:45.794743+0000","last_became_peered":"2023-11-08T05:30:45.794743+0000","last_unstale":"2023-11-08T05:30:45.795085+0000","last_undegraded":"2023-11-08T05:30:45.795085+0000","last_fullsized":"2023-11-08T05:30:45.795085+0000","mapping_epoch":458,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":459,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":454,"reported_epoch":458,"state":"active+clean","last_fresh":"2023-11-08T05:30:44.490618+0000","last_change":"2023-11-08T04:52:47.548597+0000","last_active":"2023-11-08T05:30:44.490618+0000","last_peered":"2023-11-08T05:30:44.490618+0000","last_clean":"2023-11-08T05:30:44.490618+0000","last_became_active":"2023-11-08T04:52:47.548294+0000","last_became_peered":"2023-11-08T04:52:47.548294+0000","last_unstale":"2023-11-08T05:30:44.490618+0000","last_undegraded":"2023-11-08T05:30:44.490618+0000","last_fullsized":"2023-11-08T05:30:44.490618+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":460,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490523+0000","last_change":"2023-11-08T05:28:13.545468+0000","last_active":"2023-11-08T05:30:44.490523+0000","last_peered":"2023-11-08T05:30:44.490523+0000","last_clean":"2023-11-08T05:28:01.456723+0000","last_became_active":"2023-11-08T05:28:13.545468+0000","last_became_peered":"2023-11-08T05:28:13.545468+0000","last_unstale":"2023-11-08T05:30:44.490523+0000","last_undegraded":"2023-11-08T05:30:44.490523+0000","last_fullsized":"2023-11-08T05:28:13.537207+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_clean_scrub_stamp":"2023-11-08T04:52:45.528796+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":465,"reported_epoch":458,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:44.491907+0000","last_change":"2023-11-08T05:29:44.395350+0000","last_active":"2023-11-08T05:30:44.491907+0000","last_peered":"2023-11-08T05:30:44.491907+0000","last_clean":"2023-11-08T05:30:44.491907+0000","last_became_active":"2023-11-08T05:29:44.395012+0000","last_became_peered":"2023-11-08T05:29:44.395012+0000","last_unstale":"2023-11-08T05:30:44.491907+0000","last_undegraded":"2023-11-08T05:30:44.491907+0000","last_fullsized":"2023-11-08T05:30:44.491907+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490629+0000","last_change":"2023-11-08T05:28:13.563357+0000","last_active":"2023-11-08T05:30:44.490629+0000","last_peered":"2023-11-08T05:30:44.490629+0000","last_clean":"2023-11-08T05:28:01.457691+0000","last_became_active":"2023-11-08T05:28:13.563357+0000","last_became_peered":"2023-11-08T05:28:13.563357+0000","last_unstale":"2023-11-08T05:30:44.490629+0000","last_undegraded":"2023-11-08T05:30:44.490629+0000","last_fullsized":"2023-11-08T05:28:13.541614+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490691+0000","last_change":"2023-11-08T05:28:13.550672+0000","last_active":"2023-11-08T05:30:44.490691+0000","last_peered":"2023-11-08T05:30:44.490691+0000","last_clean":"2023-11-08T05:28:01.457904+0000","last_became_active":"2023-11-08T05:28:13.550672+0000","last_became_peered":"2023-11-08T05:28:13.550672+0000","last_unstale":"2023-11-08T05:30:44.490691+0000","last_undegraded":"2023-11-08T05:30:44.490691+0000","last_fullsized":"2023-11-08T05:28:13.539595+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490805+0000","last_change":"2023-11-08T05:28:13.557968+0000","last_active":"2023-11-08T05:30:44.490805+0000","last_peered":"2023-11-08T05:30:44.490805+0000","last_clean":"2023-11-08T05:28:01.458433+0000","last_became_active":"2023-11-08T05:28:13.557968+0000","last_became_peered":"2023-11-08T05:28:13.557968+0000","last_unstale":"2023-11-08T05:30:44.490805+0000","last_undegraded":"2023-11-08T05:30:44.490805+0000","last_fullsized":"2023-11-08T05:28:13.542832+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":1,"acting":1,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":12,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738855,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":645040,"kb_used_data":5352,"kb_used_omap":275,"kb_used_meta":639404,"kb_avail":93726800,"statfs":{"total":96636764160,"available":95976243200,"internally_reserved":0,"allocated":5480448,"data_stored":2697647,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":281650,"internal_metadata":654750670},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:59 2023","interfaces":[{"interface":"back","average":{"1min":0.30499999999999999,"5min":0.439,"15min":0.36099999999999999},"min":{"1min":0.17399999999999999,"5min":0.14499999999999999,"15min":0.14499999999999999},"max":{"1min":0.60099999999999998,"5min":4.9379999999999997,"15min":4.9379999999999997},"last":0.379},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.49399999999999999,"15min":0.42399999999999999},"min":{"1min":0.221,"5min":0.19600000000000001,"15min":0.17000000000000001},"max":{"1min":0.63200000000000001,"5min":4.9859999999999998,"15min":4.9859999999999998},"last":0.40899999999999997}]},{"osd":1,"last update":"Wed Nov 8 05:32:06 2023","interfaces":[{"interface":"back","average":{"1min":0.27400000000000002,"5min":0.307,"15min":0.46800000000000003},"min":{"1min":0.17599999999999999,"5min":0.16,"15min":0.153},"max":{"1min":0.39000000000000001,"5min":1.825,"15min":23.242999999999999},"last":0.372},{"interface":"front","average":{"1min":0.28000000000000003,"5min":0.34799999999999998,"15min":0.45700000000000002},"min":{"1min":0.20200000000000001,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":0.41699999999999998,"5min":2.0419999999999998,"15min":23.251000000000001},"last":0.39500000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738798,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1339228,"kb_used_data":398304,"kb_used_omap":291,"kb_used_meta":936476,"kb_avail":93032612,"statfs":{"total":96636764160,"available":95265394688,"internally_reserved":0,"allocated":407863296,"data_stored":405103900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":298462,"internal_metadata":958951970},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.436,"15min":0.36299999999999999},"min":{"1min":0.19700000000000001,"5min":0.14699999999999999,"15min":0.13700000000000001},"max":{"1min":3.222,"5min":6.5499999999999998,"15min":6.9989999999999997},"last":0.245},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.46300000000000002,"15min":0.42299999999999999},"min":{"1min":0.17799999999999999,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":5.0030000000000001,"5min":6.0629999999999997,"15min":9.6869999999999994},"last":0.28599999999999998}]},{"osd":2,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.51600000000000001,"15min":0.47099999999999997},"min":{"1min":0.21099999999999999,"5min":0.153,"15min":0.153},"max":{"1min":5.7439999999999998,"5min":5.8280000000000003,"15min":9.7639999999999993},"last":0.252},{"interface":"front","average":{"1min":0.497,"5min":0.56200000000000006,"15min":0.42299999999999999},"min":{"1min":0.20399999999999999,"5min":0.187,"15min":0.16800000000000001},"max":{"1min":3.9319999999999999,"5min":8.8040000000000003,"15min":8.8040000000000003},"last":0.29899999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738857,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":650100,"kb_used_data":5356,"kb_used_omap":214,"kb_used_meta":644521,"kb_avail":93721740,"statfs":{"total":96636764160,"available":95971061760,"internally_reserved":0,"allocated":5484544,"data_stored":2698616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":219335,"internal_metadata":659990329},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:56 2023","interfaces":[{"interface":"back","average":{"1min":0.30399999999999999,"5min":0.34699999999999998,"15min":0.36499999999999999},"min":{"1min":0.23999999999999999,"5min":0.14199999999999999,"15min":0.14199999999999999},"max":{"1min":0.42399999999999999,"5min":1.595,"15min":7.4249999999999998},"last":0.17100000000000001},{"interface":"front","average":{"1min":0.35499999999999998,"5min":0.40899999999999997,"15min":0.376},"min":{"1min":0.23999999999999999,"5min":0.19600000000000001,"15min":0.183},"max":{"1min":0.46999999999999997,"5min":2.6859999999999999,"15min":2.6859999999999999},"last":0.23000000000000001}]},{"osd":2,"last update":"Wed Nov 8 05:33:03 2023","interfaces":[{"interface":"back","average":{"1min":0.26000000000000001,"5min":0.317,"15min":0.35999999999999999},"min":{"1min":0.18099999999999999,"5min":0.161,"15min":0.161},"max":{"1min":0.40799999999999997,"5min":2.2650000000000001,"15min":5.8209999999999997},"last":0.24399999999999999},{"interface":"front","average":{"1min":0.26900000000000002,"5min":0.28699999999999998,"15min":0.35199999999999998},"min":{"1min":0.192,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.41999999999999998,"5min":1.137,"15min":4.2350000000000003},"last":0.254}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-08T05:33:09.869 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~292s 2023-11-08T05:33:10.851 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-11-08T05:33:11.148 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T05:33:11.149 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2023-11-08T05:33:11.160 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":1669,"stamp":"2023-11-08T05:33:10.089729+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":10,"acting":13,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":109,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2634368,"kb_used_data":409012,"kb_used_omap":780,"kb_used_meta":2220403,"kb_avail":280481152,"statfs":{"total":289910292480,"available":287212699648,"internally_reserved":0,"allocated":418828288,"data_stored":410500163,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":799447,"internal_metadata":2273692969},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-536782902,"num_objects":-164,"num_object_clones":0,"num_object_copies":-328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-115,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-164,"num_whiteouts":0,"num_read":-122056,"num_read_kb":-39011942,"num_write":-413642,"num_write_kb":-151705339,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-8,"num_bytes_recovered":-16922624,"num_keys_recovered":-2,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.424817"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490644+0000","last_change":"2023-11-08T05:28:13.557071+0000","last_active":"2023-11-08T05:30:44.490644+0000","last_peered":"2023-11-08T05:30:44.490644+0000","last_clean":"2023-11-08T05:28:01.458439+0000","last_became_active":"2023-11-08T05:28:13.557071+0000","last_became_peered":"2023-11-08T05:28:13.557071+0000","last_unstale":"2023-11-08T05:30:44.490644+0000","last_undegraded":"2023-11-08T05:30:44.490644+0000","last_fullsized":"2023-11-08T05:28:13.547025+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":459,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:45.794760+0000","last_change":"2023-11-08T05:30:45.794760+0000","last_active":"2023-11-08T05:30:45.794760+0000","last_peered":"2023-11-08T05:30:45.794760+0000","last_clean":"2023-11-08T05:30:45.794760+0000","last_became_active":"2023-11-08T05:30:45.794398+0000","last_became_peered":"2023-11-08T05:30:45.794398+0000","last_unstale":"2023-11-08T05:30:45.794760+0000","last_undegraded":"2023-11-08T05:30:45.794760+0000","last_fullsized":"2023-11-08T05:30:45.794760+0000","mapping_epoch":458,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":459,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":465,"reported_epoch":459,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:45.795085+0000","last_change":"2023-11-08T05:30:45.795085+0000","last_active":"2023-11-08T05:30:45.795085+0000","last_peered":"2023-11-08T05:30:45.795085+0000","last_clean":"2023-11-08T05:30:45.795085+0000","last_became_active":"2023-11-08T05:30:45.794743+0000","last_became_peered":"2023-11-08T05:30:45.794743+0000","last_unstale":"2023-11-08T05:30:45.795085+0000","last_undegraded":"2023-11-08T05:30:45.795085+0000","last_fullsized":"2023-11-08T05:30:45.795085+0000","mapping_epoch":458,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":459,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":454,"reported_epoch":458,"state":"active+clean","last_fresh":"2023-11-08T05:30:44.490618+0000","last_change":"2023-11-08T04:52:47.548597+0000","last_active":"2023-11-08T05:30:44.490618+0000","last_peered":"2023-11-08T05:30:44.490618+0000","last_clean":"2023-11-08T05:30:44.490618+0000","last_became_active":"2023-11-08T04:52:47.548294+0000","last_became_peered":"2023-11-08T04:52:47.548294+0000","last_unstale":"2023-11-08T05:30:44.490618+0000","last_undegraded":"2023-11-08T05:30:44.490618+0000","last_fullsized":"2023-11-08T05:30:44.490618+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":460,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490523+0000","last_change":"2023-11-08T05:28:13.545468+0000","last_active":"2023-11-08T05:30:44.490523+0000","last_peered":"2023-11-08T05:30:44.490523+0000","last_clean":"2023-11-08T05:28:01.456723+0000","last_became_active":"2023-11-08T05:28:13.545468+0000","last_became_peered":"2023-11-08T05:28:13.545468+0000","last_unstale":"2023-11-08T05:30:44.490523+0000","last_undegraded":"2023-11-08T05:30:44.490523+0000","last_fullsized":"2023-11-08T05:28:13.537207+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_clean_scrub_stamp":"2023-11-08T04:52:45.528796+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":465,"reported_epoch":458,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:44.491907+0000","last_change":"2023-11-08T05:29:44.395350+0000","last_active":"2023-11-08T05:30:44.491907+0000","last_peered":"2023-11-08T05:30:44.491907+0000","last_clean":"2023-11-08T05:30:44.491907+0000","last_became_active":"2023-11-08T05:29:44.395012+0000","last_became_peered":"2023-11-08T05:29:44.395012+0000","last_unstale":"2023-11-08T05:30:44.491907+0000","last_undegraded":"2023-11-08T05:30:44.491907+0000","last_fullsized":"2023-11-08T05:30:44.491907+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490629+0000","last_change":"2023-11-08T05:28:13.563357+0000","last_active":"2023-11-08T05:30:44.490629+0000","last_peered":"2023-11-08T05:30:44.490629+0000","last_clean":"2023-11-08T05:28:01.457691+0000","last_became_active":"2023-11-08T05:28:13.563357+0000","last_became_peered":"2023-11-08T05:28:13.563357+0000","last_unstale":"2023-11-08T05:30:44.490629+0000","last_undegraded":"2023-11-08T05:30:44.490629+0000","last_fullsized":"2023-11-08T05:28:13.541614+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490691+0000","last_change":"2023-11-08T05:28:13.550672+0000","last_active":"2023-11-08T05:30:44.490691+0000","last_peered":"2023-11-08T05:30:44.490691+0000","last_clean":"2023-11-08T05:28:01.457904+0000","last_became_active":"2023-11-08T05:28:13.550672+0000","last_became_peered":"2023-11-08T05:28:13.550672+0000","last_unstale":"2023-11-08T05:30:44.490691+0000","last_undegraded":"2023-11-08T05:30:44.490691+0000","last_fullsized":"2023-11-08T05:28:13.539595+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490805+0000","last_change":"2023-11-08T05:28:13.557968+0000","last_active":"2023-11-08T05:30:44.490805+0000","last_peered":"2023-11-08T05:30:44.490805+0000","last_clean":"2023-11-08T05:28:01.458433+0000","last_became_active":"2023-11-08T05:28:13.557968+0000","last_became_peered":"2023-11-08T05:28:13.557968+0000","last_unstale":"2023-11-08T05:30:44.490805+0000","last_undegraded":"2023-11-08T05:30:44.490805+0000","last_fullsized":"2023-11-08T05:28:13.542832+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":1,"acting":1,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":12,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738856,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":645040,"kb_used_data":5352,"kb_used_omap":275,"kb_used_meta":639404,"kb_avail":93726800,"statfs":{"total":96636764160,"available":95976243200,"internally_reserved":0,"allocated":5480448,"data_stored":2697647,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":281650,"internal_metadata":654750670},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:59 2023","interfaces":[{"interface":"back","average":{"1min":0.30499999999999999,"5min":0.439,"15min":0.36099999999999999},"min":{"1min":0.17399999999999999,"5min":0.14499999999999999,"15min":0.14499999999999999},"max":{"1min":0.60099999999999998,"5min":4.9379999999999997,"15min":4.9379999999999997},"last":0.379},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.49399999999999999,"15min":0.42399999999999999},"min":{"1min":0.221,"5min":0.19600000000000001,"15min":0.17000000000000001},"max":{"1min":0.63200000000000001,"5min":4.9859999999999998,"15min":4.9859999999999998},"last":0.40899999999999997}]},{"osd":1,"last update":"Wed Nov 8 05:33:07 2023","interfaces":[{"interface":"back","average":{"1min":0.26200000000000001,"5min":0.27900000000000003,"15min":0.46500000000000002},"min":{"1min":0.187,"5min":0.16,"15min":0.153},"max":{"1min":0.372,"5min":0.68200000000000005,"15min":23.242999999999999},"last":0.28000000000000003},{"interface":"front","average":{"1min":0.28100000000000003,"5min":0.32400000000000001,"15min":0.45600000000000002},"min":{"1min":0.20300000000000001,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":0.39600000000000002,"5min":2.0419999999999998,"15min":23.251000000000001},"last":0.252}]}]},{"osd":0,"up_from":8,"seq":34359738798,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1339228,"kb_used_data":398304,"kb_used_omap":291,"kb_used_meta":936476,"kb_avail":93032612,"statfs":{"total":96636764160,"available":95265394688,"internally_reserved":0,"allocated":407863296,"data_stored":405103900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":298462,"internal_metadata":958951970},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.436,"15min":0.36299999999999999},"min":{"1min":0.19700000000000001,"5min":0.14699999999999999,"15min":0.13700000000000001},"max":{"1min":3.222,"5min":6.5499999999999998,"15min":6.9989999999999997},"last":0.245},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.46300000000000002,"15min":0.42299999999999999},"min":{"1min":0.17799999999999999,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":5.0030000000000001,"5min":6.0629999999999997,"15min":9.6869999999999994},"last":0.28599999999999998}]},{"osd":2,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.51600000000000001,"15min":0.47099999999999997},"min":{"1min":0.21099999999999999,"5min":0.153,"15min":0.153},"max":{"1min":5.7439999999999998,"5min":5.8280000000000003,"15min":9.7639999999999993},"last":0.252},{"interface":"front","average":{"1min":0.497,"5min":0.56200000000000006,"15min":0.42299999999999999},"min":{"1min":0.20399999999999999,"5min":0.187,"15min":0.16800000000000001},"max":{"1min":3.9319999999999999,"5min":8.8040000000000003,"15min":8.8040000000000003},"last":0.29899999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738857,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":650100,"kb_used_data":5356,"kb_used_omap":214,"kb_used_meta":644521,"kb_avail":93721740,"statfs":{"total":96636764160,"available":95971061760,"internally_reserved":0,"allocated":5484544,"data_stored":2698616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":219335,"internal_metadata":659990329},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:56 2023","interfaces":[{"interface":"back","average":{"1min":0.30399999999999999,"5min":0.34699999999999998,"15min":0.36499999999999999},"min":{"1min":0.23999999999999999,"5min":0.14199999999999999,"15min":0.14199999999999999},"max":{"1min":0.42399999999999999,"5min":1.595,"15min":7.4249999999999998},"last":0.17100000000000001},{"interface":"front","average":{"1min":0.35499999999999998,"5min":0.40899999999999997,"15min":0.376},"min":{"1min":0.23999999999999999,"5min":0.19600000000000001,"15min":0.183},"max":{"1min":0.46999999999999997,"5min":2.6859999999999999,"15min":2.6859999999999999},"last":0.23000000000000001}]},{"osd":2,"last update":"Wed Nov 8 05:33:03 2023","interfaces":[{"interface":"back","average":{"1min":0.26000000000000001,"5min":0.317,"15min":0.35999999999999999},"min":{"1min":0.18099999999999999,"5min":0.161,"15min":0.161},"max":{"1min":0.40799999999999997,"5min":2.2650000000000001,"15min":5.8209999999999997},"last":0.24399999999999999},{"interface":"front","average":{"1min":0.26900000000000002,"5min":0.28699999999999998,"15min":0.35199999999999998},"min":{"1min":0.192,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.41999999999999998,"5min":1.137,"15min":4.2350000000000003},"last":0.254}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-08T05:33:11.162 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-11-08T05:33:11.445 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T05:33:11.446 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2023-11-08T05:33:11.457 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":1669,"stamp":"2023-11-08T05:33:10.089729+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":10,"acting":13,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":109,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2634368,"kb_used_data":409012,"kb_used_omap":780,"kb_used_meta":2220403,"kb_avail":280481152,"statfs":{"total":289910292480,"available":287212699648,"internally_reserved":0,"allocated":418828288,"data_stored":410500163,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":799447,"internal_metadata":2273692969},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-536782902,"num_objects":-164,"num_object_clones":0,"num_object_copies":-328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-115,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-164,"num_whiteouts":0,"num_read":-122056,"num_read_kb":-39011942,"num_write":-413642,"num_write_kb":-151705339,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-8,"num_bytes_recovered":-16922624,"num_keys_recovered":-2,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.424817"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490644+0000","last_change":"2023-11-08T05:28:13.557071+0000","last_active":"2023-11-08T05:30:44.490644+0000","last_peered":"2023-11-08T05:30:44.490644+0000","last_clean":"2023-11-08T05:28:01.458439+0000","last_became_active":"2023-11-08T05:28:13.557071+0000","last_became_peered":"2023-11-08T05:28:13.557071+0000","last_unstale":"2023-11-08T05:30:44.490644+0000","last_undegraded":"2023-11-08T05:30:44.490644+0000","last_fullsized":"2023-11-08T05:28:13.547025+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":459,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:45.794760+0000","last_change":"2023-11-08T05:30:45.794760+0000","last_active":"2023-11-08T05:30:45.794760+0000","last_peered":"2023-11-08T05:30:45.794760+0000","last_clean":"2023-11-08T05:30:45.794760+0000","last_became_active":"2023-11-08T05:30:45.794398+0000","last_became_peered":"2023-11-08T05:30:45.794398+0000","last_unstale":"2023-11-08T05:30:45.794760+0000","last_undegraded":"2023-11-08T05:30:45.794760+0000","last_fullsized":"2023-11-08T05:30:45.794760+0000","mapping_epoch":458,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":459,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":465,"reported_epoch":459,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:45.795085+0000","last_change":"2023-11-08T05:30:45.795085+0000","last_active":"2023-11-08T05:30:45.795085+0000","last_peered":"2023-11-08T05:30:45.795085+0000","last_clean":"2023-11-08T05:30:45.795085+0000","last_became_active":"2023-11-08T05:30:45.794743+0000","last_became_peered":"2023-11-08T05:30:45.794743+0000","last_unstale":"2023-11-08T05:30:45.795085+0000","last_undegraded":"2023-11-08T05:30:45.795085+0000","last_fullsized":"2023-11-08T05:30:45.795085+0000","mapping_epoch":458,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":459,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":454,"reported_epoch":458,"state":"active+clean","last_fresh":"2023-11-08T05:30:44.490618+0000","last_change":"2023-11-08T04:52:47.548597+0000","last_active":"2023-11-08T05:30:44.490618+0000","last_peered":"2023-11-08T05:30:44.490618+0000","last_clean":"2023-11-08T05:30:44.490618+0000","last_became_active":"2023-11-08T04:52:47.548294+0000","last_became_peered":"2023-11-08T04:52:47.548294+0000","last_unstale":"2023-11-08T05:30:44.490618+0000","last_undegraded":"2023-11-08T05:30:44.490618+0000","last_fullsized":"2023-11-08T05:30:44.490618+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":460,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490523+0000","last_change":"2023-11-08T05:28:13.545468+0000","last_active":"2023-11-08T05:30:44.490523+0000","last_peered":"2023-11-08T05:30:44.490523+0000","last_clean":"2023-11-08T05:28:01.456723+0000","last_became_active":"2023-11-08T05:28:13.545468+0000","last_became_peered":"2023-11-08T05:28:13.545468+0000","last_unstale":"2023-11-08T05:30:44.490523+0000","last_undegraded":"2023-11-08T05:30:44.490523+0000","last_fullsized":"2023-11-08T05:28:13.537207+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_clean_scrub_stamp":"2023-11-08T04:52:45.528796+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":465,"reported_epoch":458,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:44.491907+0000","last_change":"2023-11-08T05:29:44.395350+0000","last_active":"2023-11-08T05:30:44.491907+0000","last_peered":"2023-11-08T05:30:44.491907+0000","last_clean":"2023-11-08T05:30:44.491907+0000","last_became_active":"2023-11-08T05:29:44.395012+0000","last_became_peered":"2023-11-08T05:29:44.395012+0000","last_unstale":"2023-11-08T05:30:44.491907+0000","last_undegraded":"2023-11-08T05:30:44.491907+0000","last_fullsized":"2023-11-08T05:30:44.491907+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490629+0000","last_change":"2023-11-08T05:28:13.563357+0000","last_active":"2023-11-08T05:30:44.490629+0000","last_peered":"2023-11-08T05:30:44.490629+0000","last_clean":"2023-11-08T05:28:01.457691+0000","last_became_active":"2023-11-08T05:28:13.563357+0000","last_became_peered":"2023-11-08T05:28:13.563357+0000","last_unstale":"2023-11-08T05:30:44.490629+0000","last_undegraded":"2023-11-08T05:30:44.490629+0000","last_fullsized":"2023-11-08T05:28:13.541614+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490691+0000","last_change":"2023-11-08T05:28:13.550672+0000","last_active":"2023-11-08T05:30:44.490691+0000","last_peered":"2023-11-08T05:30:44.490691+0000","last_clean":"2023-11-08T05:28:01.457904+0000","last_became_active":"2023-11-08T05:28:13.550672+0000","last_became_peered":"2023-11-08T05:28:13.550672+0000","last_unstale":"2023-11-08T05:30:44.490691+0000","last_undegraded":"2023-11-08T05:30:44.490691+0000","last_fullsized":"2023-11-08T05:28:13.539595+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490805+0000","last_change":"2023-11-08T05:28:13.557968+0000","last_active":"2023-11-08T05:30:44.490805+0000","last_peered":"2023-11-08T05:30:44.490805+0000","last_clean":"2023-11-08T05:28:01.458433+0000","last_became_active":"2023-11-08T05:28:13.557968+0000","last_became_peered":"2023-11-08T05:28:13.557968+0000","last_unstale":"2023-11-08T05:30:44.490805+0000","last_undegraded":"2023-11-08T05:30:44.490805+0000","last_fullsized":"2023-11-08T05:28:13.542832+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":1,"acting":1,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":12,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738856,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":645040,"kb_used_data":5352,"kb_used_omap":275,"kb_used_meta":639404,"kb_avail":93726800,"statfs":{"total":96636764160,"available":95976243200,"internally_reserved":0,"allocated":5480448,"data_stored":2697647,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":281650,"internal_metadata":654750670},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:59 2023","interfaces":[{"interface":"back","average":{"1min":0.30499999999999999,"5min":0.439,"15min":0.36099999999999999},"min":{"1min":0.17399999999999999,"5min":0.14499999999999999,"15min":0.14499999999999999},"max":{"1min":0.60099999999999998,"5min":4.9379999999999997,"15min":4.9379999999999997},"last":0.379},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.49399999999999999,"15min":0.42399999999999999},"min":{"1min":0.221,"5min":0.19600000000000001,"15min":0.17000000000000001},"max":{"1min":0.63200000000000001,"5min":4.9859999999999998,"15min":4.9859999999999998},"last":0.40899999999999997}]},{"osd":1,"last update":"Wed Nov 8 05:33:07 2023","interfaces":[{"interface":"back","average":{"1min":0.26200000000000001,"5min":0.27900000000000003,"15min":0.46500000000000002},"min":{"1min":0.187,"5min":0.16,"15min":0.153},"max":{"1min":0.372,"5min":0.68200000000000005,"15min":23.242999999999999},"last":0.28000000000000003},{"interface":"front","average":{"1min":0.28100000000000003,"5min":0.32400000000000001,"15min":0.45600000000000002},"min":{"1min":0.20300000000000001,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":0.39600000000000002,"5min":2.0419999999999998,"15min":23.251000000000001},"last":0.252}]}]},{"osd":0,"up_from":8,"seq":34359738798,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1339228,"kb_used_data":398304,"kb_used_omap":291,"kb_used_meta":936476,"kb_avail":93032612,"statfs":{"total":96636764160,"available":95265394688,"internally_reserved":0,"allocated":407863296,"data_stored":405103900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":298462,"internal_metadata":958951970},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.436,"15min":0.36299999999999999},"min":{"1min":0.19700000000000001,"5min":0.14699999999999999,"15min":0.13700000000000001},"max":{"1min":3.222,"5min":6.5499999999999998,"15min":6.9989999999999997},"last":0.245},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.46300000000000002,"15min":0.42299999999999999},"min":{"1min":0.17799999999999999,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":5.0030000000000001,"5min":6.0629999999999997,"15min":9.6869999999999994},"last":0.28599999999999998}]},{"osd":2,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.51600000000000001,"15min":0.47099999999999997},"min":{"1min":0.21099999999999999,"5min":0.153,"15min":0.153},"max":{"1min":5.7439999999999998,"5min":5.8280000000000003,"15min":9.7639999999999993},"last":0.252},{"interface":"front","average":{"1min":0.497,"5min":0.56200000000000006,"15min":0.42299999999999999},"min":{"1min":0.20399999999999999,"5min":0.187,"15min":0.16800000000000001},"max":{"1min":3.9319999999999999,"5min":8.8040000000000003,"15min":8.8040000000000003},"last":0.29899999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738857,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":650100,"kb_used_data":5356,"kb_used_omap":214,"kb_used_meta":644521,"kb_avail":93721740,"statfs":{"total":96636764160,"available":95971061760,"internally_reserved":0,"allocated":5484544,"data_stored":2698616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":219335,"internal_metadata":659990329},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:56 2023","interfaces":[{"interface":"back","average":{"1min":0.30399999999999999,"5min":0.34699999999999998,"15min":0.36499999999999999},"min":{"1min":0.23999999999999999,"5min":0.14199999999999999,"15min":0.14199999999999999},"max":{"1min":0.42399999999999999,"5min":1.595,"15min":7.4249999999999998},"last":0.17100000000000001},{"interface":"front","average":{"1min":0.35499999999999998,"5min":0.40899999999999997,"15min":0.376},"min":{"1min":0.23999999999999999,"5min":0.19600000000000001,"15min":0.183},"max":{"1min":0.46999999999999997,"5min":2.6859999999999999,"15min":2.6859999999999999},"last":0.23000000000000001}]},{"osd":2,"last update":"Wed Nov 8 05:33:03 2023","interfaces":[{"interface":"back","average":{"1min":0.26000000000000001,"5min":0.317,"15min":0.35999999999999999},"min":{"1min":0.18099999999999999,"5min":0.161,"15min":0.161},"max":{"1min":0.40799999999999997,"5min":2.2650000000000001,"15min":5.8209999999999997},"last":0.24399999999999999},{"interface":"front","average":{"1min":0.26900000000000002,"5min":0.28699999999999998,"15min":0.35199999999999998},"min":{"1min":0.192,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.41999999999999998,"5min":1.137,"15min":4.2350000000000003},"last":0.254}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-08T05:33:11.459 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-11-08T05:33:11.459 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 459, 'reported_epoch': 458, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:30:44.490644+0000', 'last_change': '2023-11-08T05:28:13.557071+0000', 'last_active': '2023-11-08T05:30:44.490644+0000', 'last_peered': '2023-11-08T05:30:44.490644+0000', 'last_clean': '2023-11-08T05:28:01.458439+0000', 'last_became_active': '2023-11-08T05:28:13.557071+0000', 'last_became_peered': '2023-11-08T05:28:13.557071+0000', 'last_unstale': '2023-11-08T05:30:44.490644+0000', 'last_undegraded': '2023-11-08T05:30:44.490644+0000', 'last_fullsized': '2023-11-08T05:28:13.547025+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:33:11.459 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-11-08T05:33:11.459 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 11, 'reported_epoch': 459, 'state': 'active+clean+remapped', 'last_fresh': '2023-11-08T05:30:45.794760+0000', 'last_change': '2023-11-08T05:30:45.794760+0000', 'last_active': '2023-11-08T05:30:45.794760+0000', 'last_peered': '2023-11-08T05:30:45.794760+0000', 'last_clean': '2023-11-08T05:30:45.794760+0000', 'last_became_active': '2023-11-08T05:30:45.794398+0000', 'last_became_peered': '2023-11-08T05:30:45.794398+0000', 'last_unstale': '2023-11-08T05:30:45.794760+0000', 'last_undegraded': '2023-11-08T05:30:45.794760+0000', 'last_fullsized': '2023-11-08T05:30:45.794760+0000', 'mapping_epoch': 458, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 459, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2], 'acting': [2, 1], 'avail_no_missing': ['2', '1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2023-11-08T05:33:11.460 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.1 is not active+clean 2023-11-08T05:33:11.460 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.1', 'version': "0'0", 'reported_seq': 465, 'reported_epoch': 459, 'state': 'active+clean+remapped', 'last_fresh': '2023-11-08T05:30:45.795085+0000', 'last_change': '2023-11-08T05:30:45.795085+0000', 'last_active': '2023-11-08T05:30:45.795085+0000', 'last_peered': '2023-11-08T05:30:45.795085+0000', 'last_clean': '2023-11-08T05:30:45.795085+0000', 'last_became_active': '2023-11-08T05:30:45.794743+0000', 'last_became_peered': '2023-11-08T05:30:45.794743+0000', 'last_unstale': '2023-11-08T05:30:45.795085+0000', 'last_undegraded': '2023-11-08T05:30:45.795085+0000', 'last_fullsized': '2023-11-08T05:30:45.795085+0000', 'mapping_epoch': 458, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 459, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2], 'acting': [2, 1], 'avail_no_missing': ['2', '1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2023-11-08T05:33:11.460 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-11-08T05:33:11.461 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "0'0", 'reported_seq': 460, 'reported_epoch': 458, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:30:44.490523+0000', 'last_change': '2023-11-08T05:28:13.545468+0000', 'last_active': '2023-11-08T05:30:44.490523+0000', 'last_peered': '2023-11-08T05:30:44.490523+0000', 'last_clean': '2023-11-08T05:28:01.456723+0000', 'last_became_active': '2023-11-08T05:28:13.545468+0000', 'last_became_peered': '2023-11-08T05:28:13.545468+0000', 'last_unstale': '2023-11-08T05:30:44.490523+0000', 'last_undegraded': '2023-11-08T05:30:44.490523+0000', 'last_fullsized': '2023-11-08T05:28:13.537207+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 9, 'last_epoch_clean': 10, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:45.528796+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:45.528796+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:45.528796+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:33:11.461 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.0 is not active+clean 2023-11-08T05:33:11.461 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.0', 'version': "0'0", 'reported_seq': 465, 'reported_epoch': 458, 'state': 'active+clean+remapped', 'last_fresh': '2023-11-08T05:30:44.491907+0000', 'last_change': '2023-11-08T05:29:44.395350+0000', 'last_active': '2023-11-08T05:30:44.491907+0000', 'last_peered': '2023-11-08T05:30:44.491907+0000', 'last_clean': '2023-11-08T05:30:44.491907+0000', 'last_became_active': '2023-11-08T05:29:44.395012+0000', 'last_became_peered': '2023-11-08T05:29:44.395012+0000', 'last_unstale': '2023-11-08T05:30:44.491907+0000', 'last_undegraded': '2023-11-08T05:30:44.491907+0000', 'last_fullsized': '2023-11-08T05:30:44.491907+0000', 'mapping_epoch': 455, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 456, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2], 'acting': [2, 1], 'avail_no_missing': ['2', '1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2023-11-08T05:33:11.462 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-11-08T05:33:11.462 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 459, 'reported_epoch': 458, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:30:44.490629+0000', 'last_change': '2023-11-08T05:28:13.563357+0000', 'last_active': '2023-11-08T05:30:44.490629+0000', 'last_peered': '2023-11-08T05:30:44.490629+0000', 'last_clean': '2023-11-08T05:28:01.457691+0000', 'last_became_active': '2023-11-08T05:28:13.563357+0000', 'last_became_peered': '2023-11-08T05:28:13.563357+0000', 'last_unstale': '2023-11-08T05:30:44.490629+0000', 'last_undegraded': '2023-11-08T05:30:44.490629+0000', 'last_fullsized': '2023-11-08T05:28:13.541614+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:33:11.462 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-11-08T05:33:11.463 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 459, 'reported_epoch': 458, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:30:44.490691+0000', 'last_change': '2023-11-08T05:28:13.550672+0000', 'last_active': '2023-11-08T05:30:44.490691+0000', 'last_peered': '2023-11-08T05:30:44.490691+0000', 'last_clean': '2023-11-08T05:28:01.457904+0000', 'last_became_active': '2023-11-08T05:28:13.550672+0000', 'last_became_peered': '2023-11-08T05:28:13.550672+0000', 'last_unstale': '2023-11-08T05:30:44.490691+0000', 'last_undegraded': '2023-11-08T05:30:44.490691+0000', 'last_fullsized': '2023-11-08T05:28:13.539595+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:33:11.463 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-11-08T05:33:11.463 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 459, 'reported_epoch': 458, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:30:44.490805+0000', 'last_change': '2023-11-08T05:28:13.557968+0000', 'last_active': '2023-11-08T05:30:44.490805+0000', 'last_peered': '2023-11-08T05:30:44.490805+0000', 'last_clean': '2023-11-08T05:28:01.458433+0000', 'last_became_active': '2023-11-08T05:28:13.557968+0000', 'last_became_peered': '2023-11-08T05:28:13.557968+0000', 'last_unstale': '2023-11-08T05:30:44.490805+0000', 'last_undegraded': '2023-11-08T05:30:44.490805+0000', 'last_fullsized': '2023-11-08T05:28:13.542832+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:33:11.464 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-11-08T05:33:11.811 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T05:33:11.811 INFO:teuthology.orchestra.run.smithi087.stdout:{"fsid":"ab4cbc06-449d-497d-b513-220d9db2b590","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 5 pgs undersized","count":5},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2434,"monmap":{"epoch":1,"min_mon_release_name":"pacific","num_mons":1},"osdmap":{"epoch":459,"num_osds":3,"num_up_osds":2,"osd_up_since":1699421292,"num_in_osds":3,"osd_in_since":1699419158,"num_remapped_pgs":3},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean+remapped","count":3},{"state_name":"active+clean","count":1}],"num_pgs":9,"num_pools":2,"num_objects":0,"data_bytes":0,"bytes_used":2697592832,"bytes_avail":287212699648,"bytes_total":289910292480},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":41,"modified":"2023-11-08T05:29:32.044544+0000","services":{}},"progress_events":{}} 2023-11-08T05:33:11.824 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-11-08T05:33:11.825 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-11-08T05:33:12.105 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T05:33:12.107 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2023-11-08T05:33:12.118 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":1670,"stamp":"2023-11-08T05:33:12.090087+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":10,"acting":13,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":109,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2634368,"kb_used_data":409012,"kb_used_omap":780,"kb_used_meta":2220403,"kb_avail":280481152,"statfs":{"total":289910292480,"available":287212699648,"internally_reserved":0,"allocated":418828288,"data_stored":410500163,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":799447,"internal_metadata":2273692969},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-536782902,"num_objects":-164,"num_object_clones":0,"num_object_copies":-328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-115,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-164,"num_whiteouts":0,"num_read":-122056,"num_read_kb":-39011942,"num_write":-413642,"num_write_kb":-151705339,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-8,"num_bytes_recovered":-16922624,"num_keys_recovered":-2,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.424817"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490644+0000","last_change":"2023-11-08T05:28:13.557071+0000","last_active":"2023-11-08T05:30:44.490644+0000","last_peered":"2023-11-08T05:30:44.490644+0000","last_clean":"2023-11-08T05:28:01.458439+0000","last_became_active":"2023-11-08T05:28:13.557071+0000","last_became_peered":"2023-11-08T05:28:13.557071+0000","last_unstale":"2023-11-08T05:30:44.490644+0000","last_undegraded":"2023-11-08T05:30:44.490644+0000","last_fullsized":"2023-11-08T05:28:13.547025+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":459,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:45.794760+0000","last_change":"2023-11-08T05:30:45.794760+0000","last_active":"2023-11-08T05:30:45.794760+0000","last_peered":"2023-11-08T05:30:45.794760+0000","last_clean":"2023-11-08T05:30:45.794760+0000","last_became_active":"2023-11-08T05:30:45.794398+0000","last_became_peered":"2023-11-08T05:30:45.794398+0000","last_unstale":"2023-11-08T05:30:45.794760+0000","last_undegraded":"2023-11-08T05:30:45.794760+0000","last_fullsized":"2023-11-08T05:30:45.794760+0000","mapping_epoch":458,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":459,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":465,"reported_epoch":459,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:45.795085+0000","last_change":"2023-11-08T05:30:45.795085+0000","last_active":"2023-11-08T05:30:45.795085+0000","last_peered":"2023-11-08T05:30:45.795085+0000","last_clean":"2023-11-08T05:30:45.795085+0000","last_became_active":"2023-11-08T05:30:45.794743+0000","last_became_peered":"2023-11-08T05:30:45.794743+0000","last_unstale":"2023-11-08T05:30:45.795085+0000","last_undegraded":"2023-11-08T05:30:45.795085+0000","last_fullsized":"2023-11-08T05:30:45.795085+0000","mapping_epoch":458,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":459,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":454,"reported_epoch":458,"state":"active+clean","last_fresh":"2023-11-08T05:30:44.490618+0000","last_change":"2023-11-08T04:52:47.548597+0000","last_active":"2023-11-08T05:30:44.490618+0000","last_peered":"2023-11-08T05:30:44.490618+0000","last_clean":"2023-11-08T05:30:44.490618+0000","last_became_active":"2023-11-08T04:52:47.548294+0000","last_became_peered":"2023-11-08T04:52:47.548294+0000","last_unstale":"2023-11-08T05:30:44.490618+0000","last_undegraded":"2023-11-08T05:30:44.490618+0000","last_fullsized":"2023-11-08T05:30:44.490618+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":460,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490523+0000","last_change":"2023-11-08T05:28:13.545468+0000","last_active":"2023-11-08T05:30:44.490523+0000","last_peered":"2023-11-08T05:30:44.490523+0000","last_clean":"2023-11-08T05:28:01.456723+0000","last_became_active":"2023-11-08T05:28:13.545468+0000","last_became_peered":"2023-11-08T05:28:13.545468+0000","last_unstale":"2023-11-08T05:30:44.490523+0000","last_undegraded":"2023-11-08T05:30:44.490523+0000","last_fullsized":"2023-11-08T05:28:13.537207+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_clean_scrub_stamp":"2023-11-08T04:52:45.528796+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":465,"reported_epoch":458,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:44.491907+0000","last_change":"2023-11-08T05:29:44.395350+0000","last_active":"2023-11-08T05:30:44.491907+0000","last_peered":"2023-11-08T05:30:44.491907+0000","last_clean":"2023-11-08T05:30:44.491907+0000","last_became_active":"2023-11-08T05:29:44.395012+0000","last_became_peered":"2023-11-08T05:29:44.395012+0000","last_unstale":"2023-11-08T05:30:44.491907+0000","last_undegraded":"2023-11-08T05:30:44.491907+0000","last_fullsized":"2023-11-08T05:30:44.491907+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490629+0000","last_change":"2023-11-08T05:28:13.563357+0000","last_active":"2023-11-08T05:30:44.490629+0000","last_peered":"2023-11-08T05:30:44.490629+0000","last_clean":"2023-11-08T05:28:01.457691+0000","last_became_active":"2023-11-08T05:28:13.563357+0000","last_became_peered":"2023-11-08T05:28:13.563357+0000","last_unstale":"2023-11-08T05:30:44.490629+0000","last_undegraded":"2023-11-08T05:30:44.490629+0000","last_fullsized":"2023-11-08T05:28:13.541614+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490691+0000","last_change":"2023-11-08T05:28:13.550672+0000","last_active":"2023-11-08T05:30:44.490691+0000","last_peered":"2023-11-08T05:30:44.490691+0000","last_clean":"2023-11-08T05:28:01.457904+0000","last_became_active":"2023-11-08T05:28:13.550672+0000","last_became_peered":"2023-11-08T05:28:13.550672+0000","last_unstale":"2023-11-08T05:30:44.490691+0000","last_undegraded":"2023-11-08T05:30:44.490691+0000","last_fullsized":"2023-11-08T05:28:13.539595+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490805+0000","last_change":"2023-11-08T05:28:13.557968+0000","last_active":"2023-11-08T05:30:44.490805+0000","last_peered":"2023-11-08T05:30:44.490805+0000","last_clean":"2023-11-08T05:28:01.458433+0000","last_became_active":"2023-11-08T05:28:13.557968+0000","last_became_peered":"2023-11-08T05:28:13.557968+0000","last_unstale":"2023-11-08T05:30:44.490805+0000","last_undegraded":"2023-11-08T05:30:44.490805+0000","last_fullsized":"2023-11-08T05:28:13.542832+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":1,"acting":1,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":12,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738856,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":645040,"kb_used_data":5352,"kb_used_omap":275,"kb_used_meta":639404,"kb_avail":93726800,"statfs":{"total":96636764160,"available":95976243200,"internally_reserved":0,"allocated":5480448,"data_stored":2697647,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":281650,"internal_metadata":654750670},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:59 2023","interfaces":[{"interface":"back","average":{"1min":0.30499999999999999,"5min":0.439,"15min":0.36099999999999999},"min":{"1min":0.17399999999999999,"5min":0.14499999999999999,"15min":0.14499999999999999},"max":{"1min":0.60099999999999998,"5min":4.9379999999999997,"15min":4.9379999999999997},"last":0.379},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.49399999999999999,"15min":0.42399999999999999},"min":{"1min":0.221,"5min":0.19600000000000001,"15min":0.17000000000000001},"max":{"1min":0.63200000000000001,"5min":4.9859999999999998,"15min":4.9859999999999998},"last":0.40899999999999997}]},{"osd":1,"last update":"Wed Nov 8 05:33:07 2023","interfaces":[{"interface":"back","average":{"1min":0.26200000000000001,"5min":0.27900000000000003,"15min":0.46500000000000002},"min":{"1min":0.187,"5min":0.16,"15min":0.153},"max":{"1min":0.372,"5min":0.68200000000000005,"15min":23.242999999999999},"last":0.28000000000000003},{"interface":"front","average":{"1min":0.28100000000000003,"5min":0.32400000000000001,"15min":0.45600000000000002},"min":{"1min":0.20300000000000001,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":0.39600000000000002,"5min":2.0419999999999998,"15min":23.251000000000001},"last":0.252}]}]},{"osd":0,"up_from":8,"seq":34359738798,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1339228,"kb_used_data":398304,"kb_used_omap":291,"kb_used_meta":936476,"kb_avail":93032612,"statfs":{"total":96636764160,"available":95265394688,"internally_reserved":0,"allocated":407863296,"data_stored":405103900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":298462,"internal_metadata":958951970},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.436,"15min":0.36299999999999999},"min":{"1min":0.19700000000000001,"5min":0.14699999999999999,"15min":0.13700000000000001},"max":{"1min":3.222,"5min":6.5499999999999998,"15min":6.9989999999999997},"last":0.245},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.46300000000000002,"15min":0.42299999999999999},"min":{"1min":0.17799999999999999,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":5.0030000000000001,"5min":6.0629999999999997,"15min":9.6869999999999994},"last":0.28599999999999998}]},{"osd":2,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.51600000000000001,"15min":0.47099999999999997},"min":{"1min":0.21099999999999999,"5min":0.153,"15min":0.153},"max":{"1min":5.7439999999999998,"5min":5.8280000000000003,"15min":9.7639999999999993},"last":0.252},{"interface":"front","average":{"1min":0.497,"5min":0.56200000000000006,"15min":0.42299999999999999},"min":{"1min":0.20399999999999999,"5min":0.187,"15min":0.16800000000000001},"max":{"1min":3.9319999999999999,"5min":8.8040000000000003,"15min":8.8040000000000003},"last":0.29899999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738858,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":650100,"kb_used_data":5356,"kb_used_omap":214,"kb_used_meta":644521,"kb_avail":93721740,"statfs":{"total":96636764160,"available":95971061760,"internally_reserved":0,"allocated":5484544,"data_stored":2698616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":219335,"internal_metadata":659990329},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:56 2023","interfaces":[{"interface":"back","average":{"1min":0.30399999999999999,"5min":0.34699999999999998,"15min":0.36499999999999999},"min":{"1min":0.23999999999999999,"5min":0.14199999999999999,"15min":0.14199999999999999},"max":{"1min":0.42399999999999999,"5min":1.595,"15min":7.4249999999999998},"last":0.17100000000000001},{"interface":"front","average":{"1min":0.35499999999999998,"5min":0.40899999999999997,"15min":0.376},"min":{"1min":0.23999999999999999,"5min":0.19600000000000001,"15min":0.183},"max":{"1min":0.46999999999999997,"5min":2.6859999999999999,"15min":2.6859999999999999},"last":0.23000000000000001}]},{"osd":2,"last update":"Wed Nov 8 05:33:03 2023","interfaces":[{"interface":"back","average":{"1min":0.26000000000000001,"5min":0.317,"15min":0.35999999999999999},"min":{"1min":0.18099999999999999,"5min":0.161,"15min":0.161},"max":{"1min":0.40799999999999997,"5min":2.2650000000000001,"15min":5.8209999999999997},"last":0.23100000000000001},{"interface":"front","average":{"1min":0.26900000000000002,"5min":0.28699999999999998,"15min":0.35199999999999998},"min":{"1min":0.192,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.41999999999999998,"5min":1.137,"15min":4.2350000000000003},"last":0.19500000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-08T05:33:15.119 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-11-08T05:33:15.272 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~297s 2023-11-08T05:33:15.416 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T05:33:15.418 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2023-11-08T05:33:15.429 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":1671,"stamp":"2023-11-08T05:33:14.090475+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":10,"acting":13,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":109,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2634368,"kb_used_data":409012,"kb_used_omap":780,"kb_used_meta":2220403,"kb_avail":280481152,"statfs":{"total":289910292480,"available":287212699648,"internally_reserved":0,"allocated":418828288,"data_stored":410500163,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":799447,"internal_metadata":2273692969},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-536782902,"num_objects":-164,"num_object_clones":0,"num_object_copies":-328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-115,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-164,"num_whiteouts":0,"num_read":-122056,"num_read_kb":-39011942,"num_write":-413642,"num_write_kb":-151705339,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-8,"num_bytes_recovered":-16922624,"num_keys_recovered":-2,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.424817"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490644+0000","last_change":"2023-11-08T05:28:13.557071+0000","last_active":"2023-11-08T05:30:44.490644+0000","last_peered":"2023-11-08T05:30:44.490644+0000","last_clean":"2023-11-08T05:28:01.458439+0000","last_became_active":"2023-11-08T05:28:13.557071+0000","last_became_peered":"2023-11-08T05:28:13.557071+0000","last_unstale":"2023-11-08T05:30:44.490644+0000","last_undegraded":"2023-11-08T05:30:44.490644+0000","last_fullsized":"2023-11-08T05:28:13.547025+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":459,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:45.794760+0000","last_change":"2023-11-08T05:30:45.794760+0000","last_active":"2023-11-08T05:30:45.794760+0000","last_peered":"2023-11-08T05:30:45.794760+0000","last_clean":"2023-11-08T05:30:45.794760+0000","last_became_active":"2023-11-08T05:30:45.794398+0000","last_became_peered":"2023-11-08T05:30:45.794398+0000","last_unstale":"2023-11-08T05:30:45.794760+0000","last_undegraded":"2023-11-08T05:30:45.794760+0000","last_fullsized":"2023-11-08T05:30:45.794760+0000","mapping_epoch":458,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":459,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":465,"reported_epoch":459,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:45.795085+0000","last_change":"2023-11-08T05:30:45.795085+0000","last_active":"2023-11-08T05:30:45.795085+0000","last_peered":"2023-11-08T05:30:45.795085+0000","last_clean":"2023-11-08T05:30:45.795085+0000","last_became_active":"2023-11-08T05:30:45.794743+0000","last_became_peered":"2023-11-08T05:30:45.794743+0000","last_unstale":"2023-11-08T05:30:45.795085+0000","last_undegraded":"2023-11-08T05:30:45.795085+0000","last_fullsized":"2023-11-08T05:30:45.795085+0000","mapping_epoch":458,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":459,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":454,"reported_epoch":458,"state":"active+clean","last_fresh":"2023-11-08T05:30:44.490618+0000","last_change":"2023-11-08T04:52:47.548597+0000","last_active":"2023-11-08T05:30:44.490618+0000","last_peered":"2023-11-08T05:30:44.490618+0000","last_clean":"2023-11-08T05:30:44.490618+0000","last_became_active":"2023-11-08T04:52:47.548294+0000","last_became_peered":"2023-11-08T04:52:47.548294+0000","last_unstale":"2023-11-08T05:30:44.490618+0000","last_undegraded":"2023-11-08T05:30:44.490618+0000","last_fullsized":"2023-11-08T05:30:44.490618+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":460,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490523+0000","last_change":"2023-11-08T05:28:13.545468+0000","last_active":"2023-11-08T05:30:44.490523+0000","last_peered":"2023-11-08T05:30:44.490523+0000","last_clean":"2023-11-08T05:28:01.456723+0000","last_became_active":"2023-11-08T05:28:13.545468+0000","last_became_peered":"2023-11-08T05:28:13.545468+0000","last_unstale":"2023-11-08T05:30:44.490523+0000","last_undegraded":"2023-11-08T05:30:44.490523+0000","last_fullsized":"2023-11-08T05:28:13.537207+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_clean_scrub_stamp":"2023-11-08T04:52:45.528796+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":465,"reported_epoch":458,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:44.491907+0000","last_change":"2023-11-08T05:29:44.395350+0000","last_active":"2023-11-08T05:30:44.491907+0000","last_peered":"2023-11-08T05:30:44.491907+0000","last_clean":"2023-11-08T05:30:44.491907+0000","last_became_active":"2023-11-08T05:29:44.395012+0000","last_became_peered":"2023-11-08T05:29:44.395012+0000","last_unstale":"2023-11-08T05:30:44.491907+0000","last_undegraded":"2023-11-08T05:30:44.491907+0000","last_fullsized":"2023-11-08T05:30:44.491907+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490629+0000","last_change":"2023-11-08T05:28:13.563357+0000","last_active":"2023-11-08T05:30:44.490629+0000","last_peered":"2023-11-08T05:30:44.490629+0000","last_clean":"2023-11-08T05:28:01.457691+0000","last_became_active":"2023-11-08T05:28:13.563357+0000","last_became_peered":"2023-11-08T05:28:13.563357+0000","last_unstale":"2023-11-08T05:30:44.490629+0000","last_undegraded":"2023-11-08T05:30:44.490629+0000","last_fullsized":"2023-11-08T05:28:13.541614+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490691+0000","last_change":"2023-11-08T05:28:13.550672+0000","last_active":"2023-11-08T05:30:44.490691+0000","last_peered":"2023-11-08T05:30:44.490691+0000","last_clean":"2023-11-08T05:28:01.457904+0000","last_became_active":"2023-11-08T05:28:13.550672+0000","last_became_peered":"2023-11-08T05:28:13.550672+0000","last_unstale":"2023-11-08T05:30:44.490691+0000","last_undegraded":"2023-11-08T05:30:44.490691+0000","last_fullsized":"2023-11-08T05:28:13.539595+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490805+0000","last_change":"2023-11-08T05:28:13.557968+0000","last_active":"2023-11-08T05:30:44.490805+0000","last_peered":"2023-11-08T05:30:44.490805+0000","last_clean":"2023-11-08T05:28:01.458433+0000","last_became_active":"2023-11-08T05:28:13.557968+0000","last_became_peered":"2023-11-08T05:28:13.557968+0000","last_unstale":"2023-11-08T05:30:44.490805+0000","last_undegraded":"2023-11-08T05:30:44.490805+0000","last_fullsized":"2023-11-08T05:28:13.542832+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":1,"acting":1,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":12,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738856,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":645040,"kb_used_data":5352,"kb_used_omap":275,"kb_used_meta":639404,"kb_avail":93726800,"statfs":{"total":96636764160,"available":95976243200,"internally_reserved":0,"allocated":5480448,"data_stored":2697647,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":281650,"internal_metadata":654750670},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:59 2023","interfaces":[{"interface":"back","average":{"1min":0.30499999999999999,"5min":0.439,"15min":0.36099999999999999},"min":{"1min":0.17399999999999999,"5min":0.14499999999999999,"15min":0.14499999999999999},"max":{"1min":0.60099999999999998,"5min":4.9379999999999997,"15min":4.9379999999999997},"last":0.379},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.49399999999999999,"15min":0.42399999999999999},"min":{"1min":0.221,"5min":0.19600000000000001,"15min":0.17000000000000001},"max":{"1min":0.63200000000000001,"5min":4.9859999999999998,"15min":4.9859999999999998},"last":0.40899999999999997}]},{"osd":1,"last update":"Wed Nov 8 05:33:07 2023","interfaces":[{"interface":"back","average":{"1min":0.26200000000000001,"5min":0.27900000000000003,"15min":0.46500000000000002},"min":{"1min":0.187,"5min":0.16,"15min":0.153},"max":{"1min":0.372,"5min":0.68200000000000005,"15min":23.242999999999999},"last":0.28000000000000003},{"interface":"front","average":{"1min":0.28100000000000003,"5min":0.32400000000000001,"15min":0.45600000000000002},"min":{"1min":0.20300000000000001,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":0.39600000000000002,"5min":2.0419999999999998,"15min":23.251000000000001},"last":0.252}]}]},{"osd":0,"up_from":8,"seq":34359738798,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1339228,"kb_used_data":398304,"kb_used_omap":291,"kb_used_meta":936476,"kb_avail":93032612,"statfs":{"total":96636764160,"available":95265394688,"internally_reserved":0,"allocated":407863296,"data_stored":405103900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":298462,"internal_metadata":958951970},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.436,"15min":0.36299999999999999},"min":{"1min":0.19700000000000001,"5min":0.14699999999999999,"15min":0.13700000000000001},"max":{"1min":3.222,"5min":6.5499999999999998,"15min":6.9989999999999997},"last":0.245},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.46300000000000002,"15min":0.42299999999999999},"min":{"1min":0.17799999999999999,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":5.0030000000000001,"5min":6.0629999999999997,"15min":9.6869999999999994},"last":0.28599999999999998}]},{"osd":2,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.51600000000000001,"15min":0.47099999999999997},"min":{"1min":0.21099999999999999,"5min":0.153,"15min":0.153},"max":{"1min":5.7439999999999998,"5min":5.8280000000000003,"15min":9.7639999999999993},"last":0.252},{"interface":"front","average":{"1min":0.497,"5min":0.56200000000000006,"15min":0.42299999999999999},"min":{"1min":0.20399999999999999,"5min":0.187,"15min":0.16800000000000001},"max":{"1min":3.9319999999999999,"5min":8.8040000000000003,"15min":8.8040000000000003},"last":0.29899999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738858,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":650100,"kb_used_data":5356,"kb_used_omap":214,"kb_used_meta":644521,"kb_avail":93721740,"statfs":{"total":96636764160,"available":95971061760,"internally_reserved":0,"allocated":5484544,"data_stored":2698616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":219335,"internal_metadata":659990329},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:56 2023","interfaces":[{"interface":"back","average":{"1min":0.30399999999999999,"5min":0.34699999999999998,"15min":0.36499999999999999},"min":{"1min":0.23999999999999999,"5min":0.14199999999999999,"15min":0.14199999999999999},"max":{"1min":0.42399999999999999,"5min":1.595,"15min":7.4249999999999998},"last":0.17100000000000001},{"interface":"front","average":{"1min":0.35499999999999998,"5min":0.40899999999999997,"15min":0.376},"min":{"1min":0.23999999999999999,"5min":0.19600000000000001,"15min":0.183},"max":{"1min":0.46999999999999997,"5min":2.6859999999999999,"15min":2.6859999999999999},"last":0.23000000000000001}]},{"osd":2,"last update":"Wed Nov 8 05:33:03 2023","interfaces":[{"interface":"back","average":{"1min":0.26000000000000001,"5min":0.317,"15min":0.35999999999999999},"min":{"1min":0.18099999999999999,"5min":0.161,"15min":0.161},"max":{"1min":0.40799999999999997,"5min":2.2650000000000001,"15min":5.8209999999999997},"last":0.23100000000000001},{"interface":"front","average":{"1min":0.26900000000000002,"5min":0.28699999999999998,"15min":0.35199999999999998},"min":{"1min":0.192,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.41999999999999998,"5min":1.137,"15min":4.2350000000000003},"last":0.19500000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-08T05:33:15.430 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-11-08T05:33:15.713 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T05:33:15.715 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2023-11-08T05:33:15.726 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":1671,"stamp":"2023-11-08T05:33:14.090475+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":10,"acting":13,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":109,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2634368,"kb_used_data":409012,"kb_used_omap":780,"kb_used_meta":2220403,"kb_avail":280481152,"statfs":{"total":289910292480,"available":287212699648,"internally_reserved":0,"allocated":418828288,"data_stored":410500163,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":799447,"internal_metadata":2273692969},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-536782902,"num_objects":-164,"num_object_clones":0,"num_object_copies":-328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-115,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-164,"num_whiteouts":0,"num_read":-122056,"num_read_kb":-39011942,"num_write":-413642,"num_write_kb":-151705339,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-8,"num_bytes_recovered":-16922624,"num_keys_recovered":-2,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.424817"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490644+0000","last_change":"2023-11-08T05:28:13.557071+0000","last_active":"2023-11-08T05:30:44.490644+0000","last_peered":"2023-11-08T05:30:44.490644+0000","last_clean":"2023-11-08T05:28:01.458439+0000","last_became_active":"2023-11-08T05:28:13.557071+0000","last_became_peered":"2023-11-08T05:28:13.557071+0000","last_unstale":"2023-11-08T05:30:44.490644+0000","last_undegraded":"2023-11-08T05:30:44.490644+0000","last_fullsized":"2023-11-08T05:28:13.547025+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":459,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:45.794760+0000","last_change":"2023-11-08T05:30:45.794760+0000","last_active":"2023-11-08T05:30:45.794760+0000","last_peered":"2023-11-08T05:30:45.794760+0000","last_clean":"2023-11-08T05:30:45.794760+0000","last_became_active":"2023-11-08T05:30:45.794398+0000","last_became_peered":"2023-11-08T05:30:45.794398+0000","last_unstale":"2023-11-08T05:30:45.794760+0000","last_undegraded":"2023-11-08T05:30:45.794760+0000","last_fullsized":"2023-11-08T05:30:45.794760+0000","mapping_epoch":458,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":459,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":465,"reported_epoch":459,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:45.795085+0000","last_change":"2023-11-08T05:30:45.795085+0000","last_active":"2023-11-08T05:30:45.795085+0000","last_peered":"2023-11-08T05:30:45.795085+0000","last_clean":"2023-11-08T05:30:45.795085+0000","last_became_active":"2023-11-08T05:30:45.794743+0000","last_became_peered":"2023-11-08T05:30:45.794743+0000","last_unstale":"2023-11-08T05:30:45.795085+0000","last_undegraded":"2023-11-08T05:30:45.795085+0000","last_fullsized":"2023-11-08T05:30:45.795085+0000","mapping_epoch":458,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":459,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":454,"reported_epoch":458,"state":"active+clean","last_fresh":"2023-11-08T05:30:44.490618+0000","last_change":"2023-11-08T04:52:47.548597+0000","last_active":"2023-11-08T05:30:44.490618+0000","last_peered":"2023-11-08T05:30:44.490618+0000","last_clean":"2023-11-08T05:30:44.490618+0000","last_became_active":"2023-11-08T04:52:47.548294+0000","last_became_peered":"2023-11-08T04:52:47.548294+0000","last_unstale":"2023-11-08T05:30:44.490618+0000","last_undegraded":"2023-11-08T05:30:44.490618+0000","last_fullsized":"2023-11-08T05:30:44.490618+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":460,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490523+0000","last_change":"2023-11-08T05:28:13.545468+0000","last_active":"2023-11-08T05:30:44.490523+0000","last_peered":"2023-11-08T05:30:44.490523+0000","last_clean":"2023-11-08T05:28:01.456723+0000","last_became_active":"2023-11-08T05:28:13.545468+0000","last_became_peered":"2023-11-08T05:28:13.545468+0000","last_unstale":"2023-11-08T05:30:44.490523+0000","last_undegraded":"2023-11-08T05:30:44.490523+0000","last_fullsized":"2023-11-08T05:28:13.537207+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_clean_scrub_stamp":"2023-11-08T04:52:45.528796+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":465,"reported_epoch":458,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:44.491907+0000","last_change":"2023-11-08T05:29:44.395350+0000","last_active":"2023-11-08T05:30:44.491907+0000","last_peered":"2023-11-08T05:30:44.491907+0000","last_clean":"2023-11-08T05:30:44.491907+0000","last_became_active":"2023-11-08T05:29:44.395012+0000","last_became_peered":"2023-11-08T05:29:44.395012+0000","last_unstale":"2023-11-08T05:30:44.491907+0000","last_undegraded":"2023-11-08T05:30:44.491907+0000","last_fullsized":"2023-11-08T05:30:44.491907+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490629+0000","last_change":"2023-11-08T05:28:13.563357+0000","last_active":"2023-11-08T05:30:44.490629+0000","last_peered":"2023-11-08T05:30:44.490629+0000","last_clean":"2023-11-08T05:28:01.457691+0000","last_became_active":"2023-11-08T05:28:13.563357+0000","last_became_peered":"2023-11-08T05:28:13.563357+0000","last_unstale":"2023-11-08T05:30:44.490629+0000","last_undegraded":"2023-11-08T05:30:44.490629+0000","last_fullsized":"2023-11-08T05:28:13.541614+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490691+0000","last_change":"2023-11-08T05:28:13.550672+0000","last_active":"2023-11-08T05:30:44.490691+0000","last_peered":"2023-11-08T05:30:44.490691+0000","last_clean":"2023-11-08T05:28:01.457904+0000","last_became_active":"2023-11-08T05:28:13.550672+0000","last_became_peered":"2023-11-08T05:28:13.550672+0000","last_unstale":"2023-11-08T05:30:44.490691+0000","last_undegraded":"2023-11-08T05:30:44.490691+0000","last_fullsized":"2023-11-08T05:28:13.539595+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490805+0000","last_change":"2023-11-08T05:28:13.557968+0000","last_active":"2023-11-08T05:30:44.490805+0000","last_peered":"2023-11-08T05:30:44.490805+0000","last_clean":"2023-11-08T05:28:01.458433+0000","last_became_active":"2023-11-08T05:28:13.557968+0000","last_became_peered":"2023-11-08T05:28:13.557968+0000","last_unstale":"2023-11-08T05:30:44.490805+0000","last_undegraded":"2023-11-08T05:30:44.490805+0000","last_fullsized":"2023-11-08T05:28:13.542832+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":1,"acting":1,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":12,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738856,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":645040,"kb_used_data":5352,"kb_used_omap":275,"kb_used_meta":639404,"kb_avail":93726800,"statfs":{"total":96636764160,"available":95976243200,"internally_reserved":0,"allocated":5480448,"data_stored":2697647,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":281650,"internal_metadata":654750670},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:59 2023","interfaces":[{"interface":"back","average":{"1min":0.30499999999999999,"5min":0.439,"15min":0.36099999999999999},"min":{"1min":0.17399999999999999,"5min":0.14499999999999999,"15min":0.14499999999999999},"max":{"1min":0.60099999999999998,"5min":4.9379999999999997,"15min":4.9379999999999997},"last":0.379},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.49399999999999999,"15min":0.42399999999999999},"min":{"1min":0.221,"5min":0.19600000000000001,"15min":0.17000000000000001},"max":{"1min":0.63200000000000001,"5min":4.9859999999999998,"15min":4.9859999999999998},"last":0.40899999999999997}]},{"osd":1,"last update":"Wed Nov 8 05:33:07 2023","interfaces":[{"interface":"back","average":{"1min":0.26200000000000001,"5min":0.27900000000000003,"15min":0.46500000000000002},"min":{"1min":0.187,"5min":0.16,"15min":0.153},"max":{"1min":0.372,"5min":0.68200000000000005,"15min":23.242999999999999},"last":0.28000000000000003},{"interface":"front","average":{"1min":0.28100000000000003,"5min":0.32400000000000001,"15min":0.45600000000000002},"min":{"1min":0.20300000000000001,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":0.39600000000000002,"5min":2.0419999999999998,"15min":23.251000000000001},"last":0.252}]}]},{"osd":0,"up_from":8,"seq":34359738798,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1339228,"kb_used_data":398304,"kb_used_omap":291,"kb_used_meta":936476,"kb_avail":93032612,"statfs":{"total":96636764160,"available":95265394688,"internally_reserved":0,"allocated":407863296,"data_stored":405103900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":298462,"internal_metadata":958951970},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.436,"15min":0.36299999999999999},"min":{"1min":0.19700000000000001,"5min":0.14699999999999999,"15min":0.13700000000000001},"max":{"1min":3.222,"5min":6.5499999999999998,"15min":6.9989999999999997},"last":0.245},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.46300000000000002,"15min":0.42299999999999999},"min":{"1min":0.17799999999999999,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":5.0030000000000001,"5min":6.0629999999999997,"15min":9.6869999999999994},"last":0.28599999999999998}]},{"osd":2,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.51600000000000001,"15min":0.47099999999999997},"min":{"1min":0.21099999999999999,"5min":0.153,"15min":0.153},"max":{"1min":5.7439999999999998,"5min":5.8280000000000003,"15min":9.7639999999999993},"last":0.252},{"interface":"front","average":{"1min":0.497,"5min":0.56200000000000006,"15min":0.42299999999999999},"min":{"1min":0.20399999999999999,"5min":0.187,"15min":0.16800000000000001},"max":{"1min":3.9319999999999999,"5min":8.8040000000000003,"15min":8.8040000000000003},"last":0.29899999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738858,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":650100,"kb_used_data":5356,"kb_used_omap":214,"kb_used_meta":644521,"kb_avail":93721740,"statfs":{"total":96636764160,"available":95971061760,"internally_reserved":0,"allocated":5484544,"data_stored":2698616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":219335,"internal_metadata":659990329},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:56 2023","interfaces":[{"interface":"back","average":{"1min":0.30399999999999999,"5min":0.34699999999999998,"15min":0.36499999999999999},"min":{"1min":0.23999999999999999,"5min":0.14199999999999999,"15min":0.14199999999999999},"max":{"1min":0.42399999999999999,"5min":1.595,"15min":7.4249999999999998},"last":0.17100000000000001},{"interface":"front","average":{"1min":0.35499999999999998,"5min":0.40899999999999997,"15min":0.376},"min":{"1min":0.23999999999999999,"5min":0.19600000000000001,"15min":0.183},"max":{"1min":0.46999999999999997,"5min":2.6859999999999999,"15min":2.6859999999999999},"last":0.23000000000000001}]},{"osd":2,"last update":"Wed Nov 8 05:33:03 2023","interfaces":[{"interface":"back","average":{"1min":0.26000000000000001,"5min":0.317,"15min":0.35999999999999999},"min":{"1min":0.18099999999999999,"5min":0.161,"15min":0.161},"max":{"1min":0.40799999999999997,"5min":2.2650000000000001,"15min":5.8209999999999997},"last":0.23100000000000001},{"interface":"front","average":{"1min":0.26900000000000002,"5min":0.28699999999999998,"15min":0.35199999999999998},"min":{"1min":0.192,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.41999999999999998,"5min":1.137,"15min":4.2350000000000003},"last":0.19500000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-08T05:33:15.727 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-11-08T05:33:15.728 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 459, 'reported_epoch': 458, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:30:44.490644+0000', 'last_change': '2023-11-08T05:28:13.557071+0000', 'last_active': '2023-11-08T05:30:44.490644+0000', 'last_peered': '2023-11-08T05:30:44.490644+0000', 'last_clean': '2023-11-08T05:28:01.458439+0000', 'last_became_active': '2023-11-08T05:28:13.557071+0000', 'last_became_peered': '2023-11-08T05:28:13.557071+0000', 'last_unstale': '2023-11-08T05:30:44.490644+0000', 'last_undegraded': '2023-11-08T05:30:44.490644+0000', 'last_fullsized': '2023-11-08T05:28:13.547025+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:33:15.728 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-11-08T05:33:15.728 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 11, 'reported_epoch': 459, 'state': 'active+clean+remapped', 'last_fresh': '2023-11-08T05:30:45.794760+0000', 'last_change': '2023-11-08T05:30:45.794760+0000', 'last_active': '2023-11-08T05:30:45.794760+0000', 'last_peered': '2023-11-08T05:30:45.794760+0000', 'last_clean': '2023-11-08T05:30:45.794760+0000', 'last_became_active': '2023-11-08T05:30:45.794398+0000', 'last_became_peered': '2023-11-08T05:30:45.794398+0000', 'last_unstale': '2023-11-08T05:30:45.794760+0000', 'last_undegraded': '2023-11-08T05:30:45.794760+0000', 'last_fullsized': '2023-11-08T05:30:45.794760+0000', 'mapping_epoch': 458, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 459, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2], 'acting': [2, 1], 'avail_no_missing': ['2', '1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2023-11-08T05:33:15.729 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.1 is not active+clean 2023-11-08T05:33:15.729 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.1', 'version': "0'0", 'reported_seq': 465, 'reported_epoch': 459, 'state': 'active+clean+remapped', 'last_fresh': '2023-11-08T05:30:45.795085+0000', 'last_change': '2023-11-08T05:30:45.795085+0000', 'last_active': '2023-11-08T05:30:45.795085+0000', 'last_peered': '2023-11-08T05:30:45.795085+0000', 'last_clean': '2023-11-08T05:30:45.795085+0000', 'last_became_active': '2023-11-08T05:30:45.794743+0000', 'last_became_peered': '2023-11-08T05:30:45.794743+0000', 'last_unstale': '2023-11-08T05:30:45.795085+0000', 'last_undegraded': '2023-11-08T05:30:45.795085+0000', 'last_fullsized': '2023-11-08T05:30:45.795085+0000', 'mapping_epoch': 458, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 459, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2], 'acting': [2, 1], 'avail_no_missing': ['2', '1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2023-11-08T05:33:15.729 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-11-08T05:33:15.729 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "0'0", 'reported_seq': 460, 'reported_epoch': 458, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:30:44.490523+0000', 'last_change': '2023-11-08T05:28:13.545468+0000', 'last_active': '2023-11-08T05:30:44.490523+0000', 'last_peered': '2023-11-08T05:30:44.490523+0000', 'last_clean': '2023-11-08T05:28:01.456723+0000', 'last_became_active': '2023-11-08T05:28:13.545468+0000', 'last_became_peered': '2023-11-08T05:28:13.545468+0000', 'last_unstale': '2023-11-08T05:30:44.490523+0000', 'last_undegraded': '2023-11-08T05:30:44.490523+0000', 'last_fullsized': '2023-11-08T05:28:13.537207+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 9, 'last_epoch_clean': 10, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:45.528796+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:45.528796+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:45.528796+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:33:15.730 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.0 is not active+clean 2023-11-08T05:33:15.730 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.0', 'version': "0'0", 'reported_seq': 465, 'reported_epoch': 458, 'state': 'active+clean+remapped', 'last_fresh': '2023-11-08T05:30:44.491907+0000', 'last_change': '2023-11-08T05:29:44.395350+0000', 'last_active': '2023-11-08T05:30:44.491907+0000', 'last_peered': '2023-11-08T05:30:44.491907+0000', 'last_clean': '2023-11-08T05:30:44.491907+0000', 'last_became_active': '2023-11-08T05:29:44.395012+0000', 'last_became_peered': '2023-11-08T05:29:44.395012+0000', 'last_unstale': '2023-11-08T05:30:44.491907+0000', 'last_undegraded': '2023-11-08T05:30:44.491907+0000', 'last_fullsized': '2023-11-08T05:30:44.491907+0000', 'mapping_epoch': 455, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 456, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2], 'acting': [2, 1], 'avail_no_missing': ['2', '1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2023-11-08T05:33:15.730 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-11-08T05:33:15.730 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 459, 'reported_epoch': 458, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:30:44.490629+0000', 'last_change': '2023-11-08T05:28:13.563357+0000', 'last_active': '2023-11-08T05:30:44.490629+0000', 'last_peered': '2023-11-08T05:30:44.490629+0000', 'last_clean': '2023-11-08T05:28:01.457691+0000', 'last_became_active': '2023-11-08T05:28:13.563357+0000', 'last_became_peered': '2023-11-08T05:28:13.563357+0000', 'last_unstale': '2023-11-08T05:30:44.490629+0000', 'last_undegraded': '2023-11-08T05:30:44.490629+0000', 'last_fullsized': '2023-11-08T05:28:13.541614+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:33:15.731 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-11-08T05:33:15.731 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 459, 'reported_epoch': 458, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:30:44.490691+0000', 'last_change': '2023-11-08T05:28:13.550672+0000', 'last_active': '2023-11-08T05:30:44.490691+0000', 'last_peered': '2023-11-08T05:30:44.490691+0000', 'last_clean': '2023-11-08T05:28:01.457904+0000', 'last_became_active': '2023-11-08T05:28:13.550672+0000', 'last_became_peered': '2023-11-08T05:28:13.550672+0000', 'last_unstale': '2023-11-08T05:30:44.490691+0000', 'last_undegraded': '2023-11-08T05:30:44.490691+0000', 'last_fullsized': '2023-11-08T05:28:13.539595+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:33:15.731 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-11-08T05:33:15.731 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 459, 'reported_epoch': 458, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:30:44.490805+0000', 'last_change': '2023-11-08T05:28:13.557968+0000', 'last_active': '2023-11-08T05:30:44.490805+0000', 'last_peered': '2023-11-08T05:30:44.490805+0000', 'last_clean': '2023-11-08T05:28:01.458433+0000', 'last_became_active': '2023-11-08T05:28:13.557968+0000', 'last_became_peered': '2023-11-08T05:28:13.557968+0000', 'last_unstale': '2023-11-08T05:30:44.490805+0000', 'last_undegraded': '2023-11-08T05:30:44.490805+0000', 'last_fullsized': '2023-11-08T05:28:13.542832+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:33:15.732 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-11-08T05:33:16.076 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T05:33:16.076 INFO:teuthology.orchestra.run.smithi087.stdout:{"fsid":"ab4cbc06-449d-497d-b513-220d9db2b590","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 5 pgs undersized","count":5},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2438,"monmap":{"epoch":1,"min_mon_release_name":"pacific","num_mons":1},"osdmap":{"epoch":459,"num_osds":3,"num_up_osds":2,"osd_up_since":1699421292,"num_in_osds":3,"osd_in_since":1699419158,"num_remapped_pgs":3},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean+remapped","count":3},{"state_name":"active+clean","count":1}],"num_pgs":9,"num_pools":2,"num_objects":0,"data_bytes":0,"bytes_used":2697592832,"bytes_avail":287212699648,"bytes_total":289910292480},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":41,"modified":"2023-11-08T05:29:32.044544+0000","services":{}},"progress_events":{}} 2023-11-08T05:33:16.090 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-11-08T05:33:16.090 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-11-08T05:33:16.374 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T05:33:16.375 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2023-11-08T05:33:16.386 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":1672,"stamp":"2023-11-08T05:33:16.090871+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":10,"acting":13,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":109,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2634368,"kb_used_data":409012,"kb_used_omap":780,"kb_used_meta":2220403,"kb_avail":280481152,"statfs":{"total":289910292480,"available":287212699648,"internally_reserved":0,"allocated":418828288,"data_stored":410500163,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":799447,"internal_metadata":2273692969},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-536782902,"num_objects":-164,"num_object_clones":0,"num_object_copies":-328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-115,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-164,"num_whiteouts":0,"num_read":-122056,"num_read_kb":-39011942,"num_write":-413642,"num_write_kb":-151705339,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-8,"num_bytes_recovered":-16922624,"num_keys_recovered":-2,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.424817"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490644+0000","last_change":"2023-11-08T05:28:13.557071+0000","last_active":"2023-11-08T05:30:44.490644+0000","last_peered":"2023-11-08T05:30:44.490644+0000","last_clean":"2023-11-08T05:28:01.458439+0000","last_became_active":"2023-11-08T05:28:13.557071+0000","last_became_peered":"2023-11-08T05:28:13.557071+0000","last_unstale":"2023-11-08T05:30:44.490644+0000","last_undegraded":"2023-11-08T05:30:44.490644+0000","last_fullsized":"2023-11-08T05:28:13.547025+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":459,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:45.794760+0000","last_change":"2023-11-08T05:30:45.794760+0000","last_active":"2023-11-08T05:30:45.794760+0000","last_peered":"2023-11-08T05:30:45.794760+0000","last_clean":"2023-11-08T05:30:45.794760+0000","last_became_active":"2023-11-08T05:30:45.794398+0000","last_became_peered":"2023-11-08T05:30:45.794398+0000","last_unstale":"2023-11-08T05:30:45.794760+0000","last_undegraded":"2023-11-08T05:30:45.794760+0000","last_fullsized":"2023-11-08T05:30:45.794760+0000","mapping_epoch":458,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":459,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":465,"reported_epoch":459,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:45.795085+0000","last_change":"2023-11-08T05:30:45.795085+0000","last_active":"2023-11-08T05:30:45.795085+0000","last_peered":"2023-11-08T05:30:45.795085+0000","last_clean":"2023-11-08T05:30:45.795085+0000","last_became_active":"2023-11-08T05:30:45.794743+0000","last_became_peered":"2023-11-08T05:30:45.794743+0000","last_unstale":"2023-11-08T05:30:45.795085+0000","last_undegraded":"2023-11-08T05:30:45.795085+0000","last_fullsized":"2023-11-08T05:30:45.795085+0000","mapping_epoch":458,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":459,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":454,"reported_epoch":458,"state":"active+clean","last_fresh":"2023-11-08T05:30:44.490618+0000","last_change":"2023-11-08T04:52:47.548597+0000","last_active":"2023-11-08T05:30:44.490618+0000","last_peered":"2023-11-08T05:30:44.490618+0000","last_clean":"2023-11-08T05:30:44.490618+0000","last_became_active":"2023-11-08T04:52:47.548294+0000","last_became_peered":"2023-11-08T04:52:47.548294+0000","last_unstale":"2023-11-08T05:30:44.490618+0000","last_undegraded":"2023-11-08T05:30:44.490618+0000","last_fullsized":"2023-11-08T05:30:44.490618+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":460,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490523+0000","last_change":"2023-11-08T05:28:13.545468+0000","last_active":"2023-11-08T05:30:44.490523+0000","last_peered":"2023-11-08T05:30:44.490523+0000","last_clean":"2023-11-08T05:28:01.456723+0000","last_became_active":"2023-11-08T05:28:13.545468+0000","last_became_peered":"2023-11-08T05:28:13.545468+0000","last_unstale":"2023-11-08T05:30:44.490523+0000","last_undegraded":"2023-11-08T05:30:44.490523+0000","last_fullsized":"2023-11-08T05:28:13.537207+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_clean_scrub_stamp":"2023-11-08T04:52:45.528796+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":465,"reported_epoch":458,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:44.491907+0000","last_change":"2023-11-08T05:29:44.395350+0000","last_active":"2023-11-08T05:30:44.491907+0000","last_peered":"2023-11-08T05:30:44.491907+0000","last_clean":"2023-11-08T05:30:44.491907+0000","last_became_active":"2023-11-08T05:29:44.395012+0000","last_became_peered":"2023-11-08T05:29:44.395012+0000","last_unstale":"2023-11-08T05:30:44.491907+0000","last_undegraded":"2023-11-08T05:30:44.491907+0000","last_fullsized":"2023-11-08T05:30:44.491907+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490629+0000","last_change":"2023-11-08T05:28:13.563357+0000","last_active":"2023-11-08T05:30:44.490629+0000","last_peered":"2023-11-08T05:30:44.490629+0000","last_clean":"2023-11-08T05:28:01.457691+0000","last_became_active":"2023-11-08T05:28:13.563357+0000","last_became_peered":"2023-11-08T05:28:13.563357+0000","last_unstale":"2023-11-08T05:30:44.490629+0000","last_undegraded":"2023-11-08T05:30:44.490629+0000","last_fullsized":"2023-11-08T05:28:13.541614+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490691+0000","last_change":"2023-11-08T05:28:13.550672+0000","last_active":"2023-11-08T05:30:44.490691+0000","last_peered":"2023-11-08T05:30:44.490691+0000","last_clean":"2023-11-08T05:28:01.457904+0000","last_became_active":"2023-11-08T05:28:13.550672+0000","last_became_peered":"2023-11-08T05:28:13.550672+0000","last_unstale":"2023-11-08T05:30:44.490691+0000","last_undegraded":"2023-11-08T05:30:44.490691+0000","last_fullsized":"2023-11-08T05:28:13.539595+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490805+0000","last_change":"2023-11-08T05:28:13.557968+0000","last_active":"2023-11-08T05:30:44.490805+0000","last_peered":"2023-11-08T05:30:44.490805+0000","last_clean":"2023-11-08T05:28:01.458433+0000","last_became_active":"2023-11-08T05:28:13.557968+0000","last_became_peered":"2023-11-08T05:28:13.557968+0000","last_unstale":"2023-11-08T05:30:44.490805+0000","last_undegraded":"2023-11-08T05:30:44.490805+0000","last_fullsized":"2023-11-08T05:28:13.542832+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":1,"acting":1,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":12,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738857,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":645040,"kb_used_data":5352,"kb_used_omap":275,"kb_used_meta":639404,"kb_avail":93726800,"statfs":{"total":96636764160,"available":95976243200,"internally_reserved":0,"allocated":5480448,"data_stored":2697647,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":281650,"internal_metadata":654750670},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:59 2023","interfaces":[{"interface":"back","average":{"1min":0.30499999999999999,"5min":0.439,"15min":0.36099999999999999},"min":{"1min":0.17399999999999999,"5min":0.14499999999999999,"15min":0.14499999999999999},"max":{"1min":0.60099999999999998,"5min":4.9379999999999997,"15min":4.9379999999999997},"last":0.379},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.49399999999999999,"15min":0.42399999999999999},"min":{"1min":0.221,"5min":0.19600000000000001,"15min":0.17000000000000001},"max":{"1min":0.63200000000000001,"5min":4.9859999999999998,"15min":4.9859999999999998},"last":0.40899999999999997}]},{"osd":1,"last update":"Wed Nov 8 05:33:07 2023","interfaces":[{"interface":"back","average":{"1min":0.26200000000000001,"5min":0.27900000000000003,"15min":0.46500000000000002},"min":{"1min":0.187,"5min":0.16,"15min":0.153},"max":{"1min":0.372,"5min":0.68200000000000005,"15min":23.242999999999999},"last":0.28000000000000003},{"interface":"front","average":{"1min":0.28100000000000003,"5min":0.32400000000000001,"15min":0.45600000000000002},"min":{"1min":0.20300000000000001,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":0.39600000000000002,"5min":2.0419999999999998,"15min":23.251000000000001},"last":0.252}]}]},{"osd":0,"up_from":8,"seq":34359738798,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1339228,"kb_used_data":398304,"kb_used_omap":291,"kb_used_meta":936476,"kb_avail":93032612,"statfs":{"total":96636764160,"available":95265394688,"internally_reserved":0,"allocated":407863296,"data_stored":405103900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":298462,"internal_metadata":958951970},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.436,"15min":0.36299999999999999},"min":{"1min":0.19700000000000001,"5min":0.14699999999999999,"15min":0.13700000000000001},"max":{"1min":3.222,"5min":6.5499999999999998,"15min":6.9989999999999997},"last":0.245},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.46300000000000002,"15min":0.42299999999999999},"min":{"1min":0.17799999999999999,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":5.0030000000000001,"5min":6.0629999999999997,"15min":9.6869999999999994},"last":0.28599999999999998}]},{"osd":2,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.51600000000000001,"15min":0.47099999999999997},"min":{"1min":0.21099999999999999,"5min":0.153,"15min":0.153},"max":{"1min":5.7439999999999998,"5min":5.8280000000000003,"15min":9.7639999999999993},"last":0.252},{"interface":"front","average":{"1min":0.497,"5min":0.56200000000000006,"15min":0.42299999999999999},"min":{"1min":0.20399999999999999,"5min":0.187,"15min":0.16800000000000001},"max":{"1min":3.9319999999999999,"5min":8.8040000000000003,"15min":8.8040000000000003},"last":0.29899999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738859,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":650100,"kb_used_data":5356,"kb_used_omap":214,"kb_used_meta":644521,"kb_avail":93721740,"statfs":{"total":96636764160,"available":95971061760,"internally_reserved":0,"allocated":5484544,"data_stored":2698616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":219335,"internal_metadata":659990329},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:56 2023","interfaces":[{"interface":"back","average":{"1min":0.30399999999999999,"5min":0.34699999999999998,"15min":0.36499999999999999},"min":{"1min":0.23999999999999999,"5min":0.14199999999999999,"15min":0.14199999999999999},"max":{"1min":0.42399999999999999,"5min":1.595,"15min":7.4249999999999998},"last":0.17100000000000001},{"interface":"front","average":{"1min":0.35499999999999998,"5min":0.40899999999999997,"15min":0.376},"min":{"1min":0.23999999999999999,"5min":0.19600000000000001,"15min":0.183},"max":{"1min":0.46999999999999997,"5min":2.6859999999999999,"15min":2.6859999999999999},"last":0.23000000000000001}]},{"osd":2,"last update":"Wed Nov 8 05:33:03 2023","interfaces":[{"interface":"back","average":{"1min":0.26000000000000001,"5min":0.317,"15min":0.35999999999999999},"min":{"1min":0.18099999999999999,"5min":0.161,"15min":0.161},"max":{"1min":0.40799999999999997,"5min":2.2650000000000001,"15min":5.8209999999999997},"last":0.19400000000000001},{"interface":"front","average":{"1min":0.26900000000000002,"5min":0.28699999999999998,"15min":0.35199999999999998},"min":{"1min":0.192,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.41999999999999998,"5min":1.137,"15min":4.2350000000000003},"last":0.20200000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-08T05:33:19.388 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-11-08T05:33:19.685 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T05:33:19.687 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2023-11-08T05:33:19.698 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":1673,"stamp":"2023-11-08T05:33:18.091402+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":10,"acting":13,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":109,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2634368,"kb_used_data":409012,"kb_used_omap":780,"kb_used_meta":2220403,"kb_avail":280481152,"statfs":{"total":289910292480,"available":287212699648,"internally_reserved":0,"allocated":418828288,"data_stored":410500163,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":799447,"internal_metadata":2273692969},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-536782902,"num_objects":-164,"num_object_clones":0,"num_object_copies":-328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-115,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-164,"num_whiteouts":0,"num_read":-122056,"num_read_kb":-39011942,"num_write":-413642,"num_write_kb":-151705339,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-8,"num_bytes_recovered":-16922624,"num_keys_recovered":-2,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.424817"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490644+0000","last_change":"2023-11-08T05:28:13.557071+0000","last_active":"2023-11-08T05:30:44.490644+0000","last_peered":"2023-11-08T05:30:44.490644+0000","last_clean":"2023-11-08T05:28:01.458439+0000","last_became_active":"2023-11-08T05:28:13.557071+0000","last_became_peered":"2023-11-08T05:28:13.557071+0000","last_unstale":"2023-11-08T05:30:44.490644+0000","last_undegraded":"2023-11-08T05:30:44.490644+0000","last_fullsized":"2023-11-08T05:28:13.547025+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":459,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:45.794760+0000","last_change":"2023-11-08T05:30:45.794760+0000","last_active":"2023-11-08T05:30:45.794760+0000","last_peered":"2023-11-08T05:30:45.794760+0000","last_clean":"2023-11-08T05:30:45.794760+0000","last_became_active":"2023-11-08T05:30:45.794398+0000","last_became_peered":"2023-11-08T05:30:45.794398+0000","last_unstale":"2023-11-08T05:30:45.794760+0000","last_undegraded":"2023-11-08T05:30:45.794760+0000","last_fullsized":"2023-11-08T05:30:45.794760+0000","mapping_epoch":458,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":459,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":465,"reported_epoch":459,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:45.795085+0000","last_change":"2023-11-08T05:30:45.795085+0000","last_active":"2023-11-08T05:30:45.795085+0000","last_peered":"2023-11-08T05:30:45.795085+0000","last_clean":"2023-11-08T05:30:45.795085+0000","last_became_active":"2023-11-08T05:30:45.794743+0000","last_became_peered":"2023-11-08T05:30:45.794743+0000","last_unstale":"2023-11-08T05:30:45.795085+0000","last_undegraded":"2023-11-08T05:30:45.795085+0000","last_fullsized":"2023-11-08T05:30:45.795085+0000","mapping_epoch":458,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":459,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":454,"reported_epoch":458,"state":"active+clean","last_fresh":"2023-11-08T05:30:44.490618+0000","last_change":"2023-11-08T04:52:47.548597+0000","last_active":"2023-11-08T05:30:44.490618+0000","last_peered":"2023-11-08T05:30:44.490618+0000","last_clean":"2023-11-08T05:30:44.490618+0000","last_became_active":"2023-11-08T04:52:47.548294+0000","last_became_peered":"2023-11-08T04:52:47.548294+0000","last_unstale":"2023-11-08T05:30:44.490618+0000","last_undegraded":"2023-11-08T05:30:44.490618+0000","last_fullsized":"2023-11-08T05:30:44.490618+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":460,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490523+0000","last_change":"2023-11-08T05:28:13.545468+0000","last_active":"2023-11-08T05:30:44.490523+0000","last_peered":"2023-11-08T05:30:44.490523+0000","last_clean":"2023-11-08T05:28:01.456723+0000","last_became_active":"2023-11-08T05:28:13.545468+0000","last_became_peered":"2023-11-08T05:28:13.545468+0000","last_unstale":"2023-11-08T05:30:44.490523+0000","last_undegraded":"2023-11-08T05:30:44.490523+0000","last_fullsized":"2023-11-08T05:28:13.537207+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_clean_scrub_stamp":"2023-11-08T04:52:45.528796+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":465,"reported_epoch":458,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:44.491907+0000","last_change":"2023-11-08T05:29:44.395350+0000","last_active":"2023-11-08T05:30:44.491907+0000","last_peered":"2023-11-08T05:30:44.491907+0000","last_clean":"2023-11-08T05:30:44.491907+0000","last_became_active":"2023-11-08T05:29:44.395012+0000","last_became_peered":"2023-11-08T05:29:44.395012+0000","last_unstale":"2023-11-08T05:30:44.491907+0000","last_undegraded":"2023-11-08T05:30:44.491907+0000","last_fullsized":"2023-11-08T05:30:44.491907+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490629+0000","last_change":"2023-11-08T05:28:13.563357+0000","last_active":"2023-11-08T05:30:44.490629+0000","last_peered":"2023-11-08T05:30:44.490629+0000","last_clean":"2023-11-08T05:28:01.457691+0000","last_became_active":"2023-11-08T05:28:13.563357+0000","last_became_peered":"2023-11-08T05:28:13.563357+0000","last_unstale":"2023-11-08T05:30:44.490629+0000","last_undegraded":"2023-11-08T05:30:44.490629+0000","last_fullsized":"2023-11-08T05:28:13.541614+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490691+0000","last_change":"2023-11-08T05:28:13.550672+0000","last_active":"2023-11-08T05:30:44.490691+0000","last_peered":"2023-11-08T05:30:44.490691+0000","last_clean":"2023-11-08T05:28:01.457904+0000","last_became_active":"2023-11-08T05:28:13.550672+0000","last_became_peered":"2023-11-08T05:28:13.550672+0000","last_unstale":"2023-11-08T05:30:44.490691+0000","last_undegraded":"2023-11-08T05:30:44.490691+0000","last_fullsized":"2023-11-08T05:28:13.539595+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490805+0000","last_change":"2023-11-08T05:28:13.557968+0000","last_active":"2023-11-08T05:30:44.490805+0000","last_peered":"2023-11-08T05:30:44.490805+0000","last_clean":"2023-11-08T05:28:01.458433+0000","last_became_active":"2023-11-08T05:28:13.557968+0000","last_became_peered":"2023-11-08T05:28:13.557968+0000","last_unstale":"2023-11-08T05:30:44.490805+0000","last_undegraded":"2023-11-08T05:30:44.490805+0000","last_fullsized":"2023-11-08T05:28:13.542832+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":1,"acting":1,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":12,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738857,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":645040,"kb_used_data":5352,"kb_used_omap":275,"kb_used_meta":639404,"kb_avail":93726800,"statfs":{"total":96636764160,"available":95976243200,"internally_reserved":0,"allocated":5480448,"data_stored":2697647,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":281650,"internal_metadata":654750670},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:59 2023","interfaces":[{"interface":"back","average":{"1min":0.30499999999999999,"5min":0.439,"15min":0.36099999999999999},"min":{"1min":0.17399999999999999,"5min":0.14499999999999999,"15min":0.14499999999999999},"max":{"1min":0.60099999999999998,"5min":4.9379999999999997,"15min":4.9379999999999997},"last":0.379},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.49399999999999999,"15min":0.42399999999999999},"min":{"1min":0.221,"5min":0.19600000000000001,"15min":0.17000000000000001},"max":{"1min":0.63200000000000001,"5min":4.9859999999999998,"15min":4.9859999999999998},"last":0.40899999999999997}]},{"osd":1,"last update":"Wed Nov 8 05:33:07 2023","interfaces":[{"interface":"back","average":{"1min":0.26200000000000001,"5min":0.27900000000000003,"15min":0.46500000000000002},"min":{"1min":0.187,"5min":0.16,"15min":0.153},"max":{"1min":0.372,"5min":0.68200000000000005,"15min":23.242999999999999},"last":0.28000000000000003},{"interface":"front","average":{"1min":0.28100000000000003,"5min":0.32400000000000001,"15min":0.45600000000000002},"min":{"1min":0.20300000000000001,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":0.39600000000000002,"5min":2.0419999999999998,"15min":23.251000000000001},"last":0.252}]}]},{"osd":0,"up_from":8,"seq":34359738798,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1339228,"kb_used_data":398304,"kb_used_omap":291,"kb_used_meta":936476,"kb_avail":93032612,"statfs":{"total":96636764160,"available":95265394688,"internally_reserved":0,"allocated":407863296,"data_stored":405103900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":298462,"internal_metadata":958951970},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.436,"15min":0.36299999999999999},"min":{"1min":0.19700000000000001,"5min":0.14699999999999999,"15min":0.13700000000000001},"max":{"1min":3.222,"5min":6.5499999999999998,"15min":6.9989999999999997},"last":0.245},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.46300000000000002,"15min":0.42299999999999999},"min":{"1min":0.17799999999999999,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":5.0030000000000001,"5min":6.0629999999999997,"15min":9.6869999999999994},"last":0.28599999999999998}]},{"osd":2,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.51600000000000001,"15min":0.47099999999999997},"min":{"1min":0.21099999999999999,"5min":0.153,"15min":0.153},"max":{"1min":5.7439999999999998,"5min":5.8280000000000003,"15min":9.7639999999999993},"last":0.252},{"interface":"front","average":{"1min":0.497,"5min":0.56200000000000006,"15min":0.42299999999999999},"min":{"1min":0.20399999999999999,"5min":0.187,"15min":0.16800000000000001},"max":{"1min":3.9319999999999999,"5min":8.8040000000000003,"15min":8.8040000000000003},"last":0.29899999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738859,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":650100,"kb_used_data":5356,"kb_used_omap":214,"kb_used_meta":644521,"kb_avail":93721740,"statfs":{"total":96636764160,"available":95971061760,"internally_reserved":0,"allocated":5484544,"data_stored":2698616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":219335,"internal_metadata":659990329},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:56 2023","interfaces":[{"interface":"back","average":{"1min":0.30399999999999999,"5min":0.34699999999999998,"15min":0.36499999999999999},"min":{"1min":0.23999999999999999,"5min":0.14199999999999999,"15min":0.14199999999999999},"max":{"1min":0.42399999999999999,"5min":1.595,"15min":7.4249999999999998},"last":0.17100000000000001},{"interface":"front","average":{"1min":0.35499999999999998,"5min":0.40899999999999997,"15min":0.376},"min":{"1min":0.23999999999999999,"5min":0.19600000000000001,"15min":0.183},"max":{"1min":0.46999999999999997,"5min":2.6859999999999999,"15min":2.6859999999999999},"last":0.23000000000000001}]},{"osd":2,"last update":"Wed Nov 8 05:33:03 2023","interfaces":[{"interface":"back","average":{"1min":0.26000000000000001,"5min":0.317,"15min":0.35999999999999999},"min":{"1min":0.18099999999999999,"5min":0.161,"15min":0.161},"max":{"1min":0.40799999999999997,"5min":2.2650000000000001,"15min":5.8209999999999997},"last":0.19400000000000001},{"interface":"front","average":{"1min":0.26900000000000002,"5min":0.28699999999999998,"15min":0.35199999999999998},"min":{"1min":0.192,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.41999999999999998,"5min":1.137,"15min":4.2350000000000003},"last":0.20200000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-08T05:33:19.699 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-11-08T05:33:19.982 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T05:33:19.984 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2023-11-08T05:33:19.995 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":1673,"stamp":"2023-11-08T05:33:18.091402+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":10,"acting":13,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":109,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2634368,"kb_used_data":409012,"kb_used_omap":780,"kb_used_meta":2220403,"kb_avail":280481152,"statfs":{"total":289910292480,"available":287212699648,"internally_reserved":0,"allocated":418828288,"data_stored":410500163,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":799447,"internal_metadata":2273692969},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-536782902,"num_objects":-164,"num_object_clones":0,"num_object_copies":-328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-115,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-164,"num_whiteouts":0,"num_read":-122056,"num_read_kb":-39011942,"num_write":-413642,"num_write_kb":-151705339,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-8,"num_bytes_recovered":-16922624,"num_keys_recovered":-2,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.424817"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490644+0000","last_change":"2023-11-08T05:28:13.557071+0000","last_active":"2023-11-08T05:30:44.490644+0000","last_peered":"2023-11-08T05:30:44.490644+0000","last_clean":"2023-11-08T05:28:01.458439+0000","last_became_active":"2023-11-08T05:28:13.557071+0000","last_became_peered":"2023-11-08T05:28:13.557071+0000","last_unstale":"2023-11-08T05:30:44.490644+0000","last_undegraded":"2023-11-08T05:30:44.490644+0000","last_fullsized":"2023-11-08T05:28:13.547025+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":459,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:45.794760+0000","last_change":"2023-11-08T05:30:45.794760+0000","last_active":"2023-11-08T05:30:45.794760+0000","last_peered":"2023-11-08T05:30:45.794760+0000","last_clean":"2023-11-08T05:30:45.794760+0000","last_became_active":"2023-11-08T05:30:45.794398+0000","last_became_peered":"2023-11-08T05:30:45.794398+0000","last_unstale":"2023-11-08T05:30:45.794760+0000","last_undegraded":"2023-11-08T05:30:45.794760+0000","last_fullsized":"2023-11-08T05:30:45.794760+0000","mapping_epoch":458,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":459,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":465,"reported_epoch":459,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:45.795085+0000","last_change":"2023-11-08T05:30:45.795085+0000","last_active":"2023-11-08T05:30:45.795085+0000","last_peered":"2023-11-08T05:30:45.795085+0000","last_clean":"2023-11-08T05:30:45.795085+0000","last_became_active":"2023-11-08T05:30:45.794743+0000","last_became_peered":"2023-11-08T05:30:45.794743+0000","last_unstale":"2023-11-08T05:30:45.795085+0000","last_undegraded":"2023-11-08T05:30:45.795085+0000","last_fullsized":"2023-11-08T05:30:45.795085+0000","mapping_epoch":458,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":459,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":454,"reported_epoch":458,"state":"active+clean","last_fresh":"2023-11-08T05:30:44.490618+0000","last_change":"2023-11-08T04:52:47.548597+0000","last_active":"2023-11-08T05:30:44.490618+0000","last_peered":"2023-11-08T05:30:44.490618+0000","last_clean":"2023-11-08T05:30:44.490618+0000","last_became_active":"2023-11-08T04:52:47.548294+0000","last_became_peered":"2023-11-08T04:52:47.548294+0000","last_unstale":"2023-11-08T05:30:44.490618+0000","last_undegraded":"2023-11-08T05:30:44.490618+0000","last_fullsized":"2023-11-08T05:30:44.490618+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":460,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490523+0000","last_change":"2023-11-08T05:28:13.545468+0000","last_active":"2023-11-08T05:30:44.490523+0000","last_peered":"2023-11-08T05:30:44.490523+0000","last_clean":"2023-11-08T05:28:01.456723+0000","last_became_active":"2023-11-08T05:28:13.545468+0000","last_became_peered":"2023-11-08T05:28:13.545468+0000","last_unstale":"2023-11-08T05:30:44.490523+0000","last_undegraded":"2023-11-08T05:30:44.490523+0000","last_fullsized":"2023-11-08T05:28:13.537207+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_clean_scrub_stamp":"2023-11-08T04:52:45.528796+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":465,"reported_epoch":458,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:44.491907+0000","last_change":"2023-11-08T05:29:44.395350+0000","last_active":"2023-11-08T05:30:44.491907+0000","last_peered":"2023-11-08T05:30:44.491907+0000","last_clean":"2023-11-08T05:30:44.491907+0000","last_became_active":"2023-11-08T05:29:44.395012+0000","last_became_peered":"2023-11-08T05:29:44.395012+0000","last_unstale":"2023-11-08T05:30:44.491907+0000","last_undegraded":"2023-11-08T05:30:44.491907+0000","last_fullsized":"2023-11-08T05:30:44.491907+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490629+0000","last_change":"2023-11-08T05:28:13.563357+0000","last_active":"2023-11-08T05:30:44.490629+0000","last_peered":"2023-11-08T05:30:44.490629+0000","last_clean":"2023-11-08T05:28:01.457691+0000","last_became_active":"2023-11-08T05:28:13.563357+0000","last_became_peered":"2023-11-08T05:28:13.563357+0000","last_unstale":"2023-11-08T05:30:44.490629+0000","last_undegraded":"2023-11-08T05:30:44.490629+0000","last_fullsized":"2023-11-08T05:28:13.541614+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490691+0000","last_change":"2023-11-08T05:28:13.550672+0000","last_active":"2023-11-08T05:30:44.490691+0000","last_peered":"2023-11-08T05:30:44.490691+0000","last_clean":"2023-11-08T05:28:01.457904+0000","last_became_active":"2023-11-08T05:28:13.550672+0000","last_became_peered":"2023-11-08T05:28:13.550672+0000","last_unstale":"2023-11-08T05:30:44.490691+0000","last_undegraded":"2023-11-08T05:30:44.490691+0000","last_fullsized":"2023-11-08T05:28:13.539595+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490805+0000","last_change":"2023-11-08T05:28:13.557968+0000","last_active":"2023-11-08T05:30:44.490805+0000","last_peered":"2023-11-08T05:30:44.490805+0000","last_clean":"2023-11-08T05:28:01.458433+0000","last_became_active":"2023-11-08T05:28:13.557968+0000","last_became_peered":"2023-11-08T05:28:13.557968+0000","last_unstale":"2023-11-08T05:30:44.490805+0000","last_undegraded":"2023-11-08T05:30:44.490805+0000","last_fullsized":"2023-11-08T05:28:13.542832+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":1,"acting":1,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":12,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738857,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":645040,"kb_used_data":5352,"kb_used_omap":275,"kb_used_meta":639404,"kb_avail":93726800,"statfs":{"total":96636764160,"available":95976243200,"internally_reserved":0,"allocated":5480448,"data_stored":2697647,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":281650,"internal_metadata":654750670},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:59 2023","interfaces":[{"interface":"back","average":{"1min":0.30499999999999999,"5min":0.439,"15min":0.36099999999999999},"min":{"1min":0.17399999999999999,"5min":0.14499999999999999,"15min":0.14499999999999999},"max":{"1min":0.60099999999999998,"5min":4.9379999999999997,"15min":4.9379999999999997},"last":0.379},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.49399999999999999,"15min":0.42399999999999999},"min":{"1min":0.221,"5min":0.19600000000000001,"15min":0.17000000000000001},"max":{"1min":0.63200000000000001,"5min":4.9859999999999998,"15min":4.9859999999999998},"last":0.40899999999999997}]},{"osd":1,"last update":"Wed Nov 8 05:33:07 2023","interfaces":[{"interface":"back","average":{"1min":0.26200000000000001,"5min":0.27900000000000003,"15min":0.46500000000000002},"min":{"1min":0.187,"5min":0.16,"15min":0.153},"max":{"1min":0.372,"5min":0.68200000000000005,"15min":23.242999999999999},"last":0.28000000000000003},{"interface":"front","average":{"1min":0.28100000000000003,"5min":0.32400000000000001,"15min":0.45600000000000002},"min":{"1min":0.20300000000000001,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":0.39600000000000002,"5min":2.0419999999999998,"15min":23.251000000000001},"last":0.252}]}]},{"osd":0,"up_from":8,"seq":34359738798,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1339228,"kb_used_data":398304,"kb_used_omap":291,"kb_used_meta":936476,"kb_avail":93032612,"statfs":{"total":96636764160,"available":95265394688,"internally_reserved":0,"allocated":407863296,"data_stored":405103900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":298462,"internal_metadata":958951970},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.436,"15min":0.36299999999999999},"min":{"1min":0.19700000000000001,"5min":0.14699999999999999,"15min":0.13700000000000001},"max":{"1min":3.222,"5min":6.5499999999999998,"15min":6.9989999999999997},"last":0.245},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.46300000000000002,"15min":0.42299999999999999},"min":{"1min":0.17799999999999999,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":5.0030000000000001,"5min":6.0629999999999997,"15min":9.6869999999999994},"last":0.28599999999999998}]},{"osd":2,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.51600000000000001,"15min":0.47099999999999997},"min":{"1min":0.21099999999999999,"5min":0.153,"15min":0.153},"max":{"1min":5.7439999999999998,"5min":5.8280000000000003,"15min":9.7639999999999993},"last":0.252},{"interface":"front","average":{"1min":0.497,"5min":0.56200000000000006,"15min":0.42299999999999999},"min":{"1min":0.20399999999999999,"5min":0.187,"15min":0.16800000000000001},"max":{"1min":3.9319999999999999,"5min":8.8040000000000003,"15min":8.8040000000000003},"last":0.29899999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738859,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":650100,"kb_used_data":5356,"kb_used_omap":214,"kb_used_meta":644521,"kb_avail":93721740,"statfs":{"total":96636764160,"available":95971061760,"internally_reserved":0,"allocated":5484544,"data_stored":2698616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":219335,"internal_metadata":659990329},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:56 2023","interfaces":[{"interface":"back","average":{"1min":0.30399999999999999,"5min":0.34699999999999998,"15min":0.36499999999999999},"min":{"1min":0.23999999999999999,"5min":0.14199999999999999,"15min":0.14199999999999999},"max":{"1min":0.42399999999999999,"5min":1.595,"15min":7.4249999999999998},"last":0.17100000000000001},{"interface":"front","average":{"1min":0.35499999999999998,"5min":0.40899999999999997,"15min":0.376},"min":{"1min":0.23999999999999999,"5min":0.19600000000000001,"15min":0.183},"max":{"1min":0.46999999999999997,"5min":2.6859999999999999,"15min":2.6859999999999999},"last":0.23000000000000001}]},{"osd":2,"last update":"Wed Nov 8 05:33:03 2023","interfaces":[{"interface":"back","average":{"1min":0.26000000000000001,"5min":0.317,"15min":0.35999999999999999},"min":{"1min":0.18099999999999999,"5min":0.161,"15min":0.161},"max":{"1min":0.40799999999999997,"5min":2.2650000000000001,"15min":5.8209999999999997},"last":0.19400000000000001},{"interface":"front","average":{"1min":0.26900000000000002,"5min":0.28699999999999998,"15min":0.35199999999999998},"min":{"1min":0.192,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.41999999999999998,"5min":1.137,"15min":4.2350000000000003},"last":0.20200000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-08T05:33:19.996 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-11-08T05:33:19.996 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 459, 'reported_epoch': 458, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:30:44.490644+0000', 'last_change': '2023-11-08T05:28:13.557071+0000', 'last_active': '2023-11-08T05:30:44.490644+0000', 'last_peered': '2023-11-08T05:30:44.490644+0000', 'last_clean': '2023-11-08T05:28:01.458439+0000', 'last_became_active': '2023-11-08T05:28:13.557071+0000', 'last_became_peered': '2023-11-08T05:28:13.557071+0000', 'last_unstale': '2023-11-08T05:30:44.490644+0000', 'last_undegraded': '2023-11-08T05:30:44.490644+0000', 'last_fullsized': '2023-11-08T05:28:13.547025+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:33:19.997 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-11-08T05:33:19.997 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 11, 'reported_epoch': 459, 'state': 'active+clean+remapped', 'last_fresh': '2023-11-08T05:30:45.794760+0000', 'last_change': '2023-11-08T05:30:45.794760+0000', 'last_active': '2023-11-08T05:30:45.794760+0000', 'last_peered': '2023-11-08T05:30:45.794760+0000', 'last_clean': '2023-11-08T05:30:45.794760+0000', 'last_became_active': '2023-11-08T05:30:45.794398+0000', 'last_became_peered': '2023-11-08T05:30:45.794398+0000', 'last_unstale': '2023-11-08T05:30:45.794760+0000', 'last_undegraded': '2023-11-08T05:30:45.794760+0000', 'last_fullsized': '2023-11-08T05:30:45.794760+0000', 'mapping_epoch': 458, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 459, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2], 'acting': [2, 1], 'avail_no_missing': ['2', '1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2023-11-08T05:33:19.997 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.1 is not active+clean 2023-11-08T05:33:19.998 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.1', 'version': "0'0", 'reported_seq': 465, 'reported_epoch': 459, 'state': 'active+clean+remapped', 'last_fresh': '2023-11-08T05:30:45.795085+0000', 'last_change': '2023-11-08T05:30:45.795085+0000', 'last_active': '2023-11-08T05:30:45.795085+0000', 'last_peered': '2023-11-08T05:30:45.795085+0000', 'last_clean': '2023-11-08T05:30:45.795085+0000', 'last_became_active': '2023-11-08T05:30:45.794743+0000', 'last_became_peered': '2023-11-08T05:30:45.794743+0000', 'last_unstale': '2023-11-08T05:30:45.795085+0000', 'last_undegraded': '2023-11-08T05:30:45.795085+0000', 'last_fullsized': '2023-11-08T05:30:45.795085+0000', 'mapping_epoch': 458, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 459, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2], 'acting': [2, 1], 'avail_no_missing': ['2', '1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2023-11-08T05:33:19.998 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-11-08T05:33:19.998 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "0'0", 'reported_seq': 460, 'reported_epoch': 458, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:30:44.490523+0000', 'last_change': '2023-11-08T05:28:13.545468+0000', 'last_active': '2023-11-08T05:30:44.490523+0000', 'last_peered': '2023-11-08T05:30:44.490523+0000', 'last_clean': '2023-11-08T05:28:01.456723+0000', 'last_became_active': '2023-11-08T05:28:13.545468+0000', 'last_became_peered': '2023-11-08T05:28:13.545468+0000', 'last_unstale': '2023-11-08T05:30:44.490523+0000', 'last_undegraded': '2023-11-08T05:30:44.490523+0000', 'last_fullsized': '2023-11-08T05:28:13.537207+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 9, 'last_epoch_clean': 10, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:45.528796+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:45.528796+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:45.528796+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:33:19.999 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.0 is not active+clean 2023-11-08T05:33:19.999 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.0', 'version': "0'0", 'reported_seq': 465, 'reported_epoch': 458, 'state': 'active+clean+remapped', 'last_fresh': '2023-11-08T05:30:44.491907+0000', 'last_change': '2023-11-08T05:29:44.395350+0000', 'last_active': '2023-11-08T05:30:44.491907+0000', 'last_peered': '2023-11-08T05:30:44.491907+0000', 'last_clean': '2023-11-08T05:30:44.491907+0000', 'last_became_active': '2023-11-08T05:29:44.395012+0000', 'last_became_peered': '2023-11-08T05:29:44.395012+0000', 'last_unstale': '2023-11-08T05:30:44.491907+0000', 'last_undegraded': '2023-11-08T05:30:44.491907+0000', 'last_fullsized': '2023-11-08T05:30:44.491907+0000', 'mapping_epoch': 455, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 456, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2], 'acting': [2, 1], 'avail_no_missing': ['2', '1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2023-11-08T05:33:19.999 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-11-08T05:33:19.999 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 459, 'reported_epoch': 458, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:30:44.490629+0000', 'last_change': '2023-11-08T05:28:13.563357+0000', 'last_active': '2023-11-08T05:30:44.490629+0000', 'last_peered': '2023-11-08T05:30:44.490629+0000', 'last_clean': '2023-11-08T05:28:01.457691+0000', 'last_became_active': '2023-11-08T05:28:13.563357+0000', 'last_became_peered': '2023-11-08T05:28:13.563357+0000', 'last_unstale': '2023-11-08T05:30:44.490629+0000', 'last_undegraded': '2023-11-08T05:30:44.490629+0000', 'last_fullsized': '2023-11-08T05:28:13.541614+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:33:20.000 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-11-08T05:33:20.000 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 459, 'reported_epoch': 458, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:30:44.490691+0000', 'last_change': '2023-11-08T05:28:13.550672+0000', 'last_active': '2023-11-08T05:30:44.490691+0000', 'last_peered': '2023-11-08T05:30:44.490691+0000', 'last_clean': '2023-11-08T05:28:01.457904+0000', 'last_became_active': '2023-11-08T05:28:13.550672+0000', 'last_became_peered': '2023-11-08T05:28:13.550672+0000', 'last_unstale': '2023-11-08T05:30:44.490691+0000', 'last_undegraded': '2023-11-08T05:30:44.490691+0000', 'last_fullsized': '2023-11-08T05:28:13.539595+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:33:20.000 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-11-08T05:33:20.000 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 459, 'reported_epoch': 458, 'state': 'active+undersized', 'last_fresh': '2023-11-08T05:30:44.490805+0000', 'last_change': '2023-11-08T05:28:13.557968+0000', 'last_active': '2023-11-08T05:30:44.490805+0000', 'last_peered': '2023-11-08T05:30:44.490805+0000', 'last_clean': '2023-11-08T05:28:01.458433+0000', 'last_became_active': '2023-11-08T05:28:13.557968+0000', 'last_became_peered': '2023-11-08T05:28:13.557968+0000', 'last_unstale': '2023-11-08T05:30:44.490805+0000', 'last_undegraded': '2023-11-08T05:30:44.490805+0000', 'last_fullsized': '2023-11-08T05:28:13.542832+0000', 'mapping_epoch': 445, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'last_clean_scrub_stamp': '2023-11-08T04:52:46.531964+0000', 'log_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-11-08T05:33:20.001 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-11-08T05:33:20.346 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T05:33:20.346 INFO:teuthology.orchestra.run.smithi087.stdout:{"fsid":"ab4cbc06-449d-497d-b513-220d9db2b590","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 5 pgs undersized","count":5},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2442,"monmap":{"epoch":1,"min_mon_release_name":"pacific","num_mons":1},"osdmap":{"epoch":459,"num_osds":3,"num_up_osds":2,"osd_up_since":1699421292,"num_in_osds":3,"osd_in_since":1699419158,"num_remapped_pgs":3},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean+remapped","count":3},{"state_name":"active+clean","count":1}],"num_pgs":9,"num_pools":2,"num_objects":0,"data_bytes":0,"bytes_used":2697592832,"bytes_avail":287212699648,"bytes_total":289910292480},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":41,"modified":"2023-11-08T05:29:32.044544+0000","services":{}},"progress_events":{}} 2023-11-08T05:33:20.359 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-11-08T05:33:20.360 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-11-08T05:33:20.642 INFO:teuthology.orchestra.run.smithi087.stdout: 2023-11-08T05:33:20.644 INFO:teuthology.orchestra.run.smithi087.stderr:dumped all 2023-11-08T05:33:20.655 INFO:teuthology.orchestra.run.smithi087.stdout:{"pg_ready":true,"pg_map":{"version":1674,"stamp":"2023-11-08T05:33:20.091862+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":10,"acting":13,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":109,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2634368,"kb_used_data":409012,"kb_used_omap":780,"kb_used_meta":2220403,"kb_avail":280481152,"statfs":{"total":289910292480,"available":287212699648,"internally_reserved":0,"allocated":418828288,"data_stored":410500163,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":799447,"internal_metadata":2273692969},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-536782902,"num_objects":-164,"num_object_clones":0,"num_object_copies":-328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-115,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-164,"num_whiteouts":0,"num_read":-122056,"num_read_kb":-39011942,"num_write":-413642,"num_write_kb":-151705339,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-8,"num_bytes_recovered":-16922624,"num_keys_recovered":-2,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.424817"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490644+0000","last_change":"2023-11-08T05:28:13.557071+0000","last_active":"2023-11-08T05:30:44.490644+0000","last_peered":"2023-11-08T05:30:44.490644+0000","last_clean":"2023-11-08T05:28:01.458439+0000","last_became_active":"2023-11-08T05:28:13.557071+0000","last_became_peered":"2023-11-08T05:28:13.557071+0000","last_unstale":"2023-11-08T05:30:44.490644+0000","last_undegraded":"2023-11-08T05:30:44.490644+0000","last_fullsized":"2023-11-08T05:28:13.547025+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":459,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:45.794760+0000","last_change":"2023-11-08T05:30:45.794760+0000","last_active":"2023-11-08T05:30:45.794760+0000","last_peered":"2023-11-08T05:30:45.794760+0000","last_clean":"2023-11-08T05:30:45.794760+0000","last_became_active":"2023-11-08T05:30:45.794398+0000","last_became_peered":"2023-11-08T05:30:45.794398+0000","last_unstale":"2023-11-08T05:30:45.794760+0000","last_undegraded":"2023-11-08T05:30:45.794760+0000","last_fullsized":"2023-11-08T05:30:45.794760+0000","mapping_epoch":458,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":459,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":465,"reported_epoch":459,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:45.795085+0000","last_change":"2023-11-08T05:30:45.795085+0000","last_active":"2023-11-08T05:30:45.795085+0000","last_peered":"2023-11-08T05:30:45.795085+0000","last_clean":"2023-11-08T05:30:45.795085+0000","last_became_active":"2023-11-08T05:30:45.794743+0000","last_became_peered":"2023-11-08T05:30:45.794743+0000","last_unstale":"2023-11-08T05:30:45.795085+0000","last_undegraded":"2023-11-08T05:30:45.795085+0000","last_fullsized":"2023-11-08T05:30:45.795085+0000","mapping_epoch":458,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":459,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":454,"reported_epoch":458,"state":"active+clean","last_fresh":"2023-11-08T05:30:44.490618+0000","last_change":"2023-11-08T04:52:47.548597+0000","last_active":"2023-11-08T05:30:44.490618+0000","last_peered":"2023-11-08T05:30:44.490618+0000","last_clean":"2023-11-08T05:30:44.490618+0000","last_became_active":"2023-11-08T04:52:47.548294+0000","last_became_peered":"2023-11-08T04:52:47.548294+0000","last_unstale":"2023-11-08T05:30:44.490618+0000","last_undegraded":"2023-11-08T05:30:44.490618+0000","last_fullsized":"2023-11-08T05:30:44.490618+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":460,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490523+0000","last_change":"2023-11-08T05:28:13.545468+0000","last_active":"2023-11-08T05:30:44.490523+0000","last_peered":"2023-11-08T05:30:44.490523+0000","last_clean":"2023-11-08T05:28:01.456723+0000","last_became_active":"2023-11-08T05:28:13.545468+0000","last_became_peered":"2023-11-08T05:28:13.545468+0000","last_unstale":"2023-11-08T05:30:44.490523+0000","last_undegraded":"2023-11-08T05:30:44.490523+0000","last_fullsized":"2023-11-08T05:28:13.537207+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:45.528796+0000","last_clean_scrub_stamp":"2023-11-08T04:52:45.528796+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":465,"reported_epoch":458,"state":"active+clean+remapped","last_fresh":"2023-11-08T05:30:44.491907+0000","last_change":"2023-11-08T05:29:44.395350+0000","last_active":"2023-11-08T05:30:44.491907+0000","last_peered":"2023-11-08T05:30:44.491907+0000","last_clean":"2023-11-08T05:30:44.491907+0000","last_became_active":"2023-11-08T05:29:44.395012+0000","last_became_peered":"2023-11-08T05:29:44.395012+0000","last_unstale":"2023-11-08T05:30:44.491907+0000","last_undegraded":"2023-11-08T05:30:44.491907+0000","last_fullsized":"2023-11-08T05:30:44.491907+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490629+0000","last_change":"2023-11-08T05:28:13.563357+0000","last_active":"2023-11-08T05:30:44.490629+0000","last_peered":"2023-11-08T05:30:44.490629+0000","last_clean":"2023-11-08T05:28:01.457691+0000","last_became_active":"2023-11-08T05:28:13.563357+0000","last_became_peered":"2023-11-08T05:28:13.563357+0000","last_unstale":"2023-11-08T05:30:44.490629+0000","last_undegraded":"2023-11-08T05:30:44.490629+0000","last_fullsized":"2023-11-08T05:28:13.541614+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490691+0000","last_change":"2023-11-08T05:28:13.550672+0000","last_active":"2023-11-08T05:30:44.490691+0000","last_peered":"2023-11-08T05:30:44.490691+0000","last_clean":"2023-11-08T05:28:01.457904+0000","last_became_active":"2023-11-08T05:28:13.550672+0000","last_became_peered":"2023-11-08T05:28:13.550672+0000","last_unstale":"2023-11-08T05:30:44.490691+0000","last_undegraded":"2023-11-08T05:30:44.490691+0000","last_fullsized":"2023-11-08T05:28:13.539595+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":459,"reported_epoch":458,"state":"active+undersized","last_fresh":"2023-11-08T05:30:44.490805+0000","last_change":"2023-11-08T05:28:13.557968+0000","last_active":"2023-11-08T05:30:44.490805+0000","last_peered":"2023-11-08T05:30:44.490805+0000","last_clean":"2023-11-08T05:28:01.458433+0000","last_became_active":"2023-11-08T05:28:13.557968+0000","last_became_peered":"2023-11-08T05:28:13.557968+0000","last_unstale":"2023-11-08T05:30:44.490805+0000","last_undegraded":"2023-11-08T05:30:44.490805+0000","last_fullsized":"2023-11-08T05:28:13.542832+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:46.531964+0000","last_clean_scrub_stamp":"2023-11-08T04:52:46.531964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":1,"acting":1,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":12,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738858,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":645040,"kb_used_data":5352,"kb_used_omap":275,"kb_used_meta":639404,"kb_avail":93726800,"statfs":{"total":96636764160,"available":95976243200,"internally_reserved":0,"allocated":5480448,"data_stored":2697647,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":281650,"internal_metadata":654750670},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:59 2023","interfaces":[{"interface":"back","average":{"1min":0.30499999999999999,"5min":0.439,"15min":0.36099999999999999},"min":{"1min":0.17399999999999999,"5min":0.14499999999999999,"15min":0.14499999999999999},"max":{"1min":0.60099999999999998,"5min":4.9379999999999997,"15min":4.9379999999999997},"last":0.379},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.49399999999999999,"15min":0.42399999999999999},"min":{"1min":0.221,"5min":0.19600000000000001,"15min":0.17000000000000001},"max":{"1min":0.63200000000000001,"5min":4.9859999999999998,"15min":4.9859999999999998},"last":0.40899999999999997}]},{"osd":1,"last update":"Wed Nov 8 05:33:07 2023","interfaces":[{"interface":"back","average":{"1min":0.26200000000000001,"5min":0.27900000000000003,"15min":0.46500000000000002},"min":{"1min":0.187,"5min":0.16,"15min":0.153},"max":{"1min":0.372,"5min":0.68200000000000005,"15min":23.242999999999999},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.28100000000000003,"5min":0.32400000000000001,"15min":0.45600000000000002},"min":{"1min":0.20300000000000001,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":0.39600000000000002,"5min":2.0419999999999998,"15min":23.251000000000001},"last":0.441}]}]},{"osd":0,"up_from":8,"seq":34359738798,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1339228,"kb_used_data":398304,"kb_used_omap":291,"kb_used_meta":936476,"kb_avail":93032612,"statfs":{"total":96636764160,"available":95265394688,"internally_reserved":0,"allocated":407863296,"data_stored":405103900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":298462,"internal_metadata":958951970},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.436,"15min":0.36299999999999999},"min":{"1min":0.19700000000000001,"5min":0.14699999999999999,"15min":0.13700000000000001},"max":{"1min":3.222,"5min":6.5499999999999998,"15min":6.9989999999999997},"last":0.245},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.46300000000000002,"15min":0.42299999999999999},"min":{"1min":0.17799999999999999,"5min":0.17499999999999999,"15min":0.16300000000000001},"max":{"1min":5.0030000000000001,"5min":6.0629999999999997,"15min":9.6869999999999994},"last":0.28599999999999998}]},{"osd":2,"last update":"Wed Nov 8 05:27:45 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.51600000000000001,"15min":0.47099999999999997},"min":{"1min":0.21099999999999999,"5min":0.153,"15min":0.153},"max":{"1min":5.7439999999999998,"5min":5.8280000000000003,"15min":9.7639999999999993},"last":0.252},{"interface":"front","average":{"1min":0.497,"5min":0.56200000000000006,"15min":0.42299999999999999},"min":{"1min":0.20399999999999999,"5min":0.187,"15min":0.16800000000000001},"max":{"1min":3.9319999999999999,"5min":8.8040000000000003,"15min":8.8040000000000003},"last":0.29899999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738859,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":650100,"kb_used_data":5356,"kb_used_omap":214,"kb_used_meta":644521,"kb_avail":93721740,"statfs":{"total":96636764160,"available":95971061760,"internally_reserved":0,"allocated":5484544,"data_stored":2698616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":219335,"internal_metadata":659990329},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:27:56 2023","interfaces":[{"interface":"back","average":{"1min":0.30399999999999999,"5min":0.34699999999999998,"15min":0.36499999999999999},"min":{"1min":0.23999999999999999,"5min":0.14199999999999999,"15min":0.14199999999999999},"max":{"1min":0.42399999999999999,"5min":1.595,"15min":7.4249999999999998},"last":0.17100000000000001},{"interface":"front","average":{"1min":0.35499999999999998,"5min":0.40899999999999997,"15min":0.376},"min":{"1min":0.23999999999999999,"5min":0.19600000000000001,"15min":0.183},"max":{"1min":0.46999999999999997,"5min":2.6859999999999999,"15min":2.6859999999999999},"last":0.23000000000000001}]},{"osd":2,"last update":"Wed Nov 8 05:33:03 2023","interfaces":[{"interface":"back","average":{"1min":0.26000000000000001,"5min":0.317,"15min":0.35999999999999999},"min":{"1min":0.18099999999999999,"5min":0.161,"15min":0.161},"max":{"1min":0.40799999999999997,"5min":2.2650000000000001,"15min":5.8209999999999997},"last":0.19400000000000001},{"interface":"front","average":{"1min":0.26900000000000002,"5min":0.28699999999999998,"15min":0.35199999999999998},"min":{"1min":0.192,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.41999999999999998,"5min":1.137,"15min":4.2350000000000003},"last":0.20200000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-08T05:33:20.676 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~303s 2023-11-08T05:33:20.677 INFO:tasks.daemonwatchdog.daemon_watchdog:BARK! unmounting mounts and killing all daemons 2023-11-08T05:33:21.080 INFO:tasks.ceph.cluster2.osd.1:Sent signal 15 2023-11-08T05:33:21.080 INFO:tasks.ceph.cluster2.osd.2:Sent signal 15 2023-11-08T05:33:21.081 INFO:tasks.ceph.cluster2.mon.a:Sent signal 15 2023-11-08T05:33:21.081 INFO:tasks.ceph.cluster2.mgr.x:Sent signal 15 2023-11-08T05:33:21.082 INFO:tasks.ceph.cluster2.osd.1.smithi087.stderr:2023-11-08T05:33:21.079+0000 7fbb8964e700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster cluster2 -i 1 (PID: 116177) UID: 0 2023-11-08T05:33:21.082 INFO:tasks.ceph.cluster2.osd.1.smithi087.stderr:2023-11-08T05:33:21.079+0000 7fbb8964e700 -1 osd.1 459 *** Got signal Terminated *** 2023-11-08T05:33:21.082 INFO:tasks.ceph.cluster2.osd.1.smithi087.stderr:2023-11-08T05:33:21.079+0000 7fbb8964e700 -1 osd.1 459 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-11-08T05:33:21.083 INFO:tasks.ceph.cluster2.osd.2.smithi087.stderr:2023-11-08T05:33:21.079+0000 7fa252469700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster cluster2 -i 2 (PID: 116180) UID: 0 2023-11-08T05:33:21.083 INFO:tasks.ceph.cluster2.osd.2.smithi087.stderr:2023-11-08T05:33:21.079+0000 7fa252469700 -1 osd.2 459 *** Got signal Terminated *** 2023-11-08T05:33:21.083 INFO:tasks.ceph.cluster2.osd.2.smithi087.stderr:2023-11-08T05:33:21.079+0000 7fa252469700 -1 osd.2 459 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-11-08T05:33:21.084 INFO:tasks.ceph.cluster2.mon.a.smithi087.stderr:2023-11-08T05:33:21.080+0000 7f8785fcd700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-mon -f --cluster cluster2 -i a (PID: 115804) UID: 0 2023-11-08T05:33:21.084 INFO:tasks.ceph.cluster2.mon.a.smithi087.stderr:2023-11-08T05:33:21.080+0000 7f8785fcd700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2023-11-08T05:33:21.085 INFO:tasks.ceph.cluster2.mgr.x.smithi077.stderr:2023-11-08T05:33:21.082+0000 7f171da6d700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-mgr -f --cluster cluster2 -i x (PID: 119112) UID: 0 2023-11-08T05:33:21.085 INFO:tasks.ceph.cluster2.mgr.x.smithi077.stderr:2023-11-08T05:33:21.082+0000 7f171da6d700 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2023-11-08T05:33:23.657 DEBUG:teuthology.orchestra.run.smithi087:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-11-08T05:35:23.695 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-11-08T05:35:23.696 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_cfb6fd52113be6a0af3fe7fb89b2e08474eadc0f/qa/tasks/ceph.py", line 1916, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_cfb6fd52113be6a0af3fe7fb89b2e08474eadc0f/qa/tasks/ceph_manager.py", line 2758, in wait_for_clean while not self.is_clean(): File "/home/teuthworker/src/github.com_ceph_ceph-c_cfb6fd52113be6a0af3fe7fb89b2e08474eadc0f/qa/tasks/ceph_manager.py", line 2694, in is_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_ceph_ceph-c_cfb6fd52113be6a0af3fe7fb89b2e08474eadc0f/qa/tasks/ceph_manager.py", line 2355, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph-c_cfb6fd52113be6a0af3fe7fb89b2e08474eadc0f/qa/tasks/ceph_manager.py", line 1616, in raw_cluster_cmd p = self.run_cluster_cmd(args=args, stdout=stdout, **kwargs) File "/home/teuthworker/src/github.com_ceph_ceph-c_cfb6fd52113be6a0af3fe7fb89b2e08474eadc0f/qa/tasks/ceph_manager.py", line 1609, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi087 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json' 2023-11-08T05:35:23.697 INFO:teuthology.misc:Shutting down mds daemons... 2023-11-08T05:35:23.697 INFO:teuthology.misc:Shutting down osd daemons... 2023-11-08T05:35:23.698 DEBUG:tasks.ceph.cluster2.osd.0:waiting for process to exit 2023-11-08T05:35:23.698 INFO:teuthology.orchestra.run:waiting for 300 2023-11-08T05:35:23.699 DEBUG:teuthology.orchestra.run:got remote process result: 5 2023-11-08T05:35:23.699 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_cfb6fd52113be6a0af3fe7fb89b2e08474eadc0f/qa/tasks/ceph.py", line 1451, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_cfb6fd52113be6a0af3fe7fb89b2e08474eadc0f/qa/tasks/ceph.py", line 1916, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_cfb6fd52113be6a0af3fe7fb89b2e08474eadc0f/qa/tasks/ceph_manager.py", line 2758, in wait_for_clean while not self.is_clean(): File "/home/teuthworker/src/github.com_ceph_ceph-c_cfb6fd52113be6a0af3fe7fb89b2e08474eadc0f/qa/tasks/ceph_manager.py", line 2694, in is_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_ceph_ceph-c_cfb6fd52113be6a0af3fe7fb89b2e08474eadc0f/qa/tasks/ceph_manager.py", line 2355, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph-c_cfb6fd52113be6a0af3fe7fb89b2e08474eadc0f/qa/tasks/ceph_manager.py", line 1616, in raw_cluster_cmd p = self.run_cluster_cmd(args=args, stdout=stdout, **kwargs) File "/home/teuthworker/src/github.com_ceph_ceph-c_cfb6fd52113be6a0af3fe7fb89b2e08474eadc0f/qa/tasks/ceph_manager.py", line 1609, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi087 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi087 with status 5: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster2 -i 0' 2023-11-08T05:35:23.700 INFO:tasks.ceph.cluster2.osd.0:Stopped 2023-11-08T05:35:23.700 DEBUG:tasks.ceph.cluster2.osd.1:waiting for process to exit 2023-11-08T05:35:23.700 INFO:teuthology.orchestra.run:waiting for 300 2023-11-08T05:35:23.701 INFO:tasks.ceph.cluster2.osd.1:Stopped 2023-11-08T05:35:23.701 DEBUG:tasks.ceph.cluster2.osd.2:waiting for process to exit 2023-11-08T05:35:23.702 INFO:teuthology.orchestra.run:waiting for 300 2023-11-08T05:35:23.702 INFO:tasks.ceph.cluster2.osd.2:Stopped 2023-11-08T05:35:23.703 INFO:teuthology.misc:Shutting down mgr daemons... 2023-11-08T05:35:23.703 DEBUG:tasks.ceph.cluster2.mgr.x:waiting for process to exit 2023-11-08T05:35:23.703 INFO:teuthology.orchestra.run:waiting for 300 2023-11-08T05:35:23.704 INFO:tasks.ceph.cluster2.mgr.x:Stopped 2023-11-08T05:35:23.704 INFO:teuthology.misc:Shutting down mon daemons... 2023-11-08T05:35:23.705 DEBUG:tasks.ceph.cluster2.mon.a:waiting for process to exit 2023-11-08T05:35:23.705 INFO:teuthology.orchestra.run:waiting for 300 2023-11-08T05:35:23.706 INFO:tasks.ceph.cluster2.mon.a:Stopped 2023-11-08T05:35:23.706 INFO:tasks.ceph:Checking cluster log for badness... 2023-11-08T05:35:23.707 DEBUG:teuthology.orchestra.run.smithi087:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/cluster2.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | head -n 1 2023-11-08T05:35:23.741 INFO:teuthology.orchestra.run.smithi087.stdout:2023-11-08T05:28:12.529768+0000 mon.a (mon.0) 1566 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN) 2023-11-08T05:35:23.742 WARNING:tasks.ceph:Found errors (ERR|WRN|SEC) in cluster log 2023-11-08T05:35:23.742 DEBUG:teuthology.orchestra.run.smithi087:> sudo egrep '\[SEC\]' /var/log/ceph/cluster2.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | head -n 1 2023-11-08T05:35:23.813 DEBUG:teuthology.orchestra.run.smithi087:> sudo egrep '\[ERR\]' /var/log/ceph/cluster2.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | head -n 1 2023-11-08T05:35:23.885 DEBUG:teuthology.orchestra.run.smithi087:> sudo egrep '\[WRN\]' /var/log/ceph/cluster2.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | head -n 1 2023-11-08T05:35:23.955 INFO:teuthology.orchestra.run.smithi087.stdout:2023-11-08T05:28:12.529768+0000 mon.a (mon.0) 1566 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN) 2023-11-08T05:35:23.956 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster2-0 on ubuntu@smithi087.front.sepia.ceph.com 2023-11-08T05:35:23.956 DEBUG:teuthology.orchestra.run.smithi087:> sync && sudo umount -f /var/lib/ceph/osd/cluster2-0 2023-11-08T05:35:24.194 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster2-1 on ubuntu@smithi087.front.sepia.ceph.com 2023-11-08T05:35:24.194 DEBUG:teuthology.orchestra.run.smithi087:> sync && sudo umount -f /var/lib/ceph/osd/cluster2-1 2023-11-08T05:35:24.373 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster2-2 on ubuntu@smithi087.front.sepia.ceph.com 2023-11-08T05:35:24.373 DEBUG:teuthology.orchestra.run.smithi087:> sync && sudo umount -f /var/lib/ceph/osd/cluster2-2 2023-11-08T05:35:24.553 INFO:tasks.ceph:Archiving mon data... 2023-11-08T05:35:24.554 DEBUG:teuthology.misc:Transferring archived files from smithi087:/var/lib/ceph/mon/cluster2-a to /home/teuthworker/archive/yuriw-2023-11-07_19:42:25-rbd-wip-yuri3-testing-2023-11-07-0801-pacific-distro-default-smithi/7451368/data/cluster2.mon.a.tgz 2023-11-08T05:35:24.555 DEBUG:teuthology.orchestra.run.smithi087:> mktemp 2023-11-08T05:35:24.573 INFO:teuthology.orchestra.run.smithi087.stdout:/tmp/tmp.q8SjFZY0Wr 2023-11-08T05:35:24.574 DEBUG:teuthology.orchestra.run.smithi087:> sudo tar cz -f - -C /var/lib/ceph/mon/cluster2-a -- . > /tmp/tmp.q8SjFZY0Wr 2023-11-08T05:35:24.935 DEBUG:teuthology.orchestra.run.smithi087:> sudo chmod 0666 /tmp/tmp.q8SjFZY0Wr 2023-11-08T05:35:25.022 DEBUG:teuthology.orchestra.remote:smithi087:/tmp/tmp.q8SjFZY0Wr is 1MB 2023-11-08T05:35:25.133 DEBUG:teuthology.orchestra.run.smithi087:> rm -fr /tmp/tmp.q8SjFZY0Wr 2023-11-08T05:35:25.151 INFO:tasks.ceph:Cleaning ceph cluster... 2023-11-08T05:35:25.152 DEBUG:teuthology.orchestra.run.smithi077:> sudo rm -rf -- /etc/ceph/cluster2.conf /etc/ceph/cluster2.keyring /home/ubuntu/cephtest/cluster2.data /home/ubuntu/cephtest/cluster2.monmap /home/ubuntu/cephtest/../*.pid 2023-11-08T05:35:25.155 DEBUG:teuthology.orchestra.run.smithi087:> sudo rm -rf -- /etc/ceph/cluster2.conf /etc/ceph/cluster2.keyring /home/ubuntu/cephtest/cluster2.data /home/ubuntu/cephtest/cluster2.monmap /home/ubuntu/cephtest/../*.pid 2023-11-08T05:35:25.217 ERROR:teuthology.run_tasks:Manager failed: ceph Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_cfb6fd52113be6a0af3fe7fb89b2e08474eadc0f/qa/tasks/ceph.py", line 1916, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_cfb6fd52113be6a0af3fe7fb89b2e08474eadc0f/qa/tasks/ceph_manager.py", line 2758, in wait_for_clean while not self.is_clean(): File "/home/teuthworker/src/github.com_ceph_ceph-c_cfb6fd52113be6a0af3fe7fb89b2e08474eadc0f/qa/tasks/ceph_manager.py", line 2694, in is_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_ceph_ceph-c_cfb6fd52113be6a0af3fe7fb89b2e08474eadc0f/qa/tasks/ceph_manager.py", line 2355, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph-c_cfb6fd52113be6a0af3fe7fb89b2e08474eadc0f/qa/tasks/ceph_manager.py", line 1616, in raw_cluster_cmd p = self.run_cluster_cmd(args=args, stdout=stdout, **kwargs) File "/home/teuthworker/src/github.com_ceph_ceph-c_cfb6fd52113be6a0af3fe7fb89b2e08474eadc0f/qa/tasks/ceph_manager.py", line 1609, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi087 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json' 2023-11-08T05:35:25.217 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2023-11-08T05:35:25.230 INFO:tasks.ceph.ceph_manager.cluster1:Canceling any pending splits or merges... 2023-11-08T05:35:25.230 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd dump --format=json 2023-11-08T05:35:25.524 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-11-08T05:35:25.524 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":457,"fsid":"1244e29e-ccb6-44ee-9f2f-7bbf2125940c","created":"2023-11-08T04:52:07.196830+0000","modified":"2023-11-08T05:29:29.654328+0000","last_up_change":"2023-11-08T04:52:15.170916+0000","last_in_change":"2023-11-08T04:52:08.120187+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2023-11-08T04:52:15.210778+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2023-11-08T04:52:16.003312+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"ce6698f1-7ce9-42d0-9385-7c2e92eb500a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":115421},{"type":"v1","addr":"172.21.15.77:6817","nonce":115421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":115421},{"type":"v1","addr":"172.21.15.77:6819","nonce":115421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":115421},{"type":"v1","addr":"172.21.15.77:6823","nonce":115421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":115421},{"type":"v1","addr":"172.21.15.77:6821","nonce":115421}]},"public_addr":"172.21.15.77:6817/115421","cluster_addr":"172.21.15.77:6819/115421","heartbeat_back_addr":"172.21.15.77:6823/115421","heartbeat_front_addr":"172.21.15.77:6821/115421","state":["exists","up"]},{"osd":1,"uuid":"84aa8223-3bd2-440d-ba7b-5f1675e4d25d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":115411},{"type":"v1","addr":"172.21.15.77:6809","nonce":115411}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":115411},{"type":"v1","addr":"172.21.15.77:6811","nonce":115411}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":115411},{"type":"v1","addr":"172.21.15.77:6815","nonce":115411}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":115411},{"type":"v1","addr":"172.21.15.77:6813","nonce":115411}]},"public_addr":"172.21.15.77:6809/115411","cluster_addr":"172.21.15.77:6811/115411","heartbeat_back_addr":"172.21.15.77:6815/115411","heartbeat_front_addr":"172.21.15.77:6813/115411","state":["exists","up"]},{"osd":2,"uuid":"0f508d08-7d10-451b-9213-4f36d0c2e9f1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":115412},{"type":"v1","addr":"172.21.15.77:6801","nonce":115412}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":115412},{"type":"v1","addr":"172.21.15.77:6803","nonce":115412}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":115412},{"type":"v1","addr":"172.21.15.77:6807","nonce":115412}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":115412},{"type":"v1","addr":"172.21.15.77:6805","nonce":115412}]},"public_addr":"172.21.15.77:6801/115412","cluster_addr":"172.21.15.77:6803/115412","heartbeat_back_addr":"172.21.15.77:6807/115412","heartbeat_front_addr":"172.21.15.77:6805/115412","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T04:52:13.259518+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T04:52:13.235929+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T04:52:13.212169+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-08T05:35:25.537 INFO:tasks.ceph.ceph_manager.cluster1:waiting for clean 2023-11-08T05:35:25.538 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2023-11-08T05:35:25.823 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-11-08T05:35:25.825 INFO:teuthology.orchestra.run.smithi077.stderr:dumped all 2023-11-08T05:35:25.836 INFO:teuthology.orchestra.run.smithi077.stdout:{"pg_ready":true,"pg_map":{"version":1749,"stamp":"2023-11-08T05:35:24.657686+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":927360,"kb_used_data":11112,"kb_used_omap":0,"kb_used_meta":916224,"kb_avail":282188160,"statfs":{"total":289910292480,"available":288960675840,"internally_reserved":0,"allocated":11378688,"data_stored":6141966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":938213376},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-38,"num_objects":-34,"num_object_clones":0,"num_object_copies":-68,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-34,"num_whiteouts":0,"num_read":-92153,"num_read_kb":-28964504,"num_write":-17859,"num_write_kb":-11548647,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.067863"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":452,"reported_epoch":456,"state":"active+clean","last_fresh":"2023-11-08T05:29:27.658302+0000","last_change":"2023-11-08T04:52:17.195087+0000","last_active":"2023-11-08T05:29:27.658302+0000","last_peered":"2023-11-08T05:29:27.658302+0000","last_clean":"2023-11-08T05:29:27.658302+0000","last_became_active":"2023-11-08T04:52:17.194821+0000","last_became_peered":"2023-11-08T04:52:17.194821+0000","last_unstale":"2023-11-08T05:29:27.658302+0000","last_undegraded":"2023-11-08T05:29:27.658302+0000","last_fullsized":"2023-11-08T05:29:27.658302+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:16.174854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:16.174854+0000","last_clean_scrub_stamp":"2023-11-08T04:52:16.174854+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":453,"reported_epoch":456,"state":"active+clean","last_fresh":"2023-11-08T05:29:27.910799+0000","last_change":"2023-11-08T04:52:17.194882+0000","last_active":"2023-11-08T05:29:27.910799+0000","last_peered":"2023-11-08T05:29:27.910799+0000","last_clean":"2023-11-08T05:29:27.910799+0000","last_became_active":"2023-11-08T04:52:17.194611+0000","last_became_peered":"2023-11-08T04:52:17.194611+0000","last_unstale":"2023-11-08T05:29:27.910799+0000","last_undegraded":"2023-11-08T05:29:27.910799+0000","last_fullsized":"2023-11-08T05:29:27.910799+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:16.174854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:16.174854+0000","last_clean_scrub_stamp":"2023-11-08T04:52:16.174854+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":455,"reported_epoch":456,"state":"active+clean","last_fresh":"2023-11-08T05:29:28.452318+0000","last_change":"2023-11-08T04:52:17.193579+0000","last_active":"2023-11-08T05:29:28.452318+0000","last_peered":"2023-11-08T05:29:28.452318+0000","last_clean":"2023-11-08T05:29:28.452318+0000","last_became_active":"2023-11-08T04:52:17.193313+0000","last_became_peered":"2023-11-08T04:52:17.193313+0000","last_unstale":"2023-11-08T05:29:28.452318+0000","last_undegraded":"2023-11-08T05:29:28.452318+0000","last_fullsized":"2023-11-08T05:29:28.452318+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:16.174854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:16.174854+0000","last_clean_scrub_stamp":"2023-11-08T04:52:16.174854+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":452,"reported_epoch":456,"state":"active+clean","last_fresh":"2023-11-08T05:29:27.653375+0000","last_change":"2023-11-08T04:52:17.193960+0000","last_active":"2023-11-08T05:29:27.653375+0000","last_peered":"2023-11-08T05:29:27.653375+0000","last_clean":"2023-11-08T05:29:27.653375+0000","last_became_active":"2023-11-08T04:52:17.193661+0000","last_became_peered":"2023-11-08T04:52:17.193661+0000","last_unstale":"2023-11-08T05:29:27.653375+0000","last_undegraded":"2023-11-08T05:29:27.653375+0000","last_fullsized":"2023-11-08T05:29:27.653375+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:16.174854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:16.174854+0000","last_clean_scrub_stamp":"2023-11-08T04:52:16.174854+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":452,"reported_epoch":456,"state":"active+clean","last_fresh":"2023-11-08T05:29:27.652411+0000","last_change":"2023-11-08T04:52:17.195090+0000","last_active":"2023-11-08T05:29:27.652411+0000","last_peered":"2023-11-08T05:29:27.652411+0000","last_clean":"2023-11-08T05:29:27.652411+0000","last_became_active":"2023-11-08T04:52:17.194795+0000","last_became_peered":"2023-11-08T04:52:17.194795+0000","last_unstale":"2023-11-08T05:29:27.652411+0000","last_undegraded":"2023-11-08T05:29:27.652411+0000","last_fullsized":"2023-11-08T05:29:27.652411+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:16.174854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:16.174854+0000","last_clean_scrub_stamp":"2023-11-08T04:52:16.174854+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":455,"reported_epoch":456,"state":"active+clean","last_fresh":"2023-11-08T05:29:28.451762+0000","last_change":"2023-11-08T04:52:17.193152+0000","last_active":"2023-11-08T05:29:28.451762+0000","last_peered":"2023-11-08T05:29:28.451762+0000","last_clean":"2023-11-08T05:29:28.451762+0000","last_became_active":"2023-11-08T04:52:17.192859+0000","last_became_peered":"2023-11-08T04:52:17.192859+0000","last_unstale":"2023-11-08T05:29:28.451762+0000","last_undegraded":"2023-11-08T05:29:28.451762+0000","last_fullsized":"2023-11-08T05:29:28.451762+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:16.174854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:16.174854+0000","last_clean_scrub_stamp":"2023-11-08T04:52:16.174854+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":452,"reported_epoch":456,"state":"active+clean","last_fresh":"2023-11-08T05:29:27.658113+0000","last_change":"2023-11-08T04:52:17.195685+0000","last_active":"2023-11-08T05:29:27.658113+0000","last_peered":"2023-11-08T05:29:27.658113+0000","last_clean":"2023-11-08T05:29:27.658113+0000","last_became_active":"2023-11-08T04:52:17.195400+0000","last_became_peered":"2023-11-08T04:52:17.195400+0000","last_unstale":"2023-11-08T05:29:27.658113+0000","last_undegraded":"2023-11-08T05:29:27.658113+0000","last_fullsized":"2023-11-08T05:29:27.658113+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:16.174854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:16.174854+0000","last_clean_scrub_stamp":"2023-11-08T04:52:16.174854+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":452,"reported_epoch":456,"state":"active+clean","last_fresh":"2023-11-08T05:29:27.654486+0000","last_change":"2023-11-08T04:52:17.196842+0000","last_active":"2023-11-08T05:29:27.654486+0000","last_peered":"2023-11-08T05:29:27.654486+0000","last_clean":"2023-11-08T05:29:27.654486+0000","last_became_active":"2023-11-08T04:52:17.196558+0000","last_became_peered":"2023-11-08T04:52:17.196558+0000","last_unstale":"2023-11-08T05:29:27.654486+0000","last_undegraded":"2023-11-08T05:29:27.654486+0000","last_fullsized":"2023-11-08T05:29:27.654486+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:16.174854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:16.174854+0000","last_clean_scrub_stamp":"2023-11-08T04:52:16.174854+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":452,"reported_epoch":456,"state":"active+clean","last_fresh":"2023-11-08T05:29:27.658763+0000","last_change":"2023-11-08T04:52:17.196926+0000","last_active":"2023-11-08T05:29:27.658763+0000","last_peered":"2023-11-08T05:29:27.658763+0000","last_clean":"2023-11-08T05:29:27.658763+0000","last_became_active":"2023-11-08T04:52:17.196648+0000","last_became_peered":"2023-11-08T04:52:17.196648+0000","last_unstale":"2023-11-08T05:29:27.658763+0000","last_undegraded":"2023-11-08T05:29:27.658763+0000","last_fullsized":"2023-11-08T05:29:27.658763+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:16.174854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:16.174854+0000","last_clean_scrub_stamp":"2023-11-08T04:52:16.174854+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738887,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":305024,"kb_used_data":3704,"kb_used_omap":0,"kb_used_meta":301312,"kb_avail":94066816,"statfs":{"total":96636764160,"available":96324419584,"internally_reserved":0,"allocated":3792896,"data_stored":2047322,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308543488},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:34:50 2023","interfaces":[{"interface":"back","average":{"1min":0.317,"5min":0.317,"15min":0.311},"min":{"1min":0.248,"5min":0.218,"15min":0.183},"max":{"1min":0.42399999999999999,"5min":0.48599999999999999,"15min":1.6859999999999999},"last":0.30599999999999999},{"interface":"front","average":{"1min":0.39300000000000002,"5min":0.38600000000000001,"15min":0.376},"min":{"1min":0.316,"5min":0.28799999999999998,"15min":0.217},"max":{"1min":0.51600000000000001,"5min":0.53100000000000003,"15min":1.714},"last":0.39500000000000002}]},{"osd":1,"last update":"Wed Nov 8 05:34:50 2023","interfaces":[{"interface":"back","average":{"1min":0.33400000000000002,"5min":0.33100000000000002,"15min":0.31900000000000001},"min":{"1min":0.23599999999999999,"5min":0.22800000000000001,"15min":0.217},"max":{"1min":0.48899999999999999,"5min":0.51100000000000001,"15min":0.51200000000000001},"last":0.41099999999999998},{"interface":"front","average":{"1min":0.35599999999999998,"5min":0.34799999999999998,"15min":0.34000000000000002},"min":{"1min":0.248,"5min":0.23200000000000001,"15min":0.21299999999999999},"max":{"1min":0.47899999999999998,"5min":0.51600000000000001,"15min":1.6379999999999999},"last":0.35999999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738887,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":309120,"kb_used_data":3704,"kb_used_omap":0,"kb_used_meta":305408,"kb_avail":94062720,"statfs":{"total":96636764160,"available":96320225280,"internally_reserved":0,"allocated":3792896,"data_stored":2047322,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312737792},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Nov 8 05:34:28 2023","interfaces":[{"interface":"back","average":{"1min":0.314,"5min":0.32400000000000001,"15min":0.32300000000000001},"min":{"1min":0.22600000000000001,"5min":0.22,"15min":0.153},"max":{"1min":0.47199999999999998,"5min":0.53200000000000003,"15min":2.222},"last":0.219},{"interface":"front","average":{"1min":0.35499999999999998,"5min":0.36799999999999999,"15min":0.36399999999999999},"min":{"1min":0.29399999999999998,"5min":0.27300000000000002,"15min":0.20899999999999999},"max":{"1min":0.49199999999999999,"5min":0.54000000000000004,"15min":2.1459999999999999},"last":0.309}]},{"osd":2,"last update":"Wed Nov 8 05:34:28 2023","interfaces":[{"interface":"back","average":{"1min":0.32400000000000001,"5min":0.32900000000000001,"15min":0.32400000000000001},"min":{"1min":0.25800000000000001,"5min":0.23200000000000001,"15min":0.16200000000000001},"max":{"1min":0.45600000000000002,"5min":0.49099999999999999,"15min":0.59199999999999997},"last":0.255},{"interface":"front","average":{"1min":0.38100000000000001,"5min":0.35199999999999998,"15min":0.33500000000000002},"min":{"1min":0.312,"5min":0.255,"15min":0.184},"max":{"1min":0.499,"5min":0.499,"15min":0.86799999999999999},"last":0.32400000000000001}]}]},{"osd":1,"up_from":8,"seq":34359738890,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":313216,"kb_used_data":3704,"kb_used_omap":0,"kb_used_meta":309504,"kb_avail":94058624,"statfs":{"total":96636764160,"available":96316030976,"internally_reserved":0,"allocated":3792896,"data_stored":2047322,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316932096},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:34:43 2023","interfaces":[{"interface":"back","average":{"1min":0.32300000000000001,"5min":0.34599999999999997,"15min":0.32600000000000001},"min":{"1min":0.22800000000000001,"5min":0.21299999999999999,"15min":0.159},"max":{"1min":0.43099999999999999,"5min":0.53200000000000003,"15min":0.53200000000000003},"last":0.32600000000000001},{"interface":"front","average":{"1min":0.36699999999999999,"5min":0.378,"15min":0.36199999999999999},"min":{"1min":0.28799999999999998,"5min":0.248,"15min":0.16600000000000001},"max":{"1min":0.48299999999999998,"5min":0.52000000000000002,"15min":0.92000000000000004},"last":0.23300000000000001}]},{"osd":2,"last update":"Wed Nov 8 05:34:43 2023","interfaces":[{"interface":"back","average":{"1min":0.32600000000000001,"5min":0.34100000000000003,"15min":0.32900000000000001},"min":{"1min":0.223,"5min":0.223,"15min":0.17799999999999999},"max":{"1min":0.44500000000000001,"5min":0.48599999999999999,"15min":1.127},"last":0.249},{"interface":"front","average":{"1min":0.33500000000000002,"5min":0.34999999999999998,"15min":0.33800000000000002},"min":{"1min":0.25,"5min":0.25,"15min":0.17100000000000001},"max":{"1min":0.47699999999999998,"5min":0.52500000000000002,"15min":1.6299999999999999},"last":0.28000000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-08T05:35:25.837 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2023-11-08T05:35:26.119 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-11-08T05:35:26.120 INFO:teuthology.orchestra.run.smithi077.stderr:dumped all 2023-11-08T05:35:26.131 INFO:teuthology.orchestra.run.smithi077.stdout:{"pg_ready":true,"pg_map":{"version":1749,"stamp":"2023-11-08T05:35:24.657686+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":927360,"kb_used_data":11112,"kb_used_omap":0,"kb_used_meta":916224,"kb_avail":282188160,"statfs":{"total":289910292480,"available":288960675840,"internally_reserved":0,"allocated":11378688,"data_stored":6141966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":938213376},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-38,"num_objects":-34,"num_object_clones":0,"num_object_copies":-68,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-34,"num_whiteouts":0,"num_read":-92153,"num_read_kb":-28964504,"num_write":-17859,"num_write_kb":-11548647,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.067863"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":452,"reported_epoch":456,"state":"active+clean","last_fresh":"2023-11-08T05:29:27.658302+0000","last_change":"2023-11-08T04:52:17.195087+0000","last_active":"2023-11-08T05:29:27.658302+0000","last_peered":"2023-11-08T05:29:27.658302+0000","last_clean":"2023-11-08T05:29:27.658302+0000","last_became_active":"2023-11-08T04:52:17.194821+0000","last_became_peered":"2023-11-08T04:52:17.194821+0000","last_unstale":"2023-11-08T05:29:27.658302+0000","last_undegraded":"2023-11-08T05:29:27.658302+0000","last_fullsized":"2023-11-08T05:29:27.658302+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:16.174854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:16.174854+0000","last_clean_scrub_stamp":"2023-11-08T04:52:16.174854+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":453,"reported_epoch":456,"state":"active+clean","last_fresh":"2023-11-08T05:29:27.910799+0000","last_change":"2023-11-08T04:52:17.194882+0000","last_active":"2023-11-08T05:29:27.910799+0000","last_peered":"2023-11-08T05:29:27.910799+0000","last_clean":"2023-11-08T05:29:27.910799+0000","last_became_active":"2023-11-08T04:52:17.194611+0000","last_became_peered":"2023-11-08T04:52:17.194611+0000","last_unstale":"2023-11-08T05:29:27.910799+0000","last_undegraded":"2023-11-08T05:29:27.910799+0000","last_fullsized":"2023-11-08T05:29:27.910799+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:16.174854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:16.174854+0000","last_clean_scrub_stamp":"2023-11-08T04:52:16.174854+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":455,"reported_epoch":456,"state":"active+clean","last_fresh":"2023-11-08T05:29:28.452318+0000","last_change":"2023-11-08T04:52:17.193579+0000","last_active":"2023-11-08T05:29:28.452318+0000","last_peered":"2023-11-08T05:29:28.452318+0000","last_clean":"2023-11-08T05:29:28.452318+0000","last_became_active":"2023-11-08T04:52:17.193313+0000","last_became_peered":"2023-11-08T04:52:17.193313+0000","last_unstale":"2023-11-08T05:29:28.452318+0000","last_undegraded":"2023-11-08T05:29:28.452318+0000","last_fullsized":"2023-11-08T05:29:28.452318+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:16.174854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:16.174854+0000","last_clean_scrub_stamp":"2023-11-08T04:52:16.174854+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":452,"reported_epoch":456,"state":"active+clean","last_fresh":"2023-11-08T05:29:27.653375+0000","last_change":"2023-11-08T04:52:17.193960+0000","last_active":"2023-11-08T05:29:27.653375+0000","last_peered":"2023-11-08T05:29:27.653375+0000","last_clean":"2023-11-08T05:29:27.653375+0000","last_became_active":"2023-11-08T04:52:17.193661+0000","last_became_peered":"2023-11-08T04:52:17.193661+0000","last_unstale":"2023-11-08T05:29:27.653375+0000","last_undegraded":"2023-11-08T05:29:27.653375+0000","last_fullsized":"2023-11-08T05:29:27.653375+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:16.174854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:16.174854+0000","last_clean_scrub_stamp":"2023-11-08T04:52:16.174854+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":452,"reported_epoch":456,"state":"active+clean","last_fresh":"2023-11-08T05:29:27.652411+0000","last_change":"2023-11-08T04:52:17.195090+0000","last_active":"2023-11-08T05:29:27.652411+0000","last_peered":"2023-11-08T05:29:27.652411+0000","last_clean":"2023-11-08T05:29:27.652411+0000","last_became_active":"2023-11-08T04:52:17.194795+0000","last_became_peered":"2023-11-08T04:52:17.194795+0000","last_unstale":"2023-11-08T05:29:27.652411+0000","last_undegraded":"2023-11-08T05:29:27.652411+0000","last_fullsized":"2023-11-08T05:29:27.652411+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:16.174854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:16.174854+0000","last_clean_scrub_stamp":"2023-11-08T04:52:16.174854+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":455,"reported_epoch":456,"state":"active+clean","last_fresh":"2023-11-08T05:29:28.451762+0000","last_change":"2023-11-08T04:52:17.193152+0000","last_active":"2023-11-08T05:29:28.451762+0000","last_peered":"2023-11-08T05:29:28.451762+0000","last_clean":"2023-11-08T05:29:28.451762+0000","last_became_active":"2023-11-08T04:52:17.192859+0000","last_became_peered":"2023-11-08T04:52:17.192859+0000","last_unstale":"2023-11-08T05:29:28.451762+0000","last_undegraded":"2023-11-08T05:29:28.451762+0000","last_fullsized":"2023-11-08T05:29:28.451762+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:16.174854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:16.174854+0000","last_clean_scrub_stamp":"2023-11-08T04:52:16.174854+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":452,"reported_epoch":456,"state":"active+clean","last_fresh":"2023-11-08T05:29:27.658113+0000","last_change":"2023-11-08T04:52:17.195685+0000","last_active":"2023-11-08T05:29:27.658113+0000","last_peered":"2023-11-08T05:29:27.658113+0000","last_clean":"2023-11-08T05:29:27.658113+0000","last_became_active":"2023-11-08T04:52:17.195400+0000","last_became_peered":"2023-11-08T04:52:17.195400+0000","last_unstale":"2023-11-08T05:29:27.658113+0000","last_undegraded":"2023-11-08T05:29:27.658113+0000","last_fullsized":"2023-11-08T05:29:27.658113+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:16.174854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:16.174854+0000","last_clean_scrub_stamp":"2023-11-08T04:52:16.174854+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":452,"reported_epoch":456,"state":"active+clean","last_fresh":"2023-11-08T05:29:27.654486+0000","last_change":"2023-11-08T04:52:17.196842+0000","last_active":"2023-11-08T05:29:27.654486+0000","last_peered":"2023-11-08T05:29:27.654486+0000","last_clean":"2023-11-08T05:29:27.654486+0000","last_became_active":"2023-11-08T04:52:17.196558+0000","last_became_peered":"2023-11-08T04:52:17.196558+0000","last_unstale":"2023-11-08T05:29:27.654486+0000","last_undegraded":"2023-11-08T05:29:27.654486+0000","last_fullsized":"2023-11-08T05:29:27.654486+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:16.174854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:16.174854+0000","last_clean_scrub_stamp":"2023-11-08T04:52:16.174854+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":452,"reported_epoch":456,"state":"active+clean","last_fresh":"2023-11-08T05:29:27.658763+0000","last_change":"2023-11-08T04:52:17.196926+0000","last_active":"2023-11-08T05:29:27.658763+0000","last_peered":"2023-11-08T05:29:27.658763+0000","last_clean":"2023-11-08T05:29:27.658763+0000","last_became_active":"2023-11-08T04:52:17.196648+0000","last_became_peered":"2023-11-08T04:52:17.196648+0000","last_unstale":"2023-11-08T05:29:27.658763+0000","last_undegraded":"2023-11-08T05:29:27.658763+0000","last_fullsized":"2023-11-08T05:29:27.658763+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:16.174854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:16.174854+0000","last_clean_scrub_stamp":"2023-11-08T04:52:16.174854+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738887,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":305024,"kb_used_data":3704,"kb_used_omap":0,"kb_used_meta":301312,"kb_avail":94066816,"statfs":{"total":96636764160,"available":96324419584,"internally_reserved":0,"allocated":3792896,"data_stored":2047322,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308543488},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:34:50 2023","interfaces":[{"interface":"back","average":{"1min":0.317,"5min":0.317,"15min":0.311},"min":{"1min":0.248,"5min":0.218,"15min":0.183},"max":{"1min":0.42399999999999999,"5min":0.48599999999999999,"15min":1.6859999999999999},"last":0.30599999999999999},{"interface":"front","average":{"1min":0.39300000000000002,"5min":0.38600000000000001,"15min":0.376},"min":{"1min":0.316,"5min":0.28799999999999998,"15min":0.217},"max":{"1min":0.51600000000000001,"5min":0.53100000000000003,"15min":1.714},"last":0.39500000000000002}]},{"osd":1,"last update":"Wed Nov 8 05:34:50 2023","interfaces":[{"interface":"back","average":{"1min":0.33400000000000002,"5min":0.33100000000000002,"15min":0.31900000000000001},"min":{"1min":0.23599999999999999,"5min":0.22800000000000001,"15min":0.217},"max":{"1min":0.48899999999999999,"5min":0.51100000000000001,"15min":0.51200000000000001},"last":0.41099999999999998},{"interface":"front","average":{"1min":0.35599999999999998,"5min":0.34799999999999998,"15min":0.34000000000000002},"min":{"1min":0.248,"5min":0.23200000000000001,"15min":0.21299999999999999},"max":{"1min":0.47899999999999998,"5min":0.51600000000000001,"15min":1.6379999999999999},"last":0.35999999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738887,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":309120,"kb_used_data":3704,"kb_used_omap":0,"kb_used_meta":305408,"kb_avail":94062720,"statfs":{"total":96636764160,"available":96320225280,"internally_reserved":0,"allocated":3792896,"data_stored":2047322,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312737792},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Nov 8 05:34:28 2023","interfaces":[{"interface":"back","average":{"1min":0.314,"5min":0.32400000000000001,"15min":0.32300000000000001},"min":{"1min":0.22600000000000001,"5min":0.22,"15min":0.153},"max":{"1min":0.47199999999999998,"5min":0.53200000000000003,"15min":2.222},"last":0.219},{"interface":"front","average":{"1min":0.35499999999999998,"5min":0.36799999999999999,"15min":0.36399999999999999},"min":{"1min":0.29399999999999998,"5min":0.27300000000000002,"15min":0.20899999999999999},"max":{"1min":0.49199999999999999,"5min":0.54000000000000004,"15min":2.1459999999999999},"last":0.309}]},{"osd":2,"last update":"Wed Nov 8 05:34:28 2023","interfaces":[{"interface":"back","average":{"1min":0.32400000000000001,"5min":0.32900000000000001,"15min":0.32400000000000001},"min":{"1min":0.25800000000000001,"5min":0.23200000000000001,"15min":0.16200000000000001},"max":{"1min":0.45600000000000002,"5min":0.49099999999999999,"15min":0.59199999999999997},"last":0.255},{"interface":"front","average":{"1min":0.38100000000000001,"5min":0.35199999999999998,"15min":0.33500000000000002},"min":{"1min":0.312,"5min":0.255,"15min":0.184},"max":{"1min":0.499,"5min":0.499,"15min":0.86799999999999999},"last":0.32400000000000001}]}]},{"osd":1,"up_from":8,"seq":34359738890,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":313216,"kb_used_data":3704,"kb_used_omap":0,"kb_used_meta":309504,"kb_avail":94058624,"statfs":{"total":96636764160,"available":96316030976,"internally_reserved":0,"allocated":3792896,"data_stored":2047322,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316932096},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:34:43 2023","interfaces":[{"interface":"back","average":{"1min":0.32300000000000001,"5min":0.34599999999999997,"15min":0.32600000000000001},"min":{"1min":0.22800000000000001,"5min":0.21299999999999999,"15min":0.159},"max":{"1min":0.43099999999999999,"5min":0.53200000000000003,"15min":0.53200000000000003},"last":0.32600000000000001},{"interface":"front","average":{"1min":0.36699999999999999,"5min":0.378,"15min":0.36199999999999999},"min":{"1min":0.28799999999999998,"5min":0.248,"15min":0.16600000000000001},"max":{"1min":0.48299999999999998,"5min":0.52000000000000002,"15min":0.92000000000000004},"last":0.23300000000000001}]},{"osd":2,"last update":"Wed Nov 8 05:34:43 2023","interfaces":[{"interface":"back","average":{"1min":0.32600000000000001,"5min":0.34100000000000003,"15min":0.32900000000000001},"min":{"1min":0.223,"5min":0.223,"15min":0.17799999999999999},"max":{"1min":0.44500000000000001,"5min":0.48599999999999999,"15min":1.127},"last":0.249},{"interface":"front","average":{"1min":0.33500000000000002,"5min":0.34999999999999998,"15min":0.33800000000000002},"min":{"1min":0.25,"5min":0.25,"15min":0.17100000000000001},"max":{"1min":0.47699999999999998,"5min":0.52500000000000002,"15min":1.6299999999999999},"last":0.28000000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-08T05:35:26.133 INFO:tasks.ceph.ceph_manager.cluster1:clean! 2023-11-08T05:35:26.133 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2023-11-08T05:35:27.256 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-11-08T05:35:27.259 INFO:teuthology.orchestra.run.smithi077.stderr:dumped all 2023-11-08T05:35:27.272 INFO:teuthology.orchestra.run.smithi077.stdout:{"pg_ready":true,"pg_map":{"version":1750,"stamp":"2023-11-08T05:35:26.658001+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":927360,"kb_used_data":11112,"kb_used_omap":0,"kb_used_meta":916224,"kb_avail":282188160,"statfs":{"total":289910292480,"available":288960675840,"internally_reserved":0,"allocated":11378688,"data_stored":6141966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":938213376},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-38,"num_objects":-34,"num_object_clones":0,"num_object_copies":-68,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-34,"num_whiteouts":0,"num_read":-92153,"num_read_kb":-28964504,"num_write":-17859,"num_write_kb":-11548647,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.067863"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":452,"reported_epoch":456,"state":"active+clean","last_fresh":"2023-11-08T05:29:27.658302+0000","last_change":"2023-11-08T04:52:17.195087+0000","last_active":"2023-11-08T05:29:27.658302+0000","last_peered":"2023-11-08T05:29:27.658302+0000","last_clean":"2023-11-08T05:29:27.658302+0000","last_became_active":"2023-11-08T04:52:17.194821+0000","last_became_peered":"2023-11-08T04:52:17.194821+0000","last_unstale":"2023-11-08T05:29:27.658302+0000","last_undegraded":"2023-11-08T05:29:27.658302+0000","last_fullsized":"2023-11-08T05:29:27.658302+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:16.174854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:16.174854+0000","last_clean_scrub_stamp":"2023-11-08T04:52:16.174854+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":453,"reported_epoch":456,"state":"active+clean","last_fresh":"2023-11-08T05:29:27.910799+0000","last_change":"2023-11-08T04:52:17.194882+0000","last_active":"2023-11-08T05:29:27.910799+0000","last_peered":"2023-11-08T05:29:27.910799+0000","last_clean":"2023-11-08T05:29:27.910799+0000","last_became_active":"2023-11-08T04:52:17.194611+0000","last_became_peered":"2023-11-08T04:52:17.194611+0000","last_unstale":"2023-11-08T05:29:27.910799+0000","last_undegraded":"2023-11-08T05:29:27.910799+0000","last_fullsized":"2023-11-08T05:29:27.910799+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:16.174854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:16.174854+0000","last_clean_scrub_stamp":"2023-11-08T04:52:16.174854+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":455,"reported_epoch":456,"state":"active+clean","last_fresh":"2023-11-08T05:29:28.452318+0000","last_change":"2023-11-08T04:52:17.193579+0000","last_active":"2023-11-08T05:29:28.452318+0000","last_peered":"2023-11-08T05:29:28.452318+0000","last_clean":"2023-11-08T05:29:28.452318+0000","last_became_active":"2023-11-08T04:52:17.193313+0000","last_became_peered":"2023-11-08T04:52:17.193313+0000","last_unstale":"2023-11-08T05:29:28.452318+0000","last_undegraded":"2023-11-08T05:29:28.452318+0000","last_fullsized":"2023-11-08T05:29:28.452318+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:16.174854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:16.174854+0000","last_clean_scrub_stamp":"2023-11-08T04:52:16.174854+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":452,"reported_epoch":456,"state":"active+clean","last_fresh":"2023-11-08T05:29:27.653375+0000","last_change":"2023-11-08T04:52:17.193960+0000","last_active":"2023-11-08T05:29:27.653375+0000","last_peered":"2023-11-08T05:29:27.653375+0000","last_clean":"2023-11-08T05:29:27.653375+0000","last_became_active":"2023-11-08T04:52:17.193661+0000","last_became_peered":"2023-11-08T04:52:17.193661+0000","last_unstale":"2023-11-08T05:29:27.653375+0000","last_undegraded":"2023-11-08T05:29:27.653375+0000","last_fullsized":"2023-11-08T05:29:27.653375+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:16.174854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:16.174854+0000","last_clean_scrub_stamp":"2023-11-08T04:52:16.174854+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":452,"reported_epoch":456,"state":"active+clean","last_fresh":"2023-11-08T05:29:27.652411+0000","last_change":"2023-11-08T04:52:17.195090+0000","last_active":"2023-11-08T05:29:27.652411+0000","last_peered":"2023-11-08T05:29:27.652411+0000","last_clean":"2023-11-08T05:29:27.652411+0000","last_became_active":"2023-11-08T04:52:17.194795+0000","last_became_peered":"2023-11-08T04:52:17.194795+0000","last_unstale":"2023-11-08T05:29:27.652411+0000","last_undegraded":"2023-11-08T05:29:27.652411+0000","last_fullsized":"2023-11-08T05:29:27.652411+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:16.174854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:16.174854+0000","last_clean_scrub_stamp":"2023-11-08T04:52:16.174854+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":455,"reported_epoch":456,"state":"active+clean","last_fresh":"2023-11-08T05:29:28.451762+0000","last_change":"2023-11-08T04:52:17.193152+0000","last_active":"2023-11-08T05:29:28.451762+0000","last_peered":"2023-11-08T05:29:28.451762+0000","last_clean":"2023-11-08T05:29:28.451762+0000","last_became_active":"2023-11-08T04:52:17.192859+0000","last_became_peered":"2023-11-08T04:52:17.192859+0000","last_unstale":"2023-11-08T05:29:28.451762+0000","last_undegraded":"2023-11-08T05:29:28.451762+0000","last_fullsized":"2023-11-08T05:29:28.451762+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:16.174854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:16.174854+0000","last_clean_scrub_stamp":"2023-11-08T04:52:16.174854+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":452,"reported_epoch":456,"state":"active+clean","last_fresh":"2023-11-08T05:29:27.658113+0000","last_change":"2023-11-08T04:52:17.195685+0000","last_active":"2023-11-08T05:29:27.658113+0000","last_peered":"2023-11-08T05:29:27.658113+0000","last_clean":"2023-11-08T05:29:27.658113+0000","last_became_active":"2023-11-08T04:52:17.195400+0000","last_became_peered":"2023-11-08T04:52:17.195400+0000","last_unstale":"2023-11-08T05:29:27.658113+0000","last_undegraded":"2023-11-08T05:29:27.658113+0000","last_fullsized":"2023-11-08T05:29:27.658113+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:16.174854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:16.174854+0000","last_clean_scrub_stamp":"2023-11-08T04:52:16.174854+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":452,"reported_epoch":456,"state":"active+clean","last_fresh":"2023-11-08T05:29:27.654486+0000","last_change":"2023-11-08T04:52:17.196842+0000","last_active":"2023-11-08T05:29:27.654486+0000","last_peered":"2023-11-08T05:29:27.654486+0000","last_clean":"2023-11-08T05:29:27.654486+0000","last_became_active":"2023-11-08T04:52:17.196558+0000","last_became_peered":"2023-11-08T04:52:17.196558+0000","last_unstale":"2023-11-08T05:29:27.654486+0000","last_undegraded":"2023-11-08T05:29:27.654486+0000","last_fullsized":"2023-11-08T05:29:27.654486+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:16.174854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:16.174854+0000","last_clean_scrub_stamp":"2023-11-08T04:52:16.174854+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":452,"reported_epoch":456,"state":"active+clean","last_fresh":"2023-11-08T05:29:27.658763+0000","last_change":"2023-11-08T04:52:17.196926+0000","last_active":"2023-11-08T05:29:27.658763+0000","last_peered":"2023-11-08T05:29:27.658763+0000","last_clean":"2023-11-08T05:29:27.658763+0000","last_became_active":"2023-11-08T04:52:17.196648+0000","last_became_peered":"2023-11-08T04:52:17.196648+0000","last_unstale":"2023-11-08T05:29:27.658763+0000","last_undegraded":"2023-11-08T05:29:27.658763+0000","last_fullsized":"2023-11-08T05:29:27.658763+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:16.174854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:16.174854+0000","last_clean_scrub_stamp":"2023-11-08T04:52:16.174854+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738888,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":305024,"kb_used_data":3704,"kb_used_omap":0,"kb_used_meta":301312,"kb_avail":94066816,"statfs":{"total":96636764160,"available":96324419584,"internally_reserved":0,"allocated":3792896,"data_stored":2047322,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308543488},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:34:50 2023","interfaces":[{"interface":"back","average":{"1min":0.317,"5min":0.317,"15min":0.311},"min":{"1min":0.248,"5min":0.218,"15min":0.183},"max":{"1min":0.42399999999999999,"5min":0.48599999999999999,"15min":1.6859999999999999},"last":0.36499999999999999},{"interface":"front","average":{"1min":0.39300000000000002,"5min":0.38600000000000001,"15min":0.376},"min":{"1min":0.316,"5min":0.28799999999999998,"15min":0.217},"max":{"1min":0.51600000000000001,"5min":0.53100000000000003,"15min":1.714},"last":0.45000000000000001}]},{"osd":1,"last update":"Wed Nov 8 05:34:50 2023","interfaces":[{"interface":"back","average":{"1min":0.33400000000000002,"5min":0.33100000000000002,"15min":0.31900000000000001},"min":{"1min":0.23599999999999999,"5min":0.22800000000000001,"15min":0.217},"max":{"1min":0.48899999999999999,"5min":0.51100000000000001,"15min":0.51200000000000001},"last":0.38500000000000001},{"interface":"front","average":{"1min":0.35599999999999998,"5min":0.34799999999999998,"15min":0.34000000000000002},"min":{"1min":0.248,"5min":0.23200000000000001,"15min":0.21299999999999999},"max":{"1min":0.47899999999999998,"5min":0.51600000000000001,"15min":1.6379999999999999},"last":0.40999999999999998}]}]},{"osd":0,"up_from":8,"seq":34359738888,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":309120,"kb_used_data":3704,"kb_used_omap":0,"kb_used_meta":305408,"kb_avail":94062720,"statfs":{"total":96636764160,"available":96320225280,"internally_reserved":0,"allocated":3792896,"data_stored":2047322,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312737792},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Nov 8 05:34:28 2023","interfaces":[{"interface":"back","average":{"1min":0.314,"5min":0.32400000000000001,"15min":0.32300000000000001},"min":{"1min":0.22600000000000001,"5min":0.22,"15min":0.153},"max":{"1min":0.47199999999999998,"5min":0.53200000000000003,"15min":2.222},"last":0.26300000000000001},{"interface":"front","average":{"1min":0.35499999999999998,"5min":0.36799999999999999,"15min":0.36399999999999999},"min":{"1min":0.29399999999999998,"5min":0.27300000000000002,"15min":0.20899999999999999},"max":{"1min":0.49199999999999999,"5min":0.54000000000000004,"15min":2.1459999999999999},"last":0.34599999999999997}]},{"osd":2,"last update":"Wed Nov 8 05:34:28 2023","interfaces":[{"interface":"back","average":{"1min":0.32400000000000001,"5min":0.32900000000000001,"15min":0.32400000000000001},"min":{"1min":0.25800000000000001,"5min":0.23200000000000001,"15min":0.16200000000000001},"max":{"1min":0.45600000000000002,"5min":0.49099999999999999,"15min":0.59199999999999997},"last":0.30199999999999999},{"interface":"front","average":{"1min":0.38100000000000001,"5min":0.35199999999999998,"15min":0.33500000000000002},"min":{"1min":0.312,"5min":0.255,"15min":0.184},"max":{"1min":0.499,"5min":0.499,"15min":0.86799999999999999},"last":0.32700000000000001}]}]},{"osd":1,"up_from":8,"seq":34359738891,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":313216,"kb_used_data":3704,"kb_used_omap":0,"kb_used_meta":309504,"kb_avail":94058624,"statfs":{"total":96636764160,"available":96316030976,"internally_reserved":0,"allocated":3792896,"data_stored":2047322,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316932096},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:34:43 2023","interfaces":[{"interface":"back","average":{"1min":0.32300000000000001,"5min":0.34599999999999997,"15min":0.32600000000000001},"min":{"1min":0.22800000000000001,"5min":0.21299999999999999,"15min":0.159},"max":{"1min":0.43099999999999999,"5min":0.53200000000000003,"15min":0.53200000000000003},"last":0.23999999999999999},{"interface":"front","average":{"1min":0.36699999999999999,"5min":0.378,"15min":0.36199999999999999},"min":{"1min":0.28799999999999998,"5min":0.248,"15min":0.16600000000000001},"max":{"1min":0.48299999999999998,"5min":0.52000000000000002,"15min":0.92000000000000004},"last":0.32000000000000001}]},{"osd":2,"last update":"Wed Nov 8 05:34:43 2023","interfaces":[{"interface":"back","average":{"1min":0.32600000000000001,"5min":0.34100000000000003,"15min":0.32900000000000001},"min":{"1min":0.223,"5min":0.223,"15min":0.17799999999999999},"max":{"1min":0.44500000000000001,"5min":0.48599999999999999,"15min":1.127},"last":0.27300000000000002},{"interface":"front","average":{"1min":0.33500000000000002,"5min":0.34999999999999998,"15min":0.33800000000000002},"min":{"1min":0.25,"5min":0.25,"15min":0.17100000000000001},"max":{"1min":0.47699999999999998,"5min":0.52500000000000002,"15min":1.6299999999999999},"last":0.28799999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-08T05:35:27.273 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd dump --format=json 2023-11-08T05:35:27.565 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-11-08T05:35:27.565 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":457,"fsid":"1244e29e-ccb6-44ee-9f2f-7bbf2125940c","created":"2023-11-08T04:52:07.196830+0000","modified":"2023-11-08T05:29:29.654328+0000","last_up_change":"2023-11-08T04:52:15.170916+0000","last_in_change":"2023-11-08T04:52:08.120187+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"pacific","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2023-11-08T04:52:15.210778+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2023-11-08T04:52:16.003312+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"ce6698f1-7ce9-42d0-9385-7c2e92eb500a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":115421},{"type":"v1","addr":"172.21.15.77:6817","nonce":115421}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":115421},{"type":"v1","addr":"172.21.15.77:6819","nonce":115421}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":115421},{"type":"v1","addr":"172.21.15.77:6823","nonce":115421}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":115421},{"type":"v1","addr":"172.21.15.77:6821","nonce":115421}]},"public_addr":"172.21.15.77:6817/115421","cluster_addr":"172.21.15.77:6819/115421","heartbeat_back_addr":"172.21.15.77:6823/115421","heartbeat_front_addr":"172.21.15.77:6821/115421","state":["exists","up"]},{"osd":1,"uuid":"84aa8223-3bd2-440d-ba7b-5f1675e4d25d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":115411},{"type":"v1","addr":"172.21.15.77:6809","nonce":115411}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":115411},{"type":"v1","addr":"172.21.15.77:6811","nonce":115411}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":115411},{"type":"v1","addr":"172.21.15.77:6815","nonce":115411}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":115411},{"type":"v1","addr":"172.21.15.77:6813","nonce":115411}]},"public_addr":"172.21.15.77:6809/115411","cluster_addr":"172.21.15.77:6811/115411","heartbeat_back_addr":"172.21.15.77:6815/115411","heartbeat_front_addr":"172.21.15.77:6813/115411","state":["exists","up"]},{"osd":2,"uuid":"0f508d08-7d10-451b-9213-4f36d0c2e9f1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":115412},{"type":"v1","addr":"172.21.15.77:6801","nonce":115412}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":115412},{"type":"v1","addr":"172.21.15.77:6803","nonce":115412}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":115412},{"type":"v1","addr":"172.21.15.77:6807","nonce":115412}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":115412},{"type":"v1","addr":"172.21.15.77:6805","nonce":115412}]},"public_addr":"172.21.15.77:6801/115412","cluster_addr":"172.21.15.77:6803/115412","heartbeat_back_addr":"172.21.15.77:6807/115412","heartbeat_front_addr":"172.21.15.77:6805/115412","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T04:52:13.259518+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T04:52:13.235929+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138314316775423,"old_weight":0,"last_purged_snaps_scrub":"2023-11-08T04:52:13.212169+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-08T05:35:28.579 INFO:tasks.ceph:Scrubbing cluster1.osd.0 2023-11-08T05:35:28.580 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.0 config set osd_debug_deep_scrub_sleep 0 2023-11-08T05:35:28.732 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2023-11-08T05:35:28.732 INFO:teuthology.orchestra.run.smithi077.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-11-08T05:35:28.732 INFO:teuthology.orchestra.run.smithi077.stdout:} 2023-11-08T05:35:28.744 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd deep-scrub 0 2023-11-08T05:35:29.032 INFO:teuthology.orchestra.run.smithi077.stderr:instructed osd(s) 0 to deep-scrub 2023-11-08T05:35:29.043 INFO:tasks.ceph:Scrubbing cluster1.osd.1 2023-11-08T05:35:29.043 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.1 config set osd_debug_deep_scrub_sleep 0 2023-11-08T05:35:29.181 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2023-11-08T05:35:29.181 INFO:teuthology.orchestra.run.smithi077.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-11-08T05:35:29.181 INFO:teuthology.orchestra.run.smithi077.stdout:} 2023-11-08T05:35:29.192 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd deep-scrub 1 2023-11-08T05:35:29.481 INFO:teuthology.orchestra.run.smithi077.stderr:instructed osd(s) 1 to deep-scrub 2023-11-08T05:35:29.492 INFO:tasks.ceph:Scrubbing cluster1.osd.2 2023-11-08T05:35:29.493 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.2 config set osd_debug_deep_scrub_sleep 0 2023-11-08T05:35:29.631 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2023-11-08T05:35:29.631 INFO:teuthology.orchestra.run.smithi077.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-11-08T05:35:29.631 INFO:teuthology.orchestra.run.smithi077.stdout:} 2023-11-08T05:35:29.642 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd deep-scrub 2 2023-11-08T05:35:29.931 INFO:teuthology.orchestra.run.smithi077.stderr:instructed osd(s) 2 to deep-scrub 2023-11-08T05:35:29.942 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2023-11-08T05:35:30.226 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-11-08T05:35:30.228 INFO:teuthology.orchestra.run.smithi077.stderr:dumped all 2023-11-08T05:35:30.239 INFO:teuthology.orchestra.run.smithi077.stdout:{"pg_ready":true,"pg_map":{"version":1751,"stamp":"2023-11-08T05:35:28.658439+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":927360,"kb_used_data":11112,"kb_used_omap":0,"kb_used_meta":916224,"kb_avail":282188160,"statfs":{"total":289910292480,"available":288960675840,"internally_reserved":0,"allocated":11378688,"data_stored":6141966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":938213376},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-38,"num_objects":-34,"num_object_clones":0,"num_object_copies":-68,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-34,"num_whiteouts":0,"num_read":-92153,"num_read_kb":-28964504,"num_write":-17859,"num_write_kb":-11548647,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.067863"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":452,"reported_epoch":456,"state":"active+clean","last_fresh":"2023-11-08T05:29:27.658302+0000","last_change":"2023-11-08T04:52:17.195087+0000","last_active":"2023-11-08T05:29:27.658302+0000","last_peered":"2023-11-08T05:29:27.658302+0000","last_clean":"2023-11-08T05:29:27.658302+0000","last_became_active":"2023-11-08T04:52:17.194821+0000","last_became_peered":"2023-11-08T04:52:17.194821+0000","last_unstale":"2023-11-08T05:29:27.658302+0000","last_undegraded":"2023-11-08T05:29:27.658302+0000","last_fullsized":"2023-11-08T05:29:27.658302+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:16.174854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:16.174854+0000","last_clean_scrub_stamp":"2023-11-08T04:52:16.174854+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":453,"reported_epoch":456,"state":"active+clean","last_fresh":"2023-11-08T05:29:27.910799+0000","last_change":"2023-11-08T04:52:17.194882+0000","last_active":"2023-11-08T05:29:27.910799+0000","last_peered":"2023-11-08T05:29:27.910799+0000","last_clean":"2023-11-08T05:29:27.910799+0000","last_became_active":"2023-11-08T04:52:17.194611+0000","last_became_peered":"2023-11-08T04:52:17.194611+0000","last_unstale":"2023-11-08T05:29:27.910799+0000","last_undegraded":"2023-11-08T05:29:27.910799+0000","last_fullsized":"2023-11-08T05:29:27.910799+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:16.174854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:16.174854+0000","last_clean_scrub_stamp":"2023-11-08T04:52:16.174854+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":455,"reported_epoch":456,"state":"active+clean","last_fresh":"2023-11-08T05:29:28.452318+0000","last_change":"2023-11-08T04:52:17.193579+0000","last_active":"2023-11-08T05:29:28.452318+0000","last_peered":"2023-11-08T05:29:28.452318+0000","last_clean":"2023-11-08T05:29:28.452318+0000","last_became_active":"2023-11-08T04:52:17.193313+0000","last_became_peered":"2023-11-08T04:52:17.193313+0000","last_unstale":"2023-11-08T05:29:28.452318+0000","last_undegraded":"2023-11-08T05:29:28.452318+0000","last_fullsized":"2023-11-08T05:29:28.452318+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:16.174854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:16.174854+0000","last_clean_scrub_stamp":"2023-11-08T04:52:16.174854+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":452,"reported_epoch":456,"state":"active+clean","last_fresh":"2023-11-08T05:29:27.653375+0000","last_change":"2023-11-08T04:52:17.193960+0000","last_active":"2023-11-08T05:29:27.653375+0000","last_peered":"2023-11-08T05:29:27.653375+0000","last_clean":"2023-11-08T05:29:27.653375+0000","last_became_active":"2023-11-08T04:52:17.193661+0000","last_became_peered":"2023-11-08T04:52:17.193661+0000","last_unstale":"2023-11-08T05:29:27.653375+0000","last_undegraded":"2023-11-08T05:29:27.653375+0000","last_fullsized":"2023-11-08T05:29:27.653375+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:16.174854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:16.174854+0000","last_clean_scrub_stamp":"2023-11-08T04:52:16.174854+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":452,"reported_epoch":456,"state":"active+clean","last_fresh":"2023-11-08T05:29:27.652411+0000","last_change":"2023-11-08T04:52:17.195090+0000","last_active":"2023-11-08T05:29:27.652411+0000","last_peered":"2023-11-08T05:29:27.652411+0000","last_clean":"2023-11-08T05:29:27.652411+0000","last_became_active":"2023-11-08T04:52:17.194795+0000","last_became_peered":"2023-11-08T04:52:17.194795+0000","last_unstale":"2023-11-08T05:29:27.652411+0000","last_undegraded":"2023-11-08T05:29:27.652411+0000","last_fullsized":"2023-11-08T05:29:27.652411+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:16.174854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:16.174854+0000","last_clean_scrub_stamp":"2023-11-08T04:52:16.174854+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":455,"reported_epoch":456,"state":"active+clean","last_fresh":"2023-11-08T05:29:28.451762+0000","last_change":"2023-11-08T04:52:17.193152+0000","last_active":"2023-11-08T05:29:28.451762+0000","last_peered":"2023-11-08T05:29:28.451762+0000","last_clean":"2023-11-08T05:29:28.451762+0000","last_became_active":"2023-11-08T04:52:17.192859+0000","last_became_peered":"2023-11-08T04:52:17.192859+0000","last_unstale":"2023-11-08T05:29:28.451762+0000","last_undegraded":"2023-11-08T05:29:28.451762+0000","last_fullsized":"2023-11-08T05:29:28.451762+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:16.174854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:16.174854+0000","last_clean_scrub_stamp":"2023-11-08T04:52:16.174854+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":452,"reported_epoch":456,"state":"active+clean","last_fresh":"2023-11-08T05:29:27.658113+0000","last_change":"2023-11-08T04:52:17.195685+0000","last_active":"2023-11-08T05:29:27.658113+0000","last_peered":"2023-11-08T05:29:27.658113+0000","last_clean":"2023-11-08T05:29:27.658113+0000","last_became_active":"2023-11-08T04:52:17.195400+0000","last_became_peered":"2023-11-08T04:52:17.195400+0000","last_unstale":"2023-11-08T05:29:27.658113+0000","last_undegraded":"2023-11-08T05:29:27.658113+0000","last_fullsized":"2023-11-08T05:29:27.658113+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:16.174854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:16.174854+0000","last_clean_scrub_stamp":"2023-11-08T04:52:16.174854+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":452,"reported_epoch":456,"state":"active+clean","last_fresh":"2023-11-08T05:29:27.654486+0000","last_change":"2023-11-08T04:52:17.196842+0000","last_active":"2023-11-08T05:29:27.654486+0000","last_peered":"2023-11-08T05:29:27.654486+0000","last_clean":"2023-11-08T05:29:27.654486+0000","last_became_active":"2023-11-08T04:52:17.196558+0000","last_became_peered":"2023-11-08T04:52:17.196558+0000","last_unstale":"2023-11-08T05:29:27.654486+0000","last_undegraded":"2023-11-08T05:29:27.654486+0000","last_fullsized":"2023-11-08T05:29:27.654486+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:16.174854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:16.174854+0000","last_clean_scrub_stamp":"2023-11-08T04:52:16.174854+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":452,"reported_epoch":456,"state":"active+clean","last_fresh":"2023-11-08T05:29:27.658763+0000","last_change":"2023-11-08T04:52:17.196926+0000","last_active":"2023-11-08T05:29:27.658763+0000","last_peered":"2023-11-08T05:29:27.658763+0000","last_clean":"2023-11-08T05:29:27.658763+0000","last_became_active":"2023-11-08T04:52:17.196648+0000","last_became_peered":"2023-11-08T04:52:17.196648+0000","last_unstale":"2023-11-08T05:29:27.658763+0000","last_undegraded":"2023-11-08T05:29:27.658763+0000","last_fullsized":"2023-11-08T05:29:27.658763+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T04:52:16.174854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T04:52:16.174854+0000","last_clean_scrub_stamp":"2023-11-08T04:52:16.174854+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738888,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":305024,"kb_used_data":3704,"kb_used_omap":0,"kb_used_meta":301312,"kb_avail":94066816,"statfs":{"total":96636764160,"available":96324419584,"internally_reserved":0,"allocated":3792896,"data_stored":2047322,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308543488},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:34:50 2023","interfaces":[{"interface":"back","average":{"1min":0.317,"5min":0.317,"15min":0.311},"min":{"1min":0.248,"5min":0.218,"15min":0.183},"max":{"1min":0.42399999999999999,"5min":0.48599999999999999,"15min":1.6859999999999999},"last":0.36499999999999999},{"interface":"front","average":{"1min":0.39300000000000002,"5min":0.38600000000000001,"15min":0.376},"min":{"1min":0.316,"5min":0.28799999999999998,"15min":0.217},"max":{"1min":0.51600000000000001,"5min":0.53100000000000003,"15min":1.714},"last":0.45000000000000001}]},{"osd":1,"last update":"Wed Nov 8 05:34:50 2023","interfaces":[{"interface":"back","average":{"1min":0.33400000000000002,"5min":0.33100000000000002,"15min":0.31900000000000001},"min":{"1min":0.23599999999999999,"5min":0.22800000000000001,"15min":0.217},"max":{"1min":0.48899999999999999,"5min":0.51100000000000001,"15min":0.51200000000000001},"last":0.38500000000000001},{"interface":"front","average":{"1min":0.35599999999999998,"5min":0.34799999999999998,"15min":0.34000000000000002},"min":{"1min":0.248,"5min":0.23200000000000001,"15min":0.21299999999999999},"max":{"1min":0.47899999999999998,"5min":0.51600000000000001,"15min":1.6379999999999999},"last":0.40999999999999998}]}]},{"osd":0,"up_from":8,"seq":34359738888,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":309120,"kb_used_data":3704,"kb_used_omap":0,"kb_used_meta":305408,"kb_avail":94062720,"statfs":{"total":96636764160,"available":96320225280,"internally_reserved":0,"allocated":3792896,"data_stored":2047322,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312737792},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Nov 8 05:34:28 2023","interfaces":[{"interface":"back","average":{"1min":0.314,"5min":0.32400000000000001,"15min":0.32300000000000001},"min":{"1min":0.22600000000000001,"5min":0.22,"15min":0.153},"max":{"1min":0.47199999999999998,"5min":0.53200000000000003,"15min":2.222},"last":0.26300000000000001},{"interface":"front","average":{"1min":0.35499999999999998,"5min":0.36799999999999999,"15min":0.36399999999999999},"min":{"1min":0.29399999999999998,"5min":0.27300000000000002,"15min":0.20899999999999999},"max":{"1min":0.49199999999999999,"5min":0.54000000000000004,"15min":2.1459999999999999},"last":0.34599999999999997}]},{"osd":2,"last update":"Wed Nov 8 05:34:28 2023","interfaces":[{"interface":"back","average":{"1min":0.32400000000000001,"5min":0.32900000000000001,"15min":0.32400000000000001},"min":{"1min":0.25800000000000001,"5min":0.23200000000000001,"15min":0.16200000000000001},"max":{"1min":0.45600000000000002,"5min":0.49099999999999999,"15min":0.59199999999999997},"last":0.30199999999999999},{"interface":"front","average":{"1min":0.38100000000000001,"5min":0.35199999999999998,"15min":0.33500000000000002},"min":{"1min":0.312,"5min":0.255,"15min":0.184},"max":{"1min":0.499,"5min":0.499,"15min":0.86799999999999999},"last":0.32700000000000001}]}]},{"osd":1,"up_from":8,"seq":34359738891,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":313216,"kb_used_data":3704,"kb_used_omap":0,"kb_used_meta":309504,"kb_avail":94058624,"statfs":{"total":96636764160,"available":96316030976,"internally_reserved":0,"allocated":3792896,"data_stored":2047322,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316932096},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:34:43 2023","interfaces":[{"interface":"back","average":{"1min":0.32300000000000001,"5min":0.34599999999999997,"15min":0.32600000000000001},"min":{"1min":0.22800000000000001,"5min":0.21299999999999999,"15min":0.159},"max":{"1min":0.43099999999999999,"5min":0.53200000000000003,"15min":0.53200000000000003},"last":0.23999999999999999},{"interface":"front","average":{"1min":0.36699999999999999,"5min":0.378,"15min":0.36199999999999999},"min":{"1min":0.28799999999999998,"5min":0.248,"15min":0.16600000000000001},"max":{"1min":0.48299999999999998,"5min":0.52000000000000002,"15min":0.92000000000000004},"last":0.32000000000000001}]},{"osd":2,"last update":"Wed Nov 8 05:34:43 2023","interfaces":[{"interface":"back","average":{"1min":0.32600000000000001,"5min":0.34100000000000003,"15min":0.32900000000000001},"min":{"1min":0.223,"5min":0.223,"15min":0.17799999999999999},"max":{"1min":0.44500000000000001,"5min":0.48599999999999999,"15min":1.127},"last":0.27300000000000002},{"interface":"front","average":{"1min":0.33500000000000002,"5min":0.34999999999999998,"15min":0.33800000000000002},"min":{"1min":0.25,"5min":0.25,"15min":0.17100000000000001},"max":{"1min":0.47699999999999998,"5min":0.52500000000000002,"15min":1.6299999999999999},"last":0.28799999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-08T05:35:30.244 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2023-11-08T04:52:16.174854+0000 time.struct_time(tm_year=2023, tm_mon=11, tm_mday=8, tm_hour=4, tm_min=52, tm_sec=16, tm_wday=2, tm_yday=312, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=11, tm_mday=8, tm_hour=5, tm_min=35, tm_sec=27, tm_wday=2, tm_yday=312, tm_isdst=0) 2023-11-08T05:35:30.245 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2023-11-08T04:52:16.174854+0000 time.struct_time(tm_year=2023, tm_mon=11, tm_mday=8, tm_hour=4, tm_min=52, tm_sec=16, tm_wday=2, tm_yday=312, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=11, tm_mday=8, tm_hour=5, tm_min=35, tm_sec=27, tm_wday=2, tm_yday=312, tm_isdst=0) 2023-11-08T05:35:30.245 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2023-11-08T04:52:16.174854+0000 time.struct_time(tm_year=2023, tm_mon=11, tm_mday=8, tm_hour=4, tm_min=52, tm_sec=16, tm_wday=2, tm_yday=312, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=11, tm_mday=8, tm_hour=5, tm_min=35, tm_sec=27, tm_wday=2, tm_yday=312, tm_isdst=0) 2023-11-08T05:35:30.246 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-11-08T04:52:16.174854+0000 time.struct_time(tm_year=2023, tm_mon=11, tm_mday=8, tm_hour=4, tm_min=52, tm_sec=16, tm_wday=2, tm_yday=312, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=11, tm_mday=8, tm_hour=5, tm_min=35, tm_sec=27, tm_wday=2, tm_yday=312, tm_isdst=0) 2023-11-08T05:35:30.246 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2023-11-08T04:52:16.174854+0000 time.struct_time(tm_year=2023, tm_mon=11, tm_mday=8, tm_hour=4, tm_min=52, tm_sec=16, tm_wday=2, tm_yday=312, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=11, tm_mday=8, tm_hour=5, tm_min=35, tm_sec=27, tm_wday=2, tm_yday=312, tm_isdst=0) 2023-11-08T05:35:30.247 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2023-11-08T04:52:16.174854+0000 time.struct_time(tm_year=2023, tm_mon=11, tm_mday=8, tm_hour=4, tm_min=52, tm_sec=16, tm_wday=2, tm_yday=312, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=11, tm_mday=8, tm_hour=5, tm_min=35, tm_sec=27, tm_wday=2, tm_yday=312, tm_isdst=0) 2023-11-08T05:35:30.247 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-11-08T04:52:16.174854+0000 time.struct_time(tm_year=2023, tm_mon=11, tm_mday=8, tm_hour=4, tm_min=52, tm_sec=16, tm_wday=2, tm_yday=312, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=11, tm_mday=8, tm_hour=5, tm_min=35, tm_sec=27, tm_wday=2, tm_yday=312, tm_isdst=0) 2023-11-08T05:35:30.248 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-11-08T04:52:16.174854+0000 time.struct_time(tm_year=2023, tm_mon=11, tm_mday=8, tm_hour=4, tm_min=52, tm_sec=16, tm_wday=2, tm_yday=312, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=11, tm_mday=8, tm_hour=5, tm_min=35, tm_sec=27, tm_wday=2, tm_yday=312, tm_isdst=0) 2023-11-08T05:35:30.248 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2023-11-08T04:52:16.174854+0000 time.struct_time(tm_year=2023, tm_mon=11, tm_mday=8, tm_hour=4, tm_min=52, tm_sec=16, tm_wday=2, tm_yday=312, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=11, tm_mday=8, tm_hour=5, tm_min=35, tm_sec=27, tm_wday=2, tm_yday=312, tm_isdst=0) 2023-11-08T05:35:30.248 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-11-08T05:35:50.249 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2023-11-08T05:35:50.544 INFO:teuthology.orchestra.run.smithi077.stdout: 2023-11-08T05:35:50.546 INFO:teuthology.orchestra.run.smithi077.stderr:dumped all 2023-11-08T05:35:50.557 INFO:teuthology.orchestra.run.smithi077.stdout:{"pg_ready":true,"pg_map":{"version":1761,"stamp":"2023-11-08T05:35:48.662319+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":927360,"kb_used_data":11112,"kb_used_omap":0,"kb_used_meta":916224,"kb_avail":282188160,"statfs":{"total":289910292480,"available":288960675840,"internally_reserved":0,"allocated":11378688,"data_stored":6141966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":938213376},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-38,"num_objects":-34,"num_object_clones":0,"num_object_copies":-68,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-34,"num_whiteouts":0,"num_read":-92153,"num_read_kb":-28964504,"num_write":-17859,"num_write_kb":-11548647,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.067863"},"pg_stats":[{"pgid":"2.4","version":"0'0","reported_seq":457,"reported_epoch":457,"state":"active+clean","last_fresh":"2023-11-08T05:35:34.030901+0000","last_change":"2023-11-08T05:35:34.030901+0000","last_active":"2023-11-08T05:35:34.030901+0000","last_peered":"2023-11-08T05:35:34.030901+0000","last_clean":"2023-11-08T05:35:34.030901+0000","last_became_active":"2023-11-08T04:52:17.194821+0000","last_became_peered":"2023-11-08T04:52:17.194821+0000","last_unstale":"2023-11-08T05:35:34.030901+0000","last_undegraded":"2023-11-08T05:35:34.030901+0000","last_fullsized":"2023-11-08T05:35:34.030901+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T05:35:34.030794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T05:35:34.030794+0000","last_clean_scrub_stamp":"2023-11-08T05:35:34.030794+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":458,"reported_epoch":457,"state":"active+clean","last_fresh":"2023-11-08T05:35:42.622500+0000","last_change":"2023-11-08T05:35:42.622500+0000","last_active":"2023-11-08T05:35:42.622500+0000","last_peered":"2023-11-08T05:35:42.622500+0000","last_clean":"2023-11-08T05:35:42.622500+0000","last_became_active":"2023-11-08T04:52:17.194611+0000","last_became_peered":"2023-11-08T04:52:17.194611+0000","last_unstale":"2023-11-08T05:35:42.622500+0000","last_undegraded":"2023-11-08T05:35:42.622500+0000","last_fullsized":"2023-11-08T05:35:42.622500+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T05:35:42.622370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T05:35:42.622370+0000","last_clean_scrub_stamp":"2023-11-08T05:35:42.622370+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":460,"reported_epoch":457,"state":"active+clean","last_fresh":"2023-11-08T05:35:36.232675+0000","last_change":"2023-11-08T05:35:36.232675+0000","last_active":"2023-11-08T05:35:36.232675+0000","last_peered":"2023-11-08T05:35:36.232675+0000","last_clean":"2023-11-08T05:35:36.232675+0000","last_became_active":"2023-11-08T04:52:17.193313+0000","last_became_peered":"2023-11-08T04:52:17.193313+0000","last_unstale":"2023-11-08T05:35:36.232675+0000","last_undegraded":"2023-11-08T05:35:36.232675+0000","last_fullsized":"2023-11-08T05:35:36.232675+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T05:35:36.232483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T05:35:36.232483+0000","last_clean_scrub_stamp":"2023-11-08T05:35:36.232483+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":457,"reported_epoch":457,"state":"active+clean","last_fresh":"2023-11-08T05:35:33.014508+0000","last_change":"2023-11-08T05:35:33.014508+0000","last_active":"2023-11-08T05:35:33.014508+0000","last_peered":"2023-11-08T05:35:33.014508+0000","last_clean":"2023-11-08T05:35:33.014508+0000","last_became_active":"2023-11-08T04:52:17.193661+0000","last_became_peered":"2023-11-08T04:52:17.193661+0000","last_unstale":"2023-11-08T05:35:33.014508+0000","last_undegraded":"2023-11-08T05:35:33.014508+0000","last_fullsized":"2023-11-08T05:35:33.014508+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T05:35:33.014331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T05:35:33.014331+0000","last_clean_scrub_stamp":"2023-11-08T05:35:33.014331+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":457,"reported_epoch":457,"state":"active+clean","last_fresh":"2023-11-08T05:35:30.923142+0000","last_change":"2023-11-08T05:35:30.923142+0000","last_active":"2023-11-08T05:35:30.923142+0000","last_peered":"2023-11-08T05:35:30.923142+0000","last_clean":"2023-11-08T05:35:30.923142+0000","last_became_active":"2023-11-08T04:52:17.194795+0000","last_became_peered":"2023-11-08T04:52:17.194795+0000","last_unstale":"2023-11-08T05:35:30.923142+0000","last_undegraded":"2023-11-08T05:35:30.923142+0000","last_fullsized":"2023-11-08T05:35:30.923142+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T05:35:30.923038+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T05:35:30.923038+0000","last_clean_scrub_stamp":"2023-11-08T05:35:30.923038+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":460,"reported_epoch":457,"state":"active+clean","last_fresh":"2023-11-08T05:35:33.185296+0000","last_change":"2023-11-08T05:35:33.185296+0000","last_active":"2023-11-08T05:35:33.185296+0000","last_peered":"2023-11-08T05:35:33.185296+0000","last_clean":"2023-11-08T05:35:33.185296+0000","last_became_active":"2023-11-08T04:52:17.192859+0000","last_became_peered":"2023-11-08T04:52:17.192859+0000","last_unstale":"2023-11-08T05:35:33.185296+0000","last_undegraded":"2023-11-08T05:35:33.185296+0000","last_fullsized":"2023-11-08T05:35:33.185296+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T05:35:33.185157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T05:35:33.185157+0000","last_clean_scrub_stamp":"2023-11-08T05:35:33.185157+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":457,"reported_epoch":457,"state":"active+clean","last_fresh":"2023-11-08T05:35:36.034283+0000","last_change":"2023-11-08T05:35:36.034283+0000","last_active":"2023-11-08T05:35:36.034283+0000","last_peered":"2023-11-08T05:35:36.034283+0000","last_clean":"2023-11-08T05:35:36.034283+0000","last_became_active":"2023-11-08T04:52:17.195400+0000","last_became_peered":"2023-11-08T04:52:17.195400+0000","last_unstale":"2023-11-08T05:35:36.034283+0000","last_undegraded":"2023-11-08T05:35:36.034283+0000","last_fullsized":"2023-11-08T05:35:36.034283+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T05:35:36.034142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T05:35:36.034142+0000","last_clean_scrub_stamp":"2023-11-08T05:35:36.034142+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":457,"reported_epoch":457,"state":"active+clean","last_fresh":"2023-11-08T05:35:39.981203+0000","last_change":"2023-11-08T05:35:39.981203+0000","last_active":"2023-11-08T05:35:39.981203+0000","last_peered":"2023-11-08T05:35:39.981203+0000","last_clean":"2023-11-08T05:35:39.981203+0000","last_became_active":"2023-11-08T04:52:17.196558+0000","last_became_peered":"2023-11-08T04:52:17.196558+0000","last_unstale":"2023-11-08T05:35:39.981203+0000","last_undegraded":"2023-11-08T05:35:39.981203+0000","last_fullsized":"2023-11-08T05:35:39.981203+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T05:35:39.981072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T05:35:39.981072+0000","last_clean_scrub_stamp":"2023-11-08T05:35:39.981072+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":457,"reported_epoch":457,"state":"active+clean","last_fresh":"2023-11-08T05:35:41.971513+0000","last_change":"2023-11-08T05:35:41.971513+0000","last_active":"2023-11-08T05:35:41.971513+0000","last_peered":"2023-11-08T05:35:41.971513+0000","last_clean":"2023-11-08T05:35:41.971513+0000","last_became_active":"2023-11-08T04:52:17.196648+0000","last_became_peered":"2023-11-08T04:52:17.196648+0000","last_unstale":"2023-11-08T05:35:41.971513+0000","last_undegraded":"2023-11-08T05:35:41.971513+0000","last_fullsized":"2023-11-08T05:35:41.971513+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T05:35:41.971387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T05:35:41.971387+0000","last_clean_scrub_stamp":"2023-11-08T05:35:41.971387+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":2,"acting":2,"num_store_stats":2},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738892,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":305024,"kb_used_data":3704,"kb_used_omap":0,"kb_used_meta":301312,"kb_avail":94066816,"statfs":{"total":96636764160,"available":96324419584,"internally_reserved":0,"allocated":3792896,"data_stored":2047322,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":308543488},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:34:50 2023","interfaces":[{"interface":"back","average":{"1min":0.317,"5min":0.317,"15min":0.311},"min":{"1min":0.248,"5min":0.218,"15min":0.183},"max":{"1min":0.42399999999999999,"5min":0.48599999999999999,"15min":1.6859999999999999},"last":0.29299999999999998},{"interface":"front","average":{"1min":0.39300000000000002,"5min":0.38600000000000001,"15min":0.376},"min":{"1min":0.316,"5min":0.28799999999999998,"15min":0.217},"max":{"1min":0.51600000000000001,"5min":0.53100000000000003,"15min":1.714},"last":0.377}]},{"osd":1,"last update":"Wed Nov 8 05:34:50 2023","interfaces":[{"interface":"back","average":{"1min":0.33400000000000002,"5min":0.33100000000000002,"15min":0.31900000000000001},"min":{"1min":0.23599999999999999,"5min":0.22800000000000001,"15min":0.217},"max":{"1min":0.48899999999999999,"5min":0.51100000000000001,"15min":0.51200000000000001},"last":0.39500000000000002},{"interface":"front","average":{"1min":0.35599999999999998,"5min":0.34799999999999998,"15min":0.34000000000000002},"min":{"1min":0.248,"5min":0.23200000000000001,"15min":0.21299999999999999},"max":{"1min":0.47899999999999998,"5min":0.51600000000000001,"15min":1.6379999999999999},"last":0.34799999999999998}]}]},{"osd":0,"up_from":8,"seq":34359738892,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":309120,"kb_used_data":3704,"kb_used_omap":0,"kb_used_meta":305408,"kb_avail":94062720,"statfs":{"total":96636764160,"available":96320225280,"internally_reserved":0,"allocated":3792896,"data_stored":2047322,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":312737792},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Nov 8 05:35:29 2023","interfaces":[{"interface":"back","average":{"1min":0.29599999999999999,"5min":0.314,"15min":0.32200000000000001},"min":{"1min":0.219,"5min":0.219,"15min":0.153},"max":{"1min":0.43099999999999999,"5min":0.48699999999999999,"15min":2.222},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.33800000000000002,"5min":0.35899999999999999,"15min":0.36399999999999999},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.20899999999999999},"max":{"1min":0.51900000000000002,"5min":0.54000000000000004,"15min":2.1459999999999999},"last":0.39000000000000001}]},{"osd":2,"last update":"Wed Nov 8 05:35:29 2023","interfaces":[{"interface":"back","average":{"1min":0.309,"5min":0.32200000000000001,"15min":0.32400000000000001},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.16200000000000001},"max":{"1min":0.47199999999999998,"5min":0.49099999999999999,"15min":0.59199999999999997},"last":0.374},{"interface":"front","average":{"1min":0.35899999999999999,"5min":0.35299999999999998,"15min":0.33900000000000002},"min":{"1min":0.30599999999999999,"5min":0.26100000000000001,"15min":0.184},"max":{"1min":0.496,"5min":0.499,"15min":0.86799999999999999},"last":0.46800000000000003}]}]},{"osd":1,"up_from":8,"seq":34359738895,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":313216,"kb_used_data":3704,"kb_used_omap":0,"kb_used_meta":309504,"kb_avail":94058624,"statfs":{"total":96636764160,"available":96316030976,"internally_reserved":0,"allocated":3792896,"data_stored":2047322,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":316932096},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Nov 8 05:34:43 2023","interfaces":[{"interface":"back","average":{"1min":0.32300000000000001,"5min":0.34599999999999997,"15min":0.32600000000000001},"min":{"1min":0.22800000000000001,"5min":0.21299999999999999,"15min":0.159},"max":{"1min":0.43099999999999999,"5min":0.53200000000000003,"15min":0.53200000000000003},"last":0.26900000000000002},{"interface":"front","average":{"1min":0.36699999999999999,"5min":0.378,"15min":0.36199999999999999},"min":{"1min":0.28799999999999998,"5min":0.248,"15min":0.16600000000000001},"max":{"1min":0.48299999999999998,"5min":0.52000000000000002,"15min":0.92000000000000004},"last":0.32600000000000001}]},{"osd":2,"last update":"Wed Nov 8 05:34:43 2023","interfaces":[{"interface":"back","average":{"1min":0.32600000000000001,"5min":0.34100000000000003,"15min":0.32900000000000001},"min":{"1min":0.223,"5min":0.223,"15min":0.17799999999999999},"max":{"1min":0.44500000000000001,"5min":0.48599999999999999,"15min":1.127},"last":0.25700000000000001},{"interface":"front","average":{"1min":0.33500000000000002,"5min":0.34999999999999998,"15min":0.33800000000000002},"min":{"1min":0.25,"5min":0.25,"15min":0.17100000000000001},"max":{"1min":0.47699999999999998,"5min":0.52500000000000002,"15min":1.6299999999999999},"last":0.312}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-11-08T05:35:50.558 DEBUG:teuthology.orchestra.run.smithi077:> sudo ceph --cluster cluster1 config set global mon_health_to_clog false 2023-11-08T05:35:50.860 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_cfb6fd52113be6a0af3fe7fb89b2e08474eadc0f/qa/tasks/ceph.py", line 1907, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_cfb6fd52113be6a0af3fe7fb89b2e08474eadc0f/qa/tasks/ceph.py", line 1916, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_cfb6fd52113be6a0af3fe7fb89b2e08474eadc0f/qa/tasks/ceph_manager.py", line 2758, in wait_for_clean while not self.is_clean(): File "/home/teuthworker/src/github.com_ceph_ceph-c_cfb6fd52113be6a0af3fe7fb89b2e08474eadc0f/qa/tasks/ceph_manager.py", line 2694, in is_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_ceph_ceph-c_cfb6fd52113be6a0af3fe7fb89b2e08474eadc0f/qa/tasks/ceph_manager.py", line 2355, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph-c_cfb6fd52113be6a0af3fe7fb89b2e08474eadc0f/qa/tasks/ceph_manager.py", line 1616, in raw_cluster_cmd p = self.run_cluster_cmd(args=args, stdout=stdout, **kwargs) File "/home/teuthworker/src/github.com_ceph_ceph-c_cfb6fd52113be6a0af3fe7fb89b2e08474eadc0f/qa/tasks/ceph_manager.py", line 1609, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi087 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json' 2023-11-08T05:35:50.860 INFO:teuthology.misc:Shutting down mds daemons... 2023-11-08T05:35:50.861 INFO:teuthology.misc:Shutting down osd daemons... 2023-11-08T05:35:50.862 DEBUG:tasks.ceph.cluster1.osd.0:waiting for process to exit 2023-11-08T05:35:50.862 INFO:teuthology.orchestra.run:waiting for 300 2023-11-08T05:35:50.901 INFO:tasks.ceph.cluster1.osd.0:Stopped 2023-11-08T05:35:50.902 DEBUG:tasks.ceph.cluster1.osd.1:waiting for process to exit 2023-11-08T05:35:50.903 INFO:teuthology.orchestra.run:waiting for 300 2023-11-08T05:35:50.977 INFO:tasks.ceph.cluster1.osd.1:Stopped 2023-11-08T05:35:50.978 DEBUG:tasks.ceph.cluster1.osd.2:waiting for process to exit 2023-11-08T05:35:50.979 INFO:teuthology.orchestra.run:waiting for 300 2023-11-08T05:35:51.046 INFO:tasks.ceph.cluster1.osd.2:Stopped 2023-11-08T05:35:51.047 INFO:teuthology.misc:Shutting down mgr daemons... 2023-11-08T05:35:51.047 DEBUG:tasks.ceph.cluster1.mgr.x:waiting for process to exit 2023-11-08T05:35:51.048 INFO:teuthology.orchestra.run:waiting for 300 2023-11-08T05:35:51.108 INFO:tasks.ceph.cluster1.mgr.x:Stopped 2023-11-08T05:35:51.109 INFO:teuthology.misc:Shutting down mon daemons... 2023-11-08T05:35:51.110 DEBUG:tasks.ceph.cluster1.mon.a:waiting for process to exit 2023-11-08T05:35:51.110 INFO:teuthology.orchestra.run:waiting for 300 2023-11-08T05:35:51.162 INFO:tasks.ceph.cluster1.mon.a:Stopped 2023-11-08T05:35:51.163 INFO:tasks.ceph:Checking cluster log for badness... 2023-11-08T05:35:51.164 DEBUG:teuthology.orchestra.run.smithi077:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/cluster1.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | head -n 1 2023-11-08T05:35:51.232 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster1-0 on ubuntu@smithi077.front.sepia.ceph.com 2023-11-08T05:35:51.233 DEBUG:teuthology.orchestra.run.smithi077:> sync && sudo umount -f /var/lib/ceph/osd/cluster1-0 2023-11-08T05:35:51.424 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster1-1 on ubuntu@smithi077.front.sepia.ceph.com 2023-11-08T05:35:51.424 DEBUG:teuthology.orchestra.run.smithi077:> sync && sudo umount -f /var/lib/ceph/osd/cluster1-1 2023-11-08T05:35:51.597 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster1-2 on ubuntu@smithi077.front.sepia.ceph.com 2023-11-08T05:35:51.598 DEBUG:teuthology.orchestra.run.smithi077:> sync && sudo umount -f /var/lib/ceph/osd/cluster1-2 2023-11-08T05:35:51.679 INFO:tasks.ceph:Archiving mon data... 2023-11-08T05:35:51.680 DEBUG:teuthology.misc:Transferring archived files from smithi077:/var/lib/ceph/mon/cluster1-a to /home/teuthworker/archive/yuriw-2023-11-07_19:42:25-rbd-wip-yuri3-testing-2023-11-07-0801-pacific-distro-default-smithi/7451368/data/cluster1.mon.a.tgz 2023-11-08T05:35:51.681 DEBUG:teuthology.orchestra.run.smithi077:> mktemp 2023-11-08T05:35:51.698 INFO:teuthology.orchestra.run.smithi077.stdout:/tmp/tmp.pUlhnaPDJ3 2023-11-08T05:35:51.699 DEBUG:teuthology.orchestra.run.smithi077:> sudo tar cz -f - -C /var/lib/ceph/mon/cluster1-a -- . > /tmp/tmp.pUlhnaPDJ3 2023-11-08T05:35:51.967 DEBUG:teuthology.orchestra.run.smithi077:> sudo chmod 0666 /tmp/tmp.pUlhnaPDJ3 2023-11-08T05:35:52.055 DEBUG:teuthology.orchestra.remote:smithi077:/tmp/tmp.pUlhnaPDJ3 is 657KB 2023-11-08T05:35:52.154 DEBUG:teuthology.orchestra.run.smithi077:> rm -fr /tmp/tmp.pUlhnaPDJ3 2023-11-08T05:35:52.174 INFO:tasks.ceph:Cleaning ceph cluster... 2023-11-08T05:35:52.175 DEBUG:teuthology.orchestra.run.smithi077:> sudo rm -rf -- /etc/ceph/cluster1.conf /etc/ceph/cluster1.keyring /home/ubuntu/cephtest/cluster1.data /home/ubuntu/cephtest/cluster1.monmap /home/ubuntu/cephtest/../*.pid 2023-11-08T05:35:52.216 DEBUG:teuthology.orchestra.run.smithi087:> sudo rm -rf -- /etc/ceph/cluster1.conf /etc/ceph/cluster1.keyring /home/ubuntu/cephtest/cluster1.data /home/ubuntu/cephtest/cluster1.monmap /home/ubuntu/cephtest/../*.pid 2023-11-08T05:35:52.249 INFO:tasks.ceph:Checking for errors in any valgrind logs... 2023-11-08T05:35:52.249 DEBUG:teuthology.orchestra.run.smithi077:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2023-11-08T05:35:52.282 DEBUG:teuthology.orchestra.run.smithi087:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2023-11-08T05:35:52.307 INFO:teuthology.orchestra.run.smithi077.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2023-11-08T05:35:52.322 INFO:teuthology.orchestra.run.smithi087.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2023-11-08T05:35:52.328 INFO:tasks.ceph:Archiving crash dumps... 2023-11-08T05:35:52.330 DEBUG:teuthology.misc:Transferring archived files from smithi077:/var/lib/ceph/crash to /home/teuthworker/archive/yuriw-2023-11-07_19:42:25-rbd-wip-yuri3-testing-2023-11-07-0801-pacific-distro-default-smithi/7451368/remote/smithi077/crash 2023-11-08T05:35:52.332 DEBUG:teuthology.orchestra.run.smithi077:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-11-08T05:35:52.379 DEBUG:teuthology.misc:Transferring archived files from smithi087:/var/lib/ceph/crash to /home/teuthworker/archive/yuriw-2023-11-07_19:42:25-rbd-wip-yuri3-testing-2023-11-07-0801-pacific-distro-default-smithi/7451368/remote/smithi087/crash 2023-11-08T05:35:52.381 DEBUG:teuthology.orchestra.run.smithi087:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-11-08T05:35:52.454 INFO:tasks.ceph:Compressing logs... 2023-11-08T05:35:52.455 DEBUG:teuthology.orchestra.run.smithi077:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-11-08T05:35:52.457 DEBUG:teuthology.orchestra.run.smithi087:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-11-08T05:35:52.481 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.118193.log 2023-11-08T05:35:52.481 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-mon.a.log 2023-11-08T05:35:52.487 INFO:teuthology.orchestra.run.smithi077.stderr:gzip /var/log/ceph/cluster1-client.admin.118193.log: -5 --verbose -- /var/log/ceph/cluster2-mgr.x.log 2023-11-08T05:35:52.487 INFO:teuthology.orchestra.run.smithi077.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.118193.log.gz 2023-11-08T05:35:52.488 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-mon.a.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.122150.log 2023-11-08T05:35:52.488 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.118291.log 2023-11-08T05:35:52.488 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-mgr.x.log: /var/log/ceph/cluster1-client.admin.122150.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.118340.log 2023-11-08T05:35:52.488 INFO:teuthology.orchestra.run.smithi077.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.122150.log.gz 2023-11-08T05:35:52.488 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.admin.118291.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.115237.log 2023-11-08T05:35:52.489 INFO:teuthology.orchestra.run.smithi077.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.118291.log.gz 2023-11-08T05:35:52.489 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.admin.118340.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.118340.log.gz 2023-11-08T05:35:52.489 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.118389.log 2023-11-08T05:35:52.489 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.admin.115237.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.118732.log 2023-11-08T05:35:52.489 INFO:teuthology.orchestra.run.smithi077.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.115237.log.gz 2023-11-08T05:35:52.490 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.admin.118389.log: gzip -5 --verbose -- /var/log/ceph/cluster1-mgr.x.log 2023-11-08T05:35:52.490 INFO:teuthology.orchestra.run.smithi077.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.118389.log.gz 2023-11-08T05:35:52.490 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.admin.118732.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.118732.log.gz 2023-11-08T05:35:52.490 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.118585.log 2023-11-08T05:35:52.490 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.122395.log 2023-11-08T05:35:52.490 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.admin.118585.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.118585.log.gz 2023-11-08T05:35:52.491 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.122444.log 2023-11-08T05:35:52.491 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.122297.log 2023-11-08T05:35:52.491 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.admin.122395.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.122395.log.gz 2023-11-08T05:35:52.491 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.admin.122444.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.110148.log 2023-11-08T05:35:52.491 INFO:teuthology.orchestra.run.smithi077.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.122444.log.gz 2023-11-08T05:35:52.492 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.admin.122297.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.122297.log.gz 2023-11-08T05:35:52.492 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.122052.log 2023-11-08T05:35:52.492 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.122346.log 2023-11-08T05:35:52.492 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph.tmp-client.admin.110148.log: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.110148.log.gz 2023-11-08T05:35:52.492 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.admin.122052.log: gzip -5 --verbose -- /var/log/ceph/cluster1.log 2023-11-08T05:35:52.493 INFO:teuthology.orchestra.run.smithi077.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.122052.log.gz 2023-11-08T05:35:52.493 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.admin.122346.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.122346.log.gz 2023-11-08T05:35:52.493 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.122640.log 2023-11-08T05:35:52.493 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.122248.log 2023-11-08T05:35:52.493 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1.log: /var/log/ceph/cluster1-client.admin.122640.log: gzip -5 --verbose -- /var/log/ceph/cluster1-osd.2.log 2023-11-08T05:35:52.494 INFO:teuthology.orchestra.run.smithi077.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.122640.log.gz 2023-11-08T05:35:52.494 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.admin.122248.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.122248.log.gz 2023-11-08T05:35:52.497 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-osd.2.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.118242.log 2023-11-08T05:35:52.497 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.115410.log 2023-11-08T05:35:52.498 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.admin.118242.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.118242.log.gz 2023-11-08T05:35:52.498 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.115188.log 2023-11-08T05:35:52.498 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.admin.115410.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.115410.log.gz 2023-11-08T05:35:52.498 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.admin.115188.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.115188.log.gz 2023-11-08T05:35:52.499 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.118830.log 2023-11-08T05:35:52.499 INFO:teuthology.orchestra.run.smithi077.stderr: 87.9% -- replaced with /var/log/ceph/cluster1.log.gz 2023-11-08T05:35:52.499 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.122687.log 2023-11-08T05:35:52.499 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.admin.118830.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.118781.log 2023-11-08T05:35:52.499 INFO:teuthology.orchestra.run.smithi077.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.118830.log.gz 2023-11-08T05:35:52.499 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.admin.122687.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.118487.log 2023-11-08T05:35:52.500 INFO:teuthology.orchestra.run.smithi077.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.122687.log.gz 2023-11-08T05:35:52.500 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.admin.118781.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.118781.log.gz 2023-11-08T05:35:52.500 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.admin.118487.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.118487.log.gz 2023-11-08T05:35:52.500 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.118536.log 2023-11-08T05:35:52.500 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.122493.log 2023-11-08T05:35:52.501 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.admin.118536.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.118536.log.gz 2023-11-08T05:35:52.501 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.admin.122493.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.122493.log.gz 2023-11-08T05:35:52.501 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.122101.log 2023-11-08T05:35:52.501 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.118438.log 2023-11-08T05:35:52.501 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.admin.122101.log: gzip -5 --verbose -- /var/log/ceph/cluster1-osd.0.log 0.0% 2023-11-08T05:35:52.502 INFO:teuthology.orchestra.run.smithi077.stderr: -- replaced with /var/log/ceph/cluster1-client.admin.122101.log.gz 2023-11-08T05:35:52.502 INFO:teuthology.orchestra.run.smithi077.stderr:gzip /var/log/ceph/cluster1-client.admin.118438.log: -5 --verbose -- /var/log/ceph/cluster1-client.admin.118634.log 2023-11-08T05:35:52.502 INFO:teuthology.orchestra.run.smithi077.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.118438.log.gz 2023-11-08T05:35:52.502 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-osd.0.log: gzip -5 --verbose -- /var/log/ceph/cluster1.audit.log 2023-11-08T05:35:52.502 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.admin.118634.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.118634.log.gz 2023-11-08T05:35:52.505 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1.audit.log: 93.0% -- replaced with /var/log/ceph/cluster1.audit.log.gz 2023-11-08T05:35:52.505 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.122199.log 2023-11-08T05:35:52.505 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.118144.log 2023-11-08T05:35:52.505 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.admin.122199.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.122199.log.gz 2023-11-08T05:35:52.505 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.admin.118144.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.118144.log.gz 2023-11-08T05:35:52.506 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.122542.log 2023-11-08T05:35:52.506 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.115483.log 2023-11-08T05:35:52.506 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.admin.122542.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.122542.log.gz 2023-11-08T05:35:52.506 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.admin.115483.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.115483.log.gz 2023-11-08T05:35:52.506 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.118075.log 2023-11-08T05:35:52.507 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.115002.log 2023-11-08T05:35:52.507 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.admin.118075.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.118075.log.gz 2023-11-08T05:35:52.507 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.admin.115002.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.115002.log.gz 2023-11-08T05:35:52.507 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.115284.log 2023-11-08T05:35:52.507 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-osd.1.log 2023-11-08T05:35:52.508 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.admin.115284.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.115284.log.gz 2023-11-08T05:35:52.509 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-osd.1.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.118683.log 2023-11-08T05:35:52.510 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.admin.118683.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.118683.log.gz 2023-11-08T05:35:52.510 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.122591.log 2023-11-08T05:35:52.510 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.admin.122591.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.122591.log.gz 2023-11-08T05:35:52.518 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.154174.log 2023-11-08T05:35:52.519 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.156373.log 2023-11-08T05:35:52.519 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.154174.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.131870.log 2023-11-08T05:35:52.519 INFO:teuthology.orchestra.run.smithi087.stderr: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.154174.log.gz 2023-11-08T05:35:52.519 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.156373.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.126101.log 2023-11-08T05:35:52.520 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.131870.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.150305.log 2023-11-08T05:35:52.520 INFO:teuthology.orchestra.run.smithi087.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.156373.log.gz 2023-11-08T05:35:52.520 INFO:teuthology.orchestra.run.smithi087.stderr: 79.5% -- replaced with /var/log/ceph/cluster2-client.mirror.131870.log.gz 2023-11-08T05:35:52.520 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.126101.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.153881.log 2023-11-08T05:35:52.520 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.150305.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.152353.log 2023-11-08T05:35:52.521 INFO:teuthology.orchestra.run.smithi087.stderr: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.126101.log.gz 2023-11-08T05:35:52.521 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.153881.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.120263.log 2023-11-08T05:35:52.521 INFO:teuthology.orchestra.run.smithi087.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.150305.log.gz 2023-11-08T05:35:52.521 INFO:teuthology.orchestra.run.smithi087.stderr: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.153881.log.gz 2023-11-08T05:35:52.521 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.152353.log: 1.3% -- replaced with /var/log/ceph/cluster1-client.mirror.152353.log.gz 2023-11-08T05:35:52.521 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.156740.log 2023-11-08T05:35:52.522 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.120263.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.128682.log 2023-11-08T05:35:52.522 INFO:teuthology.orchestra.run.smithi087.stderr: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.120263.log.gz 2023-11-08T05:35:52.522 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.156740.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.139872.log 2023-11-08T05:35:52.522 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.128682.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.156740.log.gz 2023-11-08T05:35:52.522 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.153544.log 2023-11-08T05:35:52.523 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.139872.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.152224.log 2023-11-08T05:35:52.523 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.153544.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.133954.log 2023-11-08T05:35:52.523 INFO:teuthology.orchestra.run.smithi087.stderr: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.153544.log.gz 2023-11-08T05:35:52.523 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.152224.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.169729.log 2023-11-08T05:35:52.523 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.133954.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.126578.log 2023-11-08T05:35:52.524 INFO:teuthology.orchestra.run.smithi087.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.152224.log.gz 2023-11-08T05:35:52.524 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.admin.169729.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.165009.log 2023-11-08T05:35:52.524 INFO:teuthology.orchestra.run.smithi087.stderr: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.139872.log.gz 2023-11-08T05:35:52.524 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.169729.log.gz 2023-11-08T05:35:52.524 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.126578.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.151438.log 2023-11-08T05:35:52.525 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.admin.165009.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.165009.log.gz 2023-11-08T05:35:52.525 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.139628.log 2023-11-08T05:35:52.525 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.132247.log 2023-11-08T05:35:52.525 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.151438.log: /var/log/ceph/cluster1-client.mirror.139628.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.150201.log 2023-11-08T05:35:52.525 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.132247.log: /var/log/ceph/cluster2-client.mirror.150201.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.139628.log.gz 2023-11-08T05:35:52.526 INFO:teuthology.orchestra.run.smithi087.stderr: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.128682.log.gz 2023-11-08T05:35:52.526 INFO:teuthology.orchestra.run.smithi087.stderr: 90.5% -- replaced with /var/log/ceph/cluster2-client.mirror.151438.log.gz 2023-11-08T05:35:52.526 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.140545.log 2023-11-08T05:35:52.526 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.149668.log 2023-11-08T05:35:52.526 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.140545.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.162845.log 2023-11-08T05:35:52.527 INFO:teuthology.orchestra.run.smithi087.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.140545.log.gz 2023-11-08T05:35:52.527 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.149668.log: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.150201.log.gz 2023-11-08T05:35:52.527 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.150367.log 2023-11-08T05:35:52.527 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.admin.162845.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.149668.log.gz 2023-11-08T05:35:52.527 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.162845.log.gz 2023-11-08T05:35:52.527 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.153634.log 2023-11-08T05:35:52.528 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.150367.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.156116.log 2023-11-08T05:35:52.528 INFO:teuthology.orchestra.run.smithi087.stderr: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.126578.log.gz 2023-11-08T05:35:52.528 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.153634.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.150367.log.gz 2023-11-08T05:35:52.528 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127073.log 2023-11-08T05:35:52.528 INFO:teuthology.orchestra.run.smithi087.stderr: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.153634.log.gz 2023-11-08T05:35:52.529 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.156116.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.156654.log 2023-11-08T05:35:52.529 INFO:teuthology.orchestra.run.smithi087.stderr: 79.7% -- replaced with /var/log/ceph/cluster2-client.mirror.156116.log.gz 2023-11-08T05:35:52.529 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.127073.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.133575.log 2023-11-08T05:35:52.529 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.156654.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127696.log 2023-11-08T05:35:52.529 INFO:teuthology.orchestra.run.smithi087.stderr: 93.2% -- replaced with /var/log/ceph/cluster1-client.mirror.127073.log.gz 2023-11-08T05:35:52.530 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.133575.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.154601.log 2023-11-08T05:35:52.530 INFO:teuthology.orchestra.run.smithi087.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.156654.log.gz 2023-11-08T05:35:52.530 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.127696.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.165989.log 2023-11-08T05:35:52.530 INFO:teuthology.orchestra.run.smithi087.stderr: 88.8% -- replaced with /var/log/ceph/cluster1-client.mirror.127696.log.gz 2023-11-08T05:35:52.530 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.154601.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.165156.log 2023-11-08T05:35:52.531 INFO:teuthology.orchestra.run.smithi087.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.154601.log.gz 2023-11-08T05:35:52.531 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.admin.165989.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.146032.log 2023-11-08T05:35:52.531 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.165989.log.gz 2023-11-08T05:35:52.531 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.129540.log 2023-11-08T05:35:52.531 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.admin.165156.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.165156.log.gz 2023-11-08T05:35:52.531 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.146032.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.139372.log 2023-11-08T05:35:52.541 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.129540.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.167418.log 2023-11-08T05:35:52.542 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.139372.log: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.146032.log.gz 2023-11-08T05:35:52.542 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.admin.167418.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.167418.log.gz 2023-11-08T05:35:52.542 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.158184.log 2023-11-08T05:35:52.542 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.116014.log 2023-11-08T05:35:52.542 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.158184.log: /var/log/ceph/cluster2-client.admin.116014.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.139372.log.gz 2023-11-08T05:35:52.543 INFO:teuthology.orchestra.run.smithi087.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.158184.log.gz 2023-11-08T05:35:52.543 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.116014.log.gz 2023-11-08T05:35:52.543 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.140003.log 2023-11-08T05:35:52.543 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.129497.log 2023-11-08T05:35:52.543 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.140003.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.139227.log 2023-11-08T05:35:52.544 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.129497.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.140003.log.gz 2023-11-08T05:35:52.544 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.139227.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.139227.log.gz 2023-11-08T05:35:52.544 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137059.log 2023-11-08T05:35:52.544 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.142463.log 2023-11-08T05:35:52.544 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.137059.log: /var/log/ceph/cluster1-client.mirror.142463.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.137059.log.gz 2023-11-08T05:35:52.545 INFO:teuthology.orchestra.run.smithi087.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.142463.log.gz 2023-11-08T05:35:52.545 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.129044.log 2023-11-08T05:35:52.545 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.128958.log 2023-11-08T05:35:52.545 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.129044.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.129540.log.gz 2023-11-08T05:35:52.545 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.128958.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.129497.log.gz 2023-11-08T05:35:52.546 INFO:teuthology.orchestra.run.smithi087.stderr: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.129044.log.gz 2023-11-08T05:35:52.546 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.168790.log 2023-11-08T05:35:52.546 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.126374.log 2023-11-08T05:35:52.546 INFO:teuthology.orchestra.run.smithi087.stderr: 93.0% -- replaced with /var/log/ceph/cluster1-client.mirror.128958.log.gz 2023-11-08T05:35:52.546 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.149482.log 2023-11-08T05:35:52.547 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.admin.168790.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.168790.log.gz 2023-11-08T05:35:52.547 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.126374.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.154579.log 2023-11-08T05:35:52.547 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.149482.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.120798.log 2023-11-08T05:35:52.547 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.154579.log: 87.9% 90.1% -- replaced with /var/log/ceph/cluster2-client.mirror.126374.log.gz -- replaced with /var/log/ceph/cluster2-client.mirror.149482.log.gz 2023-11-08T05:35:52.547 INFO:teuthology.orchestra.run.smithi087.stderr: 2023-11-08T05:35:52.548 INFO:teuthology.orchestra.run.smithi087.stderr: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.154579.log.gz 2023-11-08T05:35:52.548 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.120798.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.118957.log 2023-11-08T05:35:52.548 INFO:teuthology.orchestra.run.smithi087.stderr: 57.1% -- replaced with /var/log/ceph/cluster1-client.mirror.120798.log.gz 2023-11-08T05:35:52.548 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.152892.log 2023-11-08T05:35:52.548 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.admin.118957.log: 0.0%gzip -- replaced with /var/log/ceph/cluster2-client.admin.118957.log.gz-5 2023-11-08T05:35:52.549 INFO:teuthology.orchestra.run.smithi087.stderr:--verbose -- /var/log/ceph/cluster2-client.mirror.135137.log 2023-11-08T05:35:52.549 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.146388.log 2023-11-08T05:35:52.549 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.152892.log: /var/log/ceph/cluster2-client.mirror.135137.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.152892.log.gz 2023-11-08T05:35:52.549 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.146388.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.135137.log.gz 2023-11-08T05:35:52.549 INFO:teuthology.orchestra.run.smithi087.stderr: 93.6% -- replaced with /var/log/ceph/cluster2-client.mirror.133954.log.gz 2023-11-08T05:35:52.550 INFO:teuthology.orchestra.run.smithi087.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.146388.log.gz 2023-11-08T05:35:52.550 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.145629.log 2023-11-08T05:35:52.550 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.129302.log 2023-11-08T05:35:52.550 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.145629.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.158959.log 2023-11-08T05:35:52.550 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.129302.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.141085.log 2023-11-08T05:35:52.551 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.158959.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.119692.log 2023-11-08T05:35:52.551 INFO:teuthology.orchestra.run.smithi087.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.145629.log.gz 2023-11-08T05:35:52.551 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.141085.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.158959.log.gz 2023-11-08T05:35:52.551 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.162649.log 2023-11-08T05:35:52.551 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.admin.119692.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.119692.log.gz 2023-11-08T05:35:52.551 INFO:teuthology.orchestra.run.smithi087.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.141085.log.gz 2023-11-08T05:35:52.552 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.admin.162649.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.162649.log.gz 2023-11-08T05:35:52.552 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.151604.log 2023-11-08T05:35:52.552 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.160679.log 2023-11-08T05:35:52.552 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.151604.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.159524.log 2023-11-08T05:35:52.552 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.admin.160679.log: gzip -5 --verbose 0.0%-- /var/log/ceph/cluster1-client.mirror.136547.log -- replaced with /var/log/ceph/cluster2-client.admin.160679.log.gz 2023-11-08T05:35:52.553 INFO:teuthology.orchestra.run.smithi087.stderr: 2023-11-08T05:35:52.553 INFO:teuthology.orchestra.run.smithi087.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.151604.log.gz 2023-11-08T05:35:52.553 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.admin.159524.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.167761.log 2023-11-08T05:35:52.553 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.159524.log.gz 2023-11-08T05:35:52.553 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.136547.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.136547.log.gz 2023-11-08T05:35:52.554 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.admin.167761.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.167761.log.gz 2023-11-08T05:35:52.554 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.143527.log 2023-11-08T05:35:52.554 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.127298.log 2023-11-08T05:35:52.554 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.143527.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.170415.log 2023-11-08T05:35:52.554 INFO:teuthology.orchestra.run.smithi087.stderr: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.129302.log.gz 2023-11-08T05:35:52.555 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.127298.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.152757.log 2023-11-08T05:35:52.555 INFO:teuthology.orchestra.run.smithi087.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.143527.log.gz 2023-11-08T05:35:52.555 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.admin.170415.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.132560.log 2023-11-08T05:35:52.555 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.170415.log.gz 2023-11-08T05:35:52.555 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.152757.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126514.log 2023-11-08T05:35:52.555 INFO:teuthology.orchestra.run.smithi087.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.152757.log.gz 2023-11-08T05:35:52.556 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.132560.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.135446.log 2023-11-08T05:35:52.556 INFO:teuthology.orchestra.run.smithi087.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.132560.log.gz 2023-11-08T05:35:52.556 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.126514.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.149131.log 2023-11-08T05:35:52.556 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.135446.log: 88.1% -- replaced with /var/log/ceph/cluster1-client.mirror.126514.log.gz 2023-11-08T05:35:52.556 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.149131.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.149131.log.gz 2023-11-08T05:35:52.557 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.127337.log 2023-11-08T05:35:52.557 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.159428.log 2023-11-08T05:35:52.557 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.admin.127337.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.131643.log 2023-11-08T05:35:52.557 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.159428.log: 88.0% -- replaced with /var/log/ceph/cluster1-client.admin.127337.log.gz 2023-11-08T05:35:52.557 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.159428.log.gz 2023-11-08T05:35:52.558 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.131643.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.131643.log.gz 2023-11-08T05:35:52.558 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127624.log 2023-11-08T05:35:52.558 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.150883.log 2023-11-08T05:35:52.562 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.127624.log: /var/log/ceph/cluster2-client.mirror.150883.log: 88.8% -- replaced with /var/log/ceph/cluster1-client.mirror.127624.log.gz 2023-11-08T05:35:52.563 INFO:teuthology.orchestra.run.smithi087.stderr: 90.5% -- replaced with /var/log/ceph/cluster2-client.mirror.150883.log.gz 2023-11-08T05:35:52.563 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.145125.log 2023-11-08T05:35:52.563 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.129783.log 2023-11-08T05:35:52.563 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.145125.log: /var/log/ceph/cluster2-client.mirror.129783.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.129783.log.gz 2023-11-08T05:35:52.563 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.3.124481.log 2023-11-08T05:35:52.564 INFO:teuthology.orchestra.run.smithi087.stderr: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.145125.log.gz 2023-11-08T05:35:52.564 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.143892.log 2023-11-08T05:35:52.564 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.3.124481.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.134515.log 2023-11-08T05:35:52.566 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.143892.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.143892.log.gz 2023-11-08T05:35:52.566 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.134515.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.134515.log.gz 2023-11-08T05:35:52.566 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.116254.log 2023-11-08T05:35:52.567 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.128341.log 2023-11-08T05:35:52.567 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.admin.116254.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.116254.log.gz 2023-11-08T05:35:52.567 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.admin.128341.log: 88.2% -- replaced with /var/log/ceph/cluster1-client.admin.128341.log.gz 2023-11-08T05:35:52.567 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.132813.log 2023-11-08T05:35:52.567 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.0.131149.log 2023-11-08T05:35:52.568 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.132813.log: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.3.124481.log.gz 2023-11-08T05:35:52.568 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.0.131149.log: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.132813.log.gz 2023-11-08T05:35:52.568 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.136078.log 2023-11-08T05:35:52.568 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.142371.log 2023-11-08T05:35:52.568 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.136078.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.136078.log.gz/var/log/ceph/cluster2-client.mirror.142371.log: 2023-11-08T05:35:52.569 INFO:teuthology.orchestra.run.smithi087.stderr: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.142371.log.gz 2023-11-08T05:35:52.569 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126874.log 2023-11-08T05:35:52.569 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.141655.log 2023-11-08T05:35:52.569 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.126874.log: /var/log/ceph/cluster1-client.mirror.141655.log: 88.7% -- replaced with /var/log/ceph/cluster1-client.mirror.126874.log.gz 2023-11-08T05:35:52.569 INFO:teuthology.orchestra.run.smithi087.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.141655.log.gz 2023-11-08T05:35:52.570 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.128213.log 2023-11-08T05:35:52.570 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.146554.log 2023-11-08T05:35:52.570 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.128213.log: /var/log/ceph/cluster2-client.mirror.146554.log: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.146554.log.gz 2023-11-08T05:35:52.570 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.160140.log 2023-11-08T05:35:52.570 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.admin.160140.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.160140.log.gz 2023-11-08T05:35:52.570 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.147437.log 2023-11-08T05:35:52.571 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.147437.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.145720.log 2023-11-08T05:35:52.571 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.145720.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.145720.log.gz 2023-11-08T05:35:52.571 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.119447.log 2023-11-08T05:35:52.571 INFO:teuthology.orchestra.run.smithi087.stderr: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.147437.log.gz 2023-11-08T05:35:52.571 INFO:teuthology.orchestra.run.smithi087.stderr: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.128213.log.gz 2023-11-08T05:35:52.572 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.admin.119447.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.119447.log.gz 2023-11-08T05:35:52.572 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.162992.log 2023-11-08T05:35:52.572 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.3.120347.log 2023-11-08T05:35:52.572 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.admin.162992.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.162992.log.gz 2023-11-08T05:35:52.576 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.3.120347.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.164568.log 2023-11-08T05:35:52.576 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.admin.164568.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.164568.log.gz 2023-11-08T05:35:52.576 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125247.log 2023-11-08T05:35:52.577 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.125247.log: 92.3% -- replaced with /var/log/ceph/cluster2-client.mirror.3.120347.log.gz 2023-11-08T05:35:52.577 INFO:teuthology.orchestra.run.smithi087.stderr: 94.9% -- replaced with /var/log/ceph/cluster1-client.mirror.125247.log.gz 2023-11-08T05:35:52.577 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.141462.log 2023-11-08T05:35:52.577 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.152397.log 2023-11-08T05:35:52.577 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.141462.log: /var/log/ceph/cluster1-client.mirror.152397.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.152397.log.gz 2023-11-08T05:35:52.578 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.140051.log 2023-11-08T05:35:52.578 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.140051.log: 92.7% -- replaced with /var/log/ceph/cluster2-client.mirror.141462.log.gz 2023-11-08T05:35:52.578 INFO:teuthology.orchestra.run.smithi087.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.140051.log.gz 2023-11-08T05:35:52.578 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138858.log 2023-11-08T05:35:52.578 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.166830.log 2023-11-08T05:35:52.579 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.138858.log: /var/log/ceph/cluster2-client.admin.166830.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.166830.log.gz 2023-11-08T05:35:52.579 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.153367.log 2023-11-08T05:35:52.579 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.153367.log: 82.7% -- replaced with /var/log/ceph/cluster2-client.mirror.153367.log.gz 2023-11-08T05:35:52.579 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.133533.log 2023-11-08T05:35:52.579 INFO:teuthology.orchestra.run.smithi087.stderr: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.138858.log.gz 2023-11-08T05:35:52.580 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.133533.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.133533.log.gz 2023-11-08T05:35:52.580 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.137219.log 2023-11-08T05:35:52.580 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.145768.log 2023-11-08T05:35:52.580 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.137219.log: /var/log/ceph/cluster1-client.mirror.145768.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.137219.log.gz 2023-11-08T05:35:52.580 INFO:teuthology.orchestra.run.smithi087.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.145768.log.gz 2023-11-08T05:35:52.581 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.165646.log 2023-11-08T05:35:52.581 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.154084.log 2023-11-08T05:35:52.581 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.admin.165646.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.165646.log.gz 2023-11-08T05:35:52.581 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.154084.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.154084.log.gz 2023-11-08T05:35:52.581 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.132014.log 2023-11-08T05:35:52.581 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138065.log 2023-11-08T05:35:52.582 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.132014.log: /var/log/ceph/cluster1-client.mirror.138065.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.138065.log.gz 2023-11-08T05:35:52.582 INFO:teuthology.orchestra.run.smithi087.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.132014.log.gz 2023-11-08T05:35:52.582 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.135599.log 2023-11-08T05:35:52.582 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.151686.log 2023-11-08T05:35:52.585 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.135599.log: /var/log/ceph/cluster2-client.mirror.151686.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.135599.log.gz 2023-11-08T05:35:52.585 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.148600.log 2023-11-08T05:35:52.586 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.148600.log: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.151686.log.gz 2023-11-08T05:35:52.586 INFO:teuthology.orchestra.run.smithi087.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.148600.log.gz 2023-11-08T05:35:52.586 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.145470.log 2023-11-08T05:35:52.586 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.157875.log 2023-11-08T05:35:52.586 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.145470.log: /var/log/ceph/cluster2-client.mirror.157875.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.145470.log.gz 2023-11-08T05:35:52.587 INFO:teuthology.orchestra.run.smithi087.stderr: 80.0% -- replaced with /var/log/ceph/cluster2-client.mirror.157875.log.gz 2023-11-08T05:35:52.587 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.133004.log 2023-11-08T05:35:52.587 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.140265.log 2023-11-08T05:35:52.587 INFO:teuthology.orchestra.run.smithi087.stderr: 93.6% -- replaced with /var/log/ceph/cluster2-client.mirror.0.131149.log.gz 2023-11-08T05:35:52.587 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.133004.log: /var/log/ceph/cluster1-client.mirror.140265.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.133004.log.gz 2023-11-08T05:35:52.588 INFO:teuthology.orchestra.run.smithi087.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.140265.log.gz 2023-11-08T05:35:52.588 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.128915.log 2023-11-08T05:35:52.588 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.158694.log 2023-11-08T05:35:52.588 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.128915.log: /var/log/ceph/cluster2-client.mirror.158694.log: 91.5% -- replaced with /var/log/ceph/cluster2-client.mirror.158694.log.gz 2023-11-08T05:35:52.588 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.148713.log 2023-11-08T05:35:52.588 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.147586.log 2023-11-08T05:35:52.589 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.148713.log: /var/log/ceph/cluster1-client.mirror.147586.log: 93.0% -- replaced with /var/log/ceph/cluster1-client.mirror.128915.log.gz 2023-11-08T05:35:52.589 INFO:teuthology.orchestra.run.smithi087.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.147586.log.gz 2023-11-08T05:35:52.589 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.149358.log 2023-11-08T05:35:52.589 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.131915.log 2023-11-08T05:35:52.590 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.149358.log: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.148713.log.gz 2023-11-08T05:35:52.591 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.131915.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.149358.log.gz 2023-11-08T05:35:52.591 INFO:teuthology.orchestra.run.smithi087.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.131915.log.gz 2023-11-08T05:35:52.591 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.128150.log 2023-11-08T05:35:52.591 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.137433.log 2023-11-08T05:35:52.592 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.128150.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.162355.log 2023-11-08T05:35:52.593 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.137433.log: /var/log/ceph/cluster2-client.admin.162355.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.162355.log.gz 2023-11-08T05:35:52.593 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.146189.log 2023-11-08T05:35:52.593 INFO:teuthology.orchestra.run.smithi087.stderr: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.137433.log.gz 2023-11-08T05:35:52.593 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.165450.log 2023-11-08T05:35:52.593 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.146189.log: /var/log/ceph/cluster2-client.admin.165450.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.165450.log.gz 2023-11-08T05:35:52.594 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.127848.log 2023-11-08T05:35:52.594 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.157726.log 2023-11-08T05:35:52.594 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.127848.log: /var/log/ceph/cluster2-client.mirror.157726.log: 92.5% -- replaced with /var/log/ceph/cluster1-client.mirror.146189.log.gz 2023-11-08T05:35:52.594 INFO:teuthology.orchestra.run.smithi087.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.157726.log.gz 2023-11-08T05:35:52.594 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.158873.log 2023-11-08T05:35:52.595 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.150974.log 2023-11-08T05:35:52.595 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.158873.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.128150.log.gz 2023-11-08T05:35:52.595 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.150974.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.158873.log.gz 2023-11-08T05:35:52.595 INFO:teuthology.orchestra.run.smithi087.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.150974.log.gz 2023-11-08T05:35:52.595 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.134760.log 2023-11-08T05:35:52.595 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.131027.log 2023-11-08T05:35:52.597 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.134760.log: /var/log/ceph/cluster2-client.mirror.131027.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.134760.log.gz 2023-11-08T05:35:52.597 INFO:teuthology.orchestra.run.smithi087.stderr: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.131027.log.gz 2023-11-08T05:35:52.597 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.144740.log 2023-11-08T05:35:52.597 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.119055.log 2023-11-08T05:35:52.598 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.144740.log: /var/log/ceph/cluster2-client.admin.119055.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.119055.log.gz 2023-11-08T05:35:52.598 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.125460.log 2023-11-08T05:35:52.598 INFO:teuthology.orchestra.run.smithi087.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.144740.log.gz 2023-11-08T05:35:52.598 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.125460.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.125460.log.gz 2023-11-08T05:35:52.598 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.147758.log 2023-11-08T05:35:52.599 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.167565.log 2023-11-08T05:35:52.599 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.147758.log: /var/log/ceph/cluster2-client.admin.167565.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.147758.log.gz 2023-11-08T05:35:52.600 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.167565.log.gz 2023-11-08T05:35:52.600 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.146453.log 2023-11-08T05:35:52.600 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.1.136240.log 2023-11-08T05:35:52.600 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.146453.log: /var/log/ceph/cluster2-client.mirror.1.136240.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.146453.log.gz 2023-11-08T05:35:52.603 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.142391.log 2023-11-08T05:35:52.604 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.142391.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.142391.log.gz 2023-11-08T05:35:52.604 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.120320.log 2023-11-08T05:35:52.609 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.120320.log: 80.9% -- replaced with /var/log/ceph/cluster1-client.mirror.120320.log.gz 2023-11-08T05:35:52.609 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.149974.log 2023-11-08T05:35:52.610 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.149974.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.149974.log.gz 2023-11-08T05:35:52.610 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.158354.log 2023-11-08T05:35:52.611 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.158354.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.158354.log.gz 2023-11-08T05:35:52.611 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.125855.log 2023-11-08T05:35:52.613 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.125855.log: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.125855.log.gz 2023-11-08T05:35:52.613 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.150759.log 2023-11-08T05:35:52.615 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.150759.log: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.150759.log.gz 2023-11-08T05:35:52.615 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137083.log 2023-11-08T05:35:52.616 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.137083.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.137083.log.gz 2023-11-08T05:35:52.616 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.137104.log 2023-11-08T05:35:52.617 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.137104.log: 94.2% -- replaced with /var/log/ceph/cluster2-client.mirror.1.136240.log.gz 2023-11-08T05:35:52.618 INFO:teuthology.orchestra.run.smithi087.stderr: 92.3% -- replaced with /var/log/ceph/cluster2-client.mirror.137104.log.gz 2023-11-08T05:35:52.619 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.131578.log 2023-11-08T05:35:52.619 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.163090.log 2023-11-08T05:35:52.620 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.131578.log: /var/log/ceph/cluster2-client.admin.163090.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.131578.log.gz 2023-11-08T05:35:52.620 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.163090.log.gz 2023-11-08T05:35:52.620 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.144988.log 2023-11-08T05:35:52.620 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.143940.log 2023-11-08T05:35:52.621 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.144988.log: /var/log/ceph/cluster1-client.mirror.143940.log: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.144988.log.gz 2023-11-08T05:35:52.622 INFO:teuthology.orchestra.run.smithi087.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.143940.log.gz 2023-11-08T05:35:52.622 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.140400.log 2023-11-08T05:35:52.622 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138525.log 2023-11-08T05:35:52.623 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.140400.log: /var/log/ceph/cluster1-client.mirror.138525.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.138525.log.gz 2023-11-08T05:35:52.623 INFO:teuthology.orchestra.run.smithi087.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.140400.log.gz 2023-11-08T05:35:52.623 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.153004.log 2023-11-08T05:35:52.624 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.153004.log: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.153004.log.gz 2023-11-08T05:35:52.624 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.131284.log 2023-11-08T05:35:52.625 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.158544.log 2023-11-08T05:35:52.625 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.131284.log: 79.4% -- replaced with /var/log/ceph/cluster2-client.mirror.131284.log.gz 2023-11-08T05:35:52.625 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.158544.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.158544.log.gz 2023-11-08T05:35:52.625 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.140854.log 2023-11-08T05:35:52.626 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.159993.log 2023-11-08T05:35:52.627 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.0.140854.log: /var/log/ceph/cluster2-client.admin.159993.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.159993.log.gz 2023-11-08T05:35:52.627 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.146774.log 2023-11-08T05:35:52.627 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.146774.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.146774.log.gz 2023-11-08T05:35:52.627 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.153072.log 2023-11-08T05:35:52.628 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.153072.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.153072.log.gz 2023-11-08T05:35:52.628 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.140759.log 2023-11-08T05:35:52.629 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.140759.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.140759.log.gz 2023-11-08T05:35:52.629 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.140313.log 2023-11-08T05:35:52.629 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.140313.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.140313.log.gz 2023-11-08T05:35:52.630 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.133810.log 2023-11-08T05:35:52.630 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.133810.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.133810.log.gz 2023-11-08T05:35:52.631 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.147610.log 2023-11-08T05:35:52.631 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.147610.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.147610.log.gz 2023-11-08T05:35:52.632 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.143479.log 2023-11-08T05:35:52.632 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.143479.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.143479.log.gz 2023-11-08T05:35:52.632 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126177.log 2023-11-08T05:35:52.633 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.126177.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.126177.log.gz 2023-11-08T05:35:52.633 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.141345.log 2023-11-08T05:35:52.634 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.141345.log: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.141345.log.gz 2023-11-08T05:35:52.634 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.133633.log 2023-11-08T05:35:52.635 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.133633.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.133633.log.gz 2023-11-08T05:35:52.635 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.147875.log 2023-11-08T05:35:52.637 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.147875.log: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.147875.log.gz 2023-11-08T05:35:52.638 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.147895.log 2023-11-08T05:35:52.640 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.147895.log: 92.5% -- replaced with /var/log/ceph/cluster1-client.mirror.147895.log.gz 2023-11-08T05:35:52.640 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.165254.log 2023-11-08T05:35:52.641 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.admin.165254.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.165254.log.gz 2023-11-08T05:35:52.641 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.133046.log 2023-11-08T05:35:52.642 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.133046.log: 79.7% -- replaced with /var/log/ceph/cluster2-client.mirror.133046.log.gz 2023-11-08T05:35:52.642 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.159549.log 2023-11-08T05:35:52.642 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.admin.159549.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.159549.log.gz 2023-11-08T05:35:52.643 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.133596.log 2023-11-08T05:35:52.643 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.133596.log: 79.5% -- replaced with /var/log/ceph/cluster2-client.mirror.133596.log.gz 2023-11-08T05:35:52.644 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.156394.log 2023-11-08T05:35:52.644 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.156394.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.156394.log.gz 2023-11-08T05:35:52.644 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.142225.log 2023-11-08T05:35:52.645 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.142225.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.142225.log.gz 2023-11-08T05:35:52.645 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.168839.log 2023-11-08T05:35:52.646 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.admin.168839.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.168839.log.gz 2023-11-08T05:35:52.646 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.157661.log 2023-11-08T05:35:52.656 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.157661.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.158894.log 2023-11-08T05:35:52.664 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.158894.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.158894.log.gz 2023-11-08T05:35:52.664 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.125644.log 2023-11-08T05:35:52.670 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.125644.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.125644.log.gz 2023-11-08T05:35:52.670 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.0.135298.log 2023-11-08T05:35:52.676 INFO:teuthology.orchestra.run.smithi077.stderr: 91.3% -- replaced with /var/log/ceph/cluster2-mgr.x.log.gz 2023-11-08T05:35:52.681 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.0.135298.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.136735.log 2023-11-08T05:35:52.692 INFO:teuthology.orchestra.run.smithi087.stderr: 92.7% -- replaced with /var/log/ceph/cluster1-client.mirror.0.140854.log.gz 2023-11-08T05:35:52.703 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.136735.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.141153.log 2023-11-08T05:35:52.703 INFO:teuthology.orchestra.run.smithi087.stderr: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.136735.log.gz 2023-11-08T05:35:52.703 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.141153.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.141153.log.gz 2023-11-08T05:35:52.704 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.132495.log 2023-11-08T05:35:52.704 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.132495.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.123057.log 2023-11-08T05:35:52.718 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.admin.123057.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.128033.log 2023-11-08T05:35:52.718 INFO:teuthology.orchestra.run.smithi087.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.123057.log.gz 2023-11-08T05:35:52.719 INFO:teuthology.orchestra.run.smithi087.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.127298.log.gz 2023-11-08T05:35:52.719 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.128033.log: 89.0% -- replaced with /var/log/ceph/cluster1-client.mirror.128033.log.gz 2023-11-08T05:35:52.719 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.144864.log 2023-11-08T05:35:52.719 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.144864.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.144864.log.gz 2023-11-08T05:35:52.719 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.130083.log 2023-11-08T05:35:52.725 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.130083.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.130083.log.gz 2023-11-08T05:35:52.726 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.168349.log 2023-11-08T05:35:52.726 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.admin.168349.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.168349.log.gz 2023-11-08T05:35:52.726 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.129067.log 2023-11-08T05:35:52.726 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.129067.log: 94.2% -- replaced with /var/log/ceph/cluster2-client.mirror.0.135298.log.gz 2023-11-08T05:35:52.727 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137323.log 2023-11-08T05:35:52.727 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.137323.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.137323.log.gz 2023-11-08T05:35:52.727 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.123281.log 2023-11-08T05:35:52.727 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.admin.123281.log: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.129067.log.gz 2023-11-08T05:35:52.727 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.156133.log 2023-11-08T05:35:52.728 INFO:teuthology.orchestra.run.smithi087.stderr: 87.2% -- replaced with /var/log/ceph/cluster1-client.admin.123281.log.gz 2023-11-08T05:35:52.728 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.156133.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.131742.log 2023-11-08T05:35:52.729 INFO:teuthology.orchestra.run.smithi087.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.156133.log.gz 2023-11-08T05:35:52.729 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138745.log 2023-11-08T05:35:52.729 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.131742.log: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.131742.log.gz 2023-11-08T05:35:52.729 INFO:teuthology.orchestra.run.smithi087.stderr:gzip /var/log/ceph/cluster1-client.mirror.138745.log: -5 --verbose -- /var/log/ceph/cluster2-client.mirror.149752.log 2023-11-08T05:35:52.730 INFO:teuthology.orchestra.run.smithi087.stderr: 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.138745.log.gz 2023-11-08T05:35:52.730 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.150098.log 2023-11-08T05:35:52.730 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.149752.log: /var/log/ceph/cluster2-client.mirror.150098.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.150098.log.gz 2023-11-08T05:35:52.730 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.141821.log 2023-11-08T05:35:52.731 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.141821.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.141821.log.gz 2023-11-08T05:35:52.731 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.140197.log 2023-11-08T05:35:52.732 INFO:teuthology.orchestra.run.smithi087.stderr: 91.7% -- replaced with /var/log/ceph/cluster2-client.mirror.149752.log.gz 2023-11-08T05:35:52.732 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.153589.log 2023-11-08T05:35:52.732 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.140197.log: /var/log/ceph/cluster1-client.mirror.153589.log: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.140197.log.gz 2023-11-08T05:35:52.733 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.129983.log 2023-11-08T05:35:52.733 INFO:teuthology.orchestra.run.smithi087.stderr: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.153589.log.gz 2023-11-08T05:35:52.733 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.138838.log 2023-11-08T05:35:52.734 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.129983.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.129983.log.gz 2023-11-08T05:35:52.734 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.138838.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.158829.log 2023-11-08T05:35:52.735 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.158829.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.158829.log.gz 2023-11-08T05:35:52.735 INFO:teuthology.orchestra.run.smithi087.stderr: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.138838.log.gz 2023-11-08T05:35:52.736 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.129883.log 2023-11-08T05:35:52.736 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.150594.log 2023-11-08T05:35:52.737 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.129883.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.129883.log.gz 2023-11-08T05:35:52.737 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.150594.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.167369.log 2023-11-08T05:35:52.737 INFO:teuthology.orchestra.run.smithi087.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.150594.log.gz 2023-11-08T05:35:52.738 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.136619.log 2023-11-08T05:35:52.738 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.admin.167369.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.167369.log.gz 2023-11-08T05:35:52.738 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.156500.log 2023-11-08T05:35:52.739 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.136619.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.136619.log.gz 2023-11-08T05:35:52.739 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.123337.log 2023-11-08T05:35:52.739 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.156500.log: 79.7% -- replaced with /var/log/ceph/cluster2-client.mirror.156500.log.gz 2023-11-08T05:35:52.739 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.146148.log 2023-11-08T05:35:52.740 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.123337.log: /var/log/ceph/cluster1-client.mirror.146148.log: 90.6% -- replaced with /var/log/ceph/cluster2-client.mirror.123337.log.gz 2023-11-08T05:35:52.740 INFO:teuthology.orchestra.run.smithi087.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.146148.log.gz 2023-11-08T05:35:52.740 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.152667.log 2023-11-08T05:35:52.741 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.129740.log 2023-11-08T05:35:52.741 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.152667.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.152667.log.gz 2023-11-08T05:35:52.741 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.129740.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.148624.log 2023-11-08T05:35:52.742 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.148624.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.148624.log.gz 2023-11-08T05:35:52.742 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.153139.log 2023-11-08T05:35:52.743 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.153139.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.153139.log.gz 2023-11-08T05:35:52.744 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.169288.log 2023-11-08T05:35:52.744 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.admin.169288.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.169288.log.gz 2023-11-08T05:35:52.745 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.151459.log 2023-11-08T05:35:52.746 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.151459.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.151459.log.gz 2023-11-08T05:35:52.746 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.153252.log 2023-11-08T05:35:52.746 INFO:teuthology.orchestra.run.smithi087.stderr: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.129740.log.gz 2023-11-08T05:35:52.747 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.142312.log 2023-11-08T05:35:52.747 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.153252.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.153252.log.gz 2023-11-08T05:35:52.748 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.142312.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.153184.log 2023-11-08T05:35:52.748 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.153184.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.142312.log.gz 2023-11-08T05:35:52.748 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.1.124002.log 2023-11-08T05:35:52.748 INFO:teuthology.orchestra.run.smithi087.stderr: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.153184.log.gz 2023-11-08T05:35:52.750 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.1.124002.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.129325.log 2023-11-08T05:35:52.754 INFO:teuthology.orchestra.run.smithi077.stderr: 91.8% -- replaced with /var/log/ceph/cluster1-mgr.x.log.gz 2023-11-08T05:35:52.754 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.129325.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.121252.log 2023-11-08T05:35:52.756 INFO:teuthology.orchestra.run.smithi087.stderr: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.129325.log.gz 2023-11-08T05:35:52.757 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.121252.log: 2.3% -- replaced with /var/log/ceph/cluster2-client.mirror.121252.log.gz 2023-11-08T05:35:52.757 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.132967.log 2023-11-08T05:35:52.758 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.132967.log: 79.7% -- replaced with /var/log/ceph/cluster2-client.mirror.132967.log.gz 2023-11-08T05:35:52.758 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.115967.log 2023-11-08T05:35:52.759 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.admin.115967.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.115967.log.gz 2023-11-08T05:35:52.759 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.129388.log 2023-11-08T05:35:52.765 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.129388.log: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.129388.log.gz 2023-11-08T05:35:52.765 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.167173.log 2023-11-08T05:35:52.766 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.admin.167173.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.167173.log.gz 2023-11-08T05:35:52.766 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.154773.log 2023-11-08T05:35:52.769 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.154773.log: 79.6% -- replaced with /var/log/ceph/cluster2-client.mirror.154773.log.gz 2023-11-08T05:35:52.769 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.154466.log 2023-11-08T05:35:52.770 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.154466.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.154466.log.gz 2023-11-08T05:35:52.770 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.139955.log 2023-11-08T05:35:52.770 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.139955.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.139955.log.gz 2023-11-08T05:35:52.771 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.157573.log 2023-11-08T05:35:52.773 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.157573.log: 91.5% -- replaced with /var/log/ceph/cluster2-client.mirror.157573.log.gz 2023-11-08T05:35:52.773 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.149883.log 2023-11-08T05:35:52.775 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.149883.log: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.149883.log.gz 2023-11-08T05:35:52.776 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.129153.log 2023-11-08T05:35:52.781 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.129153.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.129153.log.gz 2023-11-08T05:35:52.782 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125484.log 2023-11-08T05:35:52.783 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.125484.log: 88.0% -- replaced with /var/log/ceph/cluster1-client.mirror.125484.log.gz 2023-11-08T05:35:52.783 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.135188.log 2023-11-08T05:35:52.784 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.135188.log: 79.6% -- replaced with /var/log/ceph/cluster2-client.mirror.135188.log.gz 2023-11-08T05:35:52.784 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.159059.log 2023-11-08T05:35:52.785 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.159059.log: 92.5% -- replaced with /var/log/ceph/cluster1-client.mirror.1.124002.log.gz 2023-11-08T05:35:52.785 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.135100.log 2023-11-08T05:35:52.786 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.135100.log: 79.4% -- replaced with /var/log/ceph/cluster2-client.mirror.135100.log.gz 2023-11-08T05:35:52.786 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.129173.log 2023-11-08T05:35:52.792 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.129173.log: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.129173.log.gz 2023-11-08T05:35:52.792 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.166136.log 2023-11-08T05:35:52.796 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.admin.166136.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.166136.log.gz 2023-11-08T05:35:52.796 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.161914.log 2023-11-08T05:35:52.797 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.admin.161914.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.161914.log.gz 2023-11-08T05:35:52.797 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.150222.log 2023-11-08T05:35:52.798 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.150222.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.150222.log.gz 2023-11-08T05:35:52.798 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.151036.log 2023-11-08T05:35:52.799 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.151036.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.151036.log.gz 2023-11-08T05:35:52.800 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.152532.log 2023-11-08T05:35:52.801 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.152532.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.152532.log.gz 2023-11-08T05:35:52.801 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.144064.log 2023-11-08T05:35:52.802 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.144064.log: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.144064.log.gz 2023-11-08T05:35:52.802 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.158917.log 2023-11-08T05:35:52.803 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.158917.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.158917.log.gz 2023-11-08T05:35:52.803 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.150801.log 2023-11-08T05:35:52.804 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.150801.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.150801.log.gz 2023-11-08T05:35:52.804 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.124923.log 2023-11-08T05:35:52.805 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.admin.124923.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.admin.124923.log.gz 2023-11-08T05:35:52.806 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.139447.log 2023-11-08T05:35:52.807 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.139447.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.139447.log.gz 2023-11-08T05:35:52.807 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.159281.log 2023-11-08T05:35:52.808 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.159281.log: 82.2% -- replaced with /var/log/ceph/cluster2-client.mirror.159281.log.gz 2023-11-08T05:35:52.808 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126036.log 2023-11-08T05:35:52.815 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.126036.log: 93.6% -- replaced with /var/log/ceph/cluster1-client.mirror.126036.log.gz 2023-11-08T05:35:52.815 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.149523.log 2023-11-08T05:35:52.817 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.149523.log: 90.5% -- replaced with /var/log/ceph/cluster2-client.mirror.149523.log.gz 2023-11-08T05:35:52.818 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.1.158247.log 2023-11-08T05:35:52.821 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.1.158247.log: 92.8% -- replaced with /var/log/ceph/cluster2-client.mirror.1.158247.log.gz 2023-11-08T05:35:52.821 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.147181.log 2023-11-08T05:35:52.823 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.147181.log: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.147181.log.gz 2023-11-08T05:35:52.824 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.168937.log 2023-11-08T05:35:52.824 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.admin.168937.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.168937.log.gz 2023-11-08T05:35:52.824 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.164225.log 2023-11-08T05:35:52.825 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.admin.164225.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.164225.log.gz 2023-11-08T05:35:52.826 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.169133.log 2023-11-08T05:35:52.826 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.admin.169133.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.169133.log.gz 2023-11-08T05:35:52.826 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.150739.log 2023-11-08T05:35:52.827 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.150739.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.150739.log.gz 2023-11-08T05:35:52.828 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.157194.log 2023-11-08T05:35:52.829 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.157194.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.157194.log.gz 2023-11-08T05:35:52.829 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.154810.log 2023-11-08T05:35:52.830 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.154810.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.154810.log.gz 2023-11-08T05:35:52.830 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.168153.log 2023-11-08T05:35:52.831 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.admin.168153.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.168153.log.gz 2023-11-08T05:35:52.831 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.143455.log 2023-11-08T05:35:52.832 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.143455.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.143455.log.gz 2023-11-08T05:35:52.833 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.132639.log 2023-11-08T05:35:52.834 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.132639.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.132639.log.gz 2023-11-08T05:35:52.834 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.145928.log 2023-11-08T05:35:52.835 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.145928.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.133575.log.gz 2023-11-08T05:35:52.835 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.157705.log 2023-11-08T05:35:52.835 INFO:teuthology.orchestra.run.smithi087.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.145928.log.gz 2023-11-08T05:35:52.836 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.148467.log 2023-11-08T05:35:52.836 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.157705.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.157705.log.gz 2023-11-08T05:35:52.838 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.152734.log 2023-11-08T05:35:52.838 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.148467.log: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.148467.log.gz 2023-11-08T05:35:52.839 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.167663.log 2023-11-08T05:35:52.839 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.152734.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.152734.log.gz 2023-11-08T05:35:52.839 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.150057.log 2023-11-08T05:35:52.840 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.admin.167663.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.167663.log.gz 2023-11-08T05:35:52.840 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.146364.log 2023-11-08T05:35:52.840 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.150057.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.150057.log.gz 2023-11-08T05:35:52.841 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.136010.log 2023-11-08T05:35:52.841 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.146364.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.146364.log.gz 2023-11-08T05:35:52.843 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.136010.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.136010.log.gz 2023-11-08T05:35:52.843 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.153656.log 2023-11-08T05:35:52.843 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.153656.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.153656.log.gz 2023-11-08T05:35:52.844 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.129803.log 2023-11-08T05:35:52.844 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.129803.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.129803.log.gz 2023-11-08T05:35:52.845 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.150995.log 2023-11-08T05:35:52.845 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.150995.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.150995.log.gz 2023-11-08T05:35:52.846 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.153049.log 2023-11-08T05:35:52.846 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.153049.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.153049.log.gz 2023-11-08T05:35:52.847 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.150387.log 2023-11-08T05:35:52.849 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.150387.log: 90.5% -- replaced with /var/log/ceph/cluster2-client.mirror.150387.log.gz 2023-11-08T05:35:52.849 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.156559.log 2023-11-08T05:35:52.855 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.156559.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.128639.log 2023-11-08T05:35:52.872 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.128639.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.128129.log 2023-11-08T05:35:52.872 INFO:teuthology.orchestra.run.smithi087.stderr: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.128639.log.gz 2023-11-08T05:35:52.880 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.128129.log: 88.4% -- replaced with /var/log/ceph/cluster1-client.mirror.128129.log.gz 2023-11-08T05:35:52.888 INFO:teuthology.orchestra.run.smithi087.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.157661.log.gz 2023-11-08T05:35:52.889 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.147538.log 2023-11-08T05:35:52.889 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.139331.log 2023-11-08T05:35:52.891 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.147538.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.147538.log.gz 2023-11-08T05:35:52.891 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.139331.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.139331.log.gz 2023-11-08T05:35:52.891 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.145084.log 2023-11-08T05:35:52.892 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.145084.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.145084.log.gz 2023-11-08T05:35:52.892 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.121269.log 2023-11-08T05:35:52.893 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.121269.log: 57.1% -- replaced with /var/log/ceph/cluster2-client.mirror.121269.log.gz 2023-11-08T05:35:52.893 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.164078.log 2023-11-08T05:35:52.894 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.admin.164078.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.164078.log.gz 2023-11-08T05:35:52.894 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.153679.log 2023-11-08T05:35:52.895 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.153679.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.153679.log.gz 2023-11-08T05:35:52.895 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.145214.log 2023-11-08T05:35:52.896 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.145214.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.145214.log.gz 2023-11-08T05:35:52.896 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.159124.log 2023-11-08T05:35:52.897 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.159124.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.159124.log.gz 2023-11-08T05:35:52.897 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.120815.log 2023-11-08T05:35:52.898 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.120815.log: 53.4% -- replaced with /var/log/ceph/cluster1-client.mirror.120815.log.gz 2023-11-08T05:35:52.898 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.128371.log 2023-11-08T05:35:52.898 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.128371.log: 87.9% -- replaced with /var/log/ceph/cluster1-client.mirror.128371.log.gz 2023-11-08T05:35:52.899 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.146124.log 2023-11-08T05:35:52.899 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.146124.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.146124.log.gz 2023-11-08T05:35:52.900 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.128574.log 2023-11-08T05:35:52.900 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.128574.log: 89.2% -- replaced with /var/log/ceph/cluster1-client.mirror.128574.log.gz 2023-11-08T05:35:52.901 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.129823.log 2023-11-08T05:35:52.901 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.129823.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.129823.log.gz 2023-11-08T05:35:52.902 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.142641.log 2023-11-08T05:35:52.902 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.142641.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.142641.log.gz 2023-11-08T05:35:52.903 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.133613.log 2023-11-08T05:35:52.904 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.133613.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.133613.log.gz 2023-11-08T05:35:52.904 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.138677.log 2023-11-08T05:35:52.905 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.138677.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.138677.log.gz 2023-11-08T05:35:52.905 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.152038.log 2023-11-08T05:35:52.906 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.152038.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.152038.log.gz 2023-11-08T05:35:52.906 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.144314.log 2023-11-08T05:35:52.907 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.144314.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.144314.log.gz 2023-11-08T05:35:52.907 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.170317.log 2023-11-08T05:35:52.908 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.admin.170317.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.170317.log.gz 2023-11-08T05:35:52.908 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.119937.log 2023-11-08T05:35:52.909 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.admin.119937.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.119937.log.gz 2023-11-08T05:35:52.909 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.132878.log 2023-11-08T05:35:52.911 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.132878.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.132247.log.gz 2023-11-08T05:35:52.911 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.135387.log 2023-11-08T05:35:52.912 INFO:teuthology.orchestra.run.smithi087.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.135446.log.gz 2023-11-08T05:35:52.912 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.135387.log: 79.7% -- replaced with /var/log/ceph/cluster2-client.mirror.135387.log.gz 2023-11-08T05:35:52.912 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.3.143290.log 2023-11-08T05:35:52.912 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.160336.log 2023-11-08T05:35:52.914 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.3.143290.log: 95.4% -- replaced with /var/log/ceph/cluster1-client.mirror.132878.log.gz 2023-11-08T05:35:52.915 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.admin.160336.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.144784.log 2023-11-08T05:35:52.915 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.160336.log.gz 2023-11-08T05:35:52.915 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.124707.log 2023-11-08T05:35:52.915 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.144784.log: /var/log/ceph/cluster1-client.mirror.124707.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.144784.log.gz 2023-11-08T05:35:52.915 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.119839.log 2023-11-08T05:35:52.916 INFO:teuthology.orchestra.run.smithi087.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.124707.log.gz 2023-11-08T05:35:52.916 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.157532.log 2023-11-08T05:35:52.916 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.admin.119839.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.119839.log.gz 2023-11-08T05:35:52.916 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.126745.log 2023-11-08T05:35:52.918 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.157532.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.157532.log.gz 2023-11-08T05:35:52.919 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.124187.log 2023-11-08T05:35:52.919 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.126745.log: /var/log/ceph/cluster1-client.mirror.124187.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.124187.log.gz 2023-11-08T05:35:52.919 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.153769.log 2023-11-08T05:35:52.919 INFO:teuthology.orchestra.run.smithi087.stderr: 91.7% -- replaced with /var/log/ceph/cluster2-client.mirror.126745.log.gz 2023-11-08T05:35:52.919 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.149606.log 2023-11-08T05:35:52.920 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.153769.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.153769.log.gz 2023-11-08T05:35:52.920 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.151562.log 2023-11-08T05:35:52.920 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.149606.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.149606.log.gz 2023-11-08T05:35:52.920 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.151562.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.151728.log 2023-11-08T05:35:52.920 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.151728.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.151728.log.gz 2023-11-08T05:35:52.920 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.134102.log 2023-11-08T05:35:52.921 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.134102.log: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.151562.log.gz 2023-11-08T05:35:52.921 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.131422.log 2023-11-08T05:35:52.922 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.131422.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.127848.log.gz 2023-11-08T05:35:52.924 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.150842.log 2023-11-08T05:35:52.925 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.150842.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.150842.log.gz 2023-11-08T05:35:52.925 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.142734.log 2023-11-08T05:35:52.927 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.142734.log: 93.3% -- replaced with /var/log/ceph/cluster1-client.mirror.3.143290.log.gz 2023-11-08T05:35:52.927 INFO:teuthology.orchestra.run.smithi087.stderr: 92.0% -- replaced with /var/log/ceph/cluster2-client.mirror.142734.log.gz 2023-11-08T05:35:52.928 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137243.log 2023-11-08T05:35:52.929 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.127872.log 2023-11-08T05:35:52.929 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.137243.log: /var/log/ceph/cluster1-client.admin.127872.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.137243.log.gz 2023-11-08T05:35:52.929 INFO:teuthology.orchestra.run.smithi087.stderr: 88.1% -- replaced with /var/log/ceph/cluster1-client.admin.127872.log.gz 2023-11-08T05:35:52.929 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.144502.log 2023-11-08T05:35:52.930 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.133731.log 2023-11-08T05:35:52.931 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.144502.log: /var/log/ceph/cluster2-client.mirror.133731.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.144502.log.gz 2023-11-08T05:35:52.931 INFO:teuthology.orchestra.run.smithi087.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.133731.log.gz 2023-11-08T05:35:52.931 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.124481.log 2023-11-08T05:35:52.931 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.151480.log 2023-11-08T05:35:52.932 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.124481.log: 81.7% -- replaced with /var/log/ceph/cluster1-client.mirror.124481.log.gz 2023-11-08T05:35:52.933 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.151480.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.151480.log.gz 2023-11-08T05:35:52.933 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.138017.log 2023-11-08T05:35:52.934 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.151057.log 2023-11-08T05:35:52.934 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.138017.log: /var/log/ceph/cluster2-client.mirror.151057.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.138017.log.gz 2023-11-08T05:35:52.934 INFO:teuthology.orchestra.run.smithi087.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.151057.log.gz 2023-11-08T05:35:52.934 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.130023.log 2023-11-08T05:35:52.934 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.143204.log 2023-11-08T05:35:52.935 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.130023.log: /var/log/ceph/cluster2-client.mirror.143204.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.130023.log.gz 2023-11-08T05:35:52.936 INFO:teuthology.orchestra.run.smithi087.stderr: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.143204.log.gz 2023-11-08T05:35:52.936 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.159453.log 2023-11-08T05:35:52.936 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.146292.log 2023-11-08T05:35:52.936 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.159453.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.159453.log.gz 2023-11-08T05:35:52.937 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.146292.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.146292.log.gz 2023-11-08T05:35:52.938 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.156675.log 2023-11-08T05:35:52.938 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.144362.log 2023-11-08T05:35:52.938 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.156675.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.156675.log.gz 2023-11-08T05:35:52.938 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.144362.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.144362.log.gz 2023-11-08T05:35:52.939 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.138921.log 2023-11-08T05:35:52.939 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.144181.log 2023-11-08T05:35:52.939 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.138921.log: /var/log/ceph/cluster2-client.mirror.144181.log: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.138921.log.gz 2023-11-08T05:35:52.940 INFO:teuthology.orchestra.run.smithi087.stderr: 92.2% -- replaced with /var/log/ceph/cluster2-client.mirror.144181.log.gz 2023-11-08T05:35:52.940 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.137146.log 2023-11-08T05:35:52.941 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.153611.log 2023-11-08T05:35:52.941 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.137146.log: /var/log/ceph/cluster1-client.mirror.153611.log: 89.7% 86.8% -- replaced with /var/log/ceph/cluster2-client.mirror.137146.log.gz -- replaced with /var/log/ceph/cluster1-client.mirror.153611.log.gz 2023-11-08T05:35:52.942 INFO:teuthology.orchestra.run.smithi087.stderr: 2023-11-08T05:35:52.942 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.135834.log 2023-11-08T05:35:52.943 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.132473.log 2023-11-08T05:35:52.943 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.135834.log: 1.3% -- replaced with /var/log/ceph/cluster1-client.mirror.135834.log.gz 2023-11-08T05:35:52.943 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.132473.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.132473.log.gz 2023-11-08T05:35:52.943 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.147300.log 2023-11-08T05:35:52.944 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.138636.log 2023-11-08T05:35:52.944 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.147300.log: /var/log/ceph/cluster2-client.mirror.138636.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.147300.log.gz 2023-11-08T05:35:52.945 INFO:teuthology.orchestra.run.smithi087.stderr: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.138636.log.gz 2023-11-08T05:35:52.945 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.135404.log 2023-11-08T05:35:52.945 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.138134.log 2023-11-08T05:35:52.946 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.135404.log: /var/log/ceph/cluster2-client.mirror.138134.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.135404.log.gz 2023-11-08T05:35:52.946 INFO:teuthology.orchestra.run.smithi087.stderr: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.138134.log.gz 2023-11-08T05:35:52.947 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.150470.log 2023-11-08T05:35:52.947 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.135298.log 2023-11-08T05:35:52.947 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.150470.log: /var/log/ceph/cluster1-client.mirror.135298.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.150470.log.gz 2023-11-08T05:35:52.950 INFO:teuthology.orchestra.run.smithi087.stderr: 95.3% -- replaced with /var/log/ceph/cluster1-client.mirror.135298.log.gz 2023-11-08T05:35:52.950 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.170366.log 2023-11-08T05:35:52.951 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.129411.log 2023-11-08T05:35:52.951 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.admin.170366.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.170366.log.gz 2023-11-08T05:35:52.953 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.129411.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.120088.log 2023-11-08T05:35:52.954 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.0.120088.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.149565.log 2023-11-08T05:35:52.956 INFO:teuthology.orchestra.run.smithi087.stderr: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.129411.log.gz 2023-11-08T05:35:52.956 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.149565.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.149565.log.gz 2023-11-08T05:35:52.957 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.124211.log 2023-11-08T05:35:52.958 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.124211.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.124211.log.gz 2023-11-08T05:35:52.958 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.125440.log 2023-11-08T05:35:52.959 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.125440.log: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.125440.log.gz 2023-11-08T05:35:52.959 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.134557.log 2023-11-08T05:35:52.969 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.134557.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.136872.log 2023-11-08T05:35:52.971 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.136872.log: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.136872.log.gz 2023-11-08T05:35:52.978 INFO:teuthology.orchestra.run.smithi087.stderr: 94.1% -- replaced with /var/log/ceph/cluster1-client.mirror.0.120088.log.gz 2023-11-08T05:35:52.980 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.150573.log 2023-11-08T05:35:52.980 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.119888.log 2023-11-08T05:35:52.981 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.150573.log: /var/log/ceph/cluster2-client.admin.119888.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.119888.log.gz 2023-11-08T05:35:52.982 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.150139.log 2023-11-08T05:35:52.983 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.150139.log: 90.5% -- replaced with /var/log/ceph/cluster2-client.mirror.150573.log.gz 2023-11-08T05:35:52.983 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.159153.log 2023-11-08T05:35:52.984 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.159153.log: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.159153.log.gz 2023-11-08T05:35:52.984 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.119545.log 2023-11-08T05:35:52.984 INFO:teuthology.orchestra.run.smithi087.stderr: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.150139.log.gz 2023-11-08T05:35:52.985 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.3.135900.log 2023-11-08T05:35:52.985 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.admin.119545.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.119545.log.gz 2023-11-08T05:35:52.986 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.3.127207.log 2023-11-08T05:35:52.990 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.3.135900.log: /var/log/ceph/cluster1-client.mirror.3.127207.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.132495.log.gz 2023-11-08T05:35:52.990 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.156952.log 2023-11-08T05:35:53.001 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.156952.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.142290.log 2023-11-08T05:35:53.003 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.142290.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.150904.log 2023-11-08T05:35:53.003 INFO:teuthology.orchestra.run.smithi087.stderr: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.142290.log.gz 2023-11-08T05:35:53.025 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.150904.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.150904.log.gz 2023-11-08T05:35:53.025 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.142927.log 2023-11-08T05:35:53.037 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.142927.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.142927.log.gz 2023-11-08T05:35:53.037 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.143224.log 2023-11-08T05:35:53.052 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.143224.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.149213.log 2023-11-08T05:35:53.052 INFO:teuthology.orchestra.run.smithi087.stderr: 92.6% -- replaced with /var/log/ceph/cluster1-client.mirror.143224.log.gz 2023-11-08T05:35:53.063 INFO:teuthology.orchestra.run.smithi087.stderr: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.3.135900.log.gz 2023-11-08T05:35:53.063 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.149213.log: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.149213.log.gz 2023-11-08T05:35:53.063 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.141607.log 2023-11-08T05:35:53.063 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.141607.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.141607.log.gz 2023-11-08T05:35:53.064 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.144386.log 2023-11-08T05:35:53.064 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.144386.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.144386.log.gz 2023-11-08T05:35:53.064 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.145815.log 2023-11-08T05:35:53.064 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.145815.log: 92.9% -- replaced with /var/log/ceph/cluster1-client.mirror.3.127207.log.gz 2023-11-08T05:35:53.064 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.123083.log 2023-11-08T05:35:53.065 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.admin.123083.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.admin.123083.log.gz 2023-11-08T05:35:53.065 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.133711.log 2023-11-08T05:35:53.074 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.133711.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.133711.log.gz 2023-11-08T05:35:53.074 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.131722.log 2023-11-08T05:35:53.074 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.131722.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.131722.log.gz 2023-11-08T05:35:53.074 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.3.120201.log 2023-11-08T05:35:53.085 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.3.120201.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.132056.log 2023-11-08T05:35:53.096 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.132056.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.148487.log 2023-11-08T05:35:53.107 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.148487.log: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.148487.log.gz 2023-11-08T05:35:53.107 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.128057.log 2023-11-08T05:35:53.117 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.128057.log: 89.1% -- replaced with /var/log/ceph/cluster1-client.mirror.128057.log.gz 2023-11-08T05:35:53.117 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.142689.log 2023-11-08T05:35:53.128 INFO:teuthology.orchestra.run.smithi087.stderr: 94.3% -- replaced with /var/log/ceph/cluster2-client.mirror.145815.log.gz 2023-11-08T05:35:53.128 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.142689.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.142689.log.gz 2023-11-08T05:35:53.129 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.147372.log 2023-11-08T05:35:53.139 INFO:teuthology.orchestra.run.smithi087.stderr: 94.3% -- replaced with /var/log/ceph/cluster1-client.mirror.3.120201.log.gz 2023-11-08T05:35:53.139 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.147372.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.147372.log.gz 2023-11-08T05:35:53.139 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.132756.log 2023-11-08T05:35:53.150 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.132756.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.153904.log 2023-11-08T05:35:53.154 INFO:teuthology.orchestra.run.smithi087.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.156559.log.gz 2023-11-08T05:35:53.154 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.153904.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.153904.log.gz 2023-11-08T05:35:53.154 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.144840.log 2023-11-08T05:35:53.155 INFO:teuthology.orchestra.run.smithi087.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.134102.log.gz 2023-11-08T05:35:53.166 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.145518.log 2023-11-08T05:35:53.168 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.144840.log: /var/log/ceph/cluster1-client.mirror.145518.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.144840.log.gz 2023-11-08T05:35:53.168 INFO:teuthology.orchestra.run.smithi087.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.145518.log.gz 2023-11-08T05:35:53.168 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.120696.log 2023-11-08T05:35:53.169 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.124591.log 2023-11-08T05:35:53.169 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.admin.120696.log: 79.9% -- replaced with /var/log/ceph/cluster1-client.admin.120696.log.gz 2023-11-08T05:35:53.169 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.124591.log: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.124591.log.gz 2023-11-08T05:35:53.170 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.161030.log 2023-11-08T05:35:53.170 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.153498.log 2023-11-08T05:35:53.171 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.admin.161030.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.161030.log.gz 2023-11-08T05:35:53.171 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.153498.log: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.153498.log.gz 2023-11-08T05:35:53.171 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.136571.log 2023-11-08T05:35:53.172 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.141242.log 2023-11-08T05:35:53.172 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.136571.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.136571.log.gz 2023-11-08T05:35:53.174 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.141242.log: 92.1% -- replaced with /var/log/ceph/cluster1-client.mirror.141242.log.gz 2023-11-08T05:35:53.174 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.145587.log 2023-11-08T05:35:53.178 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.147161.log 2023-11-08T05:35:53.178 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.145587.log: /var/log/ceph/cluster2-client.mirror.147161.log: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.145587.log.gz 2023-11-08T05:35:53.178 INFO:teuthology.orchestra.run.smithi087.stderr: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.147161.log.gz 2023-11-08T05:35:53.178 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.160532.log 2023-11-08T05:35:53.179 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.149585.log 2023-11-08T05:35:53.179 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.admin.160532.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.160532.log.gz 2023-11-08T05:35:53.180 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.149585.log: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.149585.log.gz 2023-11-08T05:35:53.180 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.134382.log 2023-11-08T05:35:53.187 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.134382.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.152712.log 2023-11-08T05:35:53.193 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.152712.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.152712.log.gz 2023-11-08T05:35:53.198 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.149503.log 2023-11-08T05:35:53.199 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.149503.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.149503.log.gz 2023-11-08T05:35:53.209 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.159103.log 2023-11-08T05:35:53.220 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.159103.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.142439.log 2023-11-08T05:35:53.221 INFO:teuthology.orchestra.run.smithi087.stderr: 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.159103.log.gz 2023-11-08T05:35:53.229 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.142439.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.142439.log.gz 2023-11-08T05:35:53.229 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.168986.log 2023-11-08T05:35:53.230 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.admin.168986.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.168986.log.gz 2023-11-08T05:35:53.230 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.150181.log 2023-11-08T05:35:53.231 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.150181.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.150181.log.gz 2023-11-08T05:35:53.232 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.166234.log 2023-11-08T05:35:53.232 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.admin.166234.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.166234.log.gz 2023-11-08T05:35:53.232 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.143868.log 2023-11-08T05:35:53.234 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.143868.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.143868.log.gz 2023-11-08T05:35:53.236 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.124438.log 2023-11-08T05:35:53.237 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.124438.log: 91.6% -- replaced with /var/log/ceph/cluster2-client.mirror.124438.log.gz 2023-11-08T05:35:53.238 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.154151.log 2023-11-08T05:35:53.238 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.154151.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.159059.log.gz 2023-11-08T05:35:53.238 INFO:teuthology.orchestra.run.smithi087.stderr: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.154151.log.gz 2023-11-08T05:35:53.238 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.133262.log 2023-11-08T05:35:53.238 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.162110.log 2023-11-08T05:35:53.239 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.133262.log: /var/log/ceph/cluster2-client.admin.162110.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.162110.log.gz 2023-11-08T05:35:53.239 INFO:teuthology.orchestra.run.smithi087.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.133262.log.gz 2023-11-08T05:35:53.240 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127720.log 2023-11-08T05:35:53.240 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.165891.log 2023-11-08T05:35:53.240 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.127720.log: 88.2% -- replaced with /var/log/ceph/cluster1-client.mirror.127720.log.gz 2023-11-08T05:35:53.240 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.1.156803.log 2023-11-08T05:35:53.240 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.admin.165891.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.165891.log.gz 2023-11-08T05:35:53.243 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.128550.log 2023-11-08T05:35:53.243 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.1.156803.log: /var/log/ceph/cluster1-client.mirror.128550.log: 89.4% -- replaced with /var/log/ceph/cluster1-client.mirror.128550.log.gz 2023-11-08T05:35:53.244 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.133889.log 2023-11-08T05:35:53.244 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.133889.log: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.133889.log.gz 2023-11-08T05:35:53.244 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.121379.log 2023-11-08T05:35:53.244 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.121379.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.121379.log.gz 2023-11-08T05:35:53.244 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.131521.log 2023-11-08T05:35:53.245 INFO:teuthology.orchestra.run.smithi087.stderr: 93.2% -- replaced with /var/log/ceph/cluster1-client.mirror.1.156803.log.gz 2023-11-08T05:35:53.245 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.131521.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.146272.log 2023-11-08T05:35:53.246 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.146272.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.146272.log.gz 2023-11-08T05:35:53.246 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.124115.log 2023-11-08T05:35:53.246 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.124115.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.124115.log.gz 2023-11-08T05:35:53.247 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.163629.log 2023-11-08T05:35:53.247 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.admin.163629.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.163629.log.gz 2023-11-08T05:35:53.248 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.135011.log 2023-11-08T05:35:53.255 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.135011.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.148735.log 2023-11-08T05:35:53.277 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.148735.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.148735.log.gz 2023-11-08T05:35:53.277 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.161522.log 2023-11-08T05:35:53.282 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.admin.161522.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.161522.log.gz 2023-11-08T05:35:53.282 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.158979.log 2023-11-08T05:35:53.283 INFO:teuthology.orchestra.run.smithi087.stderr: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.135011.log.gz 2023-11-08T05:35:53.285 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.158979.log: 91.6% -- replaced with /var/log/ceph/cluster2-client.mirror.158979.log.gz 2023-11-08T05:35:53.286 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127804.log 2023-11-08T05:35:53.291 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.127804.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.132756.log.gz 2023-11-08T05:35:53.291 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.140735.log 2023-11-08T05:35:53.291 INFO:teuthology.orchestra.run.smithi087.stderr: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.127804.log.gz 2023-11-08T05:35:53.295 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.140735.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.140735.log.gz 2023-11-08T05:35:53.295 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.154241.log 2023-11-08T05:35:53.296 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.154241.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.154241.log.gz 2023-11-08T05:35:53.296 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.153410.log 2023-11-08T05:35:53.296 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.161326.log 2023-11-08T05:35:53.297 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.153410.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.153410.log.gz 2023-11-08T05:35:53.297 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.admin.161326.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.161326.log.gz 2023-11-08T05:35:53.297 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.145147.log 2023-11-08T05:35:53.299 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.145147.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.145147.log.gz 2023-11-08T05:35:53.300 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.159218.log 2023-11-08T05:35:53.300 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.159218.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.159218.log.gz 2023-11-08T05:35:53.300 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.126967.log 2023-11-08T05:35:53.302 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.126967.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.126967.log.gz 2023-11-08T05:35:53.303 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.144290.log 2023-11-08T05:35:53.304 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.144290.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.144290.log.gz 2023-11-08T05:35:53.304 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.169680.log 2023-11-08T05:35:53.305 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.admin.169680.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.169680.log.gz 2023-11-08T05:35:53.305 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.170170.log 2023-11-08T05:35:53.305 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.admin.170170.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.170170.log.gz 2023-11-08T05:35:53.305 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.158090.log 2023-11-08T05:35:53.306 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.158090.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.158090.log.gz 2023-11-08T05:35:53.307 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137709.log 2023-11-08T05:35:53.308 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.137709.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.137709.log.gz 2023-11-08T05:35:53.308 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.147738.log 2023-11-08T05:35:53.309 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.147738.log: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.147738.log.gz 2023-11-08T05:35:53.309 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.162257.log 2023-11-08T05:35:53.310 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.admin.162257.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.162257.log.gz 2023-11-08T05:35:53.310 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.159478.log 2023-11-08T05:35:53.311 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.159478.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.159478.log.gz 2023-11-08T05:35:53.311 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.148446.log 2023-11-08T05:35:53.311 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.148446.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.148446.log.gz 2023-11-08T05:35:53.312 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.148648.log 2023-11-08T05:35:53.312 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.148648.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.148648.log.gz 2023-11-08T05:35:53.313 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.147806.log 2023-11-08T05:35:53.313 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.147806.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.147806.log.gz 2023-11-08T05:35:53.314 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.146923.log 2023-11-08T05:35:53.315 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.146923.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.146923.log.gz 2023-11-08T05:35:53.315 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.128801.log 2023-11-08T05:35:53.316 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.128801.log: 89.4% -- replaced with /var/log/ceph/cluster1-client.mirror.128801.log.gz 2023-11-08T05:35:53.316 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.148556.log 2023-11-08T05:35:53.318 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.148556.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.148556.log.gz 2023-11-08T05:35:53.318 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.140027.log 2023-11-08T05:35:53.319 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.140027.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.140027.log.gz 2023-11-08T05:35:53.320 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.130344.log 2023-11-08T05:35:53.320 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.130344.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.130344.log.gz 2023-11-08T05:35:53.320 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.161963.log 2023-11-08T05:35:53.321 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.admin.161963.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.161963.log.gz 2023-11-08T05:35:53.321 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.158852.log 2023-11-08T05:35:53.322 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.158852.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.158852.log.gz 2023-11-08T05:35:53.322 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.123247.log 2023-11-08T05:35:53.323 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.admin.123247.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.admin.123247.log.gz 2023-11-08T05:35:53.324 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.153207.log 2023-11-08T05:35:53.324 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.153207.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.153207.log.gz 2023-11-08T05:35:53.324 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.157259.log 2023-11-08T05:35:53.325 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.157259.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.157259.log.gz 2023-11-08T05:35:53.325 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.143820.log 2023-11-08T05:35:53.326 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.143820.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.143820.log.gz 2023-11-08T05:35:53.326 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.164323.log 2023-11-08T05:35:53.326 INFO:teuthology.orchestra.run.smithi087.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.131422.log.gz 2023-11-08T05:35:53.327 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.admin.164323.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.164323.log.gz 2023-11-08T05:35:53.327 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.158782.log 2023-11-08T05:35:53.328 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138113.log 2023-11-08T05:35:53.328 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.158782.log: /var/log/ceph/cluster1-client.mirror.138113.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.138113.log.gz 2023-11-08T05:35:53.329 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.154196.log 2023-11-08T05:35:53.329 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.154196.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.154196.log.gz 2023-11-08T05:35:53.329 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.151852.log 2023-11-08T05:35:53.331 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.151852.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.151852.log.gz 2023-11-08T05:35:53.331 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.2.133954.log 2023-11-08T05:35:53.336 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.2.133954.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.156054.log 2023-11-08T05:35:53.341 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.156054.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.156054.log.gz 2023-11-08T05:35:53.347 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.142879.log 2023-11-08T05:35:53.348 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.142879.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.142879.log.gz 2023-11-08T05:35:53.351 INFO:teuthology.orchestra.run.smithi087.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.134557.log.gz 2023-11-08T05:35:53.352 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.120876.log 2023-11-08T05:35:53.352 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.121342.log 2023-11-08T05:35:53.353 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.admin.120876.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.120876.log.gz 2023-11-08T05:35:53.354 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.121342.log: 79.7% -- replaced with /var/log/ceph/cluster2-client.mirror.121342.log.gz 2023-11-08T05:35:53.354 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.128105.log 2023-11-08T05:35:53.354 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.128105.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.3.151182.log 2023-11-08T05:35:53.356 INFO:teuthology.orchestra.run.smithi087.stderr: 89.0% -- replaced with /var/log/ceph/cluster1-client.mirror.128105.log.gz 2023-11-08T05:35:53.361 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.3.151182.log: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.3.151182.log.gz 2023-11-08T05:35:53.361 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.156459.log 2023-11-08T05:35:53.362 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.145450.log 2023-11-08T05:35:53.362 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.156459.log: /var/log/ceph/cluster2-client.mirror.145450.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.156459.log.gz 2023-11-08T05:35:53.363 INFO:teuthology.orchestra.run.smithi087.stderr: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.145450.log.gz 2023-11-08T05:35:53.363 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.135466.log 2023-11-08T05:35:53.363 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.153521.log 2023-11-08T05:35:53.364 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.135466.log: 79.8% -- replaced with /var/log/ceph/cluster2-client.mirror.135466.log.gz 2023-11-08T05:35:53.364 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.153521.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.153521.log.gz 2023-11-08T05:35:53.365 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.156930.log 2023-11-08T05:35:53.365 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.156930.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.156930.log.gz 2023-11-08T05:35:53.366 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.119202.log 2023-11-08T05:35:53.366 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.admin.119202.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.119202.log.gz 2023-11-08T05:35:53.366 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.135424.log 2023-11-08T05:35:53.367 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.135424.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.135424.log.gz 2023-11-08T05:35:53.367 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.125732.log 2023-11-08T05:35:53.368 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.admin.125732.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.admin.125732.log.gz 2023-11-08T05:35:53.368 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.168741.log 2023-11-08T05:35:53.369 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.admin.168741.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.168741.log.gz 2023-11-08T05:35:53.369 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.131343.log 2023-11-08T05:35:53.372 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.131343.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.157402.log 2023-11-08T05:35:53.395 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.157402.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.120781.log 2023-11-08T05:35:53.396 INFO:teuthology.orchestra.run.smithi087.stderr: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.2.133954.log.gz 2023-11-08T05:35:53.397 INFO:teuthology.orchestra.run.smithi087.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.131521.log.gz 2023-11-08T05:35:53.399 INFO:teuthology.orchestra.run.smithi087.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.134382.log.gz 2023-11-08T05:35:53.407 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.150925.log 2023-11-08T05:35:53.407 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.120781.log: 56.7% -- replaced with /var/log/ceph/cluster1-client.mirror.120781.log.gz 2023-11-08T05:35:53.407 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.160189.log 2023-11-08T05:35:53.409 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.150925.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.150925.log.gz 2023-11-08T05:35:53.409 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.159503.log 2023-11-08T05:35:53.409 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.admin.160189.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.160189.log.gz 2023-11-08T05:35:53.409 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.157170.log 2023-11-08T05:35:53.410 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.159503.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.159503.log.gz 2023-11-08T05:35:53.410 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.150945.log 2023-11-08T05:35:53.410 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.157170.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.157170.log.gz 2023-11-08T05:35:53.410 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.154489.log 2023-11-08T05:35:53.412 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.150945.log: /var/log/ceph/cluster1-client.mirror.154489.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.154489.log.gz 2023-11-08T05:35:53.412 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.162894.log 2023-11-08T05:35:53.412 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.admin.162894.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.162894.log.gz 2023-11-08T05:35:53.412 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.141482.log 2023-11-08T05:35:53.412 INFO:teuthology.orchestra.run.smithi087.stderr: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.150945.log.gz 2023-11-08T05:35:53.413 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.145036.log 2023-11-08T05:35:53.413 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.141482.log: /var/log/ceph/cluster1-client.mirror.145036.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.145036.log.gz 2023-11-08T05:35:53.413 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.140967.log 2023-11-08T05:35:53.415 INFO:teuthology.orchestra.run.smithi087.stderr: 92.5% -- replaced with /var/log/ceph/cluster1-client.mirror.141482.log.gz 2023-11-08T05:35:53.415 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.140967.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.153701.log 2023-11-08T05:35:53.416 INFO:teuthology.orchestra.run.smithi087.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.140967.log.gz 2023-11-08T05:35:53.416 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.120747.log 2023-11-08T05:35:53.416 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.153701.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.153701.log.gz 2023-11-08T05:35:53.416 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.167908.log 2023-11-08T05:35:53.416 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.120747.log: 2.3% -- replaced with /var/log/ceph/cluster1-client.mirror.120747.log.gz 2023-11-08T05:35:53.416 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.145904.log 2023-11-08T05:35:53.417 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.admin.167908.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.167908.log.gz 2023-11-08T05:35:53.417 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.132328.log 2023-11-08T05:35:53.417 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.145904.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.145904.log.gz 2023-11-08T05:35:53.417 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.132328.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.123882.log 2023-11-08T05:35:53.417 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.123882.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.123882.log.gz 2023-11-08T05:35:53.417 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.160091.log 2023-11-08T05:35:53.419 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.admin.160091.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.160091.log.gz 2023-11-08T05:35:53.419 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.140715.log 2023-11-08T05:35:53.420 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.140715.log: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.140715.log.gz 2023-11-08T05:35:53.420 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.145494.log 2023-11-08T05:35:53.420 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.145494.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.145494.log.gz 2023-11-08T05:35:53.420 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.141583.log 2023-11-08T05:35:53.420 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.141583.log: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.141583.log.gz 2023-11-08T05:35:53.420 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.160875.log 2023-11-08T05:35:53.422 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.admin.160875.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.160875.log.gz 2023-11-08T05:35:53.422 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.135784.log 2023-11-08T05:35:53.422 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.135784.log: 1.3% -- replaced with /var/log/ceph/cluster1-client.mirror.135784.log.gz 2023-11-08T05:35:53.422 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.146100.log 2023-11-08T05:35:53.423 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.146100.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.146100.log.gz 2023-11-08T05:35:53.423 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.134060.log 2023-11-08T05:35:53.423 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.134060.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.134060.log.gz 2023-11-08T05:35:53.424 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.151313.log 2023-11-08T05:35:53.424 INFO:teuthology.orchestra.run.smithi077.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-mon.a.log.gz 2023-11-08T05:35:53.426 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.151313.log: 90.5% -- replaced with /var/log/ceph/cluster2-client.mirror.151313.log.gz 2023-11-08T05:35:53.426 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.152203.log 2023-11-08T05:35:53.427 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.152203.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.152203.log.gz 2023-11-08T05:35:53.427 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.163335.log 2023-11-08T05:35:53.428 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.admin.163335.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.163335.log.gz 2023-11-08T05:35:53.428 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.145060.log 2023-11-08T05:35:53.429 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.145060.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.145060.log.gz 2023-11-08T05:35:53.429 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.123518.log 2023-11-08T05:35:53.429 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.123518.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.123518.log.gz 2023-11-08T05:35:53.430 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.154039.log 2023-11-08T05:35:53.431 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.154039.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.154039.log.gz 2023-11-08T05:35:53.431 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.165548.log 2023-11-08T05:35:53.432 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.admin.165548.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.165548.log.gz 2023-11-08T05:35:53.432 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.151872.log 2023-11-08T05:35:53.434 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.151872.log: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.151872.log.gz 2023-11-08T05:35:53.434 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.120903.log 2023-11-08T05:35:53.435 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.admin.120903.log: 79.9% -- replaced with /var/log/ceph/cluster2-client.admin.120903.log.gz 2023-11-08T05:35:53.435 INFO:teuthology.orchestra.run.smithi087.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.156952.log.gz 2023-11-08T05:35:53.435 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.143981.log 2023-11-08T05:35:53.437 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.143981.log: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.143981.log.gz 2023-11-08T05:35:53.437 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.144659.log 2023-11-08T05:35:53.437 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.149296.log 2023-11-08T05:35:53.438 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.144659.log: /var/log/ceph/cluster2-client.mirror.149296.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.149296.log.gz 2023-11-08T05:35:53.439 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.163482.log 2023-11-08T05:35:53.439 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.admin.163482.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.163482.log.gz 2023-11-08T05:35:53.440 INFO:teuthology.orchestra.run.smithi087.stderr: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.144659.log.gz 2023-11-08T05:35:53.440 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.163041.log 2023-11-08T05:35:53.440 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126922.log 2023-11-08T05:35:53.441 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.admin.163041.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.163041.log.gz 2023-11-08T05:35:53.441 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.126922.log: 88.7% -- replaced with /var/log/ceph/cluster1-client.mirror.126922.log.gz 2023-11-08T05:35:53.441 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.131260.log 2023-11-08T05:35:53.442 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.142528.log 2023-11-08T05:35:53.443 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.131260.log: /var/log/ceph/cluster1-client.mirror.142528.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.132056.log.gz 2023-11-08T05:35:53.443 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose 92.3%-- /var/log/ceph/cluster2-client.mirror.159238.log 2023-11-08T05:35:53.443 INFO:teuthology.orchestra.run.smithi087.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.142528.log.gz 2023-11-08T05:35:53.444 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.142550.log 2023-11-08T05:35:53.444 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.159238.log: /var/log/ceph/cluster2-client.mirror.142550.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.142550.log.gz 2023-11-08T05:35:53.445 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.159846.log 2023-11-08T05:35:53.446 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.admin.159846.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.159846.log.gz 2023-11-08T05:35:53.446 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.144760.log 2023-11-08T05:35:53.446 INFO:teuthology.orchestra.run.smithi087.stderr: 91.6% -- replaced with /var/log/ceph/cluster2-client.mirror.159238.log.gz 2023-11-08T05:35:53.447 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.144760.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.144760.log.gz 2023-11-08T05:35:53.447 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.134323.log 2023-11-08T05:35:53.448 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.157360.log 2023-11-08T05:35:53.448 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.134323.log: 79.6% -- replaced with /var/log/ceph/cluster2-client.mirror.134323.log.gz 2023-11-08T05:35:53.448 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.157360.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.157360.log.gz 2023-11-08T05:35:53.449 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.149275.log 2023-11-08T05:35:53.449 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.150635.log 2023-11-08T05:35:53.451 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.149275.log: /var/log/ceph/cluster2-client.mirror.150635.log: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.149275.log.gz 2023-11-08T05:35:53.451 INFO:teuthology.orchestra.run.smithi087.stderr: 90.5% -- replaced with /var/log/ceph/cluster2-client.mirror.150635.log.gz 2023-11-08T05:35:53.451 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.144618.log 2023-11-08T05:35:53.453 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.136914.log 2023-11-08T05:35:53.453 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.144618.log: /var/log/ceph/cluster2-client.mirror.136914.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.144618.log.gz 2023-11-08T05:35:53.453 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.131089.log 2023-11-08T05:35:53.453 INFO:teuthology.orchestra.run.smithi087.stderr: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.136914.log.gz 2023-11-08T05:35:53.453 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.131089.log: 79.4% -- replaced with /var/log/ceph/cluster2-client.mirror.131089.log.gz 2023-11-08T05:35:53.454 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.157982.log 2023-11-08T05:35:53.455 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.126826.log 2023-11-08T05:35:53.455 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.157982.log: /var/log/ceph/cluster2-client.mirror.126826.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.126826.log.gz 2023-11-08T05:35:53.455 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.131363.log 2023-11-08T05:35:53.456 INFO:teuthology.orchestra.run.smithi087.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.157982.log.gz 2023-11-08T05:35:53.456 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.134839.log 2023-11-08T05:35:53.456 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.131363.log: 79.4% -- replaced with /var/log/ceph/cluster2-client.mirror.131363.log.gz 2023-11-08T05:35:53.457 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.134839.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.134839.log.gz 2023-11-08T05:35:53.457 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.129259.log 2023-11-08T05:35:53.457 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.124772.log 2023-11-08T05:35:53.458 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.129259.log: /var/log/ceph/cluster1-client.mirror.124772.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.133488.log 2023-11-08T05:35:53.472 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.133488.log: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.129259.log.gz 2023-11-08T05:35:53.472 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.134469.log 2023-11-08T05:35:53.472 INFO:teuthology.orchestra.run.smithi087.stderr: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.124772.log.gz 2023-11-08T05:35:53.473 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.134469.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.134447.log 2023-11-08T05:35:53.473 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.134447.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.134447.log.gz 2023-11-08T05:35:53.473 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.130517.log 2023-11-08T05:35:53.473 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.130517.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.130517.log.gz 2023-11-08T05:35:53.473 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.140807.log 2023-11-08T05:35:53.474 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.140807.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.140807.log.gz 2023-11-08T05:35:53.474 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.141742.log 2023-11-08T05:35:53.483 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.141742.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.141742.log.gz 2023-11-08T05:35:53.483 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.157280.log 2023-11-08T05:35:53.483 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.157280.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.157280.log.gz 2023-11-08T05:35:53.484 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.160434.log 2023-11-08T05:35:53.484 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.admin.160434.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.160434.log.gz 2023-11-08T05:35:53.484 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.133332.log 2023-11-08T05:35:53.484 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.133332.log: 79.4% -- replaced with /var/log/ceph/cluster2-client.mirror.133332.log.gz 2023-11-08T05:35:53.484 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.132776.log 2023-11-08T05:35:53.485 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.132776.log: 79.5% -- replaced with /var/log/ceph/cluster2-client.mirror.132776.log.gz 2023-11-08T05:35:53.485 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.139082.log 2023-11-08T05:35:53.485 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.139082.log: 92.3% -- replaced with /var/log/ceph/cluster2-client.mirror.139082.log.gz 2023-11-08T05:35:53.485 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137035.log 2023-11-08T05:35:53.485 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.137035.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.137035.log.gz 2023-11-08T05:35:53.486 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.152937.log 2023-11-08T05:35:53.486 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.152937.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.152937.log.gz 2023-11-08T05:35:53.486 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.121320.log 2023-11-08T05:35:53.486 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.121320.log: 53.8% -- replaced with /var/log/ceph/cluster2-client.mirror.121320.log.gz 2023-11-08T05:35:53.486 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.118859.log 2023-11-08T05:35:53.487 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.admin.118859.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.118859.log.gz 2023-11-08T05:35:53.487 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126946.log 2023-11-08T05:35:53.494 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.126946.log: 88.1% -- replaced with /var/log/ceph/cluster1-client.mirror.126946.log.gz 2023-11-08T05:35:53.494 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.135205.log 2023-11-08T05:35:53.495 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.135205.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.135205.log.gz 2023-11-08T05:35:53.495 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.120730.log 2023-11-08T05:35:53.495 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.120730.log: 53.1% -- replaced with /var/log/ceph/cluster1-client.mirror.120730.log.gz 2023-11-08T05:35:53.496 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.131238.log 2023-11-08T05:35:53.496 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.131238.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.131238.log.gz 2023-11-08T05:35:53.497 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.133217.log 2023-11-08T05:35:53.497 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.133217.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.152120.log 2023-11-08T05:35:53.530 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.152120.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.168643.log 2023-11-08T05:35:53.530 INFO:teuthology.orchestra.run.smithi087.stderr: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.152120.log.gz 2023-11-08T05:35:53.541 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.admin.168643.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.168643.log.gz 2023-11-08T05:35:53.541 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.139608.log 2023-11-08T05:35:53.552 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.139608.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.140593.log 2023-11-08T05:35:53.552 INFO:teuthology.orchestra.run.smithi087.stderr: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.139608.log.gz 2023-11-08T05:35:53.563 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.140593.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.140593.log.gzgzip 2023-11-08T05:35:53.563 INFO:teuthology.orchestra.run.smithi087.stderr:-5 --verbose -- /var/log/ceph/cluster1-client.mirror.155960.log 2023-11-08T05:35:53.574 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.155960.log: 87.1% -- replaced with /var/log/ceph/cluster1-client.mirror.155960.log.gz 2023-11-08T05:35:53.574 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126418.log 2023-11-08T05:35:53.605 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.126418.log: 88.4% -- replaced with /var/log/ceph/cluster1-client.mirror.126418.log.gz 2023-11-08T05:35:53.605 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.135763.log 2023-11-08T05:35:53.616 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.135763.log: 2.6% -- replaced with /var/log/ceph/cluster2-client.mirror.135763.log.gz 2023-11-08T05:35:53.616 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.156153.log 2023-11-08T05:35:53.624 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.156153.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.156153.log.gz 2023-11-08T05:35:53.624 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.170023.log 2023-11-08T05:35:53.625 INFO:teuthology.orchestra.run.smithi087.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.158782.log.gz 2023-11-08T05:35:53.625 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.admin.170023.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.170023.log.gz 2023-11-08T05:35:53.625 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.142713.log 2023-11-08T05:35:53.626 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.142713.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.142713.log.gz 2023-11-08T05:35:53.626 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.170464.log 2023-11-08T05:35:53.627 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.admin.170464.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.170464.log.gz 2023-11-08T05:35:53.627 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.150325.log 2023-11-08T05:35:53.630 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.150325.log: 90.5% -- replaced with /var/log/ceph/cluster2-client.mirror.150325.log.gz 2023-11-08T05:35:53.630 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.133655.log 2023-11-08T05:35:53.640 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.133655.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.161571.log 2023-11-08T05:35:53.645 INFO:teuthology.orchestra.run.smithi087.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.133217.log.gz 2023-11-08T05:35:53.654 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.admin.161571.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.161571.log.gz 2023-11-08T05:35:53.662 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.133195.log 2023-11-08T05:35:53.663 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.133195.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.133195.log.gz 2023-11-08T05:35:53.673 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.130641.log 2023-11-08T05:35:53.685 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.130641.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.150160.log 2023-11-08T05:35:53.685 INFO:teuthology.orchestra.run.smithi087.stderr: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.130641.log.gz 2023-11-08T05:35:53.691 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.150160.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.150160.log.gz 2023-11-08T05:35:53.692 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.163580.log 2023-11-08T05:35:53.692 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.admin.163580.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.163580.log.gz 2023-11-08T05:35:53.692 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138388.log 2023-11-08T05:35:53.695 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.138388.log: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.138388.log.gz 2023-11-08T05:35:53.695 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.128009.log 2023-11-08T05:35:53.696 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.128009.log: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.128009.log.gz 2023-11-08T05:35:53.696 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.163286.log 2023-11-08T05:35:53.697 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.admin.163286.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.163286.log.gz 2023-11-08T05:35:53.697 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.2.123155.log 2023-11-08T05:35:53.708 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.2.123155.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.147830.log 2023-11-08T05:35:53.726 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.147830.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.147830.log.gz 2023-11-08T05:35:53.727 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.144927.log 2023-11-08T05:35:53.736 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.144927.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.142270.log 2023-11-08T05:35:53.736 INFO:teuthology.orchestra.run.smithi087.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.144927.log.gz 2023-11-08T05:35:53.736 INFO:teuthology.orchestra.run.smithi087.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.134469.log.gz 2023-11-08T05:35:53.738 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.142270.log: 92.3% -- replaced with /var/log/ceph/cluster2-client.mirror.142270.log.gz 2023-11-08T05:35:53.738 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125532.log 2023-11-08T05:35:53.740 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.125532.log: 88.1% -- replaced with /var/log/ceph/cluster1-client.mirror.125532.log.gz 2023-11-08T05:35:53.740 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.136892.log 2023-11-08T05:35:53.743 INFO:teuthology.orchestra.run.smithi087.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.132328.log.gz 2023-11-08T05:35:53.745 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.142177.log 2023-11-08T05:35:53.745 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.136892.log: /var/log/ceph/cluster1-client.mirror.142177.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.142177.log.gz 2023-11-08T05:35:53.746 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.162453.log 2023-11-08T05:35:53.746 INFO:teuthology.orchestra.run.smithi087.stderr: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.136892.log.gz 2023-11-08T05:35:53.747 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.admin.162453.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.162453.log.gz 2023-11-08T05:35:53.747 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.149399.log 2023-11-08T05:35:53.747 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125995.log 2023-11-08T05:35:53.748 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.149399.log: /var/log/ceph/cluster1-client.mirror.125995.log: 88.1% -- replaced with /var/log/ceph/cluster1-client.mirror.125995.log.gz 2023-11-08T05:35:53.748 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.141222.log 2023-11-08T05:35:53.749 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.141222.log: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.149399.log.gz 2023-11-08T05:35:53.749 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.136147.log 2023-11-08T05:35:53.750 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.136147.log: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.141222.log.gz 2023-11-08T05:35:53.751 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.162698.log 2023-11-08T05:35:53.751 INFO:teuthology.orchestra.run.smithi087.stderr: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.136147.log.gz 2023-11-08T05:35:53.752 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.admin.162698.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.162698.log.gzgzip 2023-11-08T05:35:53.753 INFO:teuthology.orchestra.run.smithi087.stderr:-5 --verbose -- /var/log/ceph/cluster2-client.mirror.137997.log 2023-11-08T05:35:53.756 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.156910.log 2023-11-08T05:35:53.756 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.137997.log: /var/log/ceph/cluster2-client.mirror.156910.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.137997.log.gz 2023-11-08T05:35:53.757 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.154421.log 2023-11-08T05:35:53.757 INFO:teuthology.orchestra.run.smithi087.stderr: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.156910.log.gz 2023-11-08T05:35:53.757 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.161718.log 2023-11-08T05:35:53.758 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.154421.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.154421.log.gz 2023-11-08T05:35:53.758 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.admin.161718.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.161718.log.gz 2023-11-08T05:35:53.759 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.118908.log 2023-11-08T05:35:53.759 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.119643.log 2023-11-08T05:35:53.759 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.admin.118908.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.118908.log.gz 2023-11-08T05:35:53.760 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.137665.log 2023-11-08T05:35:53.760 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.admin.119643.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.119643.log.gz 2023-11-08T05:35:53.760 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.157965.log 2023-11-08T05:35:53.761 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.137665.log: /var/log/ceph/cluster2-client.mirror.157965.log: 79.4% -- replaced with /var/log/ceph/cluster2-client.mirror.157965.log.gz 2023-11-08T05:35:53.761 INFO:teuthology.orchestra.run.smithi087.stderr: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.137665.log.gz 2023-11-08T05:35:53.761 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.119741.log 2023-11-08T05:35:53.765 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.admin.119741.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.119741.log.gz 2023-11-08T05:35:53.766 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138275.log 2023-11-08T05:35:53.766 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.138275.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.138275.log.gz 2023-11-08T05:35:53.767 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.137946.log 2023-11-08T05:35:53.767 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.137946.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.137946.log.gz 2023-11-08T05:35:53.767 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137525.log 2023-11-08T05:35:53.767 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.137525.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.137525.log.gz 2023-11-08T05:35:53.768 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.136595.log 2023-11-08T05:35:53.768 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.136595.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.136595.log.gz 2023-11-08T05:35:53.768 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138154.log 2023-11-08T05:35:53.768 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.138154.log: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.138154.log.gz 2023-11-08T05:35:53.769 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.134723.log 2023-11-08T05:35:53.770 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.134723.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.158609.log 2023-11-08T05:35:53.770 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.158609.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.158609.log.gz 2023-11-08T05:35:53.770 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.139539.log 2023-11-08T05:35:53.771 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.139539.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.139539.log.gz 2023-11-08T05:35:53.771 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.135525.log 2023-11-08T05:35:53.779 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.135525.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.157402.log.gz 2023-11-08T05:35:53.779 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.151707.log 2023-11-08T05:35:53.779 INFO:teuthology.orchestra.run.smithi087.stderr: 93.8% -- replaced with /var/log/ceph/cluster1-client.mirror.2.123155.log.gz 2023-11-08T05:35:53.785 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.119986.log 2023-11-08T05:35:53.796 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.151707.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.151707.log.gz 2023-11-08T05:35:53.796 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.admin.119986.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.119986.log.gz 2023-11-08T05:35:53.797 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.157343.log 2023-11-08T05:35:53.797 INFO:teuthology.orchestra.run.smithi087.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.133655.log.gz 2023-11-08T05:35:53.797 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.1.120104.log 2023-11-08T05:35:53.797 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.157343.log: 79.6% -- replaced with /var/log/ceph/cluster2-client.mirror.157343.log.gz 2023-11-08T05:35:53.797 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.147675.log 2023-11-08T05:35:53.798 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.1.120104.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.154309.log 2023-11-08T05:35:53.798 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.147675.log: /var/log/ceph/cluster1-client.mirror.154309.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.154309.log.gz 2023-11-08T05:35:53.798 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.138594.log 2023-11-08T05:35:53.798 INFO:teuthology.orchestra.run.smithi087.stderr: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.147675.log.gz 2023-11-08T05:35:53.798 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.138594.log: 92.1% -- replaced with /var/log/ceph/cluster1-client.mirror.1.120104.log.gz 2023-11-08T05:35:53.799 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.130661.log 2023-11-08T05:35:53.799 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.130661.log: 92.7% -- replaced with /var/log/ceph/cluster2-client.mirror.138594.log.gz 2023-11-08T05:35:53.799 INFO:teuthology.orchestra.run.smithi087.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.130661.log.gz 2023-11-08T05:35:53.799 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.151955.log 2023-11-08T05:35:53.800 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.136682.log 2023-11-08T05:35:53.800 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.151955.log: /var/log/ceph/cluster2-client.mirror.136682.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.151955.log.gz 2023-11-08T05:35:53.800 INFO:teuthology.orchestra.run.smithi087.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.136682.log.gz 2023-11-08T05:35:53.800 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.139467.log 2023-11-08T05:35:53.800 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.148241.log 2023-11-08T05:35:53.801 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.139467.log: /var/log/ceph/cluster2-client.mirror.148241.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.139467.log.gz 2023-11-08T05:35:53.801 INFO:teuthology.orchestra.run.smithi087.stderr: 92.2% -- replaced with /var/log/ceph/cluster2-client.mirror.148241.log.gz 2023-11-08T05:35:53.801 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.121218.log 2023-11-08T05:35:53.801 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.144136.log 2023-11-08T05:35:53.801 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.121218.log: 54.4% -- replaced with /var/log/ceph/cluster2-client.mirror.121218.log.gz 2023-11-08T05:35:53.802 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.144136.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.144136.log.gz 2023-11-08T05:35:53.802 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.146433.log 2023-11-08T05:35:53.802 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.135503.log 2023-11-08T05:35:53.807 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.146433.log: /var/log/ceph/cluster2-client.mirror.135503.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.135503.log.gz 2023-11-08T05:35:53.807 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.156610.log 2023-11-08T05:35:53.807 INFO:teuthology.orchestra.run.smithi087.stderr: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.146433.log.gz 2023-11-08T05:35:53.808 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.156610.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.156610.log.gz 2023-11-08T05:35:53.808 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.146316.log 2023-11-08T05:35:53.808 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.137802.log 2023-11-08T05:35:53.808 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.146316.log: /var/log/ceph/cluster2-client.mirror.137802.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.146316.log.gz 2023-11-08T05:35:53.808 INFO:teuthology.orchestra.run.smithi087.stderr: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.137802.log.gz 2023-11-08T05:35:53.809 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.136523.log 2023-11-08T05:35:53.809 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.133694.log 2023-11-08T05:35:53.809 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.136523.log: /var/log/ceph/cluster2-client.mirror.133694.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.136523.log.gz 2023-11-08T05:35:53.809 INFO:teuthology.orchestra.run.smithi087.stderr: 79.7% -- replaced with /var/log/ceph/cluster2-client.mirror.133694.log.gz 2023-11-08T05:35:53.809 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.139935.log 2023-11-08T05:35:53.809 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.147140.log 2023-11-08T05:35:53.810 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.139935.log: /var/log/ceph/cluster1-client.mirror.147140.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.139935.log.gz 2023-11-08T05:35:53.810 INFO:teuthology.orchestra.run.smithi087.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.147140.log.gz 2023-11-08T05:35:53.810 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126850.log 2023-11-08T05:35:53.810 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.149995.log 2023-11-08T05:35:53.811 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.126850.log: /var/log/ceph/cluster2-client.mirror.149995.log: 88.6% -- replaced with /var/log/ceph/cluster1-client.mirror.126850.log.gz 2023-11-08T05:35:53.811 INFO:teuthology.orchestra.run.smithi087.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.149995.log.gz 2023-11-08T05:35:53.811 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127500.log 2023-11-08T05:35:53.811 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.160042.log 2023-11-08T05:35:53.811 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.127500.log: /var/log/ceph/cluster2-client.admin.160042.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.127500.log.gz 2023-11-08T05:35:53.812 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.160042.log.gz 2023-11-08T05:35:53.812 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.148985.log 2023-11-08T05:35:53.812 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.161767.log 2023-11-08T05:35:53.812 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.148985.log: /var/log/ceph/cluster2-client.admin.161767.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.148985.log.gz 2023-11-08T05:35:53.812 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.161767.log.gz 2023-11-08T05:35:53.813 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.144160.log 2023-11-08T05:35:53.813 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.129368.log 2023-11-08T05:35:53.813 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.144160.log: /var/log/ceph/cluster2-client.mirror.129368.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.144160.log.gz 2023-11-08T05:35:53.815 INFO:teuthology.orchestra.run.smithi087.stderr: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.129368.log.gz 2023-11-08T05:35:53.815 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.131997.log 2023-11-08T05:35:53.816 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.169827.log 2023-11-08T05:35:53.816 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.131997.log: 79.5% -- replaced with /var/log/ceph/cluster2-client.mirror.131997.log.gz 2023-11-08T05:35:53.816 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.admin.169827.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.169827.log.gz 2023-11-08T05:35:53.817 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.143503.log 2023-11-08T05:35:53.817 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.165842.log 2023-11-08T05:35:53.817 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.143503.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.143503.log.gz 2023-11-08T05:35:53.818 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.143159.log 2023-11-08T05:35:53.818 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.admin.165842.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.165842.log.gz 2023-11-08T05:35:53.819 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.141369.log 2023-11-08T05:35:53.819 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.143159.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.143159.log.gz 2023-11-08T05:35:53.819 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.142201.log 2023-11-08T05:35:53.826 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.141369.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.141369.log.gz 2023-11-08T05:35:53.826 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.127928.log 2023-11-08T05:35:53.827 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.142201.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.142201.log.gz 2023-11-08T05:35:53.827 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.164519.log 2023-11-08T05:35:53.827 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.127928.log: /var/log/ceph/cluster2-client.admin.164519.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.164519.log.gz 2023-11-08T05:35:53.828 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.137612.log 2023-11-08T05:35:53.828 INFO:teuthology.orchestra.run.smithi087.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.131343.log.gz 2023-11-08T05:35:53.828 INFO:teuthology.orchestra.run.smithi087.stderr: 91.7% -- replaced with /var/log/ceph/cluster2-client.mirror.127928.log.gz 2023-11-08T05:35:53.829 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.137612.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.137612.log.gz 2023-11-08T05:35:53.829 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.149379.log 2023-11-08T05:35:53.829 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.167810.log 2023-11-08T05:35:53.830 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.149379.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.131850.log 2023-11-08T05:35:53.830 INFO:teuthology.orchestra.run.smithi087.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.149379.log.gz 2023-11-08T05:35:53.830 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.admin.167810.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.167810.log.gzgzip 2023-11-08T05:35:53.831 INFO:teuthology.orchestra.run.smithi087.stderr:-5 --verbose -- /var/log/ceph/cluster2-client.mirror.133349.log 2023-11-08T05:35:53.831 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.135900.log 2023-11-08T05:35:53.831 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.131850.log: /var/log/ceph/cluster2-client.mirror.133349.log: /var/log/ceph/cluster2-client.mirror.135900.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.133349.log.gz 2023-11-08T05:35:53.832 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.161669.log 2023-11-08T05:35:53.832 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.admin.161669.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.161669.log.gz 2023-11-08T05:35:53.833 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.148120.log 2023-11-08T05:35:53.833 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.148120.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.148120.log.gz 2023-11-08T05:35:53.833 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.132093.log 2023-11-08T05:35:53.834 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.132093.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.132093.log.gz 2023-11-08T05:35:53.834 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.151748.log 2023-11-08T05:35:53.841 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.151748.log: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.151748.log.gz 2023-11-08T05:35:53.842 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.127580.log 2023-11-08T05:35:53.842 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.127580.log: 90.1% -- replaced with /var/log/ceph/cluster2-client.mirror.127580.log.gz 2023-11-08T05:35:53.842 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.153027.log 2023-11-08T05:35:53.842 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.153027.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.153027.log.gz 2023-11-08T05:35:53.843 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.147518.log 2023-11-08T05:35:53.843 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.147518.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.147518.log.gz 2023-11-08T05:35:53.843 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.130559.log 2023-11-08T05:35:53.844 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.130559.log: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.131260.log.gz 2023-11-08T05:35:53.844 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.150863.log 2023-11-08T05:35:53.844 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.150863.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.150863.log.gz 2023-11-08T05:35:53.845 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.166185.log 2023-11-08T05:35:53.845 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.admin.166185.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.166185.log.gz 2023-11-08T05:35:53.846 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.146602.log 2023-11-08T05:35:53.846 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.146602.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.146602.log.gz 2023-11-08T05:35:53.846 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.164715.log 2023-11-08T05:35:53.847 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.admin.164715.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.164715.log.gz 2023-11-08T05:35:53.847 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.147068.log 2023-11-08T05:35:53.847 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.147068.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.153994.log 2023-11-08T05:35:53.848 INFO:teuthology.orchestra.run.smithi087.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.147068.log.gz 2023-11-08T05:35:53.848 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.153994.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.153994.log.gz 2023-11-08T05:35:53.848 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.163882.log 2023-11-08T05:35:53.848 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.admin.163882.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.163882.log.gz 2023-11-08T05:35:53.848 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.157770.log 2023-11-08T05:35:53.848 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.157770.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.157770.log.gz 2023-11-08T05:35:53.849 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.125601.log 2023-11-08T05:35:53.849 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.125601.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.133063.log 2023-11-08T05:35:53.852 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.133063.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.133063.log.gz 2023-11-08T05:35:53.852 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.163531.log 2023-11-08T05:35:53.852 INFO:teuthology.orchestra.run.smithi087.stderr: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.125601.log.gz 2023-11-08T05:35:53.852 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.admin.163531.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.163531.log.gz 2023-11-08T05:35:53.852 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.119251.log 2023-11-08T05:35:53.853 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.admin.119251.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.119251.log.gz 2023-11-08T05:35:53.853 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.139807.log 2023-11-08T05:35:53.853 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.139807.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.139807.log.gz 2023-11-08T05:35:53.853 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.144338.log 2023-11-08T05:35:53.853 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.144338.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.144338.log.gz 2023-11-08T05:35:53.854 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.163727.log 2023-11-08T05:35:53.854 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.admin.163727.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.163727.log.gz 2023-11-08T05:35:53.854 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.151098.log 2023-11-08T05:35:53.854 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.151098.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.151098.log.gz 2023-11-08T05:35:53.854 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.160777.log 2023-11-08T05:35:53.855 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.admin.160777.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.160777.log.gz 2023-11-08T05:35:53.855 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.150429.log 2023-11-08T05:35:53.855 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138549.log 2023-11-08T05:35:53.855 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.150429.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.150429.log.gz 2023-11-08T05:35:53.855 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.138549.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.138549.log.gz 2023-11-08T05:35:53.856 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.165695.log 2023-11-08T05:35:53.856 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.148918.log 2023-11-08T05:35:53.856 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.admin.165695.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.165695.log.gz 2023-11-08T05:35:53.856 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.136126.log 2023-11-08T05:35:53.858 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.148918.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.148918.log.gz 2023-11-08T05:35:53.858 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.136126.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.136126.log.gz 2023-11-08T05:35:53.858 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.143568.log 2023-11-08T05:35:53.859 INFO:teuthology.orchestra.run.smithi087.stderr: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.135900.log.gz 2023-11-08T05:35:53.859 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.142024.log 2023-11-08T05:35:53.859 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.143568.log: /var/log/ceph/cluster1-client.mirror.142024.log: 95.2% -- replaced with /var/log/ceph/cluster1-client.mirror.142024.log.gz 2023-11-08T05:35:53.859 INFO:teuthology.orchestra.run.smithi087.stderr: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.143568.log.gz 2023-11-08T05:35:53.859 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.130386.log 2023-11-08T05:35:53.860 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.126123.log 2023-11-08T05:35:53.860 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.130386.log: /var/log/ceph/cluster2-client.mirror.126123.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.134360.log 2023-11-08T05:35:53.860 INFO:teuthology.orchestra.run.smithi087.stderr: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.130386.log.gz 2023-11-08T05:35:53.861 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.134360.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.134360.log.gz 2023-11-08T05:35:53.861 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.129943.log 2023-11-08T05:35:53.861 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.129943.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.122115.log 2023-11-08T05:35:53.861 INFO:teuthology.orchestra.run.smithi087.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.129943.log.gz 2023-11-08T05:35:53.869 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.122115.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.140988.log 2023-11-08T05:35:53.874 INFO:teuthology.orchestra.run.smithi087.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.133488.log.gz 2023-11-08T05:35:53.884 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.140988.log: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.140988.log.gz 2023-11-08T05:35:53.884 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.150511.log 2023-11-08T05:35:53.887 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.150511.log: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.150511.log.gz 2023-11-08T05:35:53.887 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.160287.log 2023-11-08T05:35:53.888 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.admin.160287.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.160287.log.gz 2023-11-08T05:35:53.888 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.144427.log 2023-11-08T05:35:53.890 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.144427.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.144427.log.gz 2023-11-08T05:35:53.890 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.140854.log 2023-11-08T05:35:53.899 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.140854.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.128235.log 2023-11-08T05:35:53.911 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.128235.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.129110.log 2023-11-08T05:35:53.912 INFO:teuthology.orchestra.run.smithi087.stderr: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.128235.log.gz 2023-11-08T05:35:53.919 INFO:teuthology.orchestra.run.smithi087.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.134723.log.gz 2023-11-08T05:35:53.920 INFO:teuthology.orchestra.run.smithi087.stderr: 93.3% -- replaced with /var/log/ceph/cluster2-client.mirror.140854.log.gz 2023-11-08T05:35:53.920 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.119398.log 2023-11-08T05:35:53.921 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.129110.log: /var/log/ceph/cluster2-client.admin.119398.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.119398.log.gz 2023-11-08T05:35:53.921 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.147256.log 2023-11-08T05:35:53.922 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.147256.log: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.147256.log.gz 2023-11-08T05:35:53.922 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.135641.log 2023-11-08T05:35:53.925 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.135641.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.129110.log.gz 2023-11-08T05:35:53.925 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.169182.log 2023-11-08T05:35:53.926 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.admin.169182.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.169182.log.gz 2023-11-08T05:35:53.926 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.150656.log 2023-11-08T05:35:53.927 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.150656.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.150656.log.gz 2023-11-08T05:35:53.927 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.148283.log 2023-11-08T05:35:53.928 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.148283.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.148283.log.gz 2023-11-08T05:35:53.928 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.151666.log 2023-11-08T05:35:53.929 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.151666.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.151666.log.gz 2023-11-08T05:35:53.929 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.169925.log 2023-11-08T05:35:53.930 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.admin.169925.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.169925.log.gz 2023-11-08T05:35:53.930 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.131499.log 2023-11-08T05:35:53.931 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.131499.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.142665.log 2023-11-08T05:35:53.931 INFO:teuthology.orchestra.run.smithi087.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.131499.log.gz 2023-11-08T05:35:53.932 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.142835.log 2023-11-08T05:35:53.932 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.142665.log: /var/log/ceph/cluster2-client.mirror.142835.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.142665.log.gz 2023-11-08T05:35:53.933 INFO:teuthology.orchestra.run.smithi087.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.142835.log.gz 2023-11-08T05:35:53.933 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.151542.log 2023-11-08T05:35:53.934 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.151542.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.151542.log.gz 2023-11-08T05:35:53.934 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.133790.log 2023-11-08T05:35:53.935 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.133790.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.133790.log.gz 2023-11-08T05:35:53.935 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.124859.log 2023-11-08T05:35:53.940 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.124859.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.154669.log 2023-11-08T05:35:53.947 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.154669.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.154669.log.gz 2023-11-08T05:35:53.947 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.123422.log 2023-11-08T05:35:53.948 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.123422.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.123422.log.gz 2023-11-08T05:35:53.948 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.129583.log 2023-11-08T05:35:53.954 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.129583.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.129583.log.gz 2023-11-08T05:35:53.954 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.120088.log 2023-11-08T05:35:53.965 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.120088.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.166732.log 2023-11-08T05:35:53.987 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.admin.166732.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.166732.log.gz 2023-11-08T05:35:53.988 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.149193.log 2023-11-08T05:35:54.006 INFO:teuthology.orchestra.run.smithi087.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.120088.log.gz 2023-11-08T05:35:54.008 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.149193.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.149193.log.gz 2023-11-08T05:35:54.008 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.160238.log 2023-11-08T05:35:54.009 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.admin.160238.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.160238.log.gz 2023-11-08T05:35:54.009 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.154755.log 2023-11-08T05:35:54.010 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.154755.log: 1.3% -- replaced with /var/log/ceph/cluster2-client.mirror.154755.log.gz 2023-11-08T05:35:54.010 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.150449.log 2023-11-08T05:35:54.013 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.150449.log: 90.5% -- replaced with /var/log/ceph/cluster2-client.mirror.150449.log.gz 2023-11-08T05:35:54.013 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.170219.log 2023-11-08T05:35:54.014 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.admin.170219.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.170219.log.gz 2023-11-08T05:35:54.014 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.140217.log 2023-11-08T05:35:54.015 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.140217.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.140217.log.gz 2023-11-08T05:35:54.015 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.142133.log 2023-11-08T05:35:54.017 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.142133.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.142133.log.gz 2023-11-08T05:35:54.017 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.161177.log 2023-11-08T05:35:54.018 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.admin.161177.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.161177.log.gz 2023-11-08T05:35:54.018 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.143183.log 2023-11-08T05:35:54.019 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.143183.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.143183.log.gz 2023-11-08T05:35:54.019 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.165793.log 2023-11-08T05:35:54.020 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.admin.165793.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.165793.log.gz 2023-11-08T05:35:54.020 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.134636.log 2023-11-08T05:35:54.031 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.134636.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.140614.log 2023-11-08T05:35:54.047 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.140614.log: 92.2% -- replaced with /var/log/ceph/cluster2-client.mirror.140614.log.gz 2023-11-08T05:35:54.047 INFO:teuthology.orchestra.run.smithi087.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.126123.log.gz 2023-11-08T05:35:54.047 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.167320.log 2023-11-08T05:35:54.048 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.admin.167320.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.167320.log.gz 2023-11-08T05:35:54.048 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125899.log 2023-11-08T05:35:54.052 INFO:teuthology.orchestra.run.smithi087.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.135525.log.gz 2023-11-08T05:35:54.053 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.151182.log 2023-11-08T05:35:54.053 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.125899.log: 88.2% -- replaced with /var/log/ceph/cluster1-client.mirror.125899.log.gz 2023-11-08T05:35:54.054 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.151182.log: 90.5% -- replaced with /var/log/ceph/cluster1-client.mirror.151182.log.gz 2023-11-08T05:35:54.054 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.142754.log 2023-11-08T05:35:54.057 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.136197.log 2023-11-08T05:35:54.057 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.142754.log: /var/log/ceph/cluster2-client.mirror.136197.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.136197.log.gz 2023-11-08T05:35:54.057 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.152079.log 2023-11-08T05:35:54.057 INFO:teuthology.orchestra.run.smithi087.stderr: 91.8% -- replaced with /var/log/ceph/cluster1-client.mirror.142754.log.gz 2023-11-08T05:35:54.058 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.152079.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.152079.log.gz 2023-11-08T05:35:54.058 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.152244.log 2023-11-08T05:35:54.060 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.152244.log: 90.5% -- replaced with /var/log/ceph/cluster2-client.mirror.152244.log.gz 2023-11-08T05:35:54.061 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.145355.log 2023-11-08T05:35:54.063 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.145355.log: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.145355.log.gz 2023-11-08T05:35:54.063 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.143014.log 2023-11-08T05:35:54.064 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.123852.log 2023-11-08T05:35:54.064 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.143014.log: /var/log/ceph/cluster1-client.admin.123852.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.123852.log.gz 2023-11-08T05:35:54.065 INFO:teuthology.orchestra.run.smithi087.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.143014.log.gz 2023-11-08T05:35:54.065 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.151500.log 2023-11-08T05:35:54.066 INFO:teuthology.orchestra.run.smithi087.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.130559.log.gz 2023-11-08T05:35:54.068 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.148330.log 2023-11-08T05:35:54.068 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.151500.log: /var/log/ceph/cluster2-client.mirror.148330.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.122115.log.gz 2023-11-08T05:35:54.068 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137011.log 2023-11-08T05:35:54.069 INFO:teuthology.orchestra.run.smithi087.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.148330.log.gz 2023-11-08T05:35:54.069 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.137011.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.137011.log.gz 2023-11-08T05:35:54.069 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.161079.log 2023-11-08T05:35:54.070 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.147203.log 2023-11-08T05:35:54.070 INFO:teuthology.orchestra.run.smithi087.stderr: 90.5% -- replaced with /var/log/ceph/cluster2-client.mirror.151500.log.gz 2023-11-08T05:35:54.071 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.admin.161079.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.161079.log.gz 2023-11-08T05:35:54.071 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.147203.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.145310.log 2023-11-08T05:35:54.071 INFO:teuthology.orchestra.run.smithi087.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.147203.log.gz 2023-11-08T05:35:54.071 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.168055.log 2023-11-08T05:35:54.073 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.124091.log 2023-11-08T05:35:54.073 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.145310.log: /var/log/ceph/cluster2-client.admin.168055.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.145310.log.gz 2023-11-08T05:35:54.073 INFO:teuthology.orchestra.run.smithi087.stderr: 24.3% -- replaced with /var/log/ceph/cluster2-client.admin.168055.log.gz 2023-11-08T05:35:54.074 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.124091.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.124091.log.gz 2023-11-08T05:35:54.074 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.134664.log 2023-11-08T05:35:54.074 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.154129.log 2023-11-08T05:35:54.074 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.134664.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137477.log 2023-11-08T05:35:54.074 INFO:teuthology.orchestra.run.smithi087.stderr: 79.6% -- replaced with /var/log/ceph/cluster2-client.mirror.134664.log.gz 2023-11-08T05:35:54.075 INFO:teuthology.orchestra.run.smithi087.stderr:gzip /var/log/ceph/cluster1-client.mirror.154129.log: -5 --verbose -- /var/log/ceph/cluster1-client.mirror.154354.log 2023-11-08T05:35:54.075 INFO:teuthology.orchestra.run.smithi087.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.154129.log.gz 2023-11-08T05:35:54.075 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.137477.log: /var/log/ceph/cluster1-client.mirror.154354.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.137477.log.gz 2023-11-08T05:35:54.075 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.132394.log 2023-11-08T05:35:54.075 INFO:teuthology.orchestra.run.smithi087.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.154354.log.gz 2023-11-08T05:35:54.076 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.141393.log 2023-11-08T05:35:54.078 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.132394.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.157070.log 2023-11-08T05:35:54.078 INFO:teuthology.orchestra.run.smithi087.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.132394.log.gz 2023-11-08T05:35:54.078 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.141393.log: /var/log/ceph/cluster1-client.mirror.157070.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.146846.log 2023-11-08T05:35:54.078 INFO:teuthology.orchestra.run.smithi087.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.141393.log.gz 2023-11-08T05:35:54.079 INFO:teuthology.orchestra.run.smithi087.stderr: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.157070.log.gz 2023-11-08T05:35:54.079 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.125054.log 2023-11-08T05:35:54.079 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.146846.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.129677.log 2023-11-08T05:35:54.079 INFO:teuthology.orchestra.run.smithi087.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.146846.log.gz 2023-11-08T05:35:54.079 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.0.125054.log: /var/log/ceph/cluster2-client.mirror.129677.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.149627.log 2023-11-08T05:35:54.079 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.149627.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.149627.log.gz 2023-11-08T05:35:54.080 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.133832.log 2023-11-08T05:35:54.081 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.133832.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.129677.log.gz 2023-11-08T05:35:54.081 INFO:teuthology.orchestra.run.smithi087.stderr: 93.8% -- replaced with /var/log/ceph/cluster1-client.mirror.0.125054.log.gz 2023-11-08T05:35:54.081 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.144449.log 2023-11-08T05:35:54.081 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.144201.log 2023-11-08T05:35:54.082 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.144449.log: /var/log/ceph/cluster1-client.mirror.144201.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.144449.log.gz 2023-11-08T05:35:54.082 INFO:teuthology.orchestra.run.smithi087.stderr: 91.9% -- replaced with /var/log/ceph/cluster1-client.mirror.144201.log.gz 2023-11-08T05:35:54.083 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.134859.log 2023-11-08T05:35:54.084 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.160385.log 2023-11-08T05:35:54.084 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.134859.log: /var/log/ceph/cluster2-client.admin.160385.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.134859.log.gz 2023-11-08T05:35:54.084 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.160385.log.gz 2023-11-08T05:35:54.084 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126598.log 2023-11-08T05:35:54.085 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.153926.log 2023-11-08T05:35:54.086 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.126598.log: /var/log/ceph/cluster1-client.mirror.153926.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.153926.log.gz 2023-11-08T05:35:54.086 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.167467.log 2023-11-08T05:35:54.086 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.admin.167467.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.167467.log.gz 2023-11-08T05:35:54.086 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.149026.log 2023-11-08T05:35:54.088 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.149026.log: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.149026.log.gz 2023-11-08T05:35:54.088 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.151645.log 2023-11-08T05:35:54.089 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.151645.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.151645.log.gz 2023-11-08T05:35:54.089 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.139852.log 2023-11-08T05:35:54.089 INFO:teuthology.orchestra.run.smithi087.stderr: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.126598.log.gz 2023-11-08T05:35:54.090 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.129024.log 2023-11-08T05:35:54.091 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.139852.log: /var/log/ceph/cluster2-client.mirror.129024.log: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.139852.log.gz 2023-11-08T05:35:54.095 INFO:teuthology.orchestra.run.smithi087.stderr: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.129024.log.gz 2023-11-08T05:35:54.095 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.157116.log 2023-11-08T05:35:54.096 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.138231.log 2023-11-08T05:35:54.096 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.157116.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.157116.log.gz 2023-11-08T05:35:54.097 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.138231.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.138231.log.gz 2023-11-08T05:35:54.097 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.130857.log 2023-11-08T05:35:54.098 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.116061.log 2023-11-08T05:35:54.098 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.130857.log: /var/log/ceph/cluster2-client.admin.116061.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.130857.log.gz 2023-11-08T05:35:54.099 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.116061.log.gz 2023-11-08T05:35:54.099 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.141917.log 2023-11-08T05:35:54.099 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.155937.log 2023-11-08T05:35:54.100 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.141917.log: /var/log/ceph/cluster1-client.mirror.155937.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.141917.log.gz 2023-11-08T05:35:54.101 INFO:teuthology.orchestra.run.smithi087.stderr: 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.155937.log.gz 2023-11-08T05:35:54.101 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.164470.log 2023-11-08T05:35:54.101 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138769.log 2023-11-08T05:35:54.101 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.admin.164470.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.164470.log.gz 2023-11-08T05:35:54.101 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.138769.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.138769.log.gz 2023-11-08T05:35:54.102 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.149172.log 2023-11-08T05:35:54.103 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.132436.log 2023-11-08T05:35:54.103 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.149172.log: /var/log/ceph/cluster2-client.mirror.132436.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.149172.log.gz 2023-11-08T05:35:54.103 INFO:teuthology.orchestra.run.smithi087.stderr: 79.7% -- replaced with /var/log/ceph/cluster2-client.mirror.132436.log.gz 2023-11-08T05:35:54.103 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.135225.log 2023-11-08T05:35:54.104 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.154511.log 2023-11-08T05:35:54.104 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.135225.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.139894.log 2023-11-08T05:35:54.104 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.154511.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.135225.log.gz 2023-11-08T05:35:54.104 INFO:teuthology.orchestra.run.smithi087.stderr: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.154511.log.gz 2023-11-08T05:35:54.105 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.139894.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.139894.log.gz 2023-11-08T05:35:54.105 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126246.log 2023-11-08T05:35:54.106 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138573.log 2023-11-08T05:35:54.106 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.126246.log: 83.1% -- replaced with /var/log/ceph/cluster1-client.mirror.126246.log.gz 2023-11-08T05:35:54.108 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.138573.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.138573.log.gz 2023-11-08T05:35:54.108 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.123155.log 2023-11-08T05:35:54.109 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.129720.log 2023-11-08T05:35:54.114 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.123155.log: /var/log/ceph/cluster2-client.mirror.129720.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.129720.log.gz 2023-11-08T05:35:54.114 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.141938.log 2023-11-08T05:35:54.116 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.141938.log: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.141938.log.gz 2023-11-08T05:35:54.116 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.119300.log 2023-11-08T05:35:54.117 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.admin.119300.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.119300.log.gz 2023-11-08T05:35:54.117 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.130837.log 2023-11-08T05:35:54.118 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.130837.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.130837.log.gz 2023-11-08T05:35:54.118 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.167712.log 2023-11-08T05:35:54.119 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.admin.167712.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.167712.log.gz 2023-11-08T05:35:54.119 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.123470.log 2023-11-08T05:35:54.121 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.123470.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.123470.log.gz 2023-11-08T05:35:54.121 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125556.log 2023-11-08T05:35:54.122 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.125556.log: 88.0% -- replaced with /var/log/ceph/cluster1-client.mirror.125556.log.gz 2023-11-08T05:35:54.122 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.139979.log 2023-11-08T05:35:54.123 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.139979.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.139979.log.gz 2023-11-08T05:35:54.123 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.152599.log 2023-11-08T05:35:54.124 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.152599.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.152599.log.gz 2023-11-08T05:35:54.124 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.148076.log 2023-11-08T05:35:54.125 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.148076.log: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.148076.log.gz 2023-11-08T05:35:54.125 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.151335.log 2023-11-08T05:35:54.126 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.151335.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.151335.log.gz 2023-11-08T05:35:54.126 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.152141.log 2023-11-08T05:35:54.128 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.152141.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.152141.log.gz 2023-11-08T05:35:54.128 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.149441.log 2023-11-08T05:35:54.129 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.149441.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.149441.log.gz 2023-11-08T05:35:54.129 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.149110.log 2023-11-08T05:35:54.131 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.149110.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.149110.log.gz 2023-11-08T05:35:54.131 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.157553.log 2023-11-08T05:35:54.132 INFO:teuthology.orchestra.run.smithi087.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.135641.log.gz 2023-11-08T05:35:54.132 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.157553.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.157553.log.gz 2023-11-08T05:35:54.132 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.128895.log 2023-11-08T05:35:54.133 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.147116.log 2023-11-08T05:35:54.133 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.128895.log: /var/log/ceph/cluster1-client.mirror.147116.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.147116.log.gz 2023-11-08T05:35:54.134 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.128526.log 2023-11-08T05:35:54.135 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.128526.log: 89.3% -- replaced with /var/log/ceph/cluster1-client.mirror.128526.log.gz 2023-11-08T05:35:54.135 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.123908.log 2023-11-08T05:35:54.136 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.123908.log: 91.6% -- replaced with /var/log/ceph/cluster2-client.mirror.123908.log.gz 2023-11-08T05:35:54.137 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.164764.log 2023-11-08T05:35:54.138 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.admin.164764.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.164764.log.gz 2023-11-08T05:35:54.138 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.132984.log 2023-11-08T05:35:54.138 INFO:teuthology.orchestra.run.smithi087.stderr: 93.0% -- replaced with /var/log/ceph/cluster2-client.mirror.128895.log.gz 2023-11-08T05:35:54.139 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.132984.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.132984.log.gz 2023-11-08T05:35:54.139 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.2.120320.log 2023-11-08T05:35:54.140 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.130879.log 2023-11-08T05:35:54.143 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.2.120320.log: /var/log/ceph/cluster2-client.mirror.130879.log: 93.2% -- replaced with /var/log/ceph/cluster2-client.mirror.2.120320.log.gz 2023-11-08T05:35:54.143 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.168447.log 2023-11-08T05:35:54.144 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.admin.168447.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.168447.log.gz 2023-11-08T05:35:54.144 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.162404.log 2023-11-08T05:35:54.145 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.admin.162404.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.162404.log.gz 2023-11-08T05:35:54.145 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.140521.log 2023-11-08T05:35:54.145 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.140521.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.140521.log.gz 2023-11-08T05:35:54.146 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.139783.log 2023-11-08T05:35:54.147 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.139783.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.139783.log.gz 2023-11-08T05:35:54.147 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.128981.log 2023-11-08T05:35:54.152 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.128981.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.128981.log.gz 2023-11-08T05:35:54.152 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.148398.log 2023-11-08T05:35:54.153 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.148398.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.148398.log.gz 2023-11-08T05:35:54.153 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.150408.log 2023-11-08T05:35:54.154 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.150408.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.150408.log.gz 2023-11-08T05:35:54.154 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.153746.log 2023-11-08T05:35:54.155 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.153746.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.153746.log.gz 2023-11-08T05:35:54.155 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.134410.log 2023-11-08T05:35:54.156 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.134410.log: 79.6% -- replaced with /var/log/ceph/cluster2-client.mirror.134410.log.gz 2023-11-08T05:35:54.156 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127761.log 2023-11-08T05:35:54.160 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.127761.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.165499.log 2023-11-08T05:35:54.161 INFO:teuthology.orchestra.run.smithi087.stderr: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.127761.log.gz 2023-11-08T05:35:54.162 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.admin.165499.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.165499.log.gz 2023-11-08T05:35:54.162 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.151015.log 2023-11-08T05:35:54.164 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.151015.log: 90.5% -- replaced with /var/log/ceph/cluster2-client.mirror.151015.log.gz 2023-11-08T05:35:54.165 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.118741.log 2023-11-08T05:35:54.165 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.admin.118741.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.118741.log.gz 2023-11-08T05:35:54.166 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.119594.log 2023-11-08T05:35:54.166 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.admin.119594.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.119594.log.gz 2023-11-08T05:35:54.167 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137124.log 2023-11-08T05:35:54.168 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.137124.log: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.137124.log.gz 2023-11-08T05:35:54.169 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.168398.log 2023-11-08T05:35:54.169 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.admin.168398.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.168398.log.gz 2023-11-08T05:35:54.169 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.134822.log 2023-11-08T05:35:54.170 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.134822.log: 79.6% -- replaced with /var/log/ceph/cluster2-client.mirror.134822.log.gz 2023-11-08T05:35:54.171 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.146650.log 2023-11-08T05:35:54.171 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.146650.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.146650.log.gz 2023-11-08T05:35:54.171 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.133516.log 2023-11-08T05:35:54.172 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.133516.log: 79.6% -- replaced with /var/log/ceph/cluster2-client.mirror.133516.log.gz 2023-11-08T05:35:54.172 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.154399.log 2023-11-08T05:35:54.173 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.154399.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.154399.log.gz 2023-11-08T05:35:54.173 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.131380.log 2023-11-08T05:35:54.174 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.131380.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.131380.log.gz 2023-11-08T05:35:54.174 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.154444.log 2023-11-08T05:35:54.175 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.154444.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.154444.log.gz 2023-11-08T05:35:54.175 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125971.log 2023-11-08T05:35:54.176 INFO:teuthology.orchestra.run.smithi087.stderr: 94.2% -- replaced with /var/log/ceph/cluster2-client.mirror.123155.log.gz 2023-11-08T05:35:54.176 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.125971.log: 88.2% -- replaced with /var/log/ceph/cluster1-client.mirror.125971.log.gz 2023-11-08T05:35:54.176 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.165940.log 2023-11-08T05:35:54.177 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.154714.log 2023-11-08T05:35:54.177 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.admin.165940.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.165940.log.gz 2023-11-08T05:35:54.178 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.154714.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.154714.log.gz 2023-11-08T05:35:54.178 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.123398.log 2023-11-08T05:35:54.179 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.152162.log 2023-11-08T05:35:54.179 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.123398.log: /var/log/ceph/cluster2-client.mirror.152162.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.123398.log.gz 2023-11-08T05:35:54.180 INFO:teuthology.orchestra.run.smithi087.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.152162.log.gz 2023-11-08T05:35:54.180 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.136167.log 2023-11-08T05:35:54.180 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.138880.log 2023-11-08T05:35:54.181 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.136167.log: /var/log/ceph/cluster2-client.mirror.138880.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.138880.log.gz 2023-11-08T05:35:54.181 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.150119.log 2023-11-08T05:35:54.181 INFO:teuthology.orchestra.run.smithi087.stderr: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.136167.log.gz 2023-11-08T05:35:54.182 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.150119.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.150119.log.gz 2023-11-08T05:35:54.182 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.124339.log 2023-11-08T05:35:54.183 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.140289.log 2023-11-08T05:35:54.183 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.124339.log: /var/log/ceph/cluster1-client.mirror.140289.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.140289.log.gz 2023-11-08T05:35:54.183 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125947.log 2023-11-08T05:35:54.184 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.125947.log: 88.3% -- replaced with /var/log/ceph/cluster1-client.mirror.125947.log.gz 2023-11-08T05:35:54.184 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.135483.log 2023-11-08T05:35:54.185 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.135483.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.135483.log.gz 2023-11-08T05:35:54.185 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.1.142024.log 2023-11-08T05:35:54.197 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.1.142024.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125664.log 2023-11-08T05:35:54.204 INFO:teuthology.orchestra.run.smithi087.stderr: 94.2% -- replaced with /var/log/ceph/cluster2-client.mirror.1.142024.log.gz 2023-11-08T05:35:54.210 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.125664.log: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.125664.log.gz 2023-11-08T05:35:54.210 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.152959.log 2023-11-08T05:35:54.211 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.152959.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.152959.log.gz 2023-11-08T05:35:54.211 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.126620.log 2023-11-08T05:35:54.212 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.126620.log: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.126620.log.gz 2023-11-08T05:35:54.212 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.153297.log 2023-11-08T05:35:54.213 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.166332.log 2023-11-08T05:35:54.214 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.153297.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.153297.log.gz 2023-11-08T05:35:54.214 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.admin.166332.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.166332.log.gz 2023-11-08T05:35:54.214 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.121303.log 2023-11-08T05:35:54.215 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.162747.log 2023-11-08T05:35:54.215 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.121303.log: 54.5% -- replaced with /var/log/ceph/cluster2-client.mirror.121303.log.gz 2023-11-08T05:35:54.215 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.admin.162747.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.162747.log.gz 2023-11-08T05:35:54.215 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.3.132878.log 2023-11-08T05:35:54.225 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.166634.log 2023-11-08T05:35:54.225 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.3.132878.log: /var/log/ceph/cluster2-client.admin.166634.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.166634.log.gz 2023-11-08T05:35:54.226 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.147348.log 2023-11-08T05:35:54.233 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.147348.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.147348.log.gz 2023-11-08T05:35:54.233 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.157682.log 2023-11-08T05:35:54.233 INFO:teuthology.orchestra.run.smithi087.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.131850.log.gz 2023-11-08T05:35:54.234 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.157682.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.157682.log.gz 2023-11-08T05:35:54.234 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.149863.log 2023-11-08T05:35:54.234 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.167516.log 2023-11-08T05:35:54.235 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.149863.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.149863.log.gz 2023-11-08T05:35:54.235 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.admin.167516.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.167516.log.gz 2023-11-08T05:35:54.235 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.134577.log 2023-11-08T05:35:54.236 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.151397.log 2023-11-08T05:35:54.236 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.134577.log: 79.6% -- replaced with /var/log/ceph/cluster2-client.mirror.134577.log.gz 2023-11-08T05:35:54.236 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.128257.log 2023-11-08T05:35:54.236 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.151397.log: /var/log/ceph/cluster2-client.mirror.128257.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.151397.log.gz 2023-11-08T05:35:54.237 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.143800.log 2023-11-08T05:35:54.238 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.143800.log: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.143800.log.gz 2023-11-08T05:35:54.238 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.149776.log 2023-11-08T05:35:54.239 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.149776.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.149776.log.gz 2023-11-08T05:35:54.240 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.129001.log 2023-11-08T05:35:54.242 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.129001.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.124859.log.gz 2023-11-08T05:35:54.243 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.134043.log 2023-11-08T05:35:54.244 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.134043.log: 79.3% -- replaced with /var/log/ceph/cluster2-client.mirror.134043.log.gz 2023-11-08T05:35:54.244 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.129863.log 2023-11-08T05:35:54.245 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.129863.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.129863.log.gz 2023-11-08T05:35:54.245 INFO:teuthology.orchestra.run.smithi087.stderr: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.129001.log.gz 2023-11-08T05:35:54.246 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.149006.log 2023-11-08T05:35:54.246 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.149006.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.149006.log.gz 2023-11-08T05:35:54.247 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.134340.log 2023-11-08T05:35:54.248 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.134340.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.134340.log.gz 2023-11-08T05:35:54.248 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.140138.log 2023-11-08T05:35:54.249 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.140138.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.140138.log.gz 2023-11-08T05:35:54.249 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127902.log 2023-11-08T05:35:54.250 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.127902.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.127902.log.gz 2023-11-08T05:35:54.250 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.123109.log 2023-11-08T05:35:54.251 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.admin.123109.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.admin.123109.log.gz 2023-11-08T05:35:54.251 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.153365.log 2023-11-08T05:35:54.252 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.153365.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.153365.log.gz 2023-11-08T05:35:54.252 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.167026.log 2023-11-08T05:35:54.253 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.admin.167026.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.167026.log.gz 2023-11-08T05:35:54.253 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.151893.log 2023-11-08T05:35:54.254 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.151893.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.151893.log.gz 2023-11-08T05:35:54.254 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.156633.log 2023-11-08T05:35:54.255 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.156633.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.156633.log.gz 2023-11-08T05:35:54.255 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.130777.log 2023-11-08T05:35:54.260 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.130777.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126719.log 2023-11-08T05:35:54.272 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.126719.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.126719.log.gz 2023-11-08T05:35:54.282 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138299.log 2023-11-08T05:35:54.283 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.138299.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.138299.log.gz 2023-11-08T05:35:54.284 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.150491.log 2023-11-08T05:35:54.285 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.150491.log: 94.2% -- replaced with /var/log/ceph/cluster2-client.mirror.3.132878.log.gz 2023-11-08T05:35:54.285 INFO:teuthology.orchestra.run.smithi087.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.150491.log.gz 2023-11-08T05:35:54.285 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.154219.log 2023-11-08T05:35:54.286 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.154219.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.154219.log.gz 2023-11-08T05:35:54.286 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.127397.log 2023-11-08T05:35:54.287 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.admin.127397.log: 88.1% -- replaced with /var/log/ceph/cluster1-client.admin.127397.log.gz 2023-11-08T05:35:54.287 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127672.log 2023-11-08T05:35:54.288 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.127672.log: 89.0% -- replaced with /var/log/ceph/cluster1-client.mirror.127672.log.gz 2023-11-08T05:35:54.288 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.164372.log 2023-11-08T05:35:54.289 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.admin.164372.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.164372.log.gz 2023-11-08T05:35:54.289 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.124979.log 2023-11-08T05:35:54.290 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.124979.log: 91.6% -- replaced with /var/log/ceph/cluster2-client.mirror.124979.log.gz 2023-11-08T05:35:54.291 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.158632.log 2023-11-08T05:35:54.291 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.158632.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.158632.log.gz 2023-11-08T05:35:54.292 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.121359.log 2023-11-08T05:35:54.293 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.121359.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.121359.log.gz 2023-11-08T05:35:54.293 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.130683.log 2023-11-08T05:35:54.298 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.130683.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.133106.log 2023-11-08T05:35:54.320 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.133106.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.144270.log 2023-11-08T05:35:54.335 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.144270.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.144270.log.gz 2023-11-08T05:35:54.335 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.151976.log 2023-11-08T05:35:54.354 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.151976.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.151976.log.gz 2023-11-08T05:35:54.354 INFO:teuthology.orchestra.run.smithi087.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.134636.log.gz 2023-11-08T05:35:54.354 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.127207.log 2023-11-08T05:35:54.363 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.127207.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.124953.log 2023-11-08T05:35:54.379 INFO:teuthology.orchestra.run.smithi087.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.127207.log.gz 2023-11-08T05:35:54.379 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.124953.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.169582.log 2023-11-08T05:35:54.379 INFO:teuthology.orchestra.run.smithi087.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.124953.log.gz 2023-11-08T05:35:54.380 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.admin.169582.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.169582.log.gz 2023-11-08T05:35:54.380 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.147044.log 2023-11-08T05:35:54.380 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.147044.log: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.147044.log.gz 2023-11-08T05:35:54.380 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.166781.log 2023-11-08T05:35:54.380 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.admin.166781.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.166781.log.gz 2023-11-08T05:35:54.381 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125762.log 2023-11-08T05:35:54.381 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.125762.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.125762.log.gz 2023-11-08T05:35:54.381 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.148959.log 2023-11-08T05:35:54.390 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.148959.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.148959.log.gz 2023-11-08T05:35:54.390 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.124275.log 2023-11-08T05:35:54.390 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.124275.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.124275.log.gz 2023-11-08T05:35:54.390 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.146340.log 2023-11-08T05:35:54.391 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.146340.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.146340.log.gz 2023-11-08T05:35:54.391 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.144808.log 2023-11-08T05:35:54.391 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.144808.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.144808.log.gz 2023-11-08T05:35:54.391 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.110676.log 2023-11-08T05:35:54.391 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/ceph.tmp-client.admin.110676.log: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.110676.log.gz 2023-11-08T05:35:54.392 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.134780.log 2023-11-08T05:35:54.392 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.134780.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.134780.log.gz 2023-11-08T05:35:54.392 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.142972.log 2023-11-08T05:35:54.394 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.142972.log: 92.7% -- replaced with /var/log/ceph/cluster2-client.mirror.142972.log.gz 2023-11-08T05:35:54.395 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.152847.log 2023-11-08T05:35:54.395 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.152847.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.152847.log.gz 2023-11-08T05:35:54.395 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.166928.log 2023-11-08T05:35:54.395 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.admin.166928.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.166928.log.gz 2023-11-08T05:35:54.395 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.145949.log 2023-11-08T05:35:54.396 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.145949.log: 93.6% -- replaced with /var/log/ceph/cluster2-client.mirror.133106.log.gz 2023-11-08T05:35:54.396 INFO:teuthology.orchestra.run.smithi087.stderr: 92.3% -- replaced with /var/log/ceph/cluster2-client.mirror.145949.log.gz 2023-11-08T05:35:54.397 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.139515.log 2023-11-08T05:35:54.397 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.139515.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.139515.log.gz 2023-11-08T05:35:54.397 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.124163.log 2023-11-08T05:35:54.398 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.124163.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.124163.log.gz 2023-11-08T05:35:54.398 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.136660.log 2023-11-08T05:35:54.400 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.136660.log: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.136660.log.gz 2023-11-08T05:35:54.400 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.148144.log 2023-11-08T05:35:54.401 INFO:teuthology.orchestra.run.smithi087.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.130879.log.gz 2023-11-08T05:35:54.401 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.148144.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.148144.log.gz 2023-11-08T05:35:54.401 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.148838.log 2023-11-08T05:35:54.402 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.145744.log 2023-11-08T05:35:54.403 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.148838.log: /var/log/ceph/cluster1-client.mirror.145744.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.148838.log.gz 2023-11-08T05:35:54.403 INFO:teuthology.orchestra.run.smithi087.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.145744.log.gz 2023-11-08T05:35:54.404 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.134261.log 2023-11-08T05:35:54.404 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.120347.log 2023-11-08T05:35:54.404 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.134261.log: /var/log/ceph/cluster1-client.mirror.120347.log: 80.5% -- replaced with /var/log/ceph/cluster1-client.mirror.120347.log.gz 2023-11-08T05:35:54.404 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.162551.log 2023-11-08T05:35:54.404 INFO:teuthology.orchestra.run.smithi087.stderr: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.134261.log.gz 2023-11-08T05:35:54.405 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.129626.log 2023-11-08T05:35:54.405 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.admin.162551.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.162551.log.gz 2023-11-08T05:35:54.409 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.129626.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.129626.log.gz 2023-11-08T05:35:54.409 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.158760.log 2023-11-08T05:35:54.410 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.132357.log 2023-11-08T05:35:54.410 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.158760.log: /var/log/ceph/cluster2-client.mirror.132357.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.158760.log.gz 2023-11-08T05:35:54.411 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.144407.log 2023-11-08T05:35:54.411 INFO:teuthology.orchestra.run.smithi087.stderr: 79.4% -- replaced with /var/log/ceph/cluster2-client.mirror.132357.log.gz 2023-11-08T05:35:54.411 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.143111.log 2023-11-08T05:35:54.413 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.144407.log: /var/log/ceph/cluster1-client.mirror.143111.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.143111.log.gz 2023-11-08T05:35:54.413 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.128170.log 2023-11-08T05:35:54.416 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.128170.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.144407.log.gz 2023-11-08T05:35:54.416 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.133446.log 2023-11-08T05:35:54.416 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.133446.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.133446.log.gz 2023-11-08T05:35:54.417 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.120764.log 2023-11-08T05:35:54.417 INFO:teuthology.orchestra.run.smithi087.stderr: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.128170.log.gz 2023-11-08T05:35:54.417 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.134427.log 2023-11-08T05:35:54.418 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.120764.log: 3.5% -- replaced with /var/log/ceph/cluster1-client.mirror.120764.log.gz 2023-11-08T05:35:54.418 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.134427.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.134427.log.gz 2023-11-08T05:35:54.418 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.151119.log 2023-11-08T05:35:54.419 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137291.log 2023-11-08T05:35:54.419 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.151119.log: /var/log/ceph/cluster1-client.mirror.137291.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.151119.log.gz 2023-11-08T05:35:54.419 INFO:teuthology.orchestra.run.smithi087.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.137291.log.gz 2023-11-08T05:35:54.419 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.165401.log 2023-11-08T05:35:54.420 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.159797.log 2023-11-08T05:35:54.421 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.admin.165401.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.165401.log.gz 2023-11-08T05:35:54.421 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.admin.159797.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.159797.log.gz 2023-11-08T05:35:54.421 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.143431.log 2023-11-08T05:35:54.421 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.134614.log 2023-11-08T05:35:54.421 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.143431.log: /var/log/ceph/cluster2-client.mirror.134614.log: 87.2% -- replaced with /var/log/ceph/cluster2-client.mirror.143431.log.gz 2023-11-08T05:35:54.422 INFO:teuthology.orchestra.run.smithi087.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.134614.log.gz 2023-11-08T05:35:54.422 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.153859.log 2023-11-08T05:35:54.423 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.148220.log 2023-11-08T05:35:54.423 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.153859.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.153859.log.gz 2023-11-08T05:35:54.423 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.148220.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.148220.log.gz 2023-11-08T05:35:54.423 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.139124.log 2023-11-08T05:35:54.423 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.129697.log 2023-11-08T05:35:54.424 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.139124.log: /var/log/ceph/cluster1-client.mirror.129697.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.139124.log.gz 2023-11-08T05:35:54.424 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.131558.log 2023-11-08T05:35:54.426 INFO:teuthology.orchestra.run.smithi087.stderr: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.129697.log.gz 2023-11-08T05:35:54.426 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.158119.log 2023-11-08T05:35:54.427 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.131558.log: /var/log/ceph/cluster1-client.mirror.158119.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.158119.log.gz 2023-11-08T05:35:54.428 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.159574.log 2023-11-08T05:35:54.428 INFO:teuthology.orchestra.run.smithi087.stderr: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.131558.log.gz 2023-11-08T05:35:54.428 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.130537.log 2023-11-08T05:35:54.429 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.admin.159574.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.159574.log.gz 2023-11-08T05:35:54.429 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.130537.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.148886.log 2023-11-08T05:35:54.429 INFO:teuthology.orchestra.run.smithi087.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.130537.log.gz 2023-11-08T05:35:54.430 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.148886.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.148886.log.gz 2023-11-08T05:35:54.430 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.142153.log 2023-11-08T05:35:54.430 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.136827.log 2023-11-08T05:35:54.431 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.142153.log: /var/log/ceph/cluster1-client.mirror.136827.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.142153.log.gz 2023-11-08T05:35:54.431 INFO:teuthology.orchestra.run.smithi087.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.136827.log.gz 2023-11-08T05:35:54.431 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.120201.log 2023-11-08T05:35:54.436 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.132416.log 2023-11-08T05:35:54.436 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.120201.log: /var/log/ceph/cluster2-client.mirror.132416.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.147276.log 2023-11-08T05:35:54.447 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.147276.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.147276.log.gz 2023-11-08T05:35:54.447 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.165058.log 2023-11-08T05:35:54.458 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.admin.165058.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.165058.log.gz 2023-11-08T05:35:54.458 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.169386.log 2023-11-08T05:35:54.469 INFO:teuthology.orchestra.run.smithi087.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.124339.log.gz 2023-11-08T05:35:54.469 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.admin.169386.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.169386.log.gz 2023-11-08T05:35:54.469 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.167124.log 2023-11-08T05:35:54.469 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.admin.167124.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.167124.log.gz 2023-11-08T05:35:54.470 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.169035.log 2023-11-08T05:35:54.470 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.admin.169035.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.169035.log.gz 2023-11-08T05:35:54.470 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.135159.log 2023-11-08T05:35:54.480 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.135159.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.146943.log 2023-11-08T05:35:54.480 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.146943.log: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.146943.log.gz 2023-11-08T05:35:54.480 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.153117.log 2023-11-08T05:35:54.480 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.153117.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.153117.log.gz 2023-11-08T05:35:54.481 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.119153.log 2023-11-08T05:35:54.491 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.admin.119153.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.119153.log.gz 2023-11-08T05:35:54.491 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.145262.log 2023-11-08T05:35:54.491 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.145262.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.145262.log.gz 2023-11-08T05:35:54.492 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.162208.log 2023-11-08T05:35:54.492 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.admin.162208.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.162208.log.gz 2023-11-08T05:35:54.492 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.129763.log 2023-11-08T05:35:54.492 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.129763.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.129763.log.gz 2023-11-08T05:35:54.492 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.124731.log 2023-11-08T05:35:54.493 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.124731.log: 87.9% -- replaced with /var/log/ceph/cluster1-client.mirror.124731.log.gz 2023-11-08T05:35:54.493 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.131479.log 2023-11-08T05:35:54.493 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.131479.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.131479.log.gz 2023-11-08T05:35:54.493 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.144546.log 2023-11-08T05:35:54.493 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.144546.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.144546.log.gz 2023-11-08T05:35:54.494 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.148509.log 2023-11-08T05:35:54.494 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.148509.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.148509.log.gz 2023-11-08T05:35:54.494 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.154016.log 2023-11-08T05:35:54.494 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.154016.log: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.154016.log.gz 2023-11-08T05:35:54.494 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.157236.log 2023-11-08T05:35:54.495 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.157236.log: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.133832.log.gz 2023-11-08T05:35:54.495 INFO:teuthology.orchestra.run.smithi087.stderr: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.157236.log.gz 2023-11-08T05:35:54.495 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.158247.log 2023-11-08T05:35:54.496 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.158247.log: 79.3% -- replaced with /var/log/ceph/cluster1-client.mirror.158247.log.gz 2023-11-08T05:35:54.497 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.124139.log 2023-11-08T05:35:54.497 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.124139.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.124139.log.gz 2023-11-08T05:35:54.497 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.141030.log 2023-11-08T05:35:54.497 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.141030.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.141030.log.gz 2023-11-08T05:35:54.497 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.149461.log 2023-11-08T05:35:54.498 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.149461.log: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.120201.log.gz 2023-11-08T05:35:54.498 INFO:teuthology.orchestra.run.smithi087.stderr: 90.5% -- replaced with /var/log/ceph/cluster2-client.mirror.149461.log.gz 2023-11-08T05:35:54.498 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.157875.log 2023-11-08T05:35:54.498 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.130718.log 2023-11-08T05:35:54.499 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.0.157875.log: /var/log/ceph/cluster2-client.mirror.130718.log: 79.7% -- replaced with /var/log/ceph/cluster2-client.mirror.130718.log.gz 2023-11-08T05:35:54.499 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.129196.log 2023-11-08T05:35:54.500 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.129196.log: 93.1% -- replaced with /var/log/ceph/cluster1-client.mirror.0.157875.log.gz 2023-11-08T05:35:54.503 INFO:teuthology.orchestra.run.smithi087.stderr: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.129196.log.gz 2023-11-08T05:35:54.504 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.157139.log 2023-11-08T05:35:54.504 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.150532.log 2023-11-08T05:35:54.505 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.157139.log: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.157139.log.gz 2023-11-08T05:35:54.505 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.150532.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.150532.log.gz 2023-11-08T05:35:54.506 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.128662.log 2023-11-08T05:35:54.506 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.146076.log 2023-11-08T05:35:54.507 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.128662.log: /var/log/ceph/cluster1-client.mirror.146076.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.146076.log.gz 2023-11-08T05:35:54.507 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.148672.log 2023-11-08T05:35:54.508 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.148672.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.148672.log.gz 2023-11-08T05:35:54.508 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.139711.log 2023-11-08T05:35:54.509 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.139711.log: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.139711.log.gz 2023-11-08T05:35:54.509 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.143246.log 2023-11-08T05:35:54.509 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.143246.log: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.143246.log.gz 2023-11-08T05:35:54.510 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.165107.log 2023-11-08T05:35:54.511 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.admin.165107.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.165107.log.gz 2023-11-08T05:35:54.511 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.163980.log 2023-11-08T05:35:54.511 INFO:teuthology.orchestra.run.smithi087.stderr: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.128662.log.gz 2023-11-08T05:35:54.511 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.admin.163980.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.163980.log.gz 2023-11-08T05:35:54.512 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.132582.log 2023-11-08T05:35:54.512 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.131069.log 2023-11-08T05:35:54.522 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.132582.log: /var/log/ceph/cluster2-client.mirror.131069.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.128478.log 2023-11-08T05:35:54.523 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.128478.log: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.128478.log.gz 2023-11-08T05:35:54.524 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.150553.log 2023-11-08T05:35:54.535 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.150553.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.150553.log.gz 2023-11-08T05:35:54.535 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.144044.log 2023-11-08T05:35:54.537 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.144044.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.144044.log.gz 2023-11-08T05:35:54.537 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.156329.log 2023-11-08T05:35:54.538 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.156329.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.156329.log.gz 2023-11-08T05:35:54.538 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.2.125247.log 2023-11-08T05:35:54.544 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.2.125247.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.119496.log 2023-11-08T05:35:54.556 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.admin.119496.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.119496.log.gz 2023-11-08T05:35:54.556 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.151624.log 2023-11-08T05:35:54.563 INFO:teuthology.orchestra.run.smithi087.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.2.125247.log.gz 2023-11-08T05:35:54.566 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.151624.log: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.151624.log.gz 2023-11-08T05:35:54.566 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.123494.log 2023-11-08T05:35:54.567 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.123494.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.123494.log.gz 2023-11-08T05:35:54.567 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.129646.log 2023-11-08T05:35:54.572 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.129646.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.123603.log 2023-11-08T05:35:54.573 INFO:teuthology.orchestra.run.smithi087.stderr: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.129646.log.gz 2023-11-08T05:35:54.584 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.123603.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.130683.log.gz 2023-11-08T05:35:54.584 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.129216.log 2023-11-08T05:35:54.584 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.129216.log: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.130777.log.gz 2023-11-08T05:35:54.586 INFO:teuthology.orchestra.run.smithi087.stderr: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.129216.log.gz 2023-11-08T05:35:54.586 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.158069.log 2023-11-08T05:35:54.589 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.158069.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.158069.log.gz 2023-11-08T05:35:54.589 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.145105.log 2023-11-08T05:35:54.591 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.145105.log: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.145105.log.gz 2023-11-08T05:35:54.591 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.141008.log 2023-11-08T05:35:54.593 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.141008.log: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.141008.log.gz 2023-11-08T05:35:54.594 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.149088.log 2023-11-08T05:35:54.596 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.149088.log: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.149088.log.gz 2023-11-08T05:35:54.596 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.132661.log 2023-11-08T05:35:54.607 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.132661.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.154534.log 2023-11-08T05:35:54.607 INFO:teuthology.orchestra.run.smithi087.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.128257.log.gz 2023-11-08T05:35:54.608 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.154534.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.154534.log.gz 2023-11-08T05:35:54.608 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.132793.log 2023-11-08T05:35:54.618 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.146902.log 2023-11-08T05:35:54.618 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.132793.log: /var/log/ceph/cluster1-client.mirror.146902.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.132793.log.gz 2023-11-08T05:35:54.618 INFO:teuthology.orchestra.run.smithi087.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.146902.log.gz 2023-11-08T05:35:54.619 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.123959.log 2023-11-08T05:35:54.619 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.123959.log: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.123959.log.gz 2023-11-08T05:35:54.619 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.159176.log 2023-11-08T05:35:54.619 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.159176.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.159176.log.gz 2023-11-08T05:35:54.619 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.170268.log 2023-11-08T05:35:54.620 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.admin.170268.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.170268.log.gz 2023-11-08T05:35:54.620 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.131400.log 2023-11-08T05:35:54.620 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.131400.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.131400.log.gz 2023-11-08T05:35:54.620 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.158396.log 2023-11-08T05:35:54.620 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.158396.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.157791.log 2023-11-08T05:35:54.620 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.157791.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.157791.log.gz 2023-11-08T05:35:54.621 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.130500.log 2023-11-08T05:35:54.628 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.130500.log: 79.7% -- replaced with /var/log/ceph/cluster2-client.mirror.130500.log.gz 2023-11-08T05:35:54.628 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.152644.log 2023-11-08T05:35:54.628 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.152644.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.152644.log.gz 2023-11-08T05:35:54.629 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.143598.log 2023-11-08T05:35:54.629 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.143598.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.143598.log.gz 2023-11-08T05:35:54.629 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.141504.log 2023-11-08T05:35:54.629 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.141504.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.141504.log.gz 2023-11-08T05:35:54.629 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.160973.log 2023-11-08T05:35:54.630 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.admin.160973.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.160973.log.gz 2023-11-08T05:35:54.630 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.170121.log 2023-11-08T05:35:54.630 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.admin.170121.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.170121.log.gz 2023-11-08T05:35:54.630 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.132306.log 2023-11-08T05:35:54.630 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.132306.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.132306.log.gz 2023-11-08T05:35:54.631 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.140634.log 2023-11-08T05:35:54.631 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.140634.log: 92.1% -- replaced with /var/log/ceph/cluster1-client.mirror.140634.log.gz 2023-11-08T05:35:54.638 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.136851.log 2023-11-08T05:35:54.638 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.136851.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.136851.log.gz 2023-11-08T05:35:54.638 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.140943.log 2023-11-08T05:35:54.639 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.140943.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.140943.log.gz 2023-11-08T05:35:54.639 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.159000.log 2023-11-08T05:35:54.639 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.159000.log: 79.7% -- replaced with /var/log/ceph/cluster2-client.mirror.159000.log.gz 2023-11-08T05:35:54.639 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.140473.log 2023-11-08T05:35:54.639 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.140473.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.140473.log.gz 2023-11-08T05:35:54.640 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.144223.log 2023-11-08T05:35:54.640 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.144223.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.144223.log.gz 2023-11-08T05:35:54.640 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.154624.log 2023-11-08T05:35:54.640 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.154624.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.154624.log.gz 2023-11-08T05:35:54.640 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.130103.log 2023-11-08T05:35:54.641 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.130103.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.130103.log.gz 2023-11-08T05:35:54.641 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.139037.log 2023-11-08T05:35:54.641 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.139037.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.139037.log.gz 2023-11-08T05:35:54.641 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.131010.log 2023-11-08T05:35:54.642 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.131010.log: 79.4% -- replaced with /var/log/ceph/cluster2-client.mirror.131010.log.gz 2023-11-08T05:35:54.642 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125508.log 2023-11-08T05:35:54.652 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.125508.log: 88.0% -- replaced with /var/log/ceph/cluster1-client.mirror.125508.log.gz 2023-11-08T05:35:54.652 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.169631.log 2023-11-08T05:35:54.654 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.admin.169631.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.169631.log.gz 2023-11-08T05:35:54.654 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.150718.log 2023-11-08T05:35:54.655 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.150718.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.150718.log.gz 2023-11-08T05:35:54.655 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2.log 2023-11-08T05:35:54.661 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2.log: 87.6% -- replaced with /var/log/ceph/cluster2.log.gz 2023-11-08T05:35:54.661 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.138251.log 2023-11-08T05:35:54.663 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.138251.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.138251.log.gz 2023-11-08T05:35:54.663 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.120832.log 2023-11-08T05:35:54.663 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.120832.log: 54.2% -- replaced with /var/log/ceph/cluster1-client.mirror.120832.log.gz 2023-11-08T05:35:54.664 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.130820.log 2023-11-08T05:35:54.664 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.130820.log: 79.5% -- replaced with /var/log/ceph/cluster2-client.mirror.130820.log.gz 2023-11-08T05:35:54.665 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.154556.log 2023-11-08T05:35:54.666 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.154556.log: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.154556.log.gz 2023-11-08T05:35:54.666 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137364.log 2023-11-08T05:35:54.668 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.137364.log: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.137364.log.gz 2023-11-08T05:35:54.668 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.138368.log 2023-11-08T05:35:54.670 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.138368.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.138368.log.gz 2023-11-08T05:35:54.671 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.128704.log 2023-11-08T05:35:54.672 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.128704.log: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.128704.log.gz 2023-11-08T05:35:54.672 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.150780.log 2023-11-08T05:35:54.674 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.150780.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.150780.log.gz 2023-11-08T05:35:54.674 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.147655.log 2023-11-08T05:35:54.676 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.147655.log: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.147655.log.gz 2023-11-08T05:35:54.676 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.133369.log 2023-11-08T05:35:54.677 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.133369.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.133369.log.gz 2023-11-08T05:35:54.677 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.131301.log 2023-11-08T05:35:54.679 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.131301.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.131301.log.gz 2023-11-08T05:35:54.679 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.127826.log 2023-11-08T05:35:54.680 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.127826.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.127826.log.gz 2023-11-08T05:35:54.681 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.162502.log 2023-11-08T05:35:54.681 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.admin.162502.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.162502.log.gz 2023-11-08T05:35:54.682 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.164176.log 2023-11-08T05:35:54.682 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.admin.164176.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.164176.log.gz 2023-11-08T05:35:54.683 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.147917.log 2023-11-08T05:35:54.684 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.147917.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.147917.log.gz 2023-11-08T05:35:54.684 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.151831.log 2023-11-08T05:35:54.685 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.151831.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.151831.log.gz 2023-11-08T05:35:54.686 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.144088.log 2023-11-08T05:35:54.687 INFO:teuthology.orchestra.run.smithi087.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.132416.log.gz 2023-11-08T05:35:54.687 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.144088.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.144088.log.gz 2023-11-08T05:35:54.687 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.120654.log 2023-11-08T05:35:54.688 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.admin.120654.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.120654.log.gz 2023-11-08T05:35:54.688 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.116185.log 2023-11-08T05:35:54.689 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.157812.log 2023-11-08T05:35:54.689 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.150263.log 2023-11-08T05:35:54.689 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.157812.log: /var/log/ceph/cluster2-client.admin.116185.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.116185.log.gz 2023-11-08T05:35:54.689 INFO:teuthology.orchestra.run.smithi087.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.157812.log.gz 2023-11-08T05:35:54.691 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.150263.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.134909.log 2023-11-08T05:35:54.692 INFO:teuthology.orchestra.run.smithi087.stderr: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.150263.log.gz 2023-11-08T05:35:54.692 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.134909.log: 79.4% -- replaced with /var/log/ceph/cluster2-client.mirror.134909.log.gz 2023-11-08T05:35:54.692 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.162012.log 2023-11-08T05:35:54.693 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.admin.162012.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.162012.log.gz 2023-11-08T05:35:54.693 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.139650.log 2023-11-08T05:35:54.695 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.139650.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.139650.log.gz 2023-11-08T05:35:54.695 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.3.145815.log 2023-11-08T05:35:54.700 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.3.145815.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.158740.log 2023-11-08T05:35:54.711 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.158740.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.158740.log.gz 2023-11-08T05:35:54.711 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.160826.log 2023-11-08T05:35:54.722 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.admin.160826.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.160826.log.gz 2023-11-08T05:35:54.722 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.129903.log 2023-11-08T05:35:54.722 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.129903.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.129903.log.gz 2023-11-08T05:35:54.722 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.139183.log 2023-11-08T05:35:54.723 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.139183.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.139183.log.gz 2023-11-08T05:35:54.723 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.149904.log 2023-11-08T05:35:54.723 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.149904.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.149904.log.gz 2023-11-08T05:35:54.723 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.124317.log 2023-11-08T05:35:54.723 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.124317.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.124317.log.gz 2023-11-08T05:35:54.724 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.132602.log 2023-11-08T05:35:54.724 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.132602.log: 79.6% -- replaced with /var/log/ceph/cluster2-client.mirror.132602.log.gz 2023-11-08T05:35:54.724 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.134080.log 2023-11-08T05:35:54.724 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.134080.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.134080.log.gz 2023-11-08T05:35:54.724 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.3.127073.log 2023-11-08T05:35:54.733 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.3.127073.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127030.log 2023-11-08T05:35:54.744 INFO:teuthology.orchestra.run.smithi087.stderr: 93.3% -- replaced with /var/log/ceph/cluster2-client.mirror.3.127073.log.gz 2023-11-08T05:35:54.745 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.127030.log: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.131069.log.gz 2023-11-08T05:35:54.746 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.140378.log 2023-11-08T05:35:54.746 INFO:teuthology.orchestra.run.smithi087.stderr: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.127030.log.gz 2023-11-08T05:35:54.747 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.140378.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.158396.log.gz 2023-11-08T05:35:54.748 INFO:teuthology.orchestra.run.smithi087.stderr: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.140378.log.gz 2023-11-08T05:35:54.748 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.0.126246.log 2023-11-08T05:35:54.749 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.151521.log 2023-11-08T05:35:54.749 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.0.126246.log: /var/log/ceph/cluster2-client.mirror.151521.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.151521.log.gz 2023-11-08T05:35:54.749 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.160581.log 2023-11-08T05:35:54.750 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.admin.160581.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.160581.log.gz 2023-11-08T05:35:54.751 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.168104.log 2023-11-08T05:35:54.751 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.admin.168104.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.168104.log.gz 2023-11-08T05:35:54.751 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.126147.log 2023-11-08T05:35:54.752 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.admin.126147.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.admin.126147.log.gz 2023-11-08T05:35:54.752 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.157639.log 2023-11-08T05:35:54.753 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.157639.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.157639.log.gz 2023-11-08T05:35:54.753 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.157380.log 2023-11-08T05:35:54.754 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.157380.log: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.0.126246.log.gz 2023-11-08T05:35:54.754 INFO:teuthology.orchestra.run.smithi087.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.157380.log.gz 2023-11-08T05:35:54.754 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.166879.log 2023-11-08T05:35:54.754 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.164029.log 2023-11-08T05:35:54.755 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.admin.166879.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.166879.log.gz 2023-11-08T05:35:54.756 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.admin.164029.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.164029.log.gz 2023-11-08T05:35:54.756 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.154832.log 2023-11-08T05:35:54.756 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.150697.log 2023-11-08T05:35:54.757 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.154832.log: /var/log/ceph/cluster2-client.mirror.150697.log: 90.5% -- replaced with /var/log/ceph/cluster2-client.mirror.150697.log.gz 2023-11-08T05:35:54.758 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.162061.log 2023-11-08T05:35:54.758 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.admin.162061.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.162061.log.gz 2023-11-08T05:35:54.759 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.142249.log 2023-11-08T05:35:54.759 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.142249.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.142249.log.gz 2023-11-08T05:35:54.759 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.158142.log 2023-11-08T05:35:54.760 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.158142.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.158142.log.gz 2023-11-08T05:35:54.760 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.131887.log 2023-11-08T05:35:54.761 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.131887.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.131887.log.gz 2023-11-08T05:35:54.762 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126466.log 2023-11-08T05:35:54.762 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.126466.log: 88.5% -- replaced with /var/log/ceph/cluster1-client.mirror.126466.log.gz 2023-11-08T05:35:54.763 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138965.log 2023-11-08T05:35:54.763 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.138965.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.138965.log.gz 2023-11-08T05:35:54.763 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.169337.log 2023-11-08T05:35:54.764 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.admin.169337.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.169337.log.gz 2023-11-08T05:35:54.764 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.129923.log 2023-11-08T05:35:54.765 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.129923.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.129923.log.gz 2023-11-08T05:35:54.765 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.134743.log 2023-11-08T05:35:54.766 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.134743.log: 79.6% -- replaced with /var/log/ceph/cluster2-client.mirror.134743.log.gz 2023-11-08T05:35:54.766 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.142992.log 2023-11-08T05:35:54.768 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.142992.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.135159.log.gz 2023-11-08T05:35:54.769 INFO:teuthology.orchestra.run.smithi087.stderr: 92.5% -- replaced with /var/log/ceph/cluster1-client.mirror.142992.log.gz 2023-11-08T05:35:54.769 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.157832.log 2023-11-08T05:35:54.769 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.143135.log 2023-11-08T05:35:54.770 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.157832.log: /var/log/ceph/cluster1-client.mirror.143135.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.143135.log.gz 2023-11-08T05:35:54.770 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.163931.log 2023-11-08T05:35:54.771 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.admin.163931.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.163931.log.gz 2023-11-08T05:35:54.771 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.138941.log 2023-11-08T05:35:54.772 INFO:teuthology.orchestra.run.smithi087.stderr: 91.6% -- replaced with /var/log/ceph/cluster2-client.mirror.157832.log.gz 2023-11-08T05:35:54.772 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.138941.log: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.138941.log.gz 2023-11-08T05:35:54.772 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137733.log 2023-11-08T05:35:54.773 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.123378.log 2023-11-08T05:35:54.773 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.137733.log: /var/log/ceph/cluster2-client.mirror.123378.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.137733.log.gz 2023-11-08T05:35:54.773 INFO:teuthology.orchestra.run.smithi087.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.123378.log.gz 2023-11-08T05:35:54.774 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.166683.log 2023-11-08T05:35:54.775 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.119790.log 2023-11-08T05:35:54.775 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.admin.166683.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.166683.log.gz 2023-11-08T05:35:54.775 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.admin.119790.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.119790.log.gz 2023-11-08T05:35:54.775 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.120713.log 2023-11-08T05:35:54.776 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.1.133106.log 2023-11-08T05:35:54.776 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.admin.120713.log: 66.4% -- replaced with /var/log/ceph/cluster1-client.admin.120713.log.gz 2023-11-08T05:35:54.780 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.1.133106.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.165352.log 2023-11-08T05:35:54.781 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.163784.log 2023-11-08T05:35:54.781 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.admin.165352.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.165352.log.gz 2023-11-08T05:35:54.789 INFO:teuthology.orchestra.run.smithi087.stderr: 92.7% -- replaced with /var/log/ceph/cluster1-client.mirror.3.145815.log.gz 2023-11-08T05:35:54.790 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.141980.log 2023-11-08T05:35:54.790 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.admin.163784.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.163784.log.gz 2023-11-08T05:35:54.790 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.147697.log 2023-11-08T05:35:54.791 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.141980.log: /var/log/ceph/cluster2-client.mirror.147697.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.141980.log.gz 2023-11-08T05:35:54.792 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.152058.log 2023-11-08T05:35:54.792 INFO:teuthology.orchestra.run.smithi087.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.147697.log.gz 2023-11-08T05:35:54.792 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.133753.log 2023-11-08T05:35:54.793 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.152058.log: /var/log/ceph/cluster2-client.mirror.133753.log: 90.5% -- replaced with /var/log/ceph/cluster2-client.mirror.152058.log.gz 2023-11-08T05:35:54.794 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.131784.log 2023-11-08T05:35:54.794 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.131784.log: 79.3% -- replaced with /var/log/ceph/cluster2-client.mirror.131784.log.gz 2023-11-08T05:35:54.794 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.140075.log 2023-11-08T05:35:54.795 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.140075.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.140075.log.gz 2023-11-08T05:35:54.795 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.163188.log 2023-11-08T05:35:54.796 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.admin.163188.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.163188.log.gz 2023-11-08T05:35:54.796 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.124295.log 2023-11-08T05:35:54.802 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.124295.log: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.124295.log.gz 2023-11-08T05:35:54.802 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.134303.log 2023-11-08T05:35:54.812 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.134303.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.136779.log 2023-11-08T05:35:54.824 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.136779.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.150677.log 2023-11-08T05:35:54.824 INFO:teuthology.orchestra.run.smithi087.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.136779.log.gz 2023-11-08T05:35:54.837 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.150677.log: 88.1%gzip -5 -- replaced with /var/log/ceph/cluster2-client.mirror.150677.log.gz--verbose 2023-11-08T05:35:54.837 INFO:teuthology.orchestra.run.smithi087.stderr:-- /var/log/ceph/cluster1-client.mirror.145334.log 2023-11-08T05:35:54.847 INFO:teuthology.orchestra.run.smithi087.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.132582.log.gz 2023-11-08T05:35:54.850 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.145334.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.145334.log.gz 2023-11-08T05:35:54.850 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.149689.log 2023-11-08T05:35:54.851 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.149689.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.149689.log.gz 2023-11-08T05:35:54.851 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.139394.log 2023-11-08T05:35:54.852 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.139394.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.139394.log.gz 2023-11-08T05:35:54.853 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.150015.log 2023-11-08T05:35:54.855 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.150015.log: 90.5% -- replaced with /var/log/ceph/cluster2-client.mirror.150015.log.gz 2023-11-08T05:35:54.855 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.166977.log 2023-11-08T05:35:54.856 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.admin.166977.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.166977.log.gz 2023-11-08T05:35:54.856 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.123539.log 2023-11-08T05:35:54.861 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.123539.log: 93.0% -- replaced with /var/log/ceph/cluster2-client.mirror.123539.log.gz 2023-11-08T05:35:54.862 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.129560.log 2023-11-08T05:35:54.863 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.129560.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.123603.log.gz 2023-11-08T05:35:54.864 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.130321.log 2023-11-08T05:35:54.864 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.130321.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.130321.log.gz 2023-11-08T05:35:54.865 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.141417.log 2023-11-08T05:35:54.865 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.141417.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.141417.log.gz 2023-11-08T05:35:54.866 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.124382.log 2023-11-08T05:35:54.866 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.admin.124382.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.admin.124382.log.gz 2023-11-08T05:35:54.867 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.146626.log 2023-11-08T05:35:54.867 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.146626.log: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.129560.log.gz 2023-11-08T05:35:54.868 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.153162.log 2023-11-08T05:35:54.868 INFO:teuthology.orchestra.run.smithi087.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.146626.log.gz 2023-11-08T05:35:54.869 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.152802.log 2023-11-08T05:35:54.869 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.153162.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.153162.log.gz 2023-11-08T05:35:54.869 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.152802.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.152802.log.gz 2023-11-08T05:35:54.869 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.146219.log 2023-11-08T05:35:54.870 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.1.120285.log 2023-11-08T05:35:54.871 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.146219.log: /var/log/ceph/cluster2-client.mirror.1.120285.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.146219.log.gz 2023-11-08T05:35:54.871 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126987.log 2023-11-08T05:35:54.875 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.126987.log: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.126987.log.gz 2023-11-08T05:35:54.876 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.159599.log 2023-11-08T05:35:54.876 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.admin.159599.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.159599.log.gz 2023-11-08T05:35:54.877 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.160924.log 2023-11-08T05:35:54.878 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.admin.160924.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.160924.log.gz 2023-11-08T05:35:54.878 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.166585.log 2023-11-08T05:35:54.878 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.admin.166585.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.166585.log.gz 2023-11-08T05:35:54.878 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.148168.log 2023-11-08T05:35:54.879 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.148168.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.148168.log.gz 2023-11-08T05:35:54.879 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.134701.log 2023-11-08T05:35:54.880 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.134701.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.134701.log.gz 2023-11-08T05:35:54.880 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.156033.log 2023-11-08T05:35:54.881 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.156033.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.156033.log.gz 2023-11-08T05:35:54.881 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.149047.log 2023-11-08T05:35:54.882 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.149047.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.149047.log.gz 2023-11-08T05:35:54.882 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.129474.log 2023-11-08T05:35:54.887 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.129474.log: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.129474.log.gz 2023-11-08T05:35:54.887 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.140569.log 2023-11-08T05:35:54.888 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.140569.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.140569.log.gz 2023-11-08T05:35:54.888 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.166381.log 2023-11-08T05:35:54.889 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.admin.166381.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.166381.log.gz 2023-11-08T05:35:54.889 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.161620.log 2023-11-08T05:35:54.890 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.admin.161620.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.161620.log.gz 2023-11-08T05:35:54.890 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.156698.log 2023-11-08T05:35:54.891 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.156698.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.156698.log.gz 2023-11-08T05:35:54.891 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.126535.log 2023-11-08T05:35:54.892 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.126535.log: 92.7% -- replaced with /var/log/ceph/cluster2-client.mirror.1.120285.log.gz 2023-11-08T05:35:54.896 INFO:teuthology.orchestra.run.smithi087.stderr: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.126535.log.gz 2023-11-08T05:35:54.896 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.164960.log 2023-11-08T05:35:54.897 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.141801.log 2023-11-08T05:35:54.897 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.admin.164960.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.164960.log.gz 2023-11-08T05:35:54.898 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.141801.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.141801.log.gz 2023-11-08T05:35:54.898 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.146691.log 2023-11-08T05:35:54.899 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.134968.log 2023-11-08T05:35:54.900 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.146691.log: /var/log/ceph/cluster2-client.mirror.134968.log: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.146691.log.gz 2023-11-08T05:35:54.904 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.153274.log 2023-11-08T05:35:54.905 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.153274.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.153274.log.gz 2023-11-08T05:35:54.905 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.151790.log 2023-11-08T05:35:54.906 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.151790.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.151790.log.gz 2023-11-08T05:35:54.906 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.1.159281.log 2023-11-08T05:35:54.908 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.1.159281.log: 91.4% -- replaced with /var/log/ceph/cluster1-client.mirror.1.159281.log.gz 2023-11-08T05:35:54.908 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.154286.log 2023-11-08T05:35:54.909 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.154286.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.154286.log.gz 2023-11-08T05:35:54.909 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.141105.log 2023-11-08T05:35:54.910 INFO:teuthology.orchestra.run.smithi087.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.154832.log.gz 2023-11-08T05:35:54.910 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.141105.log: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.141105.log.gz 2023-11-08T05:35:54.910 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.125708.log 2023-11-08T05:35:54.911 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.126059.log 2023-11-08T05:35:54.915 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.125708.log: /var/log/ceph/cluster2-client.mirror.126059.log: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.126059.log.gz 2023-11-08T05:35:54.916 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138721.log 2023-11-08T05:35:54.916 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.138721.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.138721.log.gz 2023-11-08T05:35:54.917 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.135678.log 2023-11-08T05:35:54.918 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.135678.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.135678.log.gz 2023-11-08T05:35:54.918 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.135011.log 2023-11-08T05:35:54.924 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.0.135011.log: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.132661.log.gz 2023-11-08T05:35:54.924 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.152419.log 2023-11-08T05:35:54.925 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.152419.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.152419.log.gz 2023-11-08T05:35:54.925 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.137865.log 2023-11-08T05:35:54.934 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.137865.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.132453.log 2023-11-08T05:35:54.946 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.132453.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.139061.log 2023-11-08T05:35:54.946 INFO:teuthology.orchestra.run.smithi087.stderr: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.132453.log.gz 2023-11-08T05:35:54.959 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.139061.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.139061.log.gz 2023-11-08T05:35:54.959 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.131321.log 2023-11-08T05:35:54.960 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.131321.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.131321.log.gz 2023-11-08T05:35:54.960 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.136054.log 2023-11-08T05:35:54.962 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.136054.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.136054.log.gz 2023-11-08T05:35:54.962 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.152286.log 2023-11-08T05:35:54.963 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.152286.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.152286.log.gz 2023-11-08T05:35:54.964 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.157215.log 2023-11-08T05:35:54.964 INFO:teuthology.orchestra.run.smithi087.stderr: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.137865.log.gz 2023-11-08T05:35:54.964 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.157215.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.157215.log.gz 2023-11-08T05:35:54.965 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.146713.log 2023-11-08T05:35:54.966 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.146713.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.146713.log.gz 2023-11-08T05:35:54.966 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.158204.log 2023-11-08T05:35:54.969 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.158204.log: 91.6% -- replaced with /var/log/ceph/cluster2-client.mirror.158204.log.gz 2023-11-08T05:35:54.969 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.131600.log 2023-11-08T05:35:54.977 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.131600.log: 93.3% -- replaced with /var/log/ceph/cluster1-client.mirror.0.135011.log.gz 2023-11-08T05:35:54.980 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.131626.log 2023-11-08T05:35:54.981 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.143411.log 2023-11-08T05:35:54.981 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.131626.log: 79.7% -- replaced with /var/log/ceph/cluster2-client.mirror.131626.log.gz 2023-11-08T05:35:54.981 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.129087.log 2023-11-08T05:35:54.982 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.143411.log: /var/log/ceph/cluster1-client.mirror.129087.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.143411.log.gz 2023-11-08T05:35:54.983 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.151810.log 2023-11-08T05:35:54.985 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.151810.log: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.151810.log.gz 2023-11-08T05:35:54.985 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.141201.log 2023-11-08T05:35:54.986 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.141201.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.141201.log.gz 2023-11-08T05:35:54.987 INFO:teuthology.orchestra.run.smithi087.stderr: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.129087.log.gz 2023-11-08T05:35:54.987 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.148350.log 2023-11-08T05:35:54.988 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.128193.log 2023-11-08T05:35:54.988 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.148350.log: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.148350.log.gz 2023-11-08T05:35:54.988 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.128193.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.161375.log 2023-11-08T05:35:54.989 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.admin.161375.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.161375.log.gz 2023-11-08T05:35:54.989 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.151996.log 2023-11-08T05:35:54.992 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.151996.log: 90.5% -- replaced with /var/log/ceph/cluster2-client.mirror.151996.log.gz 2023-11-08T05:35:54.992 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.139587.log 2023-11-08T05:35:54.993 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.139587.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.128193.log.gz 2023-11-08T05:35:54.993 INFO:teuthology.orchestra.run.smithi087.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.139587.log.gz 2023-11-08T05:35:54.993 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138089.log 2023-11-08T05:35:54.994 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.145696.log 2023-11-08T05:35:54.994 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.138089.log: 87.1% -- replaced with /var/log/ceph/cluster1-client.mirror.138089.log.gz 2023-11-08T05:35:54.995 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.145696.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.145696.log.gz 2023-11-08T05:35:54.995 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.141958.log 2023-11-08T05:35:54.996 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.141958.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.141958.log.gz 2023-11-08T05:35:54.996 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.139759.log 2023-11-08T05:35:54.996 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.139759.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.139759.log.gz 2023-11-08T05:35:54.997 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.129843.log 2023-11-08T05:35:54.997 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.129843.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.129843.log.gz 2023-11-08T05:35:54.998 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.145397.log 2023-11-08T05:35:54.998 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.145397.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.145397.log.gz 2023-11-08T05:35:54.999 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.162159.log 2023-11-08T05:35:54.999 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.admin.162159.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.162159.log.gz 2023-11-08T05:35:54.999 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.156095.log 2023-11-08T05:35:55.000 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.156095.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.157093.log 2023-11-08T05:35:55.001 INFO:teuthology.orchestra.run.smithi087.stderr: 91.6% -- replaced with /var/log/ceph/cluster2-client.mirror.156095.log.gz 2023-11-08T05:35:55.002 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.157093.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.170072.log 2023-11-08T05:35:55.002 INFO:teuthology.orchestra.run.smithi087.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.157093.log.gz 2023-11-08T05:35:55.003 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.admin.170072.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.170072.log.gz 2023-11-08T05:35:55.003 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.133869.log 2023-11-08T05:35:55.004 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.133869.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.133869.log.gz 2023-11-08T05:35:55.004 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.157488.log 2023-11-08T05:35:55.004 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.157488.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.157488.log.gz 2023-11-08T05:35:55.005 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.149709.log 2023-11-08T05:35:55.006 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.149709.log: 90.5% -- replaced with /var/log/ceph/cluster2-client.mirror.149709.log.gz 2023-11-08T05:35:55.007 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.150077.log 2023-11-08T05:35:55.008 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.150077.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.127164.log 2023-11-08T05:35:55.008 INFO:teuthology.orchestra.run.smithi087.stderr: 90.5% -- replaced with /var/log/ceph/cluster2-client.mirror.150077.log.gz 2023-11-08T05:35:55.009 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.127164.log: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.133753.log.gz 2023-11-08T05:35:55.010 INFO:teuthology.orchestra.run.smithi087.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.127164.log.gz 2023-11-08T05:35:55.010 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.131149.log 2023-11-08T05:35:55.010 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138323.log 2023-11-08T05:35:55.011 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.131149.log: /var/log/ceph/cluster1-client.mirror.138323.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.138323.log.gz 2023-11-08T05:35:55.012 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.139691.log 2023-11-08T05:35:55.012 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.139691.log: 94.8% -- replaced with /var/log/ceph/cluster1-client.mirror.131149.log.gz 2023-11-08T05:35:55.012 INFO:teuthology.orchestra.run.smithi087.stderr: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.139691.log.gz 2023-11-08T05:35:55.013 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.147396.log 2023-11-08T05:35:55.013 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.147854.log 2023-11-08T05:35:55.014 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.147396.log: /var/log/ceph/cluster1-client.mirror.147854.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.147396.log.gz 2023-11-08T05:35:55.014 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.169484.log 2023-11-08T05:35:55.014 INFO:teuthology.orchestra.run.smithi087.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.147854.log.gz 2023-11-08T05:35:55.014 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.157747.log 2023-11-08T05:35:55.015 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.admin.169484.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.169484.log.gz 2023-11-08T05:35:55.015 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.167075.log 2023-11-08T05:35:55.016 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.157747.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.157747.log.gz 2023-11-08T05:35:55.016 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.admin.167075.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.167075.log.gz 2023-11-08T05:35:55.017 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.161228.log 2023-11-08T05:35:55.017 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.135582.log 2023-11-08T05:35:55.017 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.admin.161228.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.161228.log.gz 2023-11-08T05:35:55.017 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.135582.log: 79.6% -- replaced with /var/log/ceph/cluster2-client.mirror.135582.log.gz 2023-11-08T05:35:55.018 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.133553.log 2023-11-08T05:35:55.018 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.161865.log 2023-11-08T05:35:55.018 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.133553.log: /var/log/ceph/cluster2-client.admin.161865.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.133553.log.gz 2023-11-08T05:35:55.018 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.161865.log.gz 2023-11-08T05:35:55.018 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.126394.log 2023-11-08T05:35:55.020 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.168888.log 2023-11-08T05:35:55.020 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.126394.log: /var/log/ceph/cluster2-client.admin.168888.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.126394.log.gz 2023-11-08T05:35:55.020 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.168888.log.gz 2023-11-08T05:35:55.020 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.131541.log 2023-11-08T05:35:55.021 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.147782.log 2023-11-08T05:35:55.021 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.131541.log: 79.6% -- replaced with /var/log/ceph/cluster2-client.mirror.131541.log.gz 2023-11-08T05:35:55.021 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.147782.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.147782.log.gz 2023-11-08T05:35:55.021 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.135661.log 2023-11-08T05:35:55.021 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.120629.log 2023-11-08T05:35:55.022 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.135661.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.142855.log 2023-11-08T05:35:55.022 INFO:teuthology.orchestra.run.smithi087.stderr: 79.2% -- replaced with /var/log/ceph/cluster2-client.mirror.135661.log.gz 2023-11-08T05:35:55.022 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.140497.log 2023-11-08T05:35:55.022 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.admin.120629.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.120629.log.gz 2023-11-08T05:35:55.023 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.142855.log: /var/log/ceph/cluster1-client.mirror.140497.log: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.142855.log.gz 2023-11-08T05:35:55.023 INFO:teuthology.orchestra.run.smithi087.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.140497.log.gz 2023-11-08T05:35:55.024 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.133391.log 2023-11-08T05:35:55.024 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.131828.log 2023-11-08T05:35:55.024 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.133391.log: /var/log/ceph/cluster2-client.mirror.131828.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.131828.log.gz 2023-11-08T05:35:55.024 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.164813.log 2023-11-08T05:35:55.024 INFO:teuthology.orchestra.run.smithi087.stderr: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.1.133106.log.gz 2023-11-08T05:35:55.025 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.156893.log 2023-11-08T05:35:55.026 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.admin.164813.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.164813.log.gz 2023-11-08T05:35:55.026 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.156893.log: 79.5% -- replaced with /var/log/ceph/cluster2-client.mirror.156893.log.gz 2023-11-08T05:35:55.026 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.154790.log 2023-11-08T05:35:55.026 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.154264.log 2023-11-08T05:35:55.027 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.154790.log: /var/log/ceph/cluster1-client.mirror.154264.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.154264.log.gz 2023-11-08T05:35:55.027 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.144968.log 2023-11-08T05:35:55.027 INFO:teuthology.orchestra.run.smithi087.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.154790.log.gz 2023-11-08T05:35:55.028 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.144968.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.144968.log.gz 2023-11-08T05:35:55.028 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.126804.log 2023-11-08T05:35:55.028 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137590.log 2023-11-08T05:35:55.029 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.126804.log: /var/log/ceph/cluster1-client.mirror.137590.log: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.126804.log.gz 2023-11-08T05:35:55.030 INFO:teuthology.orchestra.run.smithi087.stderr: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.137590.log.gz 2023-11-08T05:35:55.031 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.130146.log 2023-11-08T05:35:55.031 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.167614.log 2023-11-08T05:35:55.036 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.130146.log: /var/log/ceph/cluster2-client.admin.167614.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.167614.log.gz 2023-11-08T05:35:55.036 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.169533.log 2023-11-08T05:35:55.037 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.admin.169533.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.169533.log.gz 2023-11-08T05:35:55.037 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.167222.log 2023-11-08T05:35:55.037 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.admin.167222.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.167222.log.gz 2023-11-08T05:35:55.037 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.130296.log 2023-11-08T05:35:55.038 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.138697.log 2023-11-08T05:35:55.038 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.130296.log: /var/log/ceph/cluster2-client.mirror.138697.log: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.138697.log.gz 2023-11-08T05:35:55.038 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.141325.log 2023-11-08T05:35:55.038 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.141325.log: 90.9% -- replaced with /var/log/ceph/cluster2-client.mirror.130296.log.gz 2023-11-08T05:35:55.038 INFO:teuthology.orchestra.run.smithi087.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.141325.log.gz 2023-11-08T05:35:55.039 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.146052.log 2023-11-08T05:35:55.039 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.146052.log: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.146052.log.gz 2023-11-08T05:35:55.040 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.146169.log 2023-11-08T05:35:55.040 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.146169.log: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.146169.log.gz 2023-11-08T05:35:55.040 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.130755.log 2023-11-08T05:35:55.041 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.130755.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.130755.log.gz 2023-11-08T05:35:55.041 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.132076.log 2023-11-08T05:35:55.042 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.132076.log: 79.5% -- replaced with /var/log/ceph/cluster2-client.mirror.132076.log.gz 2023-11-08T05:35:55.042 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.158674.log 2023-11-08T05:35:55.043 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.158674.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.158674.log.gz 2023-11-08T05:35:55.043 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.153367.log 2023-11-08T05:35:55.047 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.0.153367.log: 92.6% -- replaced with /var/log/ceph/cluster1-client.mirror.0.153367.log.gz 2023-11-08T05:35:55.047 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.147961.log 2023-11-08T05:35:55.048 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.147961.log: 94.3% -- replaced with /var/log/ceph/cluster1-client.mirror.147961.log.gz 2023-11-08T05:35:55.048 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138793.log 2023-11-08T05:35:55.049 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.138793.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.138793.log.gz 2023-11-08T05:35:55.049 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.132269.log 2023-11-08T05:35:55.050 INFO:teuthology.orchestra.run.smithi087.stderr: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.130146.log.gz 2023-11-08T05:35:55.050 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.132269.log: 79.5% -- replaced with /var/log/ceph/cluster2-client.mirror.132269.log.gz 2023-11-08T05:35:55.051 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.148192.log 2023-11-08T05:35:55.051 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.146671.log 2023-11-08T05:35:55.052 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.148192.log: /var/log/ceph/cluster2-client.mirror.146671.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.148192.log.gz 2023-11-08T05:35:55.053 INFO:teuthology.orchestra.run.smithi087.stderr: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.146671.log.gz 2023-11-08T05:35:55.053 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.128458.log 2023-11-08T05:35:55.053 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.139203.log 2023-11-08T05:35:55.054 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.128458.log: /var/log/ceph/cluster2-client.mirror.139203.log: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.139203.log.gz 2023-11-08T05:35:55.054 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.135720.log 2023-11-08T05:35:55.055 INFO:teuthology.orchestra.run.smithi087.stderr: 90.1% -- replaced with /var/log/ceph/cluster2-client.mirror.128458.log.gz 2023-11-08T05:35:55.068 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.135720.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.152306.log 2023-11-08T05:35:55.069 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.123446.log 2023-11-08T05:35:55.069 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.152306.log: /var/log/ceph/cluster1-client.mirror.123446.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.123446.log.gz 2023-11-08T05:35:55.070 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137757.log 2023-11-08T05:35:55.071 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.137757.log: 90.5% -- replaced with /var/log/ceph/cluster2-client.mirror.152306.log.gz 2023-11-08T05:35:55.071 INFO:teuthology.orchestra.run.smithi087.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.137757.log.gz 2023-11-08T05:35:55.071 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.168202.log 2023-11-08T05:35:55.072 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.admin.168202.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.168202.log.gz 2023-11-08T05:35:55.072 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.123559.log 2023-11-08T05:35:55.077 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.123559.log: 93.0% -- replaced with /var/log/ceph/cluster1-client.mirror.123559.log.gz 2023-11-08T05:35:55.078 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.157446.log 2023-11-08T05:35:55.078 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.157446.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.157446.log.gz 2023-11-08T05:35:55.079 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126898.log 2023-11-08T05:35:55.079 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.126898.log: 88.6% -- replaced with /var/log/ceph/cluster1-client.mirror.126898.log.gz 2023-11-08T05:35:55.079 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.145286.log 2023-11-08T05:35:55.080 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.145286.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.145286.log.gz 2023-11-08T05:35:55.081 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.152554.log 2023-11-08T05:35:55.081 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.152554.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.152554.log.gz 2023-11-08T05:35:55.081 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.124795.log 2023-11-08T05:35:55.085 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.124795.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.162943.log 2023-11-08T05:35:55.087 INFO:teuthology.orchestra.run.smithi087.stderr: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.124795.log.gz 2023-11-08T05:35:55.088 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.admin.162943.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.162943.log.gz 2023-11-08T05:35:55.088 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.140116.log 2023-11-08T05:35:55.090 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.140116.log: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.140116.log.gz 2023-11-08T05:35:55.090 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.150036.log 2023-11-08T05:35:55.091 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.150036.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.150036.log.gz 2023-11-08T05:35:55.091 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.156479.log 2023-11-08T05:35:55.094 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.156479.log: 91.5% -- replaced with /var/log/ceph/cluster2-client.mirror.156479.log.gz 2023-11-08T05:35:55.094 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.149337.log 2023-11-08T05:35:55.096 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.149337.log: 90.5% -- replaced with /var/log/ceph/cluster2-client.mirror.149337.log.gz 2023-11-08T05:35:55.096 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126079.log 2023-11-08T05:35:55.102 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.126079.log: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.126079.log.gz 2023-11-08T05:35:55.102 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.137344.log 2023-11-08T05:35:55.104 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.137344.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.164127.log 2023-11-08T05:35:55.104 INFO:teuthology.orchestra.run.smithi087.stderr: 92.3% -- replaced with /var/log/ceph/cluster2-client.mirror.137344.log.gz 2023-11-08T05:35:55.105 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.158723.log 2023-11-08T05:35:55.105 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.admin.164127.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.164127.log.gz 2023-11-08T05:35:55.105 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.158723.log: 79.6% -- replaced with /var/log/ceph/cluster2-client.mirror.158723.log.gz 2023-11-08T05:35:55.106 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.149945.log 2023-11-08T05:35:55.108 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.149945.log: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.149945.log.gz 2023-11-08T05:35:55.108 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.135805.log 2023-11-08T05:35:55.109 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.135805.log: 2.6% -- replaced with /var/log/ceph/cluster1-client.mirror.135805.log.gz 2023-11-08T05:35:55.109 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.125054.log 2023-11-08T05:35:55.110 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.125054.log: 73.6% -- replaced with /var/log/ceph/cluster2-client.mirror.125054.log.gz 2023-11-08T05:35:55.110 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.133026.log 2023-11-08T05:35:55.114 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.133026.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.124252.log 2023-11-08T05:35:55.118 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.124252.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.161128.log 2023-11-08T05:35:55.120 INFO:teuthology.orchestra.run.smithi087.stderr: 93.0% -- replaced with /var/log/ceph/cluster1-client.mirror.124252.log.gz 2023-11-08T05:35:55.121 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.admin.161128.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.161128.log.gz 2023-11-08T05:35:55.121 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.153971.log 2023-11-08T05:35:55.122 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.153971.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.153971.log.gz 2023-11-08T05:35:55.122 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.147324.log 2023-11-08T05:35:55.123 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.147324.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.147324.log.gz 2023-11-08T05:35:55.124 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.131047.log 2023-11-08T05:35:55.125 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.131047.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.131047.log.gz 2023-11-08T05:35:55.125 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.129282.log 2023-11-08T05:35:55.131 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.129282.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.129282.log.gz 2023-11-08T05:35:55.131 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125580.log 2023-11-08T05:35:55.132 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.125580.log: 88.0% -- replaced with /var/log/ceph/cluster1-client.mirror.125580.log.gz 2023-11-08T05:35:55.132 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138989.log 2023-11-08T05:35:55.133 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.138989.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.138989.log.gz 2023-11-08T05:35:55.133 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.163678.log 2023-11-08T05:35:55.134 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.admin.163678.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.163678.log.gz 2023-11-08T05:35:55.134 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.0.120263.log 2023-11-08T05:35:55.139 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.0.120263.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.159748.log 2023-11-08T05:35:55.142 INFO:teuthology.orchestra.run.smithi087.stderr: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.0.120263.log.gz 2023-11-08T05:35:55.143 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.admin.159748.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.159748.log.gz 2023-11-08T05:35:55.143 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.157301.log 2023-11-08T05:35:55.144 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.157301.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.157301.log.gz 2023-11-08T05:35:55.144 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.139013.log 2023-11-08T05:35:55.145 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.139013.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.139013.log.gz 2023-11-08T05:35:55.146 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.130276.log 2023-11-08T05:35:55.147 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.130276.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.130276.log.gz 2023-11-08T05:35:55.147 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.136102.log 2023-11-08T05:35:55.148 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.136102.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.136102.log.gz 2023-11-08T05:35:55.148 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.123581.log 2023-11-08T05:35:55.149 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.123581.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.123581.log.gz 2023-11-08T05:35:55.150 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.138410.log 2023-11-08T05:35:55.151 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.138410.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.138410.log.gz 2023-11-08T05:35:55.151 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126555.log 2023-11-08T05:35:55.157 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.126555.log: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.126555.log.gz 2023-11-08T05:35:55.157 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.141720.log 2023-11-08T05:35:55.159 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.141720.log: 92.0% -- replaced with /var/log/ceph/cluster1-client.mirror.141720.log.gz 2023-11-08T05:35:55.160 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.158046.log 2023-11-08T05:35:55.160 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.158046.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.158046.log.gz 2023-11-08T05:35:55.161 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.139275.log 2023-11-08T05:35:55.162 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.139275.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.139275.log.gz 2023-11-08T05:35:55.162 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.152100.log 2023-11-08T05:35:55.163 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.152100.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.152100.log.gz 2023-11-08T05:35:55.163 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.144570.log 2023-11-08T05:35:55.164 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.144570.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.144570.log.gz 2023-11-08T05:35:55.164 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.163833.log 2023-11-08T05:35:55.165 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.admin.163833.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.163833.log.gz 2023-11-08T05:35:55.165 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.154691.log 2023-11-08T05:35:55.166 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.154691.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.154691.log.gz 2023-11-08T05:35:55.167 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125621.log 2023-11-08T05:35:55.172 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.125621.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.161424.log 2023-11-08T05:35:55.172 INFO:teuthology.orchestra.run.smithi087.stderr: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.125621.log.gz 2023-11-08T05:35:55.173 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.admin.161424.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.161424.log.gz 2023-11-08T05:35:55.174 INFO:teuthology.orchestra.run.smithi087.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.125708.log.gz 2023-11-08T05:35:55.174 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.120849.log 2023-11-08T05:35:55.175 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.admin.120849.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.120849.log.gz 2023-11-08T05:35:55.175 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.130428.log 2023-11-08T05:35:55.178 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.130428.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.133304.log 2023-11-08T05:35:55.183 INFO:teuthology.orchestra.run.smithi087.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.134303.log.gz 2023-11-08T05:35:55.200 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.133304.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.152869.log 2023-11-08T05:35:55.209 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.152869.log: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.152869.log.gz 2023-11-08T05:35:55.211 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.133282.log 2023-11-08T05:35:55.219 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.133282.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.133282.log.gz 2023-11-08T05:35:55.222 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.132374.log 2023-11-08T05:35:55.228 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.132374.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.132374.log.gz 2023-11-08T05:35:55.233 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.161816.log 2023-11-08T05:35:55.237 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.admin.161816.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.161816.log.gz 2023-11-08T05:35:55.239 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.143961.log 2023-11-08T05:35:55.241 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.143961.log: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.143961.log.gz 2023-11-08T05:35:55.242 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.147092.log 2023-11-08T05:35:55.243 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.147092.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.147092.log.gz 2023-11-08T05:35:55.243 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.153319.log 2023-11-08T05:35:55.244 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.153319.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.153319.log.gz 2023-11-08T05:35:55.244 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.1.132136.log 2023-11-08T05:35:55.255 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.1.132136.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137267.log 2023-11-08T05:35:55.267 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.137267.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.137267.log.gz 2023-11-08T05:35:55.271 INFO:teuthology.orchestra.run.smithi087.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.131600.log.gz 2023-11-08T05:35:55.272 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.132034.log 2023-11-08T05:35:55.273 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.132034.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.132034.log.gz 2023-11-08T05:35:55.273 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.145194.log 2023-11-08T05:35:55.274 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.145194.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.145194.log.gz 2023-11-08T05:35:55.274 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138501.log 2023-11-08T05:35:55.275 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.138501.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.138501.log.gz 2023-11-08T05:35:55.275 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.124635.log 2023-11-08T05:35:55.276 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.124635.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.124635.log.gz 2023-11-08T05:35:55.276 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.142903.log 2023-11-08T05:35:55.277 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.142903.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.142903.log.gz 2023-11-08T05:35:55.277 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.164421.log 2023-11-08T05:35:55.278 INFO:teuthology.orchestra.run.smithi087.stderr: 94.0%/var/log/ceph/cluster2-client.admin.164421.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.164421.log.gz 2023-11-08T05:35:55.278 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.159378.log 2023-11-08T05:35:55.278 INFO:teuthology.orchestra.run.smithi087.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.1.132136.log.gz 2023-11-08T05:35:55.279 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.159378.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.159378.log.gz 2023-11-08T05:35:55.279 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.124752.log 2023-11-08T05:35:55.284 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.124752.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.124752.log.gz 2023-11-08T05:35:55.285 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.140783.log 2023-11-08T05:35:55.285 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.140783.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.140783.log.gz 2023-11-08T05:35:55.286 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.131808.log 2023-11-08T05:35:55.287 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.131808.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.131808.log.gz 2023-11-08T05:35:55.287 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.130146.log 2023-11-08T05:35:55.291 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.0.130146.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.134968.log.gz 2023-11-08T05:35:55.293 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.134681.log 2023-11-08T05:35:55.294 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.134681.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.134681.log.gz 2023-11-08T05:35:55.295 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.147459.log 2023-11-08T05:35:55.296 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.147459.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.147459.log.gz 2023-11-08T05:35:55.296 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.142415.log 2023-11-08T05:35:55.297 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.142415.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.142415.log.gz 2023-11-08T05:35:55.297 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138614.log 2023-11-08T05:35:55.299 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.138614.log: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.138614.log.gz 2023-11-08T05:35:55.299 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.141679.log 2023-11-08T05:35:55.300 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.141679.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.141679.log.gz 2023-11-08T05:35:55.300 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.128502.log 2023-11-08T05:35:55.301 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.128502.log: 89.3% -- replaced with /var/log/ceph/cluster1-client.mirror.128502.log.gz 2023-11-08T05:35:55.301 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.154061.log 2023-11-08T05:35:55.302 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.154061.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.154061.log.gz 2023-11-08T05:35:55.302 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.126642.log 2023-11-08T05:35:55.306 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.126642.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.144594.log 2023-11-08T05:35:55.310 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.144594.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.144594.log.gz 2023-11-08T05:35:55.311 INFO:teuthology.orchestra.run.smithi087.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.133391.log.gz 2023-11-08T05:35:55.312 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.154376.log 2023-11-08T05:35:55.312 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.154376.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.154376.log.gz 2023-11-08T05:35:55.313 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.158938.log 2023-11-08T05:35:55.313 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.158938.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.158938.log.gz 2023-11-08T05:35:55.314 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.159944.log 2023-11-08T05:35:55.314 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.admin.159944.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.159944.log.gz 2023-11-08T05:35:55.315 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.152265.log 2023-11-08T05:35:55.315 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.152265.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.152265.log.gz 2023-11-08T05:35:55.316 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.144522.log 2023-11-08T05:35:55.316 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.144522.log: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.144522.log.gz 2023-11-08T05:35:55.317 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.124002.log 2023-11-08T05:35:55.321 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.124002.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.146822.log 2023-11-08T05:35:55.322 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.146822.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.146822.log.gz 2023-11-08T05:35:55.322 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.169231.log 2023-11-08T05:35:55.323 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.admin.169231.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.169231.log.gz 2023-11-08T05:35:55.323 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.163139.log 2023-11-08T05:35:55.324 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.admin.163139.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.163139.log.gz 2023-11-08T05:35:55.324 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.157423.log 2023-11-08T05:35:55.325 INFO:teuthology.orchestra.run.smithi087.stderr: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.124002.log.gz 2023-11-08T05:35:55.325 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-mon.a.log 2023-11-08T05:35:55.325 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.157423.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.157423.log.gz 2023-11-08T05:35:55.326 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-mon.a.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.152914.log 2023-11-08T05:35:55.327 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.152914.log: 85.0% -- replaced with /var/log/ceph/cluster1-client.mirror.152914.log.gz 2023-11-08T05:35:55.327 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.152622.log 2023-11-08T05:35:55.328 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.152622.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.152622.log.gz 2023-11-08T05:35:55.328 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.146965.log 2023-11-08T05:35:55.329 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.146965.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.146965.log.gz 2023-11-08T05:35:55.329 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.152182.log 2023-11-08T05:35:55.332 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.152182.log: 92.9% -- replaced with /var/log/ceph/cluster1-client.mirror.0.130146.log.gz 2023-11-08T05:35:55.332 INFO:teuthology.orchestra.run.smithi087.stderr: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.152182.log.gz 2023-11-08T05:35:55.333 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.138477.log 2023-11-08T05:35:55.334 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.138477.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.138477.log.gz 2023-11-08T05:35:55.334 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.141893.log 2023-11-08T05:35:55.334 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.141129.log 2023-11-08T05:35:55.334 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.141893.log: /var/log/ceph/cluster1-client.mirror.141129.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.141893.log.gz 2023-11-08T05:35:55.335 INFO:teuthology.orchestra.run.smithi087.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.141129.log.gz 2023-11-08T05:35:55.335 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.160630.log 2023-11-08T05:35:55.336 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.admin.160630.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.160630.log.gz 2023-11-08T05:35:55.336 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.167957.log 2023-11-08T05:35:55.336 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.admin.167957.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.167957.log.gz 2023-11-08T05:35:55.336 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.145676.log 2023-11-08T05:35:55.337 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.145676.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.145676.log.gz 2023-11-08T05:35:55.337 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.128757.log 2023-11-08T05:35:55.339 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.128757.log: 90.1% -- replaced with /var/log/ceph/cluster2-client.mirror.128757.log.gz 2023-11-08T05:35:55.339 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.124611.log 2023-11-08T05:35:55.340 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.124611.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.124611.log.gz 2023-11-08T05:35:55.340 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.148422.log 2023-11-08T05:35:55.341 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.148422.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.148422.log.gz 2023-11-08T05:35:55.341 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.167271.log 2023-11-08T05:35:55.342 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.admin.167271.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.167271.log.gz 2023-11-08T05:35:55.342 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.131462.log 2023-11-08T05:35:55.343 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.131462.log: 79.7% -- replaced with /var/log/ceph/cluster2-client.mirror.131462.log.gz 2023-11-08T05:35:55.343 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-osd.2.log 2023-11-08T05:35:55.349 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-osd.2.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.165597.log 2023-11-08T05:35:55.350 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.admin.165597.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.165597.log.gz 2023-11-08T05:35:55.350 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.132225.log 2023-11-08T05:35:55.351 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.132225.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.132225.log.gz 2023-11-08T05:35:55.351 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.127600.log 2023-11-08T05:35:55.353 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.127600.log: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.127600.log.gz 2023-11-08T05:35:55.353 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.127989.log 2023-11-08T05:35:55.354 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.127989.log: 90.1% -- replaced with /var/log/ceph/cluster2-client.mirror.127989.log.gz 2023-11-08T05:35:55.354 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.156438.log 2023-11-08T05:35:55.355 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.156438.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.156438.log.gz 2023-11-08T05:35:55.356 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.146475.log 2023-11-08T05:35:55.357 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.146475.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.146475.log.gz 2023-11-08T05:35:55.357 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.139735.log 2023-11-08T05:35:55.358 INFO:teuthology.orchestra.run.smithi087.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.133026.log.gz 2023-11-08T05:35:55.358 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.139735.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.139735.log.gz 2023-11-08T05:35:55.359 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.134926.log 2023-11-08T05:35:55.359 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.134926.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.134926.log.gz 2023-11-08T05:35:55.360 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.148693.log 2023-11-08T05:35:55.362 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.148693.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.148693.log.gz 2023-11-08T05:35:55.362 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.147024.log 2023-11-08T05:35:55.363 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.147024.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.147024.log.gz 2023-11-08T05:35:55.363 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.126203.log 2023-11-08T05:35:55.365 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.126203.log: 91.6% -- replaced with /var/log/ceph/cluster2-client.mirror.126203.log.gz 2023-11-08T05:35:55.365 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.169778.log 2023-11-08T05:35:55.366 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.admin.169778.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.169778.log.gz 2023-11-08T05:35:55.366 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.149925.log 2023-11-08T05:35:55.367 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.149925.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.149925.log.gz 2023-11-08T05:35:55.367 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.136640.log 2023-11-08T05:35:55.369 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.136640.log: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.136640.log.gz 2023-11-08T05:35:55.369 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.124659.log 2023-11-08T05:35:55.370 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.124659.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.124659.log.gz 2023-11-08T05:35:55.370 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.153094.log 2023-11-08T05:35:55.371 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.153094.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.153094.log.gz 2023-11-08T05:35:55.371 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.148814.log 2023-11-08T05:35:55.372 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.148814.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.148814.log.gz 2023-11-08T05:35:55.372 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.124683.log 2023-11-08T05:35:55.373 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.124683.log: 87.9% -- replaced with /var/log/ceph/cluster1-client.mirror.124683.log.gz 2023-11-08T05:35:55.373 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.2.120152.log 2023-11-08T05:35:55.375 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.2.120152.log: 91.2% -- replaced with /var/log/ceph/cluster1-client.mirror.2.120152.log.gz 2023-11-08T05:35:55.375 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.153229.log 2023-11-08T05:35:55.376 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.153229.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.153229.log.gz 2023-11-08T05:35:55.376 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.150243.log 2023-11-08T05:35:55.377 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.150243.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.150243.log.gz 2023-11-08T05:35:55.377 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.131764.log 2023-11-08T05:35:55.388 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.131764.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.149647.log 2023-11-08T05:35:55.391 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.149647.log: 90.5% -- replaced with /var/log/ceph/cluster2-client.mirror.149647.log.gz 2023-11-08T05:35:55.391 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.156352.log 2023-11-08T05:35:55.403 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.156352.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.156352.log.gz 2023-11-08T05:35:55.403 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.129963.log 2023-11-08T05:35:55.415 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.129963.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.129963.log.gz 2023-11-08T05:35:55.415 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.127470.log 2023-11-08T05:35:55.417 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.admin.127470.log: 87.9% -- replaced with /var/log/ceph/cluster1-client.admin.127470.log.gz 2023-11-08T05:35:55.417 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.143844.log 2023-11-08T05:35:55.418 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.143844.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.143844.log.gz 2023-11-08T05:35:55.418 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.149420.log 2023-11-08T05:35:55.419 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.149420.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.149420.log.gz 2023-11-08T05:35:55.419 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.150284.log 2023-11-08T05:35:55.420 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.150284.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.150284.log.gz 2023-11-08T05:35:55.421 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.128873.log 2023-11-08T05:35:55.422 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.128873.log: 88.4% -- replaced with /var/log/ceph/cluster1-client.mirror.128873.log.gz 2023-11-08T05:35:55.422 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.143087.log 2023-11-08T05:35:55.423 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.143087.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.143087.log.gz 2023-11-08T05:35:55.423 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.130406.log 2023-11-08T05:35:55.424 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.130406.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.130406.log.gz 2023-11-08T05:35:55.424 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.125686.log 2023-11-08T05:35:55.426 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.125686.log: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.125686.log.gz 2023-11-08T05:35:55.426 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.141264.log 2023-11-08T05:35:55.427 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.141264.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.141264.log.gz 2023-11-08T05:35:55.428 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.130043.log 2023-11-08T05:35:55.429 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.130043.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.130043.log.gz 2023-11-08T05:35:55.429 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.148862.log 2023-11-08T05:35:55.430 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.148862.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.148862.log.gz 2023-11-08T05:35:55.430 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.134594.log 2023-11-08T05:35:55.432 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.134594.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.134594.log.gz 2023-11-08T05:35:55.432 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138041.log 2023-11-08T05:35:55.433 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.138041.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.138041.log.gz 2023-11-08T05:35:55.433 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.137570.log 2023-11-08T05:35:55.435 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.137570.log: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.137570.log.gz 2023-11-08T05:35:55.436 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.136967.log 2023-11-08T05:35:55.437 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.136967.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.136967.log.gz 2023-11-08T05:35:55.437 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.151583.log 2023-11-08T05:35:55.438 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.151583.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.151583.log.gz 2023-11-08T05:35:55.439 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.140358.log 2023-11-08T05:35:55.441 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.140358.log: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.140358.log.gz 2023-11-08T05:35:55.441 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.134946.log 2023-11-08T05:35:55.442 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.134946.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.134946.log.gz 2023-11-08T05:35:55.442 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.137685.log 2023-11-08T05:35:55.443 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.137685.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.137685.log.gz 2023-11-08T05:35:55.443 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.145566.log 2023-11-08T05:35:55.444 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.145566.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.145566.log.gz 2023-11-08T05:35:55.445 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.128081.log 2023-11-08T05:35:55.446 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.128081.log: 89.0% -- replaced with /var/log/ceph/cluster1-client.mirror.128081.log.gz 2023-11-08T05:35:55.446 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.133466.log 2023-11-08T05:35:55.447 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.133466.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.133466.log.gz 2023-11-08T05:35:55.447 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.126016.log 2023-11-08T05:35:55.453 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.126016.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.126016.log.gz 2023-11-08T05:35:55.453 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.151418.log 2023-11-08T05:35:55.454 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.151418.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.151418.log.gz 2023-11-08T05:35:55.454 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126442.log 2023-11-08T05:35:55.456 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.126442.log: 88.5% -- replaced with /var/log/ceph/cluster1-client.mirror.126442.log.gz 2023-11-08T05:35:55.456 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.134145.log 2023-11-08T05:35:55.460 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.134145.log: 95.0% -- replaced with /var/log/ceph/cluster1-client.mirror.134145.log.gz 2023-11-08T05:35:55.461 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.158567.log 2023-11-08T05:35:55.462 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.158567.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.158567.log.gz 2023-11-08T05:35:55.462 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.161473.log 2023-11-08T05:35:55.463 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.admin.161473.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.161473.log.gz 2023-11-08T05:35:55.463 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.159080.log 2023-11-08T05:35:55.464 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.159080.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.159080.log.gz 2023-11-08T05:35:55.465 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.154738.log 2023-11-08T05:35:55.465 INFO:teuthology.orchestra.run.smithi087.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.135720.log.gz 2023-11-08T05:35:55.465 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.154738.log: 1.3% -- replaced with /var/log/ceph/cluster1-client.mirror.154738.log.gz 2023-11-08T05:35:55.466 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.166283.log 2023-11-08T05:35:55.467 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.admin.166283.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137501.log 2023-11-08T05:35:55.467 INFO:teuthology.orchestra.run.smithi087.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.166283.log.gz 2023-11-08T05:35:55.467 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.147562.log 2023-11-08T05:35:55.467 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.137501.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.137501.log.gz 2023-11-08T05:35:55.468 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.147562.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.147562.log.gz 2023-11-08T05:35:55.468 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.137199.log 2023-11-08T05:35:55.470 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.159037.log 2023-11-08T05:35:55.471 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.137199.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.137199.log.gz 2023-11-08T05:35:55.472 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.133429.log 2023-11-08T05:35:55.473 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.159037.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.159037.log.gz 2023-11-08T05:35:55.473 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.168692.log 2023-11-08T05:35:55.474 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.133429.log: 79.6% -- replaced with /var/log/ceph/cluster2-client.mirror.133429.log.gz 2023-11-08T05:35:55.474 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.133245.log 2023-11-08T05:35:55.475 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.admin.168692.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.168692.log.gz 2023-11-08T05:35:55.476 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.133245.log: 79.4% -- replaced with /var/log/ceph/cluster2-client.mirror.133245.log.gz 2023-11-08T05:35:55.476 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.140241.log 2023-11-08T05:35:55.477 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.140241.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.140241.log.gz 2023-11-08T05:35:55.477 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.139352.log 2023-11-08T05:35:55.477 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.148939.log 2023-11-08T05:35:55.479 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.139352.log: /var/log/ceph/cluster2-client.mirror.148939.log: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.139352.log.gz 2023-11-08T05:35:55.479 INFO:teuthology.orchestra.run.smithi087.stderr: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.148939.log.gz 2023-11-08T05:35:55.480 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.136755.log 2023-11-08T05:35:55.481 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.136755.log: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.136755.log.gz 2023-11-08T05:35:55.481 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.158163.log 2023-11-08T05:35:55.482 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.158163.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.158163.log.gz 2023-11-08T05:35:55.482 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-osd.1.log 2023-11-08T05:35:55.489 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-osd.1.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.148794.log 2023-11-08T05:35:55.496 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.148794.log: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.148794.log.gz 2023-11-08T05:35:55.498 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.146754.log 2023-11-08T05:35:55.500 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.146754.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.146754.log.gz 2023-11-08T05:35:55.509 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.158588.log 2023-11-08T05:35:55.520 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.158588.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.158588.log.gz 2023-11-08T05:35:55.520 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.120152.log 2023-11-08T05:35:55.520 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.120152.log: 73.9% -- replaced with /var/log/ceph/cluster2-client.mirror.120152.log.gz 2023-11-08T05:35:55.521 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.168545.log 2023-11-08T05:35:55.522 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.admin.168545.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.168545.log.gz 2023-11-08T05:35:55.522 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.150346.log 2023-11-08T05:35:55.523 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.150346.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.150346.log.gz 2023-11-08T05:35:55.523 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137549.log 2023-11-08T05:35:55.524 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.137549.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.137549.log.gz 2023-11-08T05:35:55.524 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.168006.log 2023-11-08T05:35:55.525 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.admin.168006.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.168006.log.gz 2023-11-08T05:35:55.525 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.152824.log 2023-11-08T05:35:55.526 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.152824.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.152824.log.gz 2023-11-08T05:35:55.526 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.131106.log 2023-11-08T05:35:55.528 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.131106.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.131106.log.gz 2023-11-08T05:35:55.528 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.153949.log 2023-11-08T05:35:55.529 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.153949.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.153949.log.gz 2023-11-08T05:35:55.529 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.146534.log 2023-11-08T05:35:55.531 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.146534.log: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.146534.log.gz 2023-11-08T05:35:55.531 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.152779.log 2023-11-08T05:35:55.532 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.152779.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.152779.log.gz 2023-11-08T05:35:55.532 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.127363.log 2023-11-08T05:35:55.533 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.admin.127363.log: 87.9% -- replaced with /var/log/ceph/cluster1-client.admin.127363.log.gz 2023-11-08T05:35:55.533 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.141845.log 2023-11-08T05:35:55.534 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.141845.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.141845.log.gz 2023-11-08T05:35:55.535 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.139491.log 2023-11-08T05:35:55.536 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.139491.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.139491.log.gz 2023-11-08T05:35:55.536 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.2.149752.log 2023-11-08T05:35:55.544 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.2.149752.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.130428.log.gz 2023-11-08T05:35:55.544 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.136030.log 2023-11-08T05:35:55.555 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.136030.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.136030.log.gz 2023-11-08T05:35:55.555 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.165205.log 2023-11-08T05:35:55.555 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.admin.165205.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.165205.log.gz 2023-11-08T05:35:55.555 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.151139.log 2023-11-08T05:35:55.559 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.151139.log: 90.5% -- replaced with /var/log/ceph/cluster2-client.mirror.151139.log.gz 2023-11-08T05:35:55.559 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.158002.log 2023-11-08T05:35:55.559 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.158002.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.158002.log.gz 2023-11-08T05:35:55.559 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.145238.log 2023-11-08T05:35:55.559 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.145238.log: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.2.149752.log.gz 2023-11-08T05:35:55.560 INFO:teuthology.orchestra.run.smithi087.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.145238.log.gz 2023-11-08T05:35:55.560 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.149234.log 2023-11-08T05:35:55.561 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.149234.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.149234.log.gz 2023-11-08T05:35:55.561 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.156075.log 2023-11-08T05:35:55.562 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.156075.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.156075.log.gz 2023-11-08T05:35:55.562 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.131685.log 2023-11-08T05:35:55.565 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.131685.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.133304.log.gz 2023-11-08T05:35:55.572 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.133911.log 2023-11-08T05:35:55.572 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.149544.log 2023-11-08T05:35:55.574 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.133911.log: /var/log/ceph/cluster2-client.mirror.149544.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.149544.log.gz 2023-11-08T05:35:55.574 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.143548.log 2023-11-08T05:35:55.576 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.143548.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.143548.log.gz 2023-11-08T05:35:55.577 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.160728.log 2023-11-08T05:35:55.577 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.admin.160728.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.160728.log.gz 2023-11-08T05:35:55.577 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.160483.log 2023-11-08T05:35:55.578 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.admin.160483.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.160483.log.gz 2023-11-08T05:35:55.578 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.118810.log 2023-11-08T05:35:55.579 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.admin.118810.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.118810.log.gz 2023-11-08T05:35:55.580 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.135247.log 2023-11-08T05:35:55.590 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.135247.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.126642.log.gz 2023-11-08T05:35:55.590 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.145542.log 2023-11-08T05:35:55.612 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.145542.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.145542.log.gz 2023-11-08T05:35:55.612 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.156760.log 2023-11-08T05:35:55.614 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.156760.log: 91.6% -- replaced with /var/log/ceph/cluster2-client.mirror.156760.log.gz 2023-11-08T05:35:55.614 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.136240.log 2023-11-08T05:35:55.626 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.136240.log: 95.3% -- replaced with /var/log/ceph/cluster1-client.mirror.136240.log.gz 2023-11-08T05:35:55.626 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.152982.log 2023-11-08T05:35:55.638 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.152982.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.152982.log.gz 2023-11-08T05:35:55.638 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.136987.log 2023-11-08T05:35:55.639 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.136987.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.136987.log.gz 2023-11-08T05:35:55.639 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.140337.log 2023-11-08T05:35:55.641 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.140337.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.140337.log.gz 2023-11-08T05:35:55.641 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.144905.log 2023-11-08T05:35:55.645 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.144905.log: 92.5% -- replaced with /var/log/ceph/cluster1-client.mirror.144905.log.gz 2023-11-08T05:35:55.645 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.132523.log 2023-11-08T05:35:55.646 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.132523.log: 79.6% -- replaced with /var/log/ceph/cluster2-client.mirror.132523.log.gz 2023-11-08T05:35:55.646 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.135619.log 2023-11-08T05:35:55.646 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.135619.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.135619.log.gz 2023-11-08T05:35:55.646 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127648.log 2023-11-08T05:35:55.647 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.127648.log: 88.8% -- replaced with /var/log/ceph/cluster1-client.mirror.127648.log.gz 2023-11-08T05:35:55.647 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.168496.log 2023-11-08T05:35:55.648 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.admin.168496.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.168496.log.gz 2023-11-08T05:35:55.648 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.153836.log 2023-11-08T05:35:55.649 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.153836.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.153836.log.gz 2023-11-08T05:35:55.649 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.158024.log 2023-11-08T05:35:55.660 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.158024.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.158337.log 2023-11-08T05:35:55.682 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.158337.log: 79.4% -- replaced with /var/log/ceph/cluster2-client.mirror.158337.log.gz 2023-11-08T05:35:55.682 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.151272.log 2023-11-08T05:35:55.685 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.151272.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.151272.log.gz 2023-11-08T05:35:55.685 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.143916.log 2023-11-08T05:35:55.698 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.143916.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.143916.log.gz 2023-11-08T05:35:55.698 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.151077.log 2023-11-08T05:35:55.710 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.151077.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.128849.log 2023-11-08T05:35:55.720 INFO:teuthology.orchestra.run.smithi087.stderr: 90.5% -- replaced with /var/log/ceph/cluster2-client.mirror.151077.log.gz 2023-11-08T05:35:55.731 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.128849.log: 89.3% -- replaced with /var/log/ceph/cluster1-client.mirror.128849.log.gz 2023-11-08T05:35:55.731 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.152017.log 2023-11-08T05:35:55.744 INFO:teuthology.orchestra.run.smithi087.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.131764.log.gz 2023-11-08T05:35:55.744 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.138457.log 2023-11-08T05:35:55.746 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.138457.log: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.138457.log.gz 2023-11-08T05:35:55.746 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.130063.log 2023-11-08T05:35:55.747 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.130063.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.130063.log.gz 2023-11-08T05:35:55.747 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.141177.log 2023-11-08T05:35:55.748 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.141177.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.141177.log.gz 2023-11-08T05:35:55.748 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.128397.log 2023-11-08T05:35:55.750 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.128397.log: 91.7% -- replaced with /var/log/ceph/cluster2-client.mirror.128397.log.gz 2023-11-08T05:35:55.750 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.135857.log 2023-11-08T05:35:55.751 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.135857.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.135857.log.gz 2023-11-08T05:35:55.751 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.166038.log 2023-11-08T05:35:55.752 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.admin.166038.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.166038.log.gz 2023-11-08T05:35:55.752 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.145607.log 2023-11-08T05:35:55.754 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.145607.log: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.145607.log.gz 2023-11-08T05:35:55.754 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.140096.log 2023-11-08T05:35:55.756 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.140096.log: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.140096.log.gz 2023-11-08T05:35:55.756 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.152442.log 2023-11-08T05:35:55.757 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.152017.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.152017.log.gz 2023-11-08T05:35:55.758 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.152442.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.152442.log.gz 2023-11-08T05:35:55.758 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.169084.log 2023-11-08T05:35:55.759 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.admin.169084.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.169084.log.gz 2023-11-08T05:35:55.759 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.154646.log 2023-11-08T05:35:55.759 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.154646.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.154646.log.gz 2023-11-08T05:35:55.760 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.125788.log 2023-11-08T05:35:55.761 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.125788.log: 91.6% -- replaced with /var/log/ceph/cluster2-client.mirror.125788.log.gz 2023-11-08T05:35:55.762 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.148374.log 2023-11-08T05:35:55.762 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.148374.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.148374.log.gz 2023-11-08T05:35:55.762 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.134535.log 2023-11-08T05:35:55.763 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.134535.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.134535.log.gz 2023-11-08T05:35:55.764 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.133773.log 2023-11-08T05:35:55.764 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.133773.log: 79.6% -- replaced with /var/log/ceph/cluster2-client.mirror.133773.log.gz 2023-11-08T05:35:55.764 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.154106.log 2023-11-08T05:35:55.765 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.154106.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.154106.log.gz 2023-11-08T05:35:55.765 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.164617.log 2023-11-08T05:35:55.766 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.admin.164617.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.164617.log.gz 2023-11-08T05:35:55.766 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.132540.log 2023-11-08T05:35:55.767 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.132540.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.132540.log.gz 2023-11-08T05:35:55.767 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.130369.log 2023-11-08T05:35:55.768 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.130369.log: 79.7% -- replaced with /var/log/ceph/cluster2-client.mirror.130369.log.gz 2023-11-08T05:35:55.768 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.157511.log 2023-11-08T05:35:55.769 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.157511.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.157511.log.gz 2023-11-08T05:35:55.769 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.130255.log 2023-11-08T05:35:55.770 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.130255.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.130255.log.gz 2023-11-08T05:35:55.770 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.139102.log 2023-11-08T05:35:55.772 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.139102.log: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.139102.log.gz 2023-11-08T05:35:55.773 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.124412.log 2023-11-08T05:35:55.773 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.124412.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.124412.log.gz 2023-11-08T05:35:55.774 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.148096.log 2023-11-08T05:35:55.774 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.148096.log: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.148096.log.gz 2023-11-08T05:35:55.775 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.151769.log 2023-11-08T05:35:55.775 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.151769.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.151769.log.gz 2023-11-08T05:35:55.776 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.139251.log 2023-11-08T05:35:55.776 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.139251.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.139251.log.gz 2023-11-08T05:35:55.777 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.141563.log 2023-11-08T05:35:55.778 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.141563.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.141563.log.gz 2023-11-08T05:35:55.778 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.132734.log 2023-11-08T05:35:55.779 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.132734.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.132734.log.gz 2023-11-08T05:35:55.779 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.153814.log 2023-11-08T05:35:55.780 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.153814.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.153814.log.gz 2023-11-08T05:35:55.780 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.157619.log 2023-11-08T05:35:55.781 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.157619.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.157619.log.gz 2023-11-08T05:35:55.781 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.130235.log 2023-11-08T05:35:55.782 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.130235.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.130235.log.gz 2023-11-08T05:35:55.783 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.152464.log 2023-11-08T05:35:55.783 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.152464.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.152464.log.gz 2023-11-08T05:35:55.784 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.150821.log 2023-11-08T05:35:55.786 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.150821.log: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.150821.log.gz 2023-11-08T05:35:55.786 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.142951.log 2023-11-08T05:35:55.787 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.142951.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.142951.log.gz 2023-11-08T05:35:55.787 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.129517.log 2023-11-08T05:35:55.793 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.129517.log: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.129517.log.gz 2023-11-08T05:35:55.793 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.119006.log 2023-11-08T05:35:55.794 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.admin.119006.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.119006.log.gz 2023-11-08T05:35:55.794 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.150615.log 2023-11-08T05:35:55.795 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.150615.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.150615.log.gz 2023-11-08T05:35:55.795 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.152332.log 2023-11-08T05:35:55.795 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.152332.log: 1.3% -- replaced with /var/log/ceph/cluster1-client.mirror.152332.log.gz 2023-11-08T05:35:55.796 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.139307.log 2023-11-08T05:35:55.796 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.139307.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.139307.log.gz 2023-11-08T05:35:55.797 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.159197.log 2023-11-08T05:35:55.797 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.159197.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.159197.log.gz 2023-11-08T05:35:55.798 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.142776.log 2023-11-08T05:35:55.799 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.142776.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.142776.log.gz 2023-11-08T05:35:55.799 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.144003.log 2023-11-08T05:35:55.800 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.144003.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.144003.log.gz 2023-11-08T05:35:55.800 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.157321.log 2023-11-08T05:35:55.803 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.157321.log: 91.4% -- replaced with /var/log/ceph/cluster2-client.mirror.157321.log.gz 2023-11-08T05:35:55.803 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.127010.log 2023-11-08T05:35:55.809 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.127010.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.127010.log.gz 2023-11-08T05:35:55.809 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.127741.log 2023-11-08T05:35:55.813 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.127741.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.141700.log 2023-11-08T05:35:55.815 INFO:teuthology.orchestra.run.smithi087.stderr: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.127741.log.gz 2023-11-08T05:35:55.817 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.141700.log: 92.2% -- replaced with /var/log/ceph/cluster2-client.mirror.141700.log.gz 2023-11-08T05:35:55.817 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.146412.log 2023-11-08T05:35:55.818 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.146412.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.146412.log.gz 2023-11-08T05:35:55.819 INFO:teuthology.orchestra.run.smithi087.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.133911.log.gz 2023-11-08T05:35:55.819 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.152689.log 2023-11-08T05:35:55.820 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.152689.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.152689.log.gz 2023-11-08T05:35:55.820 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.144681.log 2023-11-08T05:35:55.821 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.144681.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.144681.log.gz 2023-11-08T05:35:55.821 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.119349.log 2023-11-08T05:35:55.821 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.admin.119349.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.119349.log.gz 2023-11-08T05:35:55.822 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.142487.log 2023-11-08T05:35:55.822 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.142487.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.142487.log.gz 2023-11-08T05:35:55.823 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.153791.log 2023-11-08T05:35:55.823 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.153791.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.153791.log.gz 2023-11-08T05:35:55.824 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.124837.log 2023-11-08T05:35:55.825 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.124837.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.124837.log.gz 2023-11-08T05:35:55.825 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.134881.log 2023-11-08T05:35:55.835 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.134881.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.164274.log 2023-11-08T05:35:55.836 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.admin.164274.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.164274.log.gz 2023-11-08T05:35:55.836 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.165303.log 2023-11-08T05:35:55.837 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.admin.165303.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.165303.log.gz 2023-11-08T05:35:55.837 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.168594.log 2023-11-08T05:35:55.838 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.admin.168594.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.168594.log.gz 2023-11-08T05:35:55.838 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.120104.log 2023-11-08T05:35:55.839 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.120104.log: 74.3% -- replaced with /var/log/ceph/cluster2-client.mirror.120104.log.gz 2023-11-08T05:35:55.839 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.131937.log 2023-11-08T05:35:55.849 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.131937.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.137386.log 2023-11-08T05:35:55.873 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.137386.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.137386.log.gz 2023-11-08T05:35:55.873 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.141631.log 2023-11-08T05:35:55.876 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.141631.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.141631.log.gz 2023-11-08T05:35:55.876 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.133852.log 2023-11-08T05:35:55.888 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.133852.log: 79.5% -- replaced with /var/log/ceph/cluster2-client.mirror.133852.log.gz 2023-11-08T05:35:55.888 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.132697.log 2023-11-08T05:35:55.900 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.132697.log: 79.4% -- replaced with /var/log/ceph/cluster2-client.mirror.132697.log.gz 2023-11-08T05:35:55.900 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.134244.log 2023-11-08T05:35:55.913 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.134244.log: 79.4% -- replaced with /var/log/ceph/cluster2-client.mirror.134244.log.gz 2023-11-08T05:35:55.913 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.163237.log 2023-11-08T05:35:55.926 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.admin.163237.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.163237.log.gz 2023-11-08T05:35:55.926 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.129454.log 2023-11-08T05:35:55.937 INFO:teuthology.orchestra.run.smithi087.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.131685.log.gz 2023-11-08T05:35:55.948 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.129454.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.131705.log 2023-11-08T05:35:55.948 INFO:teuthology.orchestra.run.smithi087.stderr: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.129454.log.gz 2023-11-08T05:35:55.948 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.131705.log: 79.6% -- replaced with /var/log/ceph/cluster2-client.mirror.131705.log.gz 2023-11-08T05:35:55.948 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.153724.log 2023-11-08T05:35:55.949 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.153724.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.153724.log.gz 2023-11-08T05:35:55.949 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.135698.log 2023-11-08T05:35:55.949 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.135698.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.135698.log.gz 2023-11-08T05:35:55.949 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.142617.log 2023-11-08T05:35:55.949 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.142617.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.142617.log.gz 2023-11-08T05:35:55.950 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.158653.log 2023-11-08T05:35:55.950 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.158653.log: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.135247.log.gz 2023-11-08T05:35:55.950 INFO:teuthology.orchestra.run.smithi087.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.158653.log.gz 2023-11-08T05:35:55.950 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.120679.log 2023-11-08T05:35:55.950 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.admin.120679.log: 79.9% -- replaced with /var/log/ceph/cluster1-client.admin.120679.log.gz 2023-11-08T05:35:55.951 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.132136.log 2023-11-08T05:35:55.951 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.132136.log: 95.1% -- replaced with /var/log/ceph/cluster1-client.mirror.132136.log.gz 2023-11-08T05:35:55.952 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.149068.log 2023-11-08T05:35:55.952 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.149068.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.149068.log.gz 2023-11-08T05:35:55.952 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137781.log 2023-11-08T05:35:55.952 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.137781.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.137781.log.gz 2023-11-08T05:35:55.952 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.168300.log 2023-11-08T05:35:55.953 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.admin.168300.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.168300.log.gz 2023-11-08T05:35:55.953 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.121286.log 2023-11-08T05:35:55.953 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.121286.log: 57.4% -- replaced with /var/log/ceph/cluster2-client.mirror.121286.log.gz 2023-11-08T05:35:55.954 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.137453.log 2023-11-08T05:35:55.954 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.137453.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.137453.log.gz 2023-11-08T05:35:55.954 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.121235.log 2023-11-08T05:35:55.955 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.121235.log: 2.3% -- replaced with /var/log/ceph/cluster2-client.mirror.121235.log.gz 2023-11-08T05:35:55.955 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.156803.log 2023-11-08T05:35:55.955 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.156803.log: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.156803.log.gz 2023-11-08T05:35:55.956 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.134802.log 2023-11-08T05:35:55.960 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.134802.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.132286.log 2023-11-08T05:35:55.968 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.132286.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.132286.log.gz 2023-11-08T05:35:55.968 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.129239.log 2023-11-08T05:35:55.972 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.129239.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.154331.log 2023-11-08T05:35:55.974 INFO:teuthology.orchestra.run.smithi087.stderr: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.129239.log.gz 2023-11-08T05:35:55.980 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.154331.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.154331.log.gz 2023-11-08T05:35:55.980 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.115807.log 2023-11-08T05:35:55.984 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.admin.115807.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.115807.log.gz 2023-11-08T05:35:55.984 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.144885.log 2023-11-08T05:35:55.986 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.144885.log: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.144885.log.gz 2023-11-08T05:35:55.986 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.127784.log 2023-11-08T05:35:55.992 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.127784.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.127784.log.gz 2023-11-08T05:35:55.992 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125923.log 2023-11-08T05:35:55.993 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.125923.log: 88.3% -- replaced with /var/log/ceph/cluster1-client.mirror.125923.log.gz 2023-11-08T05:35:55.994 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.139831.log 2023-11-08T05:35:55.995 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.139831.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.139831.log.gz 2023-11-08T05:35:55.995 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.125875.log 2023-11-08T05:35:55.996 INFO:teuthology.orchestra.run.smithi087.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.158024.log.gz 2023-11-08T05:35:55.996 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.125875.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.125875.log.gz 2023-11-08T05:35:55.996 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.143067.log 2023-11-08T05:35:55.997 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.143067.log: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.143067.log.gz 2023-11-08T05:35:55.997 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.139563.log 2023-11-08T05:35:55.998 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.139563.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.139563.log.gz 2023-11-08T05:35:55.999 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.144112.log 2023-11-08T05:35:55.999 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.144112.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.144112.log.gz 2023-11-08T05:35:55.999 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.162796.log 2023-11-08T05:35:56.000 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.admin.162796.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.162796.log.gz 2023-11-08T05:35:56.000 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.146878.log 2023-11-08T05:35:56.001 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.146878.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.146878.log.gz 2023-11-08T05:35:56.001 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.121201.log 2023-11-08T05:35:56.002 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.admin.121201.log: 66.2% -- replaced with /var/log/ceph/cluster2-client.admin.121201.log.gz 2023-11-08T05:35:56.002 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.143290.log 2023-11-08T05:35:56.009 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.143290.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.136803.log 2023-11-08T05:35:56.012 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.136803.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.136803.log.gz 2023-11-08T05:35:56.012 INFO:teuthology.orchestra.run.smithi087.stderr: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.143290.log.gz 2023-11-08T05:35:56.013 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.128598.log 2023-11-08T05:35:56.013 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.132714.log 2023-11-08T05:35:56.013 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.128598.log: /var/log/ceph/cluster2-client.mirror.132714.log: 88.4% -- replaced with /var/log/ceph/cluster1-client.mirror.128598.log.gz 2023-11-08T05:35:56.014 INFO:teuthology.orchestra.run.smithi087.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.132714.log.gz 2023-11-08T05:35:56.014 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.147634.log 2023-11-08T05:35:56.015 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.147634.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.147634.log.gz 2023-11-08T05:35:56.015 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.155981.log 2023-11-08T05:35:56.016 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.155981.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.155981.log.gz 2023-11-08T05:35:56.016 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138347.log 2023-11-08T05:35:56.017 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.138347.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.138347.log.gz 2023-11-08T05:35:56.018 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.126689.log 2023-11-08T05:35:56.018 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.admin.126689.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.admin.126689.log.gz 2023-11-08T05:35:56.018 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.119104.log 2023-11-08T05:35:56.019 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.admin.119104.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.119104.log.gz 2023-11-08T05:35:56.019 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.147417.log 2023-11-08T05:35:56.021 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.147417.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.147417.log.gz 2023-11-08T05:35:56.022 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.152374.log 2023-11-08T05:35:56.022 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.152374.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.152374.log.gz 2023-11-08T05:35:56.022 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.127436.log 2023-11-08T05:35:56.023 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.admin.127436.log: 88.0% -- replaced with /var/log/ceph/cluster1-client.admin.127436.log.gz 2023-11-08T05:35:56.023 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.164666.log 2023-11-08T05:35:56.024 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.admin.164666.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.164666.log.gz 2023-11-08T05:35:56.024 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.153566.log 2023-11-08T05:35:56.025 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.153566.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.153566.log.gz 2023-11-08T05:35:56.025 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.128825.log 2023-11-08T05:35:56.026 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.128825.log: 89.4% -- replaced with /var/log/ceph/cluster1-client.mirror.128825.log.gz 2023-11-08T05:35:56.026 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2.audit.log 2023-11-08T05:35:56.030 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2.audit.log: 92.5% -- replaced with /var/log/ceph/cluster2.audit.log.gz 2023-11-08T05:35:56.031 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.151934.log 2023-11-08T05:35:56.033 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.151934.log: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.151934.log.gz 2023-11-08T05:35:56.033 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.163384.log 2023-11-08T05:35:56.034 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.admin.163384.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.163384.log.gz 2023-11-08T05:35:56.034 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.129345.log 2023-11-08T05:35:56.040 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.129345.log: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.129345.log.gz 2023-11-08T05:35:56.040 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.151376.log 2023-11-08T05:35:56.042 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.151376.log: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.151376.log.gz 2023-11-08T05:35:56.042 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.131663.log 2023-11-08T05:35:56.043 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.131663.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.149255.log 2023-11-08T05:35:56.043 INFO:teuthology.orchestra.run.smithi087.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.131663.log.gz 2023-11-08T05:35:56.044 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.149255.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.149255.log.gz 2023-11-08T05:35:56.044 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.128938.log 2023-11-08T05:35:56.045 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.124232.log 2023-11-08T05:35:56.051 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.128938.log: /var/log/ceph/cluster2-client.mirror.124232.log: 93.0% -- replaced with /var/log/ceph/cluster2-client.mirror.128938.log.gz 2023-11-08T05:35:56.051 INFO:teuthology.orchestra.run.smithi087.stderr: 93.0% -- replaced with /var/log/ceph/cluster2-client.mirror.124232.log.gz 2023-11-08T05:35:56.051 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.130003.log 2023-11-08T05:35:56.052 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.132835.log 2023-11-08T05:35:56.052 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.130003.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.130003.log.gz 2023-11-08T05:35:56.062 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.132835.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.145012.log 2023-11-08T05:35:56.063 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.145012.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.145012.log.gz 2023-11-08T05:35:56.063 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.121176.log 2023-11-08T05:35:56.064 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.admin.121176.log: 80.2% -- replaced with /var/log/ceph/cluster2-client.admin.121176.log.gz 2023-11-08T05:35:56.065 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.123311.log 2023-11-08T05:35:56.066 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.123311.log: 87.1% -- replaced with /var/log/ceph/cluster1-client.mirror.123311.log.gz 2023-11-08T05:35:56.066 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-osd.0.log 2023-11-08T05:35:56.076 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-osd.0.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138817.log 2023-11-08T05:35:56.088 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.138817.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.138817.log.gz 2023-11-08T05:35:56.092 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.146798.log 2023-11-08T05:35:56.103 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.146798.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.146798.log.gz 2023-11-08T05:35:56.114 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.134281.log 2023-11-08T05:35:56.119 INFO:teuthology.orchestra.run.smithi087.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.131937.log.gz 2023-11-08T05:35:56.120 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.135117.log 2023-11-08T05:35:56.120 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.134281.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.134281.log.gz 2023-11-08T05:35:56.122 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.135117.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.135117.log.gz 2023-11-08T05:35:56.122 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.156175.log 2023-11-08T05:35:56.132 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.156175.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137822.log 2023-11-08T05:35:56.133 INFO:teuthology.orchestra.run.smithi087.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.134802.log.gz 2023-11-08T05:35:56.135 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.137822.log: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.137822.log.gz 2023-11-08T05:35:56.135 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.145991.log 2023-11-08T05:35:56.136 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.145991.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.145991.log.gz 2023-11-08T05:35:56.136 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.152577.log 2023-11-08T05:35:56.137 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.152577.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.152577.log.gz 2023-11-08T05:35:56.137 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.149317.log 2023-11-08T05:35:56.138 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.149317.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.149317.log.gz 2023-11-08T05:35:56.138 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.129130.log 2023-11-08T05:35:56.144 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.129130.log: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.129130.log.gz 2023-11-08T05:35:56.144 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.132619.log 2023-11-08T05:35:56.155 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.132619.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.132619.log.gz 2023-11-08T05:35:56.155 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.161277.log 2023-11-08T05:35:56.156 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.admin.161277.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.161277.log.gz 2023-11-08T05:35:56.156 INFO:teuthology.orchestra.run.smithi087.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.134881.log.gz 2023-11-08T05:35:56.156 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.166087.log 2023-11-08T05:35:56.157 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.152509.log 2023-11-08T05:35:56.158 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.admin.166087.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.166087.log.gz 2023-11-08T05:35:56.159 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.152509.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.152509.log.gz 2023-11-08T05:35:56.159 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.141869.log 2023-11-08T05:35:56.159 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.141869.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.141869.log.gz 2023-11-08T05:35:56.159 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.165744.log 2023-11-08T05:35:56.160 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.admin.165744.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.165744.log.gz 2023-11-08T05:35:56.160 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.164911.log 2023-11-08T05:35:56.161 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.admin.164911.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.164911.log.gz 2023-11-08T05:35:56.161 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.151293.log 2023-11-08T05:35:56.162 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.151293.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.151293.log.gz 2023-11-08T05:35:56.162 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.135742.log 2023-11-08T05:35:56.162 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.135742.log: 1.3% -- replaced with /var/log/ceph/cluster2-client.mirror.135742.log.gz 2023-11-08T05:35:56.162 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.159403.log 2023-11-08T05:35:56.163 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.159403.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.159403.log.gz 2023-11-08T05:35:56.163 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.130735.log 2023-11-08T05:35:56.164 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.130735.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.130735.log.gz 2023-11-08T05:35:56.164 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.166430.log 2023-11-08T05:35:56.165 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.admin.166430.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.166430.log.gz 2023-11-08T05:35:56.165 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.151914.log 2023-11-08T05:35:56.166 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.151914.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.151914.log.gz 2023-11-08T05:35:56.166 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.145375.log 2023-11-08T05:35:56.167 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.168251.log 2023-11-08T05:35:56.168 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.145375.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.145375.log.gz 2023-11-08T05:35:56.169 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.admin.168251.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.168251.log.gz 2023-11-08T05:35:56.170 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126490.log 2023-11-08T05:35:56.170 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.126490.log: 88.4% -- replaced with /var/log/ceph/cluster1-client.mirror.126490.log.gz 2023-11-08T05:35:56.171 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.142508.log 2023-11-08T05:35:56.172 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.142508.log: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.142508.log.gz 2023-11-08T05:35:56.173 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.156417.log 2023-11-08T05:35:56.173 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.156417.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.156417.log.gz 2023-11-08T05:35:56.174 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.158374.log 2023-11-08T05:35:56.174 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.158374.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.158374.log.gz 2023-11-08T05:35:56.175 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.146578.log 2023-11-08T05:35:56.175 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.146578.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.146578.log.gz 2023-11-08T05:35:56.175 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.120285.log 2023-11-08T05:35:56.177 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.120285.log: 92.7% -- replaced with /var/log/ceph/cluster1-client.mirror.120285.log.gz 2023-11-08T05:35:56.177 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.156517.log 2023-11-08T05:35:56.178 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.156517.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.156517.log.gz 2023-11-08T05:35:56.178 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.129431.log 2023-11-08T05:35:56.184 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.129431.log: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.129431.log.gz 2023-11-08T05:35:56.184 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.162600.log 2023-11-08T05:35:56.185 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.admin.162600.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.162600.log.gz 2023-11-08T05:35:56.185 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.151356.log 2023-11-08T05:35:56.186 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.151356.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.151356.log.gz 2023-11-08T05:35:56.186 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.130624.log 2023-11-08T05:35:56.187 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.130624.log: 79.5% -- replaced with /var/log/ceph/cluster2-client.mirror.130624.log.gz 2023-11-08T05:35:56.187 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.169876.log 2023-11-08T05:35:56.187 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.admin.169876.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.169876.log.gz 2023-11-08T05:35:56.188 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.156537.log 2023-11-08T05:35:56.188 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.156537.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.156537.log.gz 2023-11-08T05:35:56.189 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.138176.log 2023-11-08T05:35:56.190 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.138176.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.138176.log.gz 2023-11-08T05:35:56.190 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.134498.log 2023-11-08T05:35:56.191 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.134498.log: 79.4% -- replaced with /var/log/ceph/cluster2-client.mirror.134498.log.gz 2023-11-08T05:35:56.191 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.2.134145.log 2023-11-08T05:35:56.196 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.2.134145.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.128619.log 2023-11-08T05:35:56.207 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.128619.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.128619.log.gz 2023-11-08T05:35:56.207 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.163433.log 2023-11-08T05:35:56.208 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.admin.163433.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.163433.log.gz 2023-11-08T05:35:56.218 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.166479.log 2023-11-08T05:35:56.226 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.admin.166479.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.166479.log.gz 2023-11-08T05:35:56.229 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.159017.log 2023-11-08T05:35:56.233 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.159017.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.159017.log.gz 2023-11-08T05:35:56.240 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.149151.log 2023-11-08T05:35:56.243 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.149151.log: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.149151.log.gz 2023-11-08T05:35:56.243 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.2.137865.log 2023-11-08T05:35:56.254 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.2.137865.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.144639.log 2023-11-08T05:35:56.267 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.144639.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.144639.log.gz 2023-11-08T05:35:56.267 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.159895.log 2023-11-08T05:35:56.274 INFO:teuthology.orchestra.run.smithi087.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.156175.log.gz 2023-11-08T05:35:56.279 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.admin.159895.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.159895.log.gz 2023-11-08T05:35:56.279 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.156010.log 2023-11-08T05:35:56.280 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.156010.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.156010.log.gz 2023-11-08T05:35:56.280 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.166536.log 2023-11-08T05:35:56.281 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.admin.166536.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.166536.log.gz 2023-11-08T05:35:56.281 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.129603.log 2023-11-08T05:35:56.296 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.129603.log: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.129603.log.gz 2023-11-08T05:35:56.296 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.156719.log 2023-11-08T05:35:56.297 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.156719.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.156719.log.gz 2023-11-08T05:35:56.297 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.169974.log 2023-11-08T05:35:56.298 INFO:teuthology.orchestra.run.smithi087.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.2.134145.log.gz 2023-11-08T05:35:56.298 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.admin.169974.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.169974.log.gz 2023-11-08T05:35:56.299 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.140656.log 2023-11-08T05:35:56.299 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.148261.log 2023-11-08T05:35:56.301 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.140656.log: /var/log/ceph/cluster1-client.mirror.148261.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.140656.log.gz 2023-11-08T05:35:56.301 INFO:teuthology.orchestra.run.smithi087.stderr: 92.1% -- replaced with /var/log/ceph/cluster1-client.mirror.148261.log.gz 2023-11-08T05:35:56.302 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.140453.log 2023-11-08T05:35:56.302 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.140453.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.140453.log.gz 2023-11-08T05:35:56.303 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.1.147961.log 2023-11-08T05:35:56.310 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.1.147961.log: 93.2% -- replaced with /var/log/ceph/cluster1-client.mirror.2.137865.log.gz 2023-11-08T05:35:56.313 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.169435.log 2023-11-08T05:35:56.314 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.admin.169435.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.169435.log.gz 2023-11-08T05:35:56.314 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.167859.log 2023-11-08T05:35:56.314 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.admin.167859.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.167859.log.gz 2023-11-08T05:35:56.315 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.145969.log 2023-11-08T05:35:56.317 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.145969.log: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.145969.log.gz 2023-11-08T05:35:56.317 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.128777.log 2023-11-08T05:35:56.318 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.128777.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.128777.log.gz 2023-11-08T05:35:56.318 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.136503.log 2023-11-08T05:35:56.318 INFO:teuthology.orchestra.run.smithi087.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.1.147961.log.gz 2023-11-08T05:35:56.319 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.136503.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.136503.log.gz 2023-11-08T05:35:56.319 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.142597.log 2023-11-08T05:35:56.320 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.162306.log 2023-11-08T05:35:56.320 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.142597.log: /var/log/ceph/cluster2-client.admin.162306.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.162306.log.gz 2023-11-08T05:35:56.321 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.164862.log 2023-11-08T05:35:56.321 INFO:teuthology.orchestra.run.smithi087.stderr: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.142597.log.gz 2023-11-08T05:35:56.322 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.admin.164862.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.164862.log.gz 2023-11-08T05:35:56.322 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.148576.log 2023-11-08T05:35:56.322 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.152487.log 2023-11-08T05:35:56.323 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.148576.log: /var/log/ceph/cluster1-client.mirror.152487.log: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.148576.log.gz 2023-11-08T05:35:56.323 INFO:teuthology.orchestra.run.smithi087.stderr: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.152487.log.gz 2023-11-08T05:35:56.323 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.157467.log 2023-11-08T05:35:56.323 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.127526.log 2023-11-08T05:35:56.324 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster2-client.mirror.157467.log: /var/log/ceph/cluster2-client.mirror.127526.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.157467.log.gz 2023-11-08T05:35:56.325 INFO:teuthology.orchestra.run.smithi087.stderr: 90.7% -- replaced with /var/log/ceph/cluster2-client.mirror.127526.log.gz 2023-11-08T05:35:56.325 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.124815.log 2023-11-08T05:35:56.325 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.157602.log 2023-11-08T05:35:56.330 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.124815.log: /var/log/ceph/cluster2-client.mirror.157602.log: 79.4% -- replaced with /var/log/ceph/cluster2-client.mirror.157602.log.gz 2023-11-08T05:35:56.330 INFO:teuthology.orchestra.run.smithi087.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.141441.log 2023-11-08T05:35:56.330 INFO:teuthology.orchestra.run.smithi087.stderr:/var/log/ceph/cluster1-client.mirror.141441.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.141441.log.gz 2023-11-08T05:35:56.330 INFO:teuthology.orchestra.run.smithi087.stderr: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.124815.log.gz 2023-11-08T05:35:56.377 INFO:teuthology.orchestra.run.smithi087.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.132835.log.gz 2023-11-08T05:35:56.682 INFO:teuthology.orchestra.run.smithi087.stderr: 90.1% -- replaced with /var/log/ceph/cluster2-mon.a.log.gz 2023-11-08T05:35:59.105 INFO:teuthology.orchestra.run.smithi077.stderr: 94.5% -- replaced with /var/log/ceph/cluster1-osd.1.log.gz 2023-11-08T05:35:59.525 INFO:teuthology.orchestra.run.smithi077.stderr: 94.5% -- replaced with /var/log/ceph/cluster1-osd.0.log.gz 2023-11-08T05:35:59.642 INFO:teuthology.orchestra.run.smithi077.stderr: 94.5% -- replaced with /var/log/ceph/cluster1-osd.2.log.gz 2023-11-08T05:35:59.644 INFO:teuthology.orchestra.run.smithi077.stderr: 2023-11-08T05:35:59.644 INFO:teuthology.orchestra.run.smithi077.stderr:real 0m7.173s 2023-11-08T05:35:59.645 INFO:teuthology.orchestra.run.smithi077.stderr:user 0m21.402s 2023-11-08T05:35:59.645 INFO:teuthology.orchestra.run.smithi077.stderr:sys 0m0.820s 2023-11-08T05:36:34.255 INFO:teuthology.orchestra.run.smithi087.stderr: 93.5% -- replaced with /var/log/ceph/cluster2-osd.1.log.gz 2023-11-08T05:36:34.357 INFO:teuthology.orchestra.run.smithi087.stderr: 93.6% -- replaced with /var/log/ceph/cluster2-osd.2.log.gz 2023-11-08T05:36:47.691 INFO:teuthology.orchestra.run.smithi087.stderr: 93.5% -- replaced with /var/log/ceph/cluster2-osd.0.log.gz 2023-11-08T05:36:47.693 INFO:teuthology.orchestra.run.smithi087.stderr: 2023-11-08T05:36:47.693 INFO:teuthology.orchestra.run.smithi087.stderr:real 0m55.200s 2023-11-08T05:36:47.693 INFO:teuthology.orchestra.run.smithi087.stderr:user 2m28.810s 2023-11-08T05:36:47.693 INFO:teuthology.orchestra.run.smithi087.stderr:sys 0m6.634s 2023-11-08T05:36:47.694 INFO:tasks.ceph:Archiving logs... 2023-11-08T05:36:47.695 DEBUG:teuthology.misc:Transferring archived files from smithi077:/var/log/ceph to /home/teuthworker/archive/yuriw-2023-11-07_19:42:25-rbd-wip-yuri3-testing-2023-11-07-0801-pacific-distro-default-smithi/7451368/remote/smithi077/log 2023-11-08T05:36:47.696 DEBUG:teuthology.orchestra.run.smithi077:> sudo tar c -f - -C /var/log/ceph -- . 2023-11-08T05:36:50.749 DEBUG:teuthology.misc:Transferring archived files from smithi087:/var/log/ceph to /home/teuthworker/archive/yuriw-2023-11-07_19:42:25-rbd-wip-yuri3-testing-2023-11-07-0801-pacific-distro-default-smithi/7451368/remote/smithi087/log 2023-11-08T05:36:50.751 DEBUG:teuthology.orchestra.run.smithi087:> sudo tar c -f - -C /var/log/ceph -- . 2023-11-08T05:37:13.481 DEBUG:teuthology.run_tasks:Unwinding manager install 2023-11-08T05:37:13.510 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/task/install/__init__.py", line 626, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_cfb6fd52113be6a0af3fe7fb89b2e08474eadc0f/qa/tasks/ceph.py", line 1916, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_cfb6fd52113be6a0af3fe7fb89b2e08474eadc0f/qa/tasks/ceph_manager.py", line 2758, in wait_for_clean while not self.is_clean(): File "/home/teuthworker/src/github.com_ceph_ceph-c_cfb6fd52113be6a0af3fe7fb89b2e08474eadc0f/qa/tasks/ceph_manager.py", line 2694, in is_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_ceph_ceph-c_cfb6fd52113be6a0af3fe7fb89b2e08474eadc0f/qa/tasks/ceph_manager.py", line 2355, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph-c_cfb6fd52113be6a0af3fe7fb89b2e08474eadc0f/qa/tasks/ceph_manager.py", line 1616, in raw_cluster_cmd p = self.run_cluster_cmd(args=args, stdout=stdout, **kwargs) File "/home/teuthworker/src/github.com_ceph_ceph-c_cfb6fd52113be6a0af3fe7fb89b2e08474eadc0f/qa/tasks/ceph_manager.py", line 1609, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi087 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json' 2023-11-08T05:37:13.511 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2023-11-08T05:37:13.511 DEBUG:teuthology.orchestra.run.smithi077:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-11-08T05:37:13.517 DEBUG:teuthology.orchestra.run.smithi087:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-11-08T05:37:13.595 DEBUG:teuthology.run_tasks:Unwinding manager clock 2023-11-08T05:37:13.610 INFO:teuthology.task.clock:Checking final clock skew... 2023-11-08T05:37:13.611 DEBUG:teuthology.orchestra.run.smithi077:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-11-08T05:37:13.614 DEBUG:teuthology.orchestra.run.smithi087:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-11-08T05:37:13.631 INFO:teuthology.orchestra.run.smithi077.stderr:bash: ntpq: command not found 2023-11-08T05:37:13.633 INFO:teuthology.orchestra.run.smithi087.stderr:bash: ntpq: command not found 2023-11-08T05:37:13.635 INFO:teuthology.orchestra.run.smithi077.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-11-08T05:37:13.635 INFO:teuthology.orchestra.run.smithi077.stdout:=============================================================================== 2023-11-08T05:37:13.636 INFO:teuthology.orchestra.run.smithi077.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 23 +830us[ +830us] +/- 88ms 2023-11-08T05:37:13.636 INFO:teuthology.orchestra.run.smithi077.stdout:^* hv02.front.sepia.ceph.com 3 6 377 25 -1057us[-1060us] +/- 79ms 2023-11-08T05:37:13.636 INFO:teuthology.orchestra.run.smithi077.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 25 +478us[ +478us] +/- 75ms 2023-11-08T05:37:13.636 INFO:teuthology.orchestra.run.smithi077.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2023-11-08T05:37:13.638 INFO:teuthology.orchestra.run.smithi087.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-11-08T05:37:13.638 INFO:teuthology.orchestra.run.smithi087.stdout:=============================================================================== 2023-11-08T05:37:13.638 INFO:teuthology.orchestra.run.smithi087.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 25 +843us[ +892us] +/- 88ms 2023-11-08T05:37:13.639 INFO:teuthology.orchestra.run.smithi087.stdout:^* hv02.front.sepia.ceph.com 3 6 377 25 -1095us[-1046us] +/- 79ms 2023-11-08T05:37:13.639 INFO:teuthology.orchestra.run.smithi087.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 26 +429us[ +477us] +/- 75ms 2023-11-08T05:37:13.639 INFO:teuthology.orchestra.run.smithi087.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2023-11-08T05:37:13.640 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2023-11-08T05:37:13.655 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2023-11-08T05:37:13.656 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2023-11-08T05:37:13.680 DEBUG:teuthology.orchestra.run.smithi077:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-11-08T05:37:13.775 DEBUG:teuthology.orchestra.run.smithi087:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-11-08T05:37:13.879 DEBUG:teuthology.orchestra.run.smithi077:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-11-08T05:37:13.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-08T05:37:13.913 DEBUG:teuthology.orchestra.run.smithi087:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-11-08T05:37:13.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-08T05:37:13.947 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2023-11-08T05:37:13.960 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2023-11-08T05:37:13.980 INFO:teuthology.task.internal:Duration was 3382.972246 seconds 2023-11-08T05:37:13.980 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2023-11-08T05:37:14.014 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2023-11-08T05:37:14.014 DEBUG:teuthology.orchestra.run.smithi077:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-11-08T05:37:14.017 DEBUG:teuthology.orchestra.run.smithi087:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-11-08T05:37:14.062 INFO:teuthology.orchestra.run.smithi077.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-11-08T05:37:14.065 INFO:teuthology.orchestra.run.smithi087.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-11-08T05:37:14.236 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2023-11-08T05:37:14.237 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi077.front.sepia.ceph.com 2023-11-08T05:37:14.237 DEBUG:teuthology.orchestra.run.smithi077:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-11-08T05:37:14.260 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi087.front.sepia.ceph.com 2023-11-08T05:37:14.260 DEBUG:teuthology.orchestra.run.smithi087:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-11-08T05:37:14.284 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2023-11-08T05:37:14.285 DEBUG:teuthology.orchestra.run.smithi077:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-11-08T05:37:14.302 DEBUG:teuthology.orchestra.run.smithi087:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-11-08T05:37:14.365 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2023-11-08T05:37:14.365 DEBUG:teuthology.orchestra.run.smithi077:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-11-08T05:37:14.380 DEBUG:teuthology.orchestra.run.smithi087:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-11-08T05:37:14.490 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2023-11-08T05:37:14.504 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2023-11-08T05:37:14.505 DEBUG:teuthology.orchestra.run.smithi077:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-11-08T05:37:14.532 DEBUG:teuthology.orchestra.run.smithi087:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-11-08T05:37:14.560 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2023-11-08T05:37:14.578 DEBUG:teuthology.orchestra.run.smithi077:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-11-08T05:37:14.581 DEBUG:teuthology.orchestra.run.smithi087:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-11-08T05:37:14.605 INFO:teuthology.orchestra.run.smithi077.stdout:kernel.core_pattern = core 2023-11-08T05:37:14.627 INFO:teuthology.orchestra.run.smithi087.stdout:kernel.core_pattern = core 2023-11-08T05:37:14.640 DEBUG:teuthology.orchestra.run.smithi077:> test -e /home/ubuntu/cephtest/archive/coredump 2023-11-08T05:37:14.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-08T05:37:14.685 DEBUG:teuthology.orchestra.run.smithi087:> test -e /home/ubuntu/cephtest/archive/coredump 2023-11-08T05:37:14.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-08T05:37:14.703 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2023-11-08T05:37:14.719 INFO:teuthology.task.internal:Transferring archived files... 2023-11-08T05:37:14.721 DEBUG:teuthology.misc:Transferring archived files from smithi077:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2023-11-07_19:42:25-rbd-wip-yuri3-testing-2023-11-07-0801-pacific-distro-default-smithi/7451368/remote/smithi077 2023-11-08T05:37:14.722 DEBUG:teuthology.orchestra.run.smithi077:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-11-08T05:37:14.763 DEBUG:teuthology.misc:Transferring archived files from smithi087:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2023-11-07_19:42:25-rbd-wip-yuri3-testing-2023-11-07-0801-pacific-distro-default-smithi/7451368/remote/smithi087 2023-11-08T05:37:14.764 DEBUG:teuthology.orchestra.run.smithi087:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-11-08T05:37:14.803 INFO:teuthology.task.internal:Removing archive directory... 2023-11-08T05:37:14.803 DEBUG:teuthology.orchestra.run.smithi077:> rm -rf -- /home/ubuntu/cephtest/archive 2023-11-08T05:37:14.806 DEBUG:teuthology.orchestra.run.smithi087:> rm -rf -- /home/ubuntu/cephtest/archive 2023-11-08T05:37:14.854 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2023-11-08T05:37:14.871 INFO:teuthology.task.internal:Not uploading archives. 2023-11-08T05:37:14.872 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2023-11-08T05:37:14.898 INFO:teuthology.task.internal:Tidying up after the test... 2023-11-08T05:37:14.899 DEBUG:teuthology.orchestra.run.smithi077:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-11-08T05:37:14.901 DEBUG:teuthology.orchestra.run.smithi087:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-11-08T05:37:14.916 INFO:teuthology.orchestra.run.smithi077.stdout: 525357 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Nov 8 05:37 /home/ubuntu/cephtest 2023-11-08T05:37:14.919 INFO:teuthology.orchestra.run.smithi087.stdout: 525357 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Nov 8 05:37 /home/ubuntu/cephtest 2023-11-08T05:37:14.920 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2023-11-08T05:37:14.937 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2023-11-08T05:37:14.983 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: CommandFailedError: Command failed on smithi087 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json' 2023-11-08T05:37:14.983 INFO:teuthology.nuke:Checking targets against current locks 2023-11-08T05:37:15.010 DEBUG:teuthology.nuke:shortname: smithi077 2023-11-08T05:37:15.010 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-11-08T05:37:15.035 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi077.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-11-07_19:42:25-rbd-wip-yuri3-testing-2023-11-07-0801-pacific-distro-default-smithi/7451368', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-11-08 04:32:25.547376', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGSQ/WLcJdpJ0ClJVyBsFrcYIQ2QRbAwp2zUvz0ifkJpShDNVGje+V/Jdm6AZOafWAIrxu2EWTj6axjmMfZQozM='} 2023-11-08T05:37:15.039 DEBUG:teuthology.nuke:shortname: smithi087 2023-11-08T05:37:15.039 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-11-08T05:37:15.066 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi087.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-11-07_19:42:25-rbd-wip-yuri3-testing-2023-11-07-0801-pacific-distro-default-smithi/7451368', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-11-08 04:32:25.549289', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBD+XKU8wa6pBVq4mhdEwRNPaK1Sl9uWyjisTURFaLby1G+ogJ5JiOLEZ7VuzQt8PXKmmvgJJqHhmd6n+97zE5BE='} 2023-11-08T05:37:15.093 INFO:teuthology.orchestra.console.smithi077:Power off 2023-11-08T05:37:15.094 DEBUG:teuthology.orchestra.console.smithi077:pexpect command: ipmitool -H smithi077.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-11-08T05:37:15.119 DEBUG:teuthology.orchestra.console.smithi077:power off output: Chassis Power Control: Down/Off 2023-11-08T05:37:15.120 DEBUG:teuthology.orchestra.console.smithi077:pexpect command: ipmitool -H smithi077.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-08T05:37:15.136 INFO:teuthology.orchestra.console.smithi087:Power off 2023-11-08T05:37:15.137 DEBUG:teuthology.orchestra.console.smithi087:pexpect command: ipmitool -H smithi087.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-11-08T05:37:15.146 DEBUG:teuthology.orchestra.console.smithi077:check power output: Chassis Power is on 2023-11-08T05:37:15.159 DEBUG:teuthology.orchestra.console.smithi087:power off output: Chassis Power Control: Down/Off 2023-11-08T05:37:15.159 DEBUG:teuthology.orchestra.console.smithi087:pexpect command: ipmitool -H smithi087.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-08T05:37:15.182 DEBUG:teuthology.orchestra.console.smithi087:check power output: Chassis Power is on 2023-11-08T05:37:19.148 DEBUG:teuthology.orchestra.console.smithi077:pexpect command: ipmitool -H smithi077.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-08T05:37:19.183 DEBUG:teuthology.orchestra.console.smithi087:pexpect command: ipmitool -H smithi087.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-08T05:37:19.258 DEBUG:teuthology.orchestra.console.smithi077:check power output: Chassis Power is on 2023-11-08T05:37:19.295 DEBUG:teuthology.orchestra.console.smithi087:check power output: Chassis Power is on 2023-11-08T05:37:23.261 DEBUG:teuthology.orchestra.console.smithi077:pexpect command: ipmitool -H smithi077.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-08T05:37:23.297 DEBUG:teuthology.orchestra.console.smithi087:pexpect command: ipmitool -H smithi087.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-08T05:37:23.372 DEBUG:teuthology.orchestra.console.smithi077:check power output: Chassis Power is on 2023-11-08T05:37:23.407 DEBUG:teuthology.orchestra.console.smithi087:check power output: Chassis Power is on 2023-11-08T05:37:27.374 DEBUG:teuthology.orchestra.console.smithi077:pexpect command: ipmitool -H smithi077.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-08T05:37:27.409 DEBUG:teuthology.orchestra.console.smithi087:pexpect command: ipmitool -H smithi087.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-08T05:37:27.485 DEBUG:teuthology.orchestra.console.smithi077:check power output: Chassis Power is off 2023-11-08T05:37:27.520 DEBUG:teuthology.orchestra.console.smithi087:check power output: Chassis Power is off 2023-11-08T05:37:27.586 INFO:teuthology.orchestra.console.smithi077:Power off completed 2023-11-08T05:37:27.621 INFO:teuthology.orchestra.console.smithi087:Power off completed 2023-11-08T05:37:27.726 INFO:teuthology.run:Summary data: description: rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-low-osd-mem-target policy/simple rbd-mirror/four-per-cluster supported-random-distro$/{centos_8} workloads/rbd-mirror-snapshot-stress-workunit-exclusive-lock} duration: 3382.9722464084625 failure_reason: '"2023-11-08T05:28:12.529768+0000 mon.a (mon.0) 1566 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log' flavor: default owner: scheduled_yuriw@teuthology status: fail success: false 2023-11-08T05:37:27.726 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-11-08T05:37:27.804 INFO:teuthology.run:FAIL